[
https://issues.apache.org/jira/browse/ARROW-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17612540#comment-17612540
]
Alenka Frim commented on ARROW-17829:
-------------------------------------
The fix has to be done in a test and the code:
* {{write_to_dataset()}} in {{pyarrow/parquet/core.py}}
* {{multisourcefs()}} fixture in {{test_dataset.py}}
The change is in fact coming from Pandas 1.5.0:
* https://pandas.pydata.org/docs/whatsnew/v1.5.0.html
* https://github.com/pandas-dev/pandas/issues/42795
Am working on the PR and will submit it today.
> [Python] Avoid pandas groupby deprecation warning write_to_dataset
> ------------------------------------------------------------------
>
> Key: ARROW-17829
> URL: https://issues.apache.org/jira/browse/ARROW-17829
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Joris Van den Bossche
> Assignee: Alenka Frim
> Priority: Critical
> Fix For: 10.0.0
>
>
> I noticed the following warnings in our test builds:
> {code}
> opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py::test_make_fragment
>
> /opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py:197:
> FutureWarning: In a future version of pandas, a length 1 tuple will be
> returned when iterating over a groupby with a grouper equal to a list of
> length 1. Don't supply a list with a single grouper to avoid this warning.
> for part, chunk in df_d.groupby(["color"]):
> opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_dataset.py::test_legacy_write_to_dataset_drops_null
> opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_pandas.py::test_write_to_dataset_pandas_preserve_extensiondtypes[True]
> opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/parquet/test_pandas.py::test_write_to_dataset_pandas_preserve_index[True]
>
> /opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/parquet/core.py:3326:
> FutureWarning: In a future version of pandas, a length 1 tuple will be
> returned when iterating over a groupby with a grouper equal to a list of
> length 1. Don't supply a list with a single grouper to avoid this warning.
> for keys, subgroup in data_df.groupby(partition_keys):
> {code}
> I suppose those are coming from pandas 1.5.0. We should investigate whether
> this is something to fix in our code (or just in the tests)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
