sohailshk opened a new pull request, #41319:
URL: https://github.com/apache/beam/pull/41319
Fixes #20967
Fixes #41305
The proxy of a grouped aggregation is computed by running pandas on an empty
frame, and two aggregations return a different shape for an empty input than
for real data. `groupby(...).describe()` dropped the names and levels of the
group keys, so the proxy index was unnamed while the real result is named by
the group keys, and `to_pcollection(..., include_indexes=True)` raised
"Encountered an unnamed index". `groupby(...).skew()` kept the integer or bool
dtype, so `to_pcollection` inferred an integer schema for a float result and
encoding the rows failed. The `describe` proxy now gets an empty index with the
names and dtypes of the group keys (from `size()` of the grouped proxy), and
the integer and bool columns of the `skew` proxy are cast to float64 (float
columns keep their dtype, as in pandas). The computed result is unchanged.
Tests: `test_groupby_describe_multiple_keys` and
`test_groupby_skew_of_integer_columns` are new, `test_categorical_groupby` gets
a `describe` case, and the two SeriesGroupBy.describe tests that were skipped
for #20967 now run (`test_groupby_project_series`,
`test_series_groupby_series`). All of them fail on master and pass with the
change. `pytest apache_beam/dataframe/frames_test.py
apache_beam/dataframe/convert_test.py`: 471 passed, 17 skipped, with pandas
2.3.3 and 2.2.3. With `to_dataframe` on a `beam.Create` of rows,
`to_pcollection(df.groupby('c').skew(), include_indexes=True) |
beam.Reshuffle()` failed on master on the default runner and returns the values
after the change; `to_pcollection(df.groupby(['c', 'd']).a.describe()[['count',
'mean']], include_indexes=True)` raised the unnamed index error on master
(FnApiRunner and DirectRunner) and returns rows after the change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
