anishmehta24 opened a new issue, #41321:
URL: https://github.com/apache/beam/issues/41321
### What happened?
In the DataFrame API, grouping by a categorical column with `observed=False`
(the default in pandas 2.x) returns each category once per partition, with
nulls in all but one of the copies:
```python
import apache_beam as beam
import numpy as np
import pandas as pd
from apache_beam.dataframe.convert import to_dataframe, to_pcollection
n = 2_000_000
rng = np.random.default_rng(0)
df = pd.DataFrame({
'color': pd.Categorical(
np.array(['red', 'green', 'blue'])[rng.integers(0, 3, n)]),
'n': rng.random(n),
})
print(df.groupby('color', observed=False).n.mean().round(4))
with beam.Pipeline() as p:
beam_df = to_dataframe(p | beam.Create([df]), proxy=df[:0])
result = beam_df.groupby('color', observed=False).n.mean().round(4)
_ = (
to_pcollection(result.to_frame(), include_indexes=True)
| beam.Map(lambda row: print(row.color, row.n)))
```
pandas:
```
color
blue 0.5002
green 0.5001
red 0.4998
Name: n, dtype: float64
```
Beam:
```
blue 0.5002
green None
red 0.4998
blue None
green 0.5001
red None
```
sum, median, std, var, count, size, nunique and the other groupby
aggregations do the same. Locally a small input usually stays in one batch and
comes out right, which is why the example is large; on a distributed runner any
size can hit it.
Grouping by a categorical index level (`groupby(level=0, observed=False)`)
is handled: it requires Singleton partitioning because of #21827. For columns,
`_liftable_agg` and `_unliftable_agg` in `frames.py` only look at
`self._grouping_indexes`, the grouping keys that were already index levels.
That list is empty when grouping by columns, so the post-aggregation runs in
parallel with `observed=False` and every partition emits every category. #13256
checked every level of the grouped index; #13843 narrowed it to
`_grouping_indexes`.
Seen on Beam 2.77.0 and on master (e87570388b6), pandas 2.3.3, Python 3.12.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
