anishmehta24 opened a new issue, #40503:
URL: https://github.com/apache/beam/issues/40503
### What happened?
`DeferredSeries.value_counts` has a non-parallel path for `bins=` (its
docstring says it needs to collect the whole dataset), but it can't be used at
all. The argument list `[self._expr]` ended up inside the lambda, so creating
the expression fails right away:
```python
import apache_beam as beam
import pandas as pd
from apache_beam.dataframe.convert import to_dataframe, to_pcollection
values = [1.0, 2.0, 2.0, 3.0, 5.0, 8.0, 8.0, 9.0]
print(pd.Series(values, name='x').value_counts(bins=3))
with beam.Pipeline() as p:
df = to_dataframe(p | beam.Create([beam.Row(x=v) for v in values]))
with beam.dataframe.allow_non_parallel_operations():
counts = df.x.value_counts(bins=3)
_ = (
to_pcollection(counts, include_indexes=True, yield_elements='pandas')
| beam.Map(print))
```
pandas:
```
(0.991, 3.667] 4
(6.333, 9.0] 3
(3.667, 6.333] 1
Name: count, dtype: int64
```
Beam:
```
TypeError: ComputedExpression.__init__() missing 1 required positional
argument: 'args'
```
Fixing the argument list alone isn't enough, because the proxy is computed
from an empty Series and pandas can't compute bins for empty data (`ValueError:
Cannot cut empty array`).
This is different from #20903, which asks for a distributed implementation.
Seen on Beam 2.77.0 and on master (7d93ef8e76a), pandas 2.3.3, Python 3.12.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
