ghostiee-11 commented on issue #28985:
URL: https://github.com/apache/arrow/issues/28985#issuecomment-5295288970
could this be reopened? it was closed by the stale bot rather than by a
decision, and the last human comment was a deferral rather than a no.
concrete impact: `hash_approximate_median` on a two element group returns
the lower value instead of the mean.
```python
import pyarrow as pa
t = pa.table({"g": ["a", "a"], "v": [10.0, 20.0]})
print(t.group_by("g").aggregate([("v", "approximate_median")]).to_pydict())
# {'g': ['a'], 'v_approximate_median': [10.0]} exact median is 15.0
```
`hash_tdigest` returns 10.0 too.
narwhals maps its `median()` onto these, so a grouped median is silently
wrong there for small groups: narwhals-dev/narwhals#3853. the scalar path can
be fixed with `quantile(q=0.5)` since that one is exact, but there is no
`hash_quantile` to switch to for grouped.
one data point on cost, scalar only: `pc.quantile(q=0.5,
interpolation="linear")` measured 3 to 4x faster than `pc.approximate_median`
from n=2 up to n=10M here. that says nothing about what a grouped
implementation would cost, and the in memory accumulation concern from 2021
still stands.
pyarrow 23.0.1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
