TPDeramus opened a new issue, #38747: URL: https://github.com/apache/arrow/issues/38747
### Describe the usage question you have. Please include as many useful details as possible. I am so sorry to bug you for something minor, but I can't seem to locate it in the documentation. I have some data I'm reading in as part of a `parquet` dataset that includes a chunked array (https://arrow.apache.org/docs/python/generated/pyarrow.ChunkedArray.html), which looks something like the following: `table = pa.table({'IDs': ["A", "A", "A", "B", "B", "C", "C", "C", "C", "D", "D"], 'ChunkedArray':[[7,0,5,8,0],[7,9,1,9,4],[7,1,0,10,7],[8,3,4,0,5],[5,4,5,6,2],[0,5,5,1,6],[8,9,0,10,7],[0,5,10,5,10],[9,6,9,5,10],[8,7,7,3,6],[5,4,10,2,5]]})` I am trying to apply the `group_by` function to the arrays to get arrays of the `maximum`, `mean`, and most frequent (`mode`) values of each chunked array by ID. Along the lines of: ``` IDs ChunkedArray A [7,9,5,10,7] B [8,4,5,6,5] C [9,9,10,10,10] D [8,7,10,3,6] IDs ChunkedArray A [7,3.333333333,2,9,3.666666667] B [6.5,3.5,4.5,3,3.5] C [4.25,6.25,6,5.25,8.25] D [6.5,5.5,8.5,2.5,5.5] IDs ChunkedArray A [7,NaN,NaN,NaN,NaN] B [NaN,NaN,NaN,NaN,NaN] C [0,5,NaN,5,10] D [NaN,NaN,NaN,NaN,NaN] ``` They can be either separate tables, or all part of the same table or dataframe (whichever is easier to implement), or have their own columns, but the actual dataset is extremely large so I was concerned this might lead to memory read issues. Calling `aggregate` on the Chunked arrays directly throws errors because the data is a list: ``` >>> table.group_by("IDs").aggregate([("ChunkedArray", "mean")]) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "pyarrow/table.pxi", line 5320, in pyarrow.lib.TableGroupBy.aggregate File "/usr/Python/mambaforge/lib/python3.10/site-packages/pyarrow/acero.py", line 308, in _group_by return decl.to_table(use_threads=True) File "pyarrow/_acero.pyx", line 511, in pyarrow._acero.Declaration.to_table File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status pyarrow.lib.ArrowNotImplementedError: Function 'hash_mean' has no kernel matching input types (list<item: int64>, uint32) ``` And all my previous attempts at applying flatten on the ChunkedArray before or after calling the functions of interest had not been successful until I tried the result from the following (https://stackoverflow.com/questions/77489901/apply-an-operation-function-to-a-chunkedarray-in-pyarrow): ``` columns = {f'c{i}': pc.list_element(table['ChunkedArray'], i) for i in range(5)} flat_table = pa.table(columns | {'IDs': table['IDs']}) flat_table.group_by('IDs').aggregate([('c0', 'mean')]) ``` However, I'm still short on understanding 2 things. 1. How to apply a function to multiple columns at once in `group_by` and `aggregate` in one go. 2. How I could get the most frequent value, or `mode` of the data using this approach. I see that `aggregate` outside of `group_by` has the mode function: https://arrow.apache.org/docs/python/generated/pyarrow.compute.mode.html#pyarrow.compute.mode But within `table.group_by`: https://arrow.apache.org/docs/python/compute.html#grouped-aggregations I see that there are ways to make custom functions: https://arrow.apache.org/docs/python/compute.html#user-defined-functions I see that has been discussed previously on custom functions: https://github.com/apache/arrow/issues/14860 But this is technically built in outside of `group_by`. Would appreciate any suggestions you are willing to provide. ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
