pearu commented on issue #50842: URL: https://github.com/apache/arrow/issues/50842#issuecomment-6078075488
@rok @jorisvandenbossche following the suggestion at the dev sync, I looked at whether the `deduplicate_objects` machinery could solve this. It can't as it stands, for two reasons: 1. **It runs too late.** `ConvertChunkedArrayToPandas` first decodes the dictionary into a dense Arrow array (`DecodeDictionaries` → `compute::Cast(dictionary → value_type)`, which is a `take` of the dictionary by the indices). Only that dense `string` array then reaches the object writer, `ConvertAsPyObjects`, where `deduplicate_objects` lives. The decode is exactly the step that overflows the 32-bit offsets, so deduplication never gets to run. 2. **On the `np.asarray` / `to_numpy` path it isn't enabled at all.** `to_numpy` sets `decode_dictionaries`, `zero_copy_only` and `to_numpy`, and leaves `deduplicate_objects` at its C++ default of `false`. (`to_pandas` enables it, but reason 1 still applies.) **Approach:** let the object writer consume the dictionary array directly instead of a decoded copy: convert each dictionary value to a Python object once, then fill the output by `Py_INCREF`-ing the object for each index. This is the `unique_values` + `Py_INCREF` part of `deduplicate_objects`, minus the memo table, because the dictionary already is the set of unique values. - Reason 1 goes away because there is no dense intermediate any more; nothing before the object stage can overflow. - Reason 2 goes away because the dictionary path doesn't depend on `deduplicate_objects` being enabled; it applies whenever a binary-like dictionary is decoded into an object array, which covers `to_numpy` as is. **Plan:** one PR in `arrow_to_pandas.cc`, with regression tests for the two entry points that hit the decode: `to_numpy`/`np.asarray`, and `to_pandas` of nested dictionaries (e.g. `list<dictionary<string>>`). I'll prepare the PR unless there are objections to the direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
