pearu commented on PR #52547: URL: https://github.com/apache/arrow/pull/52547#issuecomment-6152421172
> I find it very weird to vary the output type depending on the input _size_. Why can't we generate a CATEGORICAL without converting to dense first? Agreed, the size gate is a premature optimization. I'll reconsider the approach: convert string/binary dictionaries from the dictionary whenever an object array is requested, with no dense intermediate, and drop the overflow predictor. I'll update the PR once I have the implementation and measurements. On categorical: the two paths this PR touches produce NumPy arrays (`Array.to_numpy`, and the children of nested types, which are sliced into rows as ndarrays), and NumPy has no categorical dtype. The non-dense NumPy form of a dictionary is an object array built from the dictionary's Python objects, which is what `ChunkedArray.to_numpy` already does. Top-level `to_pandas` keeps producing a Categorical and is untouched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
