pearu commented on issue #50842: URL: https://github.com/apache/arrow/issues/50842#issuecomment-6082509827
Follow-up on the plan above: the PR (#52547) narrows it. Benchmarking the always-on variant (dictionary path for every `string`/`binary` dictionary) against the current decode showed that it is faster and far lighter for ordinary dictionaries (10²–10⁴ entries: 2–4× faster, ~100× less memory), but worse in three situations: dictionaries beyond ~10⁵ entries (the table of Python objects no longer fits in cache, up to 3× slower), many chunks sharing one large dictionary (the dictionary is converted once per chunk; 1000 chunks × 10³ rows over a 10⁵-entry dictionary took 10 s instead of ~50 ms), and small slices of arrays with large dictionaries. So the PR takes the dictionary path only when decoding would actually overflow: a cheap check on the dictionary offsets and indices predicts whether `Take` would reject the decode, and everything else goes through the existing code unchanged (measured at parity on 10⁶–10⁷ rows with 10²–10⁶-entry dictionaries). Using the dictionary path more widely, where it is the better choice, can be a separate change. --- _🤖 Drafted by Claude Code (an AI agent) and reviewed & approved by pearu._ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
