Rich-T-kid opened a new issue, #12115: URL: https://github.com/apache/arrow-rs/issues/12115
## Problem The dictionary-cache profitability check added in #11982 calls: ```rust use_dictionary_cache(dictionary, values.len()) ``` `values.len()` includes null rows, but `write_gather` only encodes the gathered dictionary indices. For a highly nullable dictionary array, a new cache with one slot per dictionary value can therefore pass the gate even when very few, or no, non-null values will use it. This can allocate approximately `8 * dictionary.len()` bytes without enough encoded rows to amortize the cache. ## Expected behavior The profitability decision for a new dictionary cache should account for the rows that can actually use the cache, while preserving reuse of an existing cache for the same dictionary values. The exact policy still needs design. Possible inputs include the gathered indices and dictionary-key validity, without adding an expensive scan that erases the optimization's benefit. Originally noted in <https://github.com/apache/arrow-rs/pull/11982#discussion_r4482187914>. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
