Rich-T-kid commented on PR #26315: URL: https://github.com/apache/datafusion/pull/26315#issuecomment-6834270573
After extensive investigation and benchmarking, I do not think we should revive this PR or add a standalone NDV threshold. NDV by itself is not a reliable predictor of whether dictionary reads will help. Even at medium or high cardinality, dictionaries can be beneficial when data pages remain dictionary encoded. Conversely, when a column chunk mixes `RLE_DICTIONARY` and plain pages, the Arrow reader may need to expand and rebuild the dictionary during encoding transitions, which can erase the benefit or cause regressions. More context on that reader behavior is tracked in apache/arrow-rs#12195. I think the narrower encoding-aware direction is worth pursuing, but it belongs as part of the broader optimization work in #24117 rather than as a user-configured NDV threshold. I am closing #24113 in favor of that direction. cc @asolimando -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
