Rich-T-kid commented on issue #24113: URL: https://github.com/apache/datafusion/issues/24113#issuecomment-6834271535
After extensive investigation and benchmarking, I think we should close this without adding a standalone NDV threshold. NDV by itself is not a reliable predictor of whether dictionary reads will help. Even at medium or high cardinality, dictionaries can be beneficial when data pages remain dictionary encoded. Conversely, when a column chunk mixes `RLE_DICTIONARY` and plain pages, the Arrow reader may need to expand and rebuild the dictionary during encoding transitions, which can erase the benefit or cause regressions. More context on that reader behavior is tracked in apache/arrow-rs#12195. I think the narrower encoding-aware direction is worth pursuing, but it belongs as part of the broader optimization work in #24117 rather than as a user-configured NDV threshold. Closing this specific issue in favor of that direction. cc @asolimando -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
