asolimando commented on issue #24111: URL: https://github.com/apache/datafusion/issues/24111#issuecomment-5372759576
Apologies for the late reply. I agree with @alamb that it's fine to start with a configuration option to let the user activate when they know it helps, and work our way towards automatic activation. We synced offline with @Rich-T-kid and indeed what will make automatic activation a little tricky, is that there is an overhead for the dictionary encoding per record batch. We don't have any statistic estimating that at the moment, and cardinality/row count isn't really a good proxy, but empirical testing will tell, it might be possible that we can still find some reasonable cost-model for this. This said, I am fully supportive of the initiative and the direction that emerged from the discussion looks good to me! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
