alamb commented on PR #23369: URL: https://github.com/apache/datafusion/pull/23369#issuecomment-4924954842
> Good call on data-type weighting — I'll experiment with weighting non-filter columns by an estimated byte size per row (e.g. Utf8 / Binary / List count much more than Int32) so the gate captures the actual wide-decode saving. Will also make the threshold configurable so it's tunable per workload if the heuristic misfires on someone's data. Let's not go too crazy with tuning knobs as we are hoping for something more dynamic in the future What I think we can defend easily is "a simple heuristic that works for most workloads" If we make the heuristic too complicated then we risk it overfitting / being hard to tune and understand I think we should save our compleixty budget for the adaptive behavior -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
