zhuqi-lucas commented on PR #23369: URL: https://github.com/apache/datafusion/pull/23369#issuecomment-4924812648
> > Basically no regression, and it seems resolve the pushdown_filters regression, but Adaptive predicate evaluation should be better. > > This seems like a very nice find (a simple heuristic) @zhuqi-lucas -- would you be willing to make a new PR with just this change? I think we should consider doing it. > > The heuristic might also want to take into account data type (e.g. projecting string columns might count more than projecting an int column) > > I would like to run some more tests to see that it works across other query benchmarks, but I think we should do it Thanks @alamb! Yes, let me open a separate PR with just the heuristic gate so it can be reviewed and merged on its own, then follow up with the pushdown_filters=true default flip once we have the extended benchmark data. Good call on data-type weighting — I'll experiment with weighting non-filter columns by an estimated byte size per row (e.g. Utf8 / Binary / List count much more than Int32) so the gate captures the actual wide-decode saving. Will also make the threshold configurable so it's tunable per workload if the heuristic misfires on someone's data. I'll post the new PR shortly! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
