mattp5657 commented on issue #24280: URL: https://github.com/apache/datafusion/issues/24280#issuecomment-5468932739
@adriangb I've opened a PR for this. To be honest, the actual effect is tiny: an isolated microbenchmark shows the eval-time saving is real (0.33–1.84µs, scaling up with row-group count) but there's also a comparable one-time construction cost for the pass itself (~2.09µs). Net, that only pays off above roughly 4,000+ row groups per file — below that it's closer to a wash or even a slight loss. TPC-H wall-clock testing shows no measurable difference either way, which checks out given the numbers. Still might be worth merging as a correctness cleanup (it removes real duplicate work), just don't expect it to move any needles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
