mohitgurav20 commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6486070934
@alamb Thanks for taking a look at this. You’re right about the I/O pattern — yes, the filters are now requested one column at a time and we stop as soon as a row group can be rejected. That was the main goal, but I agree we should make that behavior more visible and testable, especially for object store cases where latency matters. I’ll clean up the overlapping logic with `prune_by_bloom_filters` (there is some duplication right now) and look into adding the kind of I/O snapshot tests you linked from arrow-rs so we can clearly see the difference in requests. Also looping in @haohuaijin since he opened the original issue — would be good to hear if this matches what he had in mind. I’ll push an update with the cleanup and better tests soon. Thanks again for the feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
