haohuaijin commented on PR #25811: URL: https://github.com/apache/datafusion/pull/25811#issuecomment-6580437523
> @haohuaijin -- since you filed the original request, can you please help review this PR and see it is what you have in mind? @alamb , I noticed this optimization when do some benchmark. my concern was that interleaving Bloom filter reads and checks changes the timing of I/O requests and maybe affect performance. I tried a PoC, but don’t remember the results now. Since I couldn’t find a good approach that addressed this concern, I opened the issue to discuss possible solutions. For example: ``` Before: read A → read B → read C → evaluate After: read A → check A → read B → check B → … ``` With 100 predicate columns and no early rejection, we still read every filter, but each subsequent read now waits for the previous column’s check. That’s the scheduling change I was concerned about. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
