mbutrovich commented on issue #3053: URL: https://github.com/apache/datafusion-comet/issues/3053#issuecomment-3719969379
We rely on Spark's planning and capabilities. One challenge with dynamic filters becomes "how do you get plan data from one part of the plan to another within Spark's communication semantics?" Spark supports some of this with bloom filters (albeit above scans). Any dynamism that we introduce to query execution should work within the context of Spark's adaptivity, mostly AQE, DPP, and bloom filter injection (maybe we could push the later into the scan with Comet and that would be a decent improvement). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
