dwsmith1983 commented on PR #6270: URL: https://github.com/apache/datafusion-comet/pull/6270#issuecomment-6145127705
> Once #5365 lands, could you merge main and strip every DPP filter from the Delta scan's `originalPlan` the same way? Done in 5cf465440, on top of a main merge that brings in #5365. `CometDeltaNativeScanExec.doCanonicalize` now drops every `DynamicPruningExpression` from `originalPlan`, and `runtimeFilters` still carries the scan's DPP identity. `CometDeltaNativeScanSuite` has two new tests: - A Delta version of the SPARK-32509 test: a self-join over a partitioned Delta table with AQE on and no broadcast join. Without the change Comet reuses no exchange where Spark reuses one; with it the counts match. - A Delta port of the #6264 UNION ALL test (91a3ae17f), for the other direction. Reverting `filterUnusedDynamicPruningExpressions` to main's version, or stripping DPP filters from the Delta scan's `runtimeFilters`, makes one branch reuse the other's shuffle and return 7 rows where Spark returns 28. Both pass on Spark 3.5, 4.0 and 4.1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
