dwsmith1983 opened a new pull request, #6902: URL: https://github.com/apache/datafusion-comet/pull/6902
## Which issue does this PR close? Closes #6901. ## Rationale for this change `DPP - single-split sort-merge join (AQE=true, pruned=a|b)` fails on Spark 3.5. With AQE on, once DPP prunes one shuffled input, `AQEPropagateEmptyRelation` replaces the join with an empty relation. The test expected a `CometEmptyRelationExec` on 3.5 and later, but the node behind it, `EmptyRelation`/`EmptyRelationExec`, only exists from Spark 4.0 (SPARK-48466). On 3.5 the rule produces an empty `LocalRelation`, which plans as `LocalTableScanExec`. `spark.comet.exec.localTableScan.enabled` is off by default, so that scan stays in Spark. The query result is correct; only the plan check was wrong for 3.5. ## What changes are included in this PR? Test only. On Spark 4.0+ the test still requires `CometEmptyRelationExec`. On Spark 3.5 it requires an empty `LocalTableScanExec`, or a `CometLocalTableScanExec` if local table scans are converted. Spark 3.4 keeps its existing checks, since there both scans stay in one native block. ## How are these changes tested? Ran the four `DPP - single-split sort-merge join` cases in `CometIcebergNativeSuite` on each profile: - `-Pspark-3.5 -Pscala-2.13` (the two AQE cases fail on main with this profile, as in the CI runs linked from #6901) - `-Pspark-3.5 -Pscala-2.12` - `-Pspark-3.4` - `-Pspark-4.0` - default (Spark 4.1) All four pass on each. `./mvnw test-compile -Pspark-3.5 -Pstrict-warnings -DskipTests` and `spotless:check` pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
