manuzhang opened a new pull request, #7625: URL: https://github.com/apache/datafusion-comet/pull/7625
## Which issue does this PR close? Part of #4329. Supersedes #5358 after #7521 removed the Spark 3.4 build profile and CI coverage and #7560 removed the Spark 3.4 shim source sets. ## Rationale for this change Spark 3.4 is no longer a supported build target, but compatibility branches, fallbacks, tests, generated diffs, and version-specific test assets remained in the codebase. Keeping those dead paths makes the supported Spark 3.5 and 4.x implementations harder to follow and maintain. The separately tracked `get_json_object_spark34` behavior is intentionally preserved. ## What changes are included in this PR? - Remove the remaining Spark 3.4 compatibility gates and fallback code from Comet, Delta, Iceberg, scan, shuffle, expression, and test paths. - Delete obsolete Spark 3.4 and Iceberg 1.8.1 generated diffs and Spark 3.4-only tests and shims. - Promote the Spark 3.5 TPC-DS plan baselines to the shared baseline and remove the now-redundant Spark 3.5 override files. - Rename `CometExec3_4PlusSuite` to `CometExecCompatibilitySuite` and update CI suite lists and active comments to describe the supported versions. ## How are these changes tested? - `./mvnw -Pspark-3.5 -DskipTests test-compile` - `./mvnw -Pspark-4.0 -DskipTests test-compile` - `./mvnw -Pspark-4.1 -DskipTests test-compile` - `./mvnw -Pspark-4.2 -DskipTests test-compile` - `python3 dev/ci/check-ci-config.py` - `python3 dev/ci/check-versioned-source-parity.py` - `python3 dev/ci/check-suites.py` - `git diff --check` The focused Spark 3.5 SQL parser and plan-stability suites compiled locally, but the runner could not start because no compatible Spark 3.5 checkout is configured in `SPARK_HOME` or `spark.test.home`. The `run-spark-3.5-tests` label is requested for hosted validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
