andygrove opened a new pull request, #7521: URL: https://github.com/apache/datafusion-comet/pull/7521
## Which issue does this PR close? Part of #4329. ## Rationale for this change Spark 3.4 has been deprecated since 1.0.0 and no longer gates a merge (#5885). This is the first, minimal step of dropping it: stop building it and stop testing it. The Spark 3.4 shims are left alone for a follow-up so this change stays easy to review. ## What changes are included in this PR? - Remove the `spark-3.4` profile from `pom.xml` and `spark/pom.xml`, and from the `contrib-delta` enforcer's profile list. - Remove the `spark_3_4` Spark SQL job and the `iceberg_1_8` job (Iceberg 1.8.1 ran against Spark 3.4.3) from `ci.yml`, along with their `compute-changes.py` filters and `POLICY` entries and the matching `check-ci-config.py` cases. The `run-spark-3.4-tests` label no longer gates anything. - Remove Spark 3.4 from the Linux test matrix (`linux-test-profiles.py`), the Lint Java matrix and the snapshot publish variants. - Remove Spark 3.4 from the release, docs-generation and golden-file scripts (`dev/release/*`, `docs/build.sh`, `dev/generate-release-docs.sh`, `dev/regenerate-golden-files.sh`). - Update the contributor and user docs (CI tiers, workflow README, Spark SQL and Iceberg test guides, installation, compatibility pages) and drop the Spark 3.4 expression compatibility index page. Deliberately not touched: - `spark/src/main/spark-3.4/`, `spark/src/test/spark-3.4/` and any other shim. Nothing builds them now; they can be deleted in a follow-up. - `dev/diffs/3.4.3.diff` and `dev/diffs/iceberg/1.8.1.diff`. Nothing uses them now. - The `!spark/src/main/spark-3.4/**` path excludes in `compute-changes.py`, which keep edits to the orphaned shim directory from triggering other jobs. Follow-ups this leaves: - The shared `approved-plans-v1_4` / `approved-plans-v2_7` golden-file base tier was only regenerable on the 3.4 profile (the other profiles write to their own `-sparkN_M` directory). The files still resolve correctly, so nothing needs regenerating now, but a future change that alters plans for every version has no way to update the base. Decide whether 3.5 becomes the base or the base is folded into `-spark3_5`. - Branch protection should not require any 3.4 check; none is a distinct required context, as `Required Checks` aggregates the jobs. ## How are these changes tested? - `python3 dev/ci/check-ci-config.py`, `check-versioned-source-parity.py` and `check-suites.py` pass. - `./mvnw -N help:evaluate -Dexpression=spark.version -Pspark-3.5` still resolves (3.5.9), and the root pom parses. - The modified workflow files parse under `actionlint` (only the existing shellcheck style notes remain). - Not run: a full Maven build or the Spark SQL suites. CI on this PR exercises the remaining profiles. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
