LuciferYang opened a new pull request, #12902: URL: https://github.com/apache/gluten/pull/12902
## What changes are proposed in this pull request? Removes the Spark 3.3 build surface. This covers three items in #12807: the Spark-33 unit tests, the Spark-33 shim source tree, and the building scripts. Two follow-ups remain, described at the end. Seven directories go, 328 files and about 41k lines: `shims/spark33`, `gluten-ut/spark33`, `backends-clickhouse/src-delta23`, `gluten-delta/src-delta23`, `backends-clickhouse/src-iceberg-spark33`, `gluten-iceberg/src-iceberg5`, `gluten-iceberg/src-iceberg3`. The `spark-3.3` profile is gone from the six poms that declared one, and the root pom's `requireActiveProfile` list now names only the four surviving versions, so `-Pspark-3.3` fails with `Missing spark version profile` instead of quietly resolving to whatever the defaults are. `gluten-iceberg/src-iceberg3` needs a word since its name does not mention 3.3: no profile ever set `iceberg.binary.version=3`, so no build could reach it, and at the base commit its two files were byte-identical to `src-iceberg5`'s. It was already dead, and `src-iceberg5` was reachable only from the 3.3 profile, so both go together. `LICENSE` carried twelve entries pointing into `shims/spark33/` and `src-delta23/`. Those are removed, and the remaining 24 paths were checked to still resolve. Four CI jobs ran only on Spark 3.3 and are the sole coverage for what they exercise, so they are retargeted to 3.5 rather than deleted: `tpc-test-ubuntu-oom`, `tpc-test-ubuntu-randomkill`, `tpc-test-centos8-uniffle`, `tpc-test-ubuntu-2204-celeborn`. For celeborn the gluten-it install command moved with it, and it now takes the profile from `matrix.spark` like the Gluten build two steps above it. `spark-test-spark33` and `spark-test-spark33-slow` are deleted outright: 3.4 and 3.5 run the same profile set (`-Piceberg -Pdelta -Phudi -Ppaimon`), so no combination is orphaned. The `shims33` change-detection flag and its path filter go with them. The ClickHouse release chain hardcoded 3.3 end to end: `ep/build-clickhouse/src/package.sh` built only that version and `bin/gluten.sh` only recognised `jars/spark33`, so leaving it would have shipped a tarball whose launcher cannot find its own jars. Both move to 3.5, which also settles the `# TODO: support spark 3.5 later` sitting next to it. I have no ClickHouse environment, so that part rests on reading and `bash -n`. The ClickHouse CI triggers itself here (this PR touches `pom.xml`, `backends-clickhouse/**`, `shims/**` and more, which the trigger workflow's path filters match), but a CH maintainer's eyes on the packaging change would still help. Three things reviewers should know. `install-spark-resources.sh` no longer installs the 3.3 distribution, so `docker_image.yml` will rebuild the images and `/opt/shims/spark33/spark_home` disappears from them. Nothing in the repo references that path, but a branch that still does will break once the new image is published. `dev/release/build-release.sh` had a "Java 8 is required" gate whose only purpose was the Spark 3.3 build directly below it. With that build gone nothing in the script needs Java 8 (the four remaining builds all pass `-Pjava-17`, and the script installs Corretto 17 a few lines later), while the gate still rejects a release manager on JDK 17, and it does so after the longest step in the script. The design note for this work listed it as out of scope; I removed it here instead, since this change is what made it vestigial. `dev/release/package-release.sh` needed the same treatment for a different reason: its `SPARK_VERSIONS` list still demanded a 3.3 bundle jar that can no longer be built, and it hard-fails on a missing jar before it packages anything, so a release cut would have produced zero artifacts. Five test cases are pinned to 3.3 with `testWithMaxSparkVersion` / `testWithSpecifiedSparkVersion`, and those helpers skip the `test()` call entirely when no version matches, so from this PR until PR 3 they run nowhere and leave no skip marker: `VeloxParquetWriteForHiveSuite`'s `hive.exec.compress.output` case, `ClickhouseGlutenKafkaScanSuite`'s only case, and three in `GlutenClickHouseNativeWriteTableSuite`. Two carry TODOs saying they fail on 3.5, so each needs a decision (port or delete plus an issue) rather than a substitution, which is why they belong to PR 3. Deliberately left alone: the nineteen `ignoreClass` entries in `package/pom.xml` under "The overridden class list by Gluten" are now inert, because all five classes they name were overridden only in `shims/spark33`, but `banDuplicateClasses` runs with `fail=true` and dropping the entries could surface a duplicate from a dependency jar, so that is its own call. `tools/gluten-it/pom.xml` is a standalone pom without `requireActiveProfile`, so `-Pspark-3.3` there now warns and builds 3.5 rather than failing; adding the same gate would also outlaw the profile-less build that works today. And `tools/workload/benchmark_velox/initialize.ipynb` still provisions Spark 3.3.1 across five coupled lines including a symlink; I cannot test that notebook, so I left it rather than half-update it. Follow-ups under #12807: PR 2 lifts the shim methods that existed only for 3.3, PR 3 removes the residual version checks (`eqSpark33`, `AlignExpandOutputTypes`, the `NativeWritePostRule` registration, the `spark33` helper in the CH tests) and handles the five pinned cases above. ## How was this patch tested? Local verification, all against this branch: | check | result | |-|-| | `mvn validate -Pspark-3.3` | fails with `Missing spark version profile`, as intended | | `test-compile` for 3.4, 3.5, 4.0, 4.1 | all pass | | full-feature builds (`-Pceleborn,uniffle -Piceberg,delta,hudi,paimon`) | pass | | `-Pbackends-clickhouse` build | passes, which is what shows `src-delta` does not reference anything that lived only in `src-delta23` | | `dev/format-scala-code.sh --check` | passes, no `Missing spark version profile` | | `docker buildx build --check` on `Dockerfile.ubuntu22-m2-cache` | one `NoEmptyContinuation` warning, byte-for-byte the same warning the pre-change file produces | | `bash -n` on every changed script | passes | Two caveats on the local runs. The 3.5 builds needed `-Pscala-2.13` to get around corrupted `spark-*_2.12/3.5.5` jars in my local `~/.m2`, and that combination has to skip `-Ppaimon` because upstream never published `paimon-spark-3.5_2.13`; the 2.12 path is what CI exercises. `dev/check.py` could not run locally (it needs the `regex` module), but this PR adds no new files, so its license-header check has no subject. Unit tests are what CI settles, not a local run. The ClickHouse packaging changes are the part I am least able to verify myself. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude claude-opus-5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
