SteNicholas opened a new pull request, #3743: URL: https://github.com/apache/celeborn/pull/3743
### What changes were proposed in this pull request? This PR removes Spark 2.4 client support from Celeborn. Specifically: - Delete the `client-spark/spark-2` and `client-spark/spark-2-shaded` modules, including the shuffle manager/reader/writer implementations (`SparkShuffleManager`, `CelebornShuffleReader`, `HashBasedShuffleWriter`, `SortBasedShuffleWriter`), `SparkUtils`, the fallback policy runner, shuffle handle, license/notice resources, and test suites. - Remove the `spark-2.4` Maven profile from `pom.xml` (and the related `tests/spark-it` profile) and the `Spark24` SBT project from `project/CelebornBuild.scala`. - Drop `spark-2.4` from the build and release scripts (`build/make-distribution.sh`, `build/release/release.sh`, `dev/dependencies.sh`, `dev/reformat`) and from CI workflows (`deps.yml`, `license.yml`, `maven.yml`, `sbt.yml`, `style.yml`). - Update documentation (`README.md`, `docs/developers/sbt.md`) to reflect the supported Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0). ### Why are the changes needed? Spark 2.4 is an old release that is no longer maintained upstream. Dropping it reduces the build/test matrix, maintenance burden, and release artifacts, allowing the project to focus on currently supported Spark versions (3.0 and above). ### Does this PR resolve a correctness bug? <!-- Check if yes. The `correctness` label will be added/removed automatically. --> - [ ] Yes ### Does this PR introduce _any_ user-facing change? <!-- Check if yes. --> - [x] Yes Celeborn no longer ships or builds the Spark 2.4 client jar. Users running Spark 2.4 must either upgrade to a supported Spark version (3.0+) or stay on a previous Celeborn release. ### How was this patch tested? Existing CI for the remaining Spark versions (3.0/3.1/3.2/3.3/3.4/3.5/4.0) via the updated Maven/SBT/style/license/deps workflows. No new functionality is introduced; the change only removes code, build profiles, and references. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
