LuciferYang opened a new issue, #2750: URL: https://github.com/apache/uniffle/issues/2750
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/incubator-uniffle/issues?q=is%3Aissue) and found no similar issues. ### What would you like to be improved? Following #1805 (Spark 4.0.2 support, now closed), Uniffle should also build and run against **Spark 4.1**. Spark 4.1 introduces two binary-incompatible additions to APIs called from `client-spark/spark4`: 1. `org.apache.spark.scheduler.MapStatus.apply` — gained a trailing `checksumVal: Long` parameter. 2. `org.apache.spark.util.collection.ExternalSorter` — constructor gained a trailing `rowBasedChecksums: Array[RowBasedChecksum]` parameter. Scala default parameters do not surface to Java callers, so the current `client-spark/spark4` module compiled against Spark 4.0.2 cannot be linked against Spark 4.1.x without code changes. ### How should we improve? Reuse the existing `client-spark/spark4` module via a new `spark4.1` Maven profile, using a thin `Spark4Compat` shim selected at build time: - Add two parallel source roots: - `src/main/java-spark4_0/.../Spark4Compat.java` — 3-arg `MapStatus.apply`, 5-arg `ExternalSorter` ctor (current Spark 4.0 shape). - `src/main/java-spark4_1/.../Spark4Compat.java` — 4-arg `MapStatus.apply` (`checksumVal=0L`), 6-arg `ExternalSorter` ctor (empty `RowBasedChecksum[]`). - `build-helper-maven-plugin` selects the source root via `\${spark4.compat.source.dir}`; the existing `spark4` profile keeps the 4.0 default, the new `spark4.1` profile overrides it. - `RssShuffleReader` / `RssShuffleWriter` call into `Spark4Compat` instead of constructing these Spark types directly. - The `client-spark/extension` (Spark UI) module already uses `scala-jakarta` under the `spark4` profile; Spark 4.1's `WebUIPage.render` keeps the same `jakarta.servlet.http.HttpServletRequest` signature, so the existing source root works for `spark4.1` as well. - CI: add `spark4.1` (java-version 17) to both `parallel.yml` matrix and `sequential.yml`. `spark4` and `spark4.1` are mutually exclusive at build time — they share a Maven coordinate but produce incompatible bytecode against different Spark majors. ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
