LuciferYang opened a new pull request, #2751:
URL: https://github.com/apache/uniffle/pull/2751
### What changes were proposed in this pull request?
Add a `spark4.1` Maven profile that reuses the existing
`client-spark/spark4` module to compile and run against **Spark 4.1.1**.
Spark 4.1 introduced two binary-incompatible additions to APIs called from
this module:
- `org.apache.spark.scheduler.MapStatus.apply` — gained a trailing
`checksumVal: Long` parameter.
- `org.apache.spark.util.collection.ExternalSorter` — constructor gained a
trailing `rowBasedChecksums: Array[RowBasedChecksum]` parameter.
Scala default parameters do not surface to Java callers, so a single
`Spark4Compat.java` cannot satisfy both 4.0.x and 4.1.x signatures. Two
parallel source roots ship a matching `Spark4Compat`:
- `src/main/java-spark4_0/` — 3-arg `mapStatus`, 5-arg `ExternalSorter` ctor
(current Spark 4.0 shape).
- `src/main/java-spark4_1/` — 4-arg `mapStatus` (`checksumVal = 0L`), 6-arg
`ExternalSorter` ctor (empty `RowBasedChecksum[]`).
`build-helper-maven-plugin` selects the source root via
`${spark4.compat.source.dir}`; the existing `spark4` profile keeps the 4.0
default, the new `spark4.1` profile overrides it to the 4.1 variant.
`RssShuffleReader` now calls `Spark4Compat.newExternalSorter(...)` and
`RssShuffleWriter` calls `Spark4Compat.mapStatus(...)` instead of constructing
those Spark types directly.
The `client-spark/extension` (Spark UI) module already uses `scala-jakarta`
under the `spark4` profile; Spark 4.1's `WebUIPage.render` keeps the same
`jakarta.servlet.http.HttpServletRequest` signature, so the existing source
root works for `spark4.1` as well — only a `spark4.1` profile body that mirrors
the `spark4` one is added.
`spark4` and `spark4.1` are mutually exclusive at build time; they share a
Maven coordinate but produce incompatible bytecode against different Spark
majors. Run `mvn clean` between profile switches locally.
### Why are the changes needed?
Following #1805 (Spark 4.0.2 support, now closed), users on Spark 4.1
currently cannot link Uniffle's Spark 4 client because of the API additions
above. Closes #2750.
### Does this PR introduce _any_ user-facing change?
No (build / packaging only). A new `-Pspark4.1` build flag is available;
default builds and the existing `-Pspark4` flag are unchanged.
### How was this patch tested?
- Local: `mvn clean package -Pspark4.1 -DskipTests` against Spark 4.1.1 —
passes.
- Local: `mvn clean package -Pspark4 -DskipTests` against Spark 4.0.2 —
still passes.
- Verified the resulting `Spark4Compat.class` differs across the two
profiles (different bytecode size / arity), confirming the multi-source-root
selection works.
- Verified API signatures via `javap -p` on Spark 4.0.2 / 4.1.1 jars from
Maven Central match the shim implementations.
- CI: `parallel.yml` now runs `-Pspark4.1` (java-version 17) in the matrix;
`sequential.yml` adds an `Execute -Pspark4.1` step gated on java-version 17.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
