This is an automated email from the ASF dual-hosted git repository.
baibaichen pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 4cda2fbc98 [VL] Spark 4.2: align profile dependency versions with
Spark 4.2.0 (#13161)
4cda2fbc98 is described below
commit 4cda2fbc9843c99587b4ab80f411e8a84d4f589f
Author: Akshay Tayal <[email protected]>
AuthorDate: Wed Sep 30 10:19:33 2026 -0700
[VL] Spark 4.2: align profile dependency versions with Spark 4.2.0 (#13161)
Align every Spark-provided library the Gluten spark-4.2 build resolves with
the
versions Spark 4.2.0 ships (verified against apache/spark v4.2.0 pom.xml
and via
mvn help:evaluate -Pspark-4.2,scala-2.13), plus fix the compile break the
Hadoop
bump introduces.
Effective versions for a spark-4.2 build, now all matching Spark 4.2.0:
- Scala 2.13.17 -> 2.13.18 (scoped to spark-4.2 profile; overrides
the
shared scala-2.13 profile only for 4.2, other Spark
versions keep 2.13.17)
- Hadoop 3.4.1 -> 3.5.0
- Jackson 2.18.2 -> 2.21.2 (annotations pinned to 2.21; no 2.21.2
patch exists)
- slf4j 2.0.16 -> 2.0.17
- Guava 33.4.0-jre -> 33.6.0-jre
- log4j 2.24.3 -> 2.25.4
- commons-lang3 3.17.0 -> 3.20.0
- antlr4 4.13.1, arrow 19.0.0: already matched, unchanged.
jackson-annotations has no 2.21.2 patch release, so it is pinned separately
to
2.21 via a new fasterxml.annotations.version property (defaults to
${fasterxml.version} globally -> no-op for other profiles).
VeloxSparkPlanExecApi.scala: Hadoop 3.5.0 drops javax.ws.rs from the compile
classpath, so the spill-path URI rewrite switches from
javax.ws.rs.core.UriBuilder
to java.net.URI (JDK-native, same encoded result).
Delta/Iceberg/Hudi are external integrations (not Spark-bundled and not in
Spark's
release), so they are intentionally left unchanged.
Co-authored-by: Akshay Tayal <[email protected]>
---
.../backendsapi/velox/VeloxSparkPlanExecApi.scala | 8 ++-----
pom.xml | 25 ++++++++++++++--------
2 files changed, 18 insertions(+), 15 deletions(-)
diff --git
a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
index b862107075..38f2bdbd38 100644
---
a/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
+++
b/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxSparkPlanExecApi.scala
@@ -66,8 +66,7 @@ import org.apache.spark.task.TaskResources
import io.substrait.proto.JoinRel
import org.apache.commons.lang3.ClassUtils
-import javax.ws.rs.core.UriBuilder
-
+import java.net.URI
import java.util.{ArrayList => JArrayList, List => JList}
import java.util.Locale
@@ -1379,10 +1378,7 @@ class VeloxSparkPlanExecApi extends SparkPlanExecApi
with Logging {
case "local" =>
path
case "heap-over-local" =>
- val rewritten = UriBuilder
- .fromPath(path)
- .scheme("jol")
- .toString
+ val rewritten = new URI("jol", null, path, null, null).toString
rewritten
case other =>
throw new IllegalStateException(s"Unsupported fs: $other")
diff --git a/pom.xml b/pom.xml
index e01cc34d9a..98025b89e9 100644
--- a/pom.xml
+++ b/pom.xml
@@ -124,6 +124,7 @@
<scala.release.version>${java.version}</scala.release.version>
<!-- For unit tests -->
<fasterxml.version>2.22</fasterxml.version>
+
<fasterxml.annotations.version>${fasterxml.version}</fasterxml.annotations.version>
<junit.version>4.13.1</junit.version>
<!-- Maven version for build/mvn wrapper script -->
<maven.version>3.9.16</maven.version>
@@ -462,7 +463,7 @@
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
- <version>${fasterxml.version}</version>
+ <version>${fasterxml.annotations.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
@@ -1451,6 +1452,9 @@
<sparkbundle.version>4.2</sparkbundle.version>
<sparkshim.artifactId>spark-sql-columnar-shims-spark42</sparkshim.artifactId>
<spark.version>4.2.0</spark.version>
+ <!-- Spark 4.2.0 uses Scala 2.13.18; override the shared scala-2.13
+ profile (2.13.17) for Spark 4.2 builds only. -->
+ <scala.version>2.13.18</scala.version>
<!-- No Spark 4.2 artifacts published for Iceberg/Delta/Hudi yet.
These values are
placeholders inherited from spark-4.1; the delta/iceberg/hudi
profiles are
opt-in (activeByDefault=false) and are NOT enabled on the Spark
4.2 CI lane. -->
@@ -1459,15 +1463,18 @@
<delta.version>4.1.0</delta.version>
<delta.binary.version>40</delta.binary.version>
<hudi.version>1.2.0</hudi.version>
- <fasterxml.version>2.18.2</fasterxml.version>
- <fasterxml.jackson.version>2.18.2</fasterxml.jackson.version>
-
<fasterxml.jackson.databind.version>2.18.2</fasterxml.jackson.databind.version>
- <hadoop.version>3.4.1</hadoop.version>
+ <!-- Spark 4.2.0 ships Jackson 2.21.2; jackson-annotations has no
2.21.2
+ patch release, so it is pinned separately to 2.21. -->
+ <fasterxml.version>2.21.2</fasterxml.version>
+ <fasterxml.annotations.version>2.21</fasterxml.annotations.version>
+ <fasterxml.jackson.version>2.21.2</fasterxml.jackson.version>
+
<fasterxml.jackson.databind.version>2.21.2</fasterxml.jackson.databind.version>
+ <hadoop.version>3.5.0</hadoop.version>
<antlr4.version>4.13.1</antlr4.version>
- <guava.version>33.4.0-jre</guava.version>
- <slf4j.version>2.0.16</slf4j.version>
- <log4j.version>2.24.3</log4j.version>
- <commons-lang3.version>3.17.0</commons-lang3.version>
+ <guava.version>33.6.0-jre</guava.version>
+ <slf4j.version>2.0.17</slf4j.version>
+ <log4j.version>2.25.4</log4j.version>
+ <commons-lang3.version>3.20.0</commons-lang3.version>
<!-- Must track Spark 4.2's own Arrow (19.0.0): scope is provided and
Gluten
deliberately lets Spark's Arrow through. -->
<arrow.version>19.0.0</arrow.version>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
