This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new f1cf1e42d [KYUUBI #6131] Simplify Maven dependency management after
dropping building support for Spark 3.1
f1cf1e42d is described below
commit f1cf1e42dea401ef275d1e56663df12c33bdfb1b
Author: Cheng Pan <[email protected]>
AuthorDate: Wed Mar 6 22:31:06 2024 +0800
[KYUUBI #6131] Simplify Maven dependency management after dropping building
support for Spark 3.1
# :mag: Description
## Issue References ๐
SPARK-33212 (fixed in 3.2.0) moves from `hadoop-client` to shaded hadoop
client, to simplify the dependency management, previously , we add some
workaround to handle Spark 3.1 dependency issues. As we removed building
support for Spark 3.1 now, we can remove those workaround to simplify `pom.xml`
## Describe Your Solution ๐ง
As above.
## Types of changes :bookmark:
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
Pass GA.
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6131 from pan3793/3-1-cleanup.
Closes #6131
1341065a7 [Cheng Pan] nit
1d7323f6e [Cheng Pan] fix
9e2e3b747 [Cheng Pan] nit
271166b58 [Cheng Pan] test
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.../spark/kyuubi-extension-spark-3-2/pom.xml | 6 ----
.../spark/kyuubi-extension-spark-3-3/pom.xml | 6 ----
.../spark/kyuubi-extension-spark-3-4/pom.xml | 6 ----
.../spark/kyuubi-extension-spark-3-5/pom.xml | 6 ----
.../spark/kyuubi-extension-spark-common/pom.xml | 6 ----
.../spark/kyuubi-spark-connector-hive/pom.xml | 6 ----
.../spark/kyuubi-spark-connector-tpcds/pom.xml | 6 ----
.../spark/kyuubi-spark-connector-tpch/pom.xml | 6 ----
pom.xml | 35 ++++------------------
9 files changed, 5 insertions(+), 78 deletions(-)
diff --git a/extensions/spark/kyuubi-extension-spark-3-2/pom.xml
b/extensions/spark/kyuubi-extension-spark-3-2/pom.xml
index 6cabd9a1d..e2139ecb7 100644
--- a/extensions/spark/kyuubi-extension-spark-3-2/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-2/pom.xml
@@ -103,12 +103,6 @@
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
diff --git a/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
b/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
index 9d2330309..1e744f304 100644
--- a/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-3/pom.xml
@@ -118,12 +118,6 @@
<artifactId>hadoop-client-runtime</artifactId>
<scope>test</scope>
</dependency>
-
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
diff --git a/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
b/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
index 59232e588..c872daee7 100644
--- a/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-4/pom.xml
@@ -105,12 +105,6 @@
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
diff --git a/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
b/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
index 09e04609a..96fbd4d16 100644
--- a/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-3-5/pom.xml
@@ -105,12 +105,6 @@
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
diff --git a/extensions/spark/kyuubi-extension-spark-common/pom.xml
b/extensions/spark/kyuubi-extension-spark-common/pom.xml
index a5cc6333b..83d4869dc 100644
--- a/extensions/spark/kyuubi-extension-spark-common/pom.xml
+++ b/extensions/spark/kyuubi-extension-spark-common/pom.xml
@@ -88,12 +88,6 @@
<scope>test</scope>
</dependency>
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
-
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-1.2-api</artifactId>
diff --git a/extensions/spark/kyuubi-spark-connector-hive/pom.xml
b/extensions/spark/kyuubi-spark-connector-hive/pom.xml
index c74bab609..c270344f8 100644
--- a/extensions/spark/kyuubi-spark-connector-hive/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-hive/pom.xml
@@ -110,12 +110,6 @@
<artifactId>hadoop-client-runtime</artifactId>
<scope>test</scope>
</dependency>
-
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
diff --git a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
index 1dd3d664d..842355ca0 100644
--- a/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpcds/pom.xml
@@ -128,12 +128,6 @@
<artifactId>hadoop-client-runtime</artifactId>
<scope>test</scope>
</dependency>
-
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
diff --git a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
index e73e1d5ba..83f768f5f 100644
--- a/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
+++ b/extensions/spark/kyuubi-spark-connector-tpch/pom.xml
@@ -127,12 +127,6 @@
<artifactId>hadoop-client-runtime</artifactId>
<scope>test</scope>
</dependency>
-
- <dependency>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
diff --git a/pom.xml b/pom.xml
index c3e01b5cb..026ec6c6e 100644
--- a/pom.xml
+++ b/pom.xml
@@ -438,11 +438,6 @@
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
<exclusions>
- <!-- Use Hadoop Shaded Client to gain more clean
transitive dependencies -->
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- </exclusion>
<!-- Use log4j2 -->
<exclusion>
<groupId>log4j</groupId>
@@ -476,13 +471,6 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${spark.version}</version>
- <exclusions>
- <!-- Use Hadoop Shaded Client to gain more clean
transitive dependencies -->
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </exclusion>
- </exclusions>
</dependency>
<dependency>
@@ -491,11 +479,6 @@
<version>${spark.version}</version>
<type>test-jar</type>
<exclusions>
- <!-- Use Hadoop Shaded Client to gain more clean
transitive dependencies -->
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
- </exclusion>
<!-- Use log4j2 -->
<exclusion>
<groupId>log4j</groupId>
@@ -1221,10 +1204,8 @@
</dependency>
<!--
- Required for JDK 11. Previous as a transitive dependency of
hadoop-client,
- since SPARK-33212(will be included in Apache Spark 3.2), Apache
Spark moved
- to Hadoop Shaded Client, and Kyuubi always uses Hadoop Shaded
Client, thus
- we need explicitly add this dependency.
+ Required by Hadoop on JDK 11 or later. Could be omitted on
modules already
+ pull Spark dependencies as it is already a transitive dependency
of spark-core.
-->
<dependency>
<groupId>jakarta.xml.bind</groupId>
@@ -1233,8 +1214,8 @@
</dependency>
<!--
- Required for Yarn Mini Cluster on JDK 11, it is already a
transitive dependency
- of spark-core, thus only should explicitly add on Kyuubi Server
Main test scope.
+ Required by YARN Mini Cluster on JDK 11 or later. Could be
omitted on modules already
+ pull Spark dependencies as it is already a transitive dependency
of spark-core.
-->
<dependency>
<groupId>jakarta.activation</groupId>
@@ -1242,7 +1223,7 @@
<version>${jakarta.activation.version}</version>
</dependency>
- <!-- Zookeeper Cli required -->
+ <!-- Required by Zookeeper Cli and Beeline -->
<dependency>
<groupId>jline</groupId>
<artifactId>jline</artifactId>
@@ -1291,12 +1272,6 @@
<groupId>org.glassfish.jersey.core</groupId>
<artifactId>jersey-server</artifactId>
<version>${jersey.version}</version>
- <exclusions>
- <exclusion>
- <groupId>jakarta.xml.bind</groupId>
- <artifactId>jakarta.xml.bind-api</artifactId>
- </exclusion>
- </exclusions>
</dependency>
<dependency>
