This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 3b9f25b62 [KYUUBI #6197] Revise dependency management of Spark authZ
plugin
3b9f25b62 is described below
commit 3b9f25b62d43600274af89fccfec0ecc31283f1c
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Mar 22 10:30:30 2024 +0800
[KYUUBI #6197] Revise dependency management of Spark authZ plugin
# :mag: Description
## Issue References ๐
The POM of `kyuubi-spark-authz-shaded` is redundant, just pull
`kyuubi-spark-authz` is necessary.
The current dependency management does not work on Ranger 2.1.0, this patch
cleans up the POM definition and fixes the compatibility with Ranger 2.1.0
## Describe Your Solution ๐ง
Carefully revise the dependency list and exclusion.
## Types of changes :bookmark:
- [x] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
## Test Plan ๐งช
perform packing kyuubi-spark-authz-shaded module.
```
build/mvn clean install -pl extensions/spark/kyuubi-spark-authz-shaded -am
-DskipTests
```
before
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default)
kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including
org.apache.kyuubi:kyuubi-spark-authz_2.12:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including
org.apache.kyuubi:kyuubi-util-scala_2.12:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the
shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the
shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the
shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the
shaded jar.
```
after
```
[INFO] --- maven-shade-plugin:3.5.2:shade (default)
kyuubi-spark-authz-shaded_2.12 ---
[INFO] Including
org.apache.kyuubi:kyuubi-spark-authz_2.12:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including
org.apache.kyuubi:kyuubi-util-scala_2.12:jar:1.10.0-SNAPSHOT in the shaded jar.
[INFO] Including org.apache.kyuubi:kyuubi-util:jar:1.10.0-SNAPSHOT in the
shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-common:jar:2.4.0 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the
shaded jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the
shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-cred:jar:2.4.0 in the
shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.19.4 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-core:jar:1.19.4 in the shaded jar.
[INFO] Including com.kstruct:gethostname4j:jar:1.0.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna:jar:5.7.0 in the shaded jar.
[INFO] Including net.java.dev.jna:jna-platform:jar:5.7.0 in the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugin-classloader:jar:2.4.0 in
the shaded jar.
[INFO] Including org.apache.ranger:ranger-plugins-audit:jar:2.4.0 in the
shaded jar.
```
---
# Checklist ๐
- [x] This patch was not authored or co-authored using [Generative
Tooling](https://www.apache.org/legal/generative-tooling.html)
**Be nice. Be informative.**
Closes #6197 from pan3793/authz-dep.
Closes #6197
d0becabce [Cheng Pan] 2.4
47e38502a [Cheng Pan] ranger 2.4
af01f7ed5 [Cheng Pan] test ranger 2.1
203aff3b3 [Cheng Pan] ranger-plugins-cred
974d76b03 [Cheng Pan] Resive dependency management of authz
e5154f30f [Cheng Pan] improve authz deps
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
extensions/spark/kyuubi-spark-authz-shaded/pom.xml | 210 +--------------------
extensions/spark/kyuubi-spark-authz/pom.xml | 85 ++++++---
pom.xml | 1 -
3 files changed, 60 insertions(+), 236 deletions(-)
diff --git a/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
b/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
index 4ff50edbc..d7bfd3db8 100644
--- a/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
+++ b/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
@@ -30,197 +30,12 @@
<name>Kyuubi Dev Spark Authorization Extension Shaded</name>
<url>https://kyuubi.apache.org/</url>
- <properties>
- <!-- the following components' version may need to tune to align w/
the ranger.version-->
- <gethostname4j.version>1.0.0</gethostname4j.version>
- <jersey.client.version>1.19.4</jersey.client.version>
- <jna.version>5.7.0</jna.version>
- </properties>
-
<dependencies>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
- <dependency>
- <groupId>org.apache.kyuubi</groupId>
- <artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
- <version>${project.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.ranger</groupId>
- <artifactId>ranger-plugins-common</artifactId>
- <version>${ranger.version}</version>
- <exclusions>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-bundle</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.ranger</groupId>
- <artifactId>ranger-plugin-classloader</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.ranger</groupId>
- <artifactId>ranger-plugins-audit</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>ch.qos.logback</groupId>
- <artifactId>logback-classic</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-lang</groupId>
- <artifactId>commons-lang</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-logging</groupId>
- <artifactId>commons-logging</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>javax.ws.rs</groupId>
- <artifactId>jsr311-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.kstruct</groupId>
- <artifactId>gethostname4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>net.java.dev.jna</groupId>
- <artifactId>jna</artifactId>
- </exclusion>
- <exclusion>
- <groupId>net.java.dev.jna</groupId>
- <artifactId>jna-platform</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-client</artifactId>
- <version>${jersey.client.version}</version>
- <exclusions>
- <exclusion>
- <groupId>javax.ws.rs</groupId>
- <artifactId>jsr311-api</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
-
- <dependency>
- <groupId>com.kstruct</groupId>
- <artifactId>gethostname4j</artifactId>
- <version>${gethostname4j.version}</version>
- </dependency>
-
- <dependency>
- <groupId>net.java.dev.jna</groupId>
- <artifactId>jna</artifactId>
- <version>${jna.version}</version>
- </dependency>
-
- <dependency>
- <groupId>net.java.dev.jna</groupId>
- <artifactId>jna-platform</artifactId>
- <version>${jna.version}</version>
- </dependency>
-
- <dependency>
- <groupId>org.apache.ranger</groupId>
- <artifactId>ranger-plugins-audit</artifactId>
- <version>${ranger.version}</version>
- <exclusions>
- <exclusion>
- <groupId>org.apache.ranger</groupId>
- <artifactId>ranger-plugins-cred</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.kafka</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.solr</groupId>
- <artifactId>solr-solrj</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.elasticsearch</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.elasticsearch.client</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.elasticsearch.plugin</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.lucene</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>log4j</groupId>
- <artifactId>log4j</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-lang</groupId>
- <artifactId>commons-lang</artifactId>
- </exclusion>
- <exclusion>
- <groupId>commons-logging</groupId>
- <artifactId>commons-logging</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.carrotsearch</groupId>
- <artifactId>hppc</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hive</groupId>
- <artifactId>hive-storage-api</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.orc</groupId>
- <artifactId>orc-core</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-common</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.google.guava</groupId>
- <artifactId>guava</artifactId>
- </exclusion>
- <exclusion>
- <groupId>joda-time</groupId>
- <artifactId>joda-time</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>*</artifactId>
- </exclusion>
- <exclusion>
- <groupId>com.amazonaws</groupId>
- <artifactId>aws-java-sdk-bundle</artifactId>
- </exclusion>
- </exclusions>
- </dependency>
</dependencies>
<build>
@@ -232,18 +47,12 @@
<shadedArtifactAttached>false</shadedArtifactAttached>
<artifactSet>
<includes>
-
<include>org.apache.kyuubi:kyuubi-util-scala_${scala.binary.version}</include>
-
<include>org.apache.kyuubi:kyuubi-spark-authz_${scala.binary.version}</include>
- <include>org.apache.kyuubi:kyuubi-util</include>
-
<include>org.apache.ranger:ranger-plugins-common</include>
-
<include>org.apache.ranger:ranger-plugins-audit</include>
-
<include>org.apache.ranger:ranger-plugins-cred</include>
-
<include>org.codehaus.jackson:jackson-jaxrs</include>
-
<include>org.codehaus.jackson:jackson-core-asl</include>
-
<include>org.codehaus.jackson:jackson-mapper-asl</include>
- <include>com.sun.jersey:jersey-client</include>
- <include>com.sun.jersey:jersey-core</include>
+ <include>org.apache.kyuubi:*</include>
+ <include>org.apache.ranger:*</include>
+ <include>org.codehaus.jackson:*</include>
+ <include>com.sun.jersey:*</include>
<include>com.kstruct:gethostname4j</include>
+ <!-- JNA is the transitive dependency of
gethostname4j -->
<include>net.java.dev.jna:jna</include>
<include>net.java.dev.jna:jna-platform</include>
</includes>
@@ -287,15 +96,6 @@
<pattern>com.kstruct.gethostname4j</pattern>
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
</relocation>
- <relocation>
- <pattern>org.apache.hadoop.security</pattern>
-
<shadedPattern>${kyuubi.shade.packageName}.org.apache.hadoop.security</shadedPattern>
- <includes>
-
<include>org.apache.hadoop.security.KrbPasswordSaverLoginModule</include>
-
<include>org.apache.hadoop.security.SecureClientLogin</include>
-
<include>org.apache.hadoop.security.SecureClientLoginConfiguration</include>
- </includes>
- </relocation>
</relocations>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
diff --git a/extensions/spark/kyuubi-spark-authz/pom.xml
b/extensions/spark/kyuubi-spark-authz/pom.xml
index 7c6e30d1d..978dad143 100644
--- a/extensions/spark/kyuubi-spark-authz/pom.xml
+++ b/extensions/spark/kyuubi-spark-authz/pom.xml
@@ -32,6 +32,7 @@
<url>https://kyuubi.apache.org/</url>
<properties>
+ <ranger.version>2.4.0</ranger.version>
<!-- the following components' version may need to tune to align w/
the ranger.version-->
<gethostname4j.version>1.0.0</gethostname4j.version>
<jersey.client.version>1.19.4</jersey.client.version>
@@ -49,10 +50,6 @@
<artifactId>ranger-plugins-common</artifactId>
<version>${ranger.version}</version>
<exclusions>
- <exclusion>
- <groupId>com.sun.jersey</groupId>
- <artifactId>jersey-bundle</artifactId>
- </exclusion>
<exclusion>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugin-classloader</artifactId>
@@ -61,6 +58,11 @@
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
</exclusion>
+ <!-- this is going to be replaced with jersey-client -->
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-bundle</artifactId>
+ </exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
@@ -70,17 +72,13 @@
<artifactId>logback-classic</artifactId>
</exclusion>
<exclusion>
- <groupId>commons-lang</groupId>
- <artifactId>commons-lang</artifactId>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-configuration2</artifactId>
</exclusion>
<exclusion>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</exclusion>
- <exclusion>
- <groupId>org.apache.httpcomponents</groupId>
- <artifactId>*</artifactId>
- </exclusion>
<exclusion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
@@ -89,14 +87,6 @@
<groupId>javax.ws.rs</groupId>
<artifactId>jsr311-api</artifactId>
</exclusion>
- <exclusion>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-core-asl</artifactId>
- </exclusion>
- <exclusion>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- </exclusion>
<exclusion>
<groupId>com.kstruct</groupId>
<artifactId>gethostname4j</artifactId>
@@ -142,6 +132,12 @@
<version>${jna.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugin-classloader</artifactId>
+ <version>${ranger.version}</version>
+ </dependency>
+
<dependency>
<groupId>org.apache.ranger</groupId>
<artifactId>ranger-plugins-audit</artifactId>
@@ -226,6 +222,35 @@
</exclusions>
</dependency>
+ <dependency>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugins-cred</artifactId>
+ <version>${ranger.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.commons</groupId>
+ <artifactId>commons-configuration2</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <!-- they were removed in RANGER-3184 (2.2.0) -->
+ <exclusion>
+ <groupId>org.eclipse.persistence</groupId>
+ <artifactId>javax.persistence</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.eclipse.persistence</groupId>
+ <artifactId>eclipselink</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
@@ -256,18 +281,6 @@
<scope>provided</scope>
</dependency>
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-api</artifactId>
- <scope>provided</scope>
- </dependency>
-
- <dependency>
- <groupId>org.apache.logging.log4j</groupId>
- <artifactId>log4j-core</artifactId>
- <scope>provided</scope>
- </dependency>
-
<dependency>
<groupId>commons-collections</groupId>
<artifactId>commons-collections</artifactId>
@@ -294,6 +307,18 @@
<!-- for hive related test only -->
</dependency>
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-api</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>log4j-core</artifactId>
+ <scope>test</scope>
+ </dependency>
+
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j-impl</artifactId>
diff --git a/pom.xml b/pom.xml
index 3f6075626..8aeebd4df 100644
--- a/pom.xml
+++ b/pom.xml
@@ -186,7 +186,6 @@
<prometheus.version>0.16.0</prometheus.version>
<protobuf.version>3.21.7</protobuf.version>
<py4j.version>0.10.7</py4j.version>
- <ranger.version>2.4.0</ranger.version>
<scalatest.version>3.2.16</scalatest.version>
<scalatestplus.version>3.2.16.0</scalatestplus.version>
<scopt.version>4.1.0</scopt.version>
