This is an automated email from the ASF dual-hosted git repository.
yikaifei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new 59c25b985 [KYUUBI #5427] [AUTHZ] Shade spark authz plugin
59c25b985 is described below
commit 59c25b9851ba73a585b3b622404f81f60ebc7c28
Author: yikaifei <[email protected]>
AuthorDate: Fri Oct 20 20:10:34 2023 +0800
[KYUUBI #5427] [AUTHZ] Shade spark authz plugin
### _Why are the changes needed?_
This PR aims to shade the kyuubi spark authz plugin to simplify the user's
use.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/contributing/code/testing.html#running-tests)
locally before make a pull request
### _Was this patch authored or co-authored using generative AI tooling?_
No
Closes #5427 from Yikf/shade-authz.
Closes #5427
d2f7ea8d1 [yikaifei] fix
695133de4 [Kent Yao] Update docs/security/authorization/spark/install.md
f3a653133 [Kent Yao] Update docs/security/authorization/spark/build.md
963cab372 [yikaifei] bundle
2068c98fc [yikaifei] relocation
6c6e50ea7 [yikaifei] Shade spark authz plugin
Lead-authored-by: yikaifei <[email protected]>
Co-authored-by: Kent Yao <[email protected]>
Signed-off-by: yikaifei <[email protected]>
---
docs/security/authorization/spark/build.md | 13 +
docs/security/authorization/spark/install.md | 2 +-
extensions/spark/kyuubi-spark-authz-shaded/pom.xml | 317 +++++++++++++++++++++
pom.xml | 1 +
4 files changed, 332 insertions(+), 1 deletion(-)
diff --git a/docs/security/authorization/spark/build.md
b/docs/security/authorization/spark/build.md
index aa7fc18da..17e8e00f4 100644
--- a/docs/security/authorization/spark/build.md
+++ b/docs/security/authorization/spark/build.md
@@ -31,6 +31,19 @@ After a while, if everything goes well, you will get the
plugin finally in two p
- The main plugin jar, which is under
`./extensions/spark/kyuubi-spark-authz/target/kyuubi-spark-authz_${scala.binary.version}-${project.version}.jar`
- The least transitive dependencies needed, which are under
`./extensions/spark/kyuubi-spark-authz/target/scala-${scala.binary.version}/jars`
+## Build shaded jar with Apache Maven
+
+Apache Kyuubi also provides the shaded jar for the Spark AuthZ plugin, You can
run the AuthZ plugin using just a shaded jar without the additional dependency
of jars,
+To build it, `cd` to the root direct of kyuubi project and run:
+
+```shell
+build/mvn clean package -pl :kyuubi-spark-authz-shaded_2.12 -DskipTests -am
+```
+
+After a while, if everything goes well, you will get the plugin finally:
+
+- The shaded AuthZ plugin jar, which is under
`./extensions/spark/kyuubi-spark-authz-shaded/target/kyuubi-spark-authz-shaded_${scala.binary.version}-${project.version}.jar`
+
### Build against Different Apache Spark Versions
The maven option `spark.version` is used for specifying Spark version to
compile with and generate corresponding transitive dependencies.
diff --git a/docs/security/authorization/spark/install.md
b/docs/security/authorization/spark/install.md
index f820f53c4..ff4131c6f 100644
--- a/docs/security/authorization/spark/install.md
+++ b/docs/security/authorization/spark/install.md
@@ -31,7 +31,7 @@
## Install
-With the `kyuubi-spark-authz_*.jar` and its transitive dependencies available
for spark runtime classpath, such as
+Use either the shaded jar `kyuubi-spark-authz-shaded_*.jar` or the
`kyuubi-spark-authz_*.jar` with its transitive dependencies available for spark
runtime classpath, such as
- Copied to `$SPARK_HOME/jars`, or
- Specified to `spark.jars` configuration
diff --git a/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
b/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
new file mode 100644
index 000000000..b135a1d7c
--- /dev/null
+++ b/extensions/spark/kyuubi-spark-authz-shaded/pom.xml
@@ -0,0 +1,317 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one or more
+ ~ contributor license agreements. See the NOTICE file distributed with
+ ~ this work for additional information regarding copyright ownership.
+ ~ The ASF licenses this file to You under the Apache License, Version 2.0
+ ~ (the "License"); you may not use this file except in compliance with
+ ~ the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing, software
+ ~ distributed under the License is distributed on an "AS IS" BASIS,
+ ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ ~ See the License for the specific language governing permissions and
+ ~ limitations under the License.
+ -->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <parent>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-parent</artifactId>
+ <version>1.9.0-SNAPSHOT</version>
+ <relativePath>../../../pom.xml</relativePath>
+ </parent>
+
+ <artifactId>kyuubi-spark-authz-shaded_${scala.binary.version}</artifactId>
+ <packaging>jar</packaging>
+ <name>Kyuubi Dev Spark Authorization Extension Shaded</name>
+ <url>https://kyuubi.apache.org/</url>
+
+ <properties>
+ <!-- the following components' version may need to tune to align w/
the ranger.version-->
+ <gethostname4j.version>1.0.0</gethostname4j.version>
+ <jersey.client.version>1.19.4</jersey.client.version>
+ <jna.version>5.7.0</jna.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-util-scala_${scala.binary.version}</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugins-common</artifactId>
+ <version>${ranger.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-bundle</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugin-classloader</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugins-audit</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>ch.qos.logback</groupId>
+ <artifactId>logback-classic</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>javax.ws.rs</groupId>
+ <artifactId>jsr311-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-mapper-asl</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.kstruct</groupId>
+ <artifactId>gethostname4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>jna</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>jna-platform</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>com.sun.jersey</groupId>
+ <artifactId>jersey-client</artifactId>
+ <version>${jersey.client.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>javax.ws.rs</groupId>
+ <artifactId>jsr311-api</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+
+ <dependency>
+ <groupId>com.kstruct</groupId>
+ <artifactId>gethostname4j</artifactId>
+ <version>${gethostname4j.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>jna</artifactId>
+ <version>${jna.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>net.java.dev.jna</groupId>
+ <artifactId>jna-platform</artifactId>
+ <version>${jna.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugins-audit</artifactId>
+ <version>${ranger.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.ranger</groupId>
+ <artifactId>ranger-plugins-cred</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.kafka</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.solr</groupId>
+ <artifactId>solr-solrj</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.elasticsearch</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.elasticsearch.client</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.elasticsearch.plugin</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.lucene</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>log4j</groupId>
+ <artifactId>log4j</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>commons-logging</groupId>
+ <artifactId>commons-logging</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.carrotsearch</groupId>
+ <artifactId>hppc</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.httpcomponents</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-storage-api</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.orc</groupId>
+ <artifactId>orc-core</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.hadoop</groupId>
+ <artifactId>hadoop-common</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.google.guava</groupId>
+ <artifactId>guava</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>joda-time</groupId>
+ <artifactId>joda-time</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.apache.logging.log4j</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-java-sdk-bundle</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <configuration>
+ <shadedArtifactAttached>false</shadedArtifactAttached>
+ <artifactSet>
+ <includes>
+
<include>org.apache.kyuubi:kyuubi-util-scala_${scala.binary.version}</include>
+
<include>org.apache.kyuubi:kyuubi-spark-authz_${scala.binary.version}</include>
+ <include>org.apache.kyuubi:kyuubi-util</include>
+
<include>org.apache.ranger:ranger-plugins-common</include>
+
<include>org.apache.ranger:ranger-plugins-audit</include>
+
<include>org.codehaus.jackson:jackson-jaxrs</include>
+ <include>com.sun.jersey:jersey-client</include>
+ <include>com.sun.jersey:jersey-core</include>
+ <include>com.kstruct:gethostname4j</include>
+ <include>net.java.dev.jna:jna</include>
+ <include>net.java.dev.jna:jna-platform</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>**/*.proto</exclude>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ <exclude>META-INF/DEPENDENCIES</exclude>
+ <exclude>META-INF/LICENSE.txt</exclude>
+ <exclude>META-INF/NOTICE.txt</exclude>
+ <exclude>META-INF/maven/**</exclude>
+ <exclude>LICENSE.txt</exclude>
+ <exclude>NOTICE.txt</exclude>
+ <exclude>mozilla/**</exclude>
+ <exclude>**/module-info.class</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+ <pattern>org.codehaus.jackson.jaxrs</pattern>
+
<shadedPattern>${kyuubi.shade.packageName}.org.codehaus.jackson.jaxrs</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.sun.jersey</pattern>
+
<shadedPattern>${kyuubi.shade.packageName}.com.sun.jersey</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.sun.ws.rs.ext</pattern>
+
<shadedPattern>${kyuubi.shade.packageName}.com.sun.ws.rs.ext</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>com.kstruct.gethostname4j</pattern>
+
<shadedPattern>${kyuubi.shade.packageName}.com.kstruct.gethostname4j</shadedPattern>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.hadoop.security</pattern>
+
<shadedPattern>${kyuubi.shade.packageName}.org.apache.hadoop.security</shadedPattern>
+ <includes>
+
<include>org.apache.hadoop.security.KrbPasswordSaverLoginModule</include>
+
<include>org.apache.hadoop.security.SecureClientLogin</include>
+
<include>org.apache.hadoop.security.SecureClientLoginConfiguration</include>
+ </includes>
+ </relocation>
+ </relocations>
+ <transformers>
+ <transformer
implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"></transformer>
+ </transformers>
+ </configuration>
+ <executions>
+ <execution>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <phase>package</phase>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+</project>
diff --git a/pom.xml b/pom.xml
index 1f3fca53b..417800be1 100644
--- a/pom.xml
+++ b/pom.xml
@@ -56,6 +56,7 @@
<module>extensions/server/kyuubi-server-plugin</module>
<module>extensions/spark/kyuubi-extension-spark-jdbc-dialect</module>
<module>extensions/spark/kyuubi-spark-authz</module>
+ <module>extensions/spark/kyuubi-spark-authz-shaded</module>
<module>extensions/spark/kyuubi-spark-connector-common</module>
<module>extensions/spark/kyuubi-spark-connector-tpcds</module>
<module>extensions/spark/kyuubi-spark-connector-tpch</module>
