This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/kyuubi-shaded.git
The following commit(s) were added to refs/heads/master by this push:
new 93536fa [KYUUBI-SHADED #20] Introduce `kyuubi-shaded-hive-service-rpc`
93536fa is described below
commit 93536fa9fcb57e030619ef23c238b34efe45f564
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Dec 1 19:03:47 2023 +0800
[KYUUBI-SHADED #20] Introduce `kyuubi-shaded-hive-service-rpc`
### _Why are the changes needed?_
This PR aims to introduce a new pre-shaded module
`kyuubi-shaded-hive-service-rpc` for Kyuubi project, which contains
- org.apache.hive:hive-service-rpc:3.1.3
- org.apache.thrift:libfb303:0.9.3
- org.apache.thrift:libthrift:0.9.3-1
Kyuubi uses Hive Service RPC (a thrift-based protocol) as the internal RPC
protocol, currently, we do shading during each engine jar packaging, it works
well for most engines except for Hive engines, such shading would corrupt the
Hive method invocation because the same package name, i.e.
```
import org.apache.hive.service.rpc.thrift.TGetInfoReq
def method(
// this is part of Kyuubi server-engine internal RPC
getInfoReq: TGetInfoReq) {
// it's a Hive internal method invocation
hiveObj.hiveMethod(getInfoReq)
}
```
if we switch to the pre-shaded `kyuubi-shaded-hive-service-rpc`, it could
be solved as
```
import org.apache.kyuubi.shaded.hive.service.rpc.thrift.TGetInfoReq
import org.apache.hive.service.rpc.thrift.{TGetInfoReq => HiveTGetInfoReq}
def method(
// this is part of Kyuubi server-engine internal RPC
getInfoReq: TGetInfoReq) {
// it's a Hive internal method invocation
val hiveGetInfoReq: HiveTGetInfoReq = HiveRpcUtils.asHive(getInfoReq)
hiveObj.hiveMethod(hiveGetInfoReq)
}
```
In addition, `kyuubi-shaded-hive-service-rpc` also shades the Thrift
runtime classes. The upcoming Hive 2.3.10, 3.2.0, 4.0.0 upgrade Thrift to
0.14+, has known incompatible changes with the current 0.9.3, pre-shaded
classes also resolve the class conflicts.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #20 from pan3793/thrift.
f6126cc [Cheng Pan] short name
49cd216 [Cheng Pan] Introduce kyuubi-shaded-hive-service-rpc
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
---
.rat-excludes | 2 +
kyuubi-shaded-hive-service-rpc/pom.xml | 151 +++++++++++++++++++++
.../src/main/resources/META-INF/NOTICE | 11 ++
kyuubi-shaded-zookeeper-parent/pom.xml | 19 +++
pom.xml | 1 +
5 files changed, 184 insertions(+)
diff --git a/.rat-excludes b/.rat-excludes
index 61118d9..74cc616 100644
--- a/.rat-excludes
+++ b/.rat-excludes
@@ -21,6 +21,8 @@
**/*.log
**/*.md
**/*.iml
+**/*.tmp
+**/*.temp
**/target/**
**/out/**
**/licenses/LICENSE*
diff --git a/kyuubi-shaded-hive-service-rpc/pom.xml
b/kyuubi-shaded-hive-service-rpc/pom.xml
new file mode 100644
index 0000000..92bc958
--- /dev/null
+++ b/kyuubi-shaded-hive-service-rpc/pom.xml
@@ -0,0 +1,151 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.kyuubi</groupId>
+ <artifactId>kyuubi-shaded-parent</artifactId>
+ <version>0.2.0-SNAPSHOT</version>
+ </parent>
+
+ <artifactId>kyuubi-shaded-hive-service-rpc</artifactId>
+ <name>kyuubi-shaded-hive-service-rpc</name>
+
+ <properties>
+ <hive.service.rpc.version>3.1.3</hive.service.rpc.version>
+ <fb303.version>0.9.3</fb303.version>
+ <thrift.version>0.9.3-1</thrift.version>
+ </properties>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.hive</groupId>
+ <artifactId>hive-service-rpc</artifactId>
+ <version>${hive.service.rpc.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libfb303</artifactId>
+ <version>${fb303.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.thrift</groupId>
+ <artifactId>libthrift</artifactId>
+ <version>${thrift.version}</version>
+ <exclusions>
+ <exclusion>
+ <groupId>*</groupId>
+ <artifactId>*</artifactId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+
+ <build>
+ <plugins>
+ <plugin>
+ <groupId>org.apache.maven.plugins</groupId>
+ <artifactId>maven-shade-plugin</artifactId>
+ <executions>
+ <execution>
+ <id>shade-kyuubi</id>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <phase>package</phase>
+ <configuration>
+
<createDependencyReducedPom>true</createDependencyReducedPom>
+
<dependencyReducedPomLocation>${project.basedir}/target/dependency-reduced-pom.xml</dependencyReducedPomLocation>
+ <artifactSet>
+ <includes>
+ <include>*</include>
+ </includes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>**/*.proto</exclude>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+
<exclude>META-INF/DEPENDENCIES</exclude>
+ <exclude>META-INF/LICENSE.txt</exclude>
+ <exclude>META-INF/NOTICE.txt</exclude>
+ <exclude>META-INF/maven/**</exclude>
+ <exclude>LICENSE.txt</exclude>
+ <exclude>NOTICE.txt</exclude>
+ <exclude>mozilla/**</exclude>
+ <exclude>**/module-info.class</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <relocations>
+ <relocation>
+
<pattern>org.apache.hive.service.rpc.thrift</pattern>
+
<shadedPattern>${shading.prefix}.hive.service.rpc.thrift</shadedPattern>
+ <includes>
+
<include>org.apache.hive.service.rpc.thrift.**</include>
+ </includes>
+ </relocation>
+ <relocation>
+ <pattern>com.facebook.fb303</pattern>
+
<shadedPattern>${shading.prefix}.fb303</shadedPattern>
+ <includes>
+
<include>com.facebook.fb303.**</include>
+ </includes>
+ </relocation>
+ <relocation>
+ <pattern>org.apache.thrift</pattern>
+
<shadedPattern>${shading.prefix}.thrift</shadedPattern>
+ <includes>
+ <include>org.apache.thrift.**</include>
+ </includes>
+ </relocation>
+ </relocations>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+
+ <plugin>
+ <!-- Used to resolve variables in the 'version' tag -->
+ <groupId>org.codehaus.mojo</groupId>
+ <artifactId>flatten-maven-plugin</artifactId>
+ </plugin>
+ </plugins>
+ </build>
+
+</project>
diff --git a/kyuubi-shaded-hive-service-rpc/src/main/resources/META-INF/NOTICE
b/kyuubi-shaded-hive-service-rpc/src/main/resources/META-INF/NOTICE
new file mode 100644
index 0000000..3a16699
--- /dev/null
+++ b/kyuubi-shaded-hive-service-rpc/src/main/resources/META-INF/NOTICE
@@ -0,0 +1,11 @@
+kyuubi-shaded-hive-service-rpc
+Copyright 2023 The Apache Software Foundation
+
+This product includes software developed at
+The Apache Software Foundation (https://www.apache.org/).
+
+This project bundles the following dependencies under the Apache Software
License 2.0 (https://www.apache.org/licenses/LICENSE-2.0.txt)
+
+- org.apache.hive:hive-service-rpc:3.1.3
+- org.apache.thrift:libfb303:0.9.3
+- org.apache.thrift:libthrift:0.9.3-1
diff --git a/kyuubi-shaded-zookeeper-parent/pom.xml
b/kyuubi-shaded-zookeeper-parent/pom.xml
index 1a238e8..45b7c5a 100644
--- a/kyuubi-shaded-zookeeper-parent/pom.xml
+++ b/kyuubi-shaded-zookeeper-parent/pom.xml
@@ -57,6 +57,25 @@ under the License.
<include>*</include>
</includes>
</artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>**/*.proto</exclude>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+
<exclude>META-INF/DEPENDENCIES</exclude>
+ <exclude>META-INF/LICENSE.txt</exclude>
+ <exclude>META-INF/NOTICE.txt</exclude>
+ <exclude>META-INF/maven/**</exclude>
+ <exclude>LICENSE.txt</exclude>
+ <exclude>NOTICE.txt</exclude>
+ <exclude>mozilla/**</exclude>
+ <exclude>**/module-info.class</exclude>
+ </excludes>
+ </filter>
+ </filters>
<relocations>
<relocation>
<pattern>org.apache.zookeeper</pattern>
diff --git a/pom.xml b/pom.xml
index bf77b86..e7391d6 100644
--- a/pom.xml
+++ b/pom.xml
@@ -52,6 +52,7 @@
<modules>
<module>kyuubi-shaded-force-shading</module>
+ <module>kyuubi-shaded-hive-service-rpc</module>
<module>kyuubi-shaded-zookeeper-parent</module>
</modules>
