This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.10
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.10 by this push:
new c65f04848d [KYUUBI #6836] Ship `kafka-clients` in binary distribution
tarball without compression libs
c65f04848d is described below
commit c65f04848df8e7a992900cdc531a53cb37f3d803
Author: Cheng Pan <[email protected]>
AuthorDate: Thu Dec 5 13:59:29 2024 +0800
[KYUUBI #6836] Ship `kafka-clients` in binary distribution tarball without
compression libs
### Why are the changes needed?
I'd like to include `kafka-clients` in the Kyuubi binary distribution
tarball to enable the out-of-box support for sinking Kyuubi events to Kafka.
- Kafka is an important component in modern data platforms, and is a
defacto message queue implementation, especially in the big data domain
- `kafka-clients` is released under Apache License V2, has no legal issue
- `kafka-clients` is quite a light lib, has no third-party deps except for
`slf4j-api` and a few optional compression libs
- `kafka-clients` uses "none" compression as default, in practice,
"gzip"(delegate to JDK gzip algorithm, no additional libs are required) already
performs well for non-extreme cases
Additionally, LOG4J2 has a built-in `KafkaAppender` that supports sinking
logging to Kafka, which also requires `kafka-clients` in the classpath, I have
some initial ideas to forward both Kyuubi server's and engine bootstrap
processes log to Kafka in a structured format, will send another PR to add docs
to guide users in configuring that.
### How was this patch tested?
Review
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #6836 from pan3793/kafka-lib.
Closes #6836
b069eb199 [Cheng Pan] Ship kafka-clients in binary distribution tarball
without compression libs
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit f2cacd3f1875c97d9d14532e780aee62ce57ab7d)
Signed-off-by: Cheng Pan <[email protected]>
---
LICENSE-binary | 3 ---
NOTICE-binary | 16 ----------------
dev/dependencyList | 3 ---
pom.xml | 15 ++++++++++++++-
4 files changed, 14 insertions(+), 23 deletions(-)
diff --git a/LICENSE-binary b/LICENSE-binary
index e78047f548..6b4c15fa77 100644
--- a/LICENSE-binary
+++ b/LICENSE-binary
@@ -306,8 +306,6 @@ io.vertx:vertx-grpc
com.squareup.retrofit2:retrofit
com.squareup.okhttp3:okhttp
org.apache.kafka:kafka-clients
-org.lz4:lz4-java
-org.xerial.snappy:snappy-java
org.xerial:sqlite-jdbc
BSD
@@ -319,7 +317,6 @@ jline:jline
com.thoughtworks.paranamer:paranamer
com.google.protobuf:protobuf-java-util
com.google.protobuf:protobuf-java
-com.github.luben:zstd-jni
org.postgresql:postgresql
Eclipse Distribution License - v 1.0
diff --git a/NOTICE-binary b/NOTICE-binary
index b9bcdb2146..708624e3f3 100644
--- a/NOTICE-binary
+++ b/NOTICE-binary
@@ -1063,19 +1063,3 @@ which can be obtained at:
* license/LICENSE.kafka.txt (Apache License 2.0)
* HOMEPAGE:
* https://github.com/apache/kafka
-
-This product optionally depends on 'snappy-java', Snappy compression and
-decompression for Java, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.snappy-java.txt (Apache License 2.0)
- * HOMEPAGE:
- * https://github.com/xerial/snappy-java
-
-This product optionally depends on 'lz4-java', Lz4 compression and
-decompression for Java, which can be obtained at:
-
- * LICENSE:
- * license/LICENSE.lz4-java.txt (Apache License 2.0)
- * HOMEPAGE:
- * https://github.com/lz4/lz4-java
diff --git a/dev/dependencyList b/dev/dependencyList
index 3a4dbdbd04..31cabe8aff 100644
--- a/dev/dependencyList
+++ b/dev/dependencyList
@@ -128,7 +128,6 @@ log4j-api/2.20.0//log4j-api-2.20.0.jar
log4j-core/2.20.0//log4j-core-2.20.0.jar
log4j-slf4j-impl/2.20.0//log4j-slf4j-impl-2.20.0.jar
logging-interceptor/3.12.12//logging-interceptor-3.12.12.jar
-lz4-java/1.8.0//lz4-java-1.8.0.jar
metrics-core/4.2.26//metrics-core-4.2.26.jar
metrics-jmx/4.2.26//metrics-jmx-4.2.26.jar
metrics-json/4.2.26//metrics-json-4.2.26.jar
@@ -173,7 +172,6 @@
simpleclient_tracer_otel_agent/0.16.0//simpleclient_tracer_otel_agent-0.16.0.jar
slf4j-api/1.7.36//slf4j-api-1.7.36.jar
snakeyaml-engine/2.7//snakeyaml-engine-2.7.jar
snakeyaml/2.2//snakeyaml-2.2.jar
-snappy-java/1.1.10.5//snappy-java-1.1.10.5.jar
sqlite-jdbc/3.46.1.3//sqlite-jdbc-3.46.1.3.jar
swagger-annotations/2.2.1//swagger-annotations-2.2.1.jar
swagger-core/2.2.1//swagger-core-2.2.1.jar
@@ -185,4 +183,3 @@ units/1.7//units-1.7.jar
vertx-core/4.5.3//vertx-core-4.5.3.jar
vertx-grpc/4.5.3//vertx-grpc-4.5.3.jar
zjsonpatch/0.3.0//zjsonpatch-0.3.0.jar
-zstd-jni/1.5.5-1//zstd-jni-1.5.5-1.jar
diff --git a/pom.xml b/pom.xml
index 3383cbcb8f..f15b0c2987 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1052,7 +1052,20 @@
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.version}</version>
- <optional>true</optional>
+ <exclusions>
+ <exclusion>
+ <groupId>com.github.luben</groupId>
+ <artifactId>zstd-jni</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.lz4</groupId>
+ <artifactId>lz4-java</artifactId>
+ </exclusion>
+ <exclusion>
+ <groupId>org.xerial.snappy</groupId>
+ <artifactId>snappy-java</artifactId>
+ </exclusion>
+ </exclusions>
</dependency>
<dependency>
