This is an automated email from the ASF dual-hosted git repository.
zhangzc pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-gluten.git
The following commit(s) were added to refs/heads/main by this push:
new d38ea9add [GLUTEN-5108][CH] Fix the classes in the hadoop-common
conflict when running ut local (#5109)
d38ea9add is described below
commit d38ea9adda812e575167267254e673a81c9f2e8c
Author: Zhichao Zhang <[email protected]>
AuthorDate: Tue Mar 26 12:07:10 2024 +0800
[GLUTEN-5108][CH] Fix the classes in the hadoop-common conflict when
running ut local (#5109)
Fix the classes in the hadoop-common conflict when running ut local.
Env: Idea + Spark 3.3 + Hadoop 3.3.1
UT: GlutenClickHouseMergeTreeWriteOnS3Suite /
GlutenClickHouseMergeTreeWriteOnHDFSSuite
Error:
```
java.lang.NoClassDefFoundError: xxxxx
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:634)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:619)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
```
RC:
The jar `hive-hcatalog-core` includes the hadoop-common 2.7.2 and
hadoop-hdfs 2.7.2, which conflict with the hadoop 3.3.1 when running ut with
hadoop-3.3 profile.
Close #5108.
---
backends-clickhouse/pom.xml | 8 ++++++++
gluten-celeborn/clickhouse/pom.xml | 12 ++++++++++++
2 files changed, 20 insertions(+)
diff --git a/backends-clickhouse/pom.xml b/backends-clickhouse/pom.xml
index f173a61c9..dd4240a3a 100644
--- a/backends-clickhouse/pom.xml
+++ b/backends-clickhouse/pom.xml
@@ -158,6 +158,14 @@
<artifactId>guava</artifactId>
<groupId>com.google.guava</groupId>
</exclusion>
+ <exclusion>
+ <artifactId>hadoop-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-hdfs</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
diff --git a/gluten-celeborn/clickhouse/pom.xml
b/gluten-celeborn/clickhouse/pom.xml
index ed4784212..b44d0963c 100755
--- a/gluten-celeborn/clickhouse/pom.xml
+++ b/gluten-celeborn/clickhouse/pom.xml
@@ -96,6 +96,18 @@
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
</exclusion>
+ <exclusion>
+ <artifactId>guava</artifactId>
+ <groupId>com.google.guava</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-common</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>hadoop-hdfs</artifactId>
+ <groupId>org.apache.hadoop</groupId>
+ </exclusion>
</exclusions>
</dependency>
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
