This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 7baf9dad196 [SPARK-44719][SQL] Fix NoClassDefFoundError when using
Hive UDF
7baf9dad196 is described below
commit 7baf9dad1962e7e44138c0b66422cca6e347f800
Author: Yuming Wang <[email protected]>
AuthorDate: Sat Aug 12 11:41:22 2023 +0800
[SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
### What changes were proposed in this pull request?
This PR changes jackson-mapper-asl's scope from `test` to
`${hive.deps.scope}`.
### Why are the changes needed?
Fix `NoClassDefFoundError` when using Hive UDF:
```
spark-sql (default)> add jar
/Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as
'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT
long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
...
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
Closes #42446 from wangyum/SPARK-44719.
Authored-by: Yuming Wang <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 ++
pom.xml | 8 +++++++-
sql/hive-thriftserver/pom.xml | 9 ---------
3 files changed, 9 insertions(+), 10 deletions(-)
diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3
b/dev/deps/spark-deps-hadoop-3-hive-2.3
index 2a5a6741bdd..3e8131b2734 100644
--- a/dev/deps/spark-deps-hadoop-3-hive-2.3
+++ b/dev/deps/spark-deps-hadoop-3-hive-2.3
@@ -99,11 +99,13 @@ ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.1//ivy-2.5.1.jar
jackson-annotations/2.15.2//jackson-annotations-2.15.2.jar
+jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.15.2//jackson-core-2.15.2.jar
jackson-databind/2.15.2//jackson-databind-2.15.2.jar
jackson-dataformat-cbor/2.15.2//jackson-dataformat-cbor-2.15.2.jar
jackson-dataformat-yaml/2.15.2//jackson-dataformat-yaml-2.15.2.jar
jackson-datatype-jsr310/2.15.2//jackson-datatype-jsr310-2.15.2.jar
+jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-scala_2.12/2.15.2//jackson-module-scala_2.12-2.15.2.jar
jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
diff --git a/pom.xml b/pom.xml
index 30357f1b27f..f25fb6b5f62 100644
--- a/pom.xml
+++ b/pom.xml
@@ -1834,11 +1834,17 @@
</exclusions>
</dependency>
<!-- Hive 2.3 need this to init Hive's FunctionRegistry -->
+ <dependency>
+ <groupId>org.codehaus.jackson</groupId>
+ <artifactId>jackson-core-asl</artifactId>
+ <version>${codehaus.jackson.version}</version>
+ <scope>${hive.deps.scope}</scope>
+ </dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
- <scope>test</scope>
+ <scope>${hive.deps.scope}</scope>
</dependency>
<dependency>
<groupId>${hive.group}</groupId>
diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml
index 3d9f6e3b2e9..76a1037f1cb 100644
--- a/sql/hive-thriftserver/pom.xml
+++ b/sql/hive-thriftserver/pom.xml
@@ -147,15 +147,6 @@
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</dependency>
- <!--
- SPARK-44601: Add this test dependency to ensure that `hive-thriftserver`
module
- can be tested using Maven
- -->
- <dependency>
- <groupId>org.codehaus.jackson</groupId>
- <artifactId>jackson-mapper-asl</artifactId>
- <scope>test</scope>
- </dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
