This is an automated email from the ASF dual-hosted git repository. yao pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push: new e12a79b6ebe [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF e12a79b6ebe is described below commit e12a79b6ebe0bd5ef9220e0e0979d6579638f3ae Author: Yuming Wang <yumw...@ebay.com> AuthorDate: Tue Aug 15 11:42:23 2023 +0800 [SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF ### What changes were proposed in this pull request? This PR partially reverts SPARK-43225. Added jackson-core-asl and jackson-mapper-asl back to pre-built distributions. ### Why are the changes needed? Fix `NoClassDefFoundError` when using Hive UDF: ``` spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar; Time taken: 0.413 seconds spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP'; Time taken: 0.038 seconds spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10); 23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)] java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test. Closes #42447 from wangyum/SPARK-44719-branch-3.5. Authored-by: Yuming Wang <yumw...@ebay.com> Signed-off-by: Kent Yao <y...@apache.org> --- core/pom.xml | 8 ++++++++ dev/deps/spark-deps-hadoop-3-hive-2.3 | 2 ++ pom.xml | 13 +++++++++++-- sql/hive-thriftserver/pom.xml | 9 --------- 4 files changed, 21 insertions(+), 11 deletions(-) diff --git a/core/pom.xml b/core/pom.xml index 674005c96ee..eb4a563f1f3 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -481,6 +481,14 @@ <groupId>commons-logging</groupId> <artifactId>commons-logging</artifactId> </exclusion> + <exclusion> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-mapper-asl</artifactId> + </exclusion> + <exclusion> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-core-asl</artifactId> + </exclusion> <exclusion> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-core</artifactId> diff --git a/dev/deps/spark-deps-hadoop-3-hive-2.3 b/dev/deps/spark-deps-hadoop-3-hive-2.3 index d182f39f882..14af8ea340f 100644 --- a/dev/deps/spark-deps-hadoop-3-hive-2.3 +++ b/dev/deps/spark-deps-hadoop-3-hive-2.3 @@ -100,11 +100,13 @@ ini4j/0.5.4//ini4j-0.5.4.jar istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar ivy/2.5.1//ivy-2.5.1.jar jackson-annotations/2.15.2//jackson-annotations-2.15.2.jar +jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar jackson-core/2.15.2//jackson-core-2.15.2.jar jackson-databind/2.15.2//jackson-databind-2.15.2.jar jackson-dataformat-cbor/2.15.2//jackson-dataformat-cbor-2.15.2.jar jackson-dataformat-yaml/2.15.2//jackson-dataformat-yaml-2.15.2.jar jackson-datatype-jsr310/2.15.2//jackson-datatype-jsr310-2.15.2.jar +jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar jackson-module-scala_2.12/2.15.2//jackson-module-scala_2.12-2.15.2.jar jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar diff --git a/pom.xml b/pom.xml index 9438a112c24..d0c3bfc361a 100644 --- a/pom.xml +++ b/pom.xml @@ -1321,6 +1321,10 @@ <groupId>asm</groupId> <artifactId>asm</artifactId> </exclusion> + <exclusion> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-mapper-asl</artifactId> + </exclusion> <exclusion> <groupId>org.ow2.asm</groupId> <artifactId>asm</artifactId> @@ -1821,12 +1825,17 @@ </exclusion> </exclusions> </dependency> - <!-- Hive 2.3 need this to init Hive's FunctionRegistry --> + <dependency> + <groupId>org.codehaus.jackson</groupId> + <artifactId>jackson-core-asl</artifactId> + <version>${codehaus.jackson.version}</version> + <scope>${hadoop.deps.scope}</scope> + </dependency> <dependency> <groupId>org.codehaus.jackson</groupId> <artifactId>jackson-mapper-asl</artifactId> <version>${codehaus.jackson.version}</version> - <scope>test</scope> + <scope>${hadoop.deps.scope}</scope> </dependency> <dependency> <groupId>${hive.group}</groupId> diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml index 7bcfb8e1908..3659a0f846a 100644 --- a/sql/hive-thriftserver/pom.xml +++ b/sql/hive-thriftserver/pom.xml @@ -147,15 +147,6 @@ <groupId>org.apache.httpcomponents</groupId> <artifactId>httpcore</artifactId> </dependency> - <!-- - SPARK-44601: Add this test dependency to ensure that `hive-thriftserver` module - can be tested using Maven - --> - <dependency> - <groupId>org.codehaus.jackson</groupId> - <artifactId>jackson-mapper-asl</artifactId> - <scope>test</scope> - </dependency> </dependencies> <build> <outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory> --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org