[
https://issues.apache.org/jira/browse/SPARK-30260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chenliang updated SPARK-30260:
------------------------------
Fix Version/s: 2.3.0
> Spark-Shell throw ClassNotFoundException exception for more than one
> statement to use UDF jar
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-30260
> URL: https://issues.apache.org/jira/browse/SPARK-30260
> Project: Spark
> Issue Type: Bug
> Components: Spark Shell, SQL
> Affects Versions: 2.2.0, 2.3.0, 2.4.3, 2.4.4
> Reporter: chenliang
> Priority: Major
> Fix For: 2.3.0, 2.4.3
>
>
> When we start spark-shell and use the udf for the first statement ,it's ok.
> But for the other statements it failed to load jar to current classpath and
> would throw ClassNotFoundException,the problem can be reproduced as described
> in the below.
> {code:java}
> scala> val res = spark.sql("select bigdata_test.Add(1,2)").show()
> ----------------------
> |bigdata_test.Add(1, 2)|
> ----------------------
> | 3|
> ----------------------
> scala> val res = spark.sql("select bigdata_test.Add(1,2)").show()
> org.apache.spark.sql.AnalysisException: No handler for UDF/UDAF/UDTF
> 'scala.didi.udf.Add': java.lang.ClassNotFoundException: scala.didi.udf.Add;
> line 1 pos 8
> at
> scala.reflect.internal.util.AbstractFileClassLoader.findClass(AbstractFileClassLoader.scala:62)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> at
> org.apache.spark.sql.hive.HiveShim$HiveFunctionWrapper.createFunction(HiveShim.scala:251)
> at
> org.apache.spark.sql.hive.HiveSimpleUDF.function$lzycompute(hiveUDFs.scala:56)
> at org.apache.spark.sql.hive.HiveSimpleUDF.function(hiveUDFs.scala:56)
> at
> org.apache.spark.sql.hive.HiveSimpleUDF.method$lzycompute(hiveUDFs.scala:60)
> at org.apache.spark.sql.hive.HiveSimpleUDF.method(hiveUDFs.scala:59)
> at
> org.apache.spark.sql.hive.HiveSimpleUDF.dataType$lzycompute(hiveUDFs.scala:77)
> at org.apache.spark.sql.hive.HiveSimpleUDF.dataType(hiveUDFs.scala:77)
> at
> org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionExpression$3.apply(HiveSessionCatalog.scala:79)
> at
> org.apache.spark.sql.hive.HiveSessionCatalog$$anonfun$makeFunctionExpression$3.apply(HiveSessionCatalog.scala:71)
> at scala.util.Try.getOrElse(Try.scala:79)
> at
> org.apache.spark.sql.hive.HiveSessionCatalog.makeFunctionExpression(HiveSessionCatalog.scala:71)
> at
> org.apache.spark.sql.catalyst.catalog.SessionCatalog$$anonfun$org$apache$spark$sql$catalyst$catalog$SessionCatalog$$makeFunctionBuilder$1.apply(SessionCatalog.scala:1133){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
