[
https://issues.apache.org/jira/browse/SPARK-56764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18082988#comment-18082988
]
Anupam Yadav commented on SPARK-56764:
--------------------------------------
Clarification on scope: the original description mentions Thrift Server, but
that scenario works correctly via parent classloader delegation (SparkSubmit
adds spark.jars to a MutableURLClassLoader which becomes jarClassLoader's
parent).
The actual bug is in programmatic SparkSession creation (without SparkSubmit):
- SparkContext.addJar() only registers jars for executor distribution, does NOT
add to any driver classloader
- SharedState.jarClassLoader is created empty with the app classloader as parent
- spark.jars are never loaded into any driver-side classloader
So persistent UDF classes from spark.jars are not resolvable when SparkSession
is created programmatically.
PR #55869 fixes this by adding spark.jars to jarClassLoader at creation time.
TDD verified: test fails without fix, passes with fix.
> CANNOT_LOAD_FUNCTION_CLASS in a new session
> -------------------------------------------
>
> Key: SPARK-56764
> URL: https://issues.apache.org/jira/browse/SPARK-56764
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.1
> Reporter: pin_zhang
> Priority: Major
> Labels: pull-request-available
> Attachments: udf.jar
>
>
> 1. ./sbin/start-thriftserver.sh with async=false and create a UDF
> spark.sql.hive.thriftServer.async false
> spark.jars
> file:///home/test/spark-4.1.1-bin-hadoop3-connect/custom/udf.jar
> CREATE FUNCTION IF NOT EXISTS hello AS 'com.test.HelloUDF';
> 2. restart thriftserver
> ./bin/beeline -u jdbc:hive2://localhost:10000 -e 'select 1;'
> ./bin/beeline -u jdbc:hive2://localhost:10000 -e 'select hello("a");'
> Connecting to jdbc:hive2://localhost:10000
> Connected to: Spark SQL (version 4.1.1)
> Driver: Hive JDBC (version 2.3.10)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> Error: Error running query: [CANNOT_LOAD_FUNCTION_CLASS]
> org.apache.spark.sql.AnalysisException: [CANNOT_LOAD_FUNCTION_CLASS]
> Cannot load class com.test.HelloUDF when registering the function
> `spark_catalog`.`default`.`hello`, please make sure it is on the classpath.
> SQLSTATE: 46103; line 1 pos 7 (state=46103,code=0)
> Closing: 0: jdbc:hive2://localhost:10000
> 3. why set async=false, because pyhive client collect to hive in sync mode
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
