Cheng Pan created SPARK-60811:
---------------------------------
Summary: Generated code cannot resolve classes from jars added at
runtime when the driver thread's context classloader does not include them
Key: SPARK-60811
URL: https://issues.apache.org/jira/browse/SPARK-60811
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Cheng Pan
Spark compiles generated code with the context classloader of the driver thread
that runs the query. Classes from jars added at runtime ({{ADD JAR}}, {{CREATE
FUNCTION ... USING JAR}}) are only visible through it when the jar loading
switched that thread's context classloader. When it did not, or the classloader
was reset (for example, on a later line in spark-shell, on another driver
thread, or in a Spark Connect session without
{{spark.sql.artifact.isolation.alwaysApplyClassloader=true}}), compiling fails
with {{Cannot determine simple type name}}. Whole-stage codegen falls back to
interpreted execution by default, but operators without a fallback, such as
{{ORDER BY f(c) LIMIT 10}}, fail the query.
One way to hit this is a DataSource V2 function's magic {{invoke}} method,
whose class is referenced by name in generated code.
Repro (spark-shell):
{code}
scala> sql("CREATE TEMPORARY FUNCTION f AS 'com.example.StrLen' USING JAR
'/tmp/strlen.jar'")
scala> sql("SELECT f(c) FROM t ORDER BY f(c) LIMIT 10").show() // fails
{code}
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
