yadavay-amzn commented on PR #55869: URL: https://github.com/apache/spark/pull/55869#issuecomment-4523087898
@sarutak You're right -- for SparkSubmit-launched scenarios (spark-shell, Thrift Server), parent delegation handles `spark.jars` because SparkSubmit creates a `MutableURLClassLoader` with the jars and sets it as context classloader. The actual gap is in programmatic SparkSession creation (without SparkSubmit): - `SparkContext.addJar()` only registers jars for executor distribution (`addedJars` map), does NOT add to any driver classloader - `SharedState.jarClassLoader` is created empty with the app classloader as parent - `spark.jars` are never loaded into any driver-side classloader I verified this locally: the test fails without the fix (jar not in classloader) and passes with it. Updated the PR description and added a comment on the JIRA to clarify the actual scope. The original JIRA description mentioning Thrift Server is misleading -- the real bug is the programmatic path. Does this make sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
