shardulm94 commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-783969152
I thought about this a bit more. I think the main concern here is where/how to find the ivySettings file. I believe this becomes tricky because the same code and properties are currently being used to load the ivySettings file in SparkSubmit (where it makes more sense to read the file directly since it will be local and its too early to be put on the classpath) and inside the driver (where it would be better to read this off the classpath, the user provided local path can be invalid). We could set an internal config like `spark.yarn.xxx` but this property will need to be used inside `core` and I don't see any precedent for yarn specific properties being used in such a way. @xkrogen We could enhance the path to support custom schemes like `classpath://`, but since the code also needs to work inside SparkSubmit where the classpath is still being constructed, this would not work for both SparkSubmit and the driver together. I guess one thing we can do is modify `spark.jars.ivySettings` to convert it from a file entry to a classpath entry before passing it to the driver? Also not sure if we should worry about backward compatibility too much here. Being able to use addJar in Spark context is a fairly new feature and hasn't been released yet, its set for 3.2.0. Another option is to have separate code for loading ivySettings file in SparkSubmit v/s driver and call the appropriate method, or pass an explicit parameter to the `loadIvySettings` function from the driver which instructs it to only search the classpath. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
