xkrogen commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-784411877
> We could enhance the path to support custom schemes like classpath://, but since the code also needs to work inside SparkSubmit where the classpath is still being constructed, this would not work for both SparkSubmit and the driver together. There are mechanisms to adjust the classpath that the client sees, but the impetus would be on the user/administrator to configure this properly, so it does push up some extra work. That being said, I think your recent proposal about using `__spark_conf__` looks cleaner. > Also not sure if we should worry about backward compatibility too much here. Being able to use addJar in Spark context is a fairly new feature and hasn't been released yet, its set for 3.2.0. Totally agreed that for reading `ivysettings.xml` from the driver, it doesn't need backwards-compatibility. But for client-side cases it should be backwards-compatible. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
