shardulm94 commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-782351781
@xkrogen > (2) I know we discussed the possibility of using an approach where the settings file is in the Spark conf zip, did you try this approach and discover any issues / complexity? I did try and there are some tradeoffs. I was planning to implement this similar to how we ship `log4j.properties` as part of Spark conf. https://github.com/apache/spark/blob/484a83e73e8b9420ffd42b4eb95dcdf4cf66ebf0/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L762 Then we can use `classloader.getResource` to read the ivy file on the driver classpath. However, this approach for shipping the file would be specific to YARN, and won't work with other resource managers with cluster mode unless we make similar change in all resource managers. There is also possibility that `classloader.getResource` may get resources from other jars/zip (not spark_conf.zip) depending on classpath ordering. One alternative can be to search for the unzipped file in spark_conf i.e `__spark_conf__/ivysettings.xml` but again that is specific to YARN's spark conf naming. I am very new to Spark, so there might be obvious ways of resolving the above concerns that I am missing. Please let me know, I would be happy to try them out. > (1) There will be a conflict if the user specifies any file to --files with the same name as their spark.jars.ivySettings. This seems like an odd case, so I am not worried about it. Yes, that is possible, however even in the alternative approaches I thought of (e.g. the one above), there was always a possibility of this happening. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
