shardulm94 commented on pull request #31591: URL: https://github.com/apache/spark/pull/31591#issuecomment-800622075
@tgravescs Sorry for the delay, time flies! This PR started off try to address this issue in the general cluster case using `--files` to distribute the ivy setting file for all resource managers. However one of the concerns was that we would have to look for the ivy file different based on whether the driver was running locally v/s on the cluster, since the codepath is shared between SparkSubmit and sc.addJar. https://github.com/apache/spark/blob/0d62c5cd023095afd7081355ec2ff1a8558fed7f/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala#L1290 There is more context at https://github.com/apache/spark/pull/31591#issuecomment-782351781 and https://github.com/apache/spark/pull/31591#issuecomment-782351781 So I decided to make this specific to YARN since I am not familiar with the other resource managers. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
