shardulm94 commented on a change in pull request #31591:
URL: https://github.com/apache/spark/pull/31591#discussion_r580946284
##########
File path: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
##########
@@ -1278,11 +1287,22 @@ private[spark] object SparkSubmitUtils {
remoteRepos: Option[String],
ivyPath: Option[String]): IvySettings = {
val file = new File(settingsFile)
- require(file.exists(), s"Ivy settings file $file does not exist")
- require(file.isFile(), s"Ivy settings file $file is not a normal file")
Review comment:
Yeah, as you pointed out it only works if I have the copy in the current
working directory. However we cannot control which directory our users launch
spark-submit from. So ideally we would want something which works without user
intervention.
What do you think of this? In Yarn Client, we can add the ivySettings file
to `spark.yarn.dist.files` or `__spark__conf__.zip` and then we can modify the
property `spark.jars.ivySettings` to change it to `./ivysettings.xml` or
`__spark__conf__/ivysettings.xml` within Yarn Client.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
