xkrogen commented on a change in pull request #31591:
URL: https://github.com/apache/spark/pull/31591#discussion_r600644547
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -131,6 +131,18 @@ private[spark] class Client(
require(keytab == null || !Utils.isLocalUri(keytab), "Keytab should
reference a local file.")
+ private val ivySettings = sparkConf.getOption("spark.jars.ivySettings")
+ private val ivySettingsLocalizedFileName: Option[String] = ivySettings match
{
+ case Some(ivySettingsPath) if isClusterMode &&
!Utils.isLocalUri(ivySettingsPath) =>
+ val ivySettingsFile = new File(ivySettingsPath)
+ require(ivySettingsFile.exists(), s"Ivy settings file $ivySettingsFile
not found")
Review comment:
+1
It would be much nicer if we could rely on `distribute` instead of trying to
figure out for ourselves here if it needs to be uploaded, e.g. `distribute`
will handle the local URI for us.
I think the reason we have to do this here is because `createConfArchive`
doesn't give us a way to pass in the `ivySettingsLocalizedFileName`
information. Can we just update `createConfArchive` to accept a parameter like
`configOverrides: Map[String, String]` which provides a way to override configs
that need to be different for the YARN container environment? Then we can move
this within `prepareLocalResources` and rely more on the logic of `distribute`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
