xkrogen commented on a change in pull request #31591:
URL: https://github.com/apache/spark/pull/31591#discussion_r598857412
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -793,6 +793,25 @@ private[spark] class Client(
// distributed file.
amKeytabFileName.foreach { kt => props.setProperty(KEYTAB.key, kt) }
+ // Upload user provided ivysettings.xml file to the distributed cache
+ val ivySettings = sparkConf.getOption("spark.jars.ivySettings")
+ if (isClusterMode && ivySettings.isDefined) {
+ val ivySettingsFile = new File(ivySettings.get)
+ require(ivySettingsFile.exists(), s"Ivy settings file $ivySettingsFile
not found")
+ require(ivySettingsFile.isFile(),
+ s"Ivy settings file $ivySettingsFile is not a normal file")
+ // Generate a file name that can be used for the ivySettings file,
that does not conflict
+ // with any other conf file.
+ val amIvySettingsFileName = ivySettingsFile.getName() + "-" +
UUID.randomUUID().toString
+ confStream.putNextEntry(new ZipEntry(amIvySettingsFileName))
Review comment:
> I get that its kind of like a conf file but its not, there is no
template file in the conf directory for this, it's a user specified config and
not automatically picked up ... Its a bit unfortunately we are not consistent
with these things.
The same is (mostly) true for examples like `metrics.properties`, which is
an external/non-Spark file, the only difference being that it is automatically
picked up vs. being requested by a user. I don't have much opinion on whether
or not this is how it should be -- I agree with you that more clear
guidelines/consistency in this area would be nice.
I don't have too much opinion between storing in the conf object and
leveraging `distribute`, I do agree that we should support remote file systems
(e.g. HDFS) so this is a good point. @HyukjinKwon do you have any opinion here
since you participated in some of the earlier conversations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
