[ https://issues.apache.org/jira/browse/SPARK-30365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
wangzhun updated SPARK-30365: ----------------------------- Priority: Major (was: Minor) > When deploy mode is a client, why doesn't it support remote "spark.files" > download? > ----------------------------------------------------------------------------------- > > Key: SPARK-30365 > URL: https://issues.apache.org/jira/browse/SPARK-30365 > Project: Spark > Issue Type: Question > Components: Spark Submit > Affects Versions: 2.3.2 > Environment: {code:java} > ./bin/spark-submit \ > --master yarn \ > --deploy-mode client \ > ......{code} > Reporter: wangzhun > Priority: Major > > {code:java} > // In client mode, download remote files. > var localPrimaryResource: String = null > var localJars: String = null > var localPyFiles: String = null > if (deployMode == CLIENT) { > localPrimaryResource = Option(args.primaryResource).map { > downloadFile(_, targetDir, sparkConf, hadoopConf, secMgr) > }.orNull > localJars = Option(args.jars).map { > downloadFileList(_, targetDir, sparkConf, hadoopConf, secMgr) > }.orNull > localPyFiles = Option(args.pyFiles).map { > downloadFileList(_, targetDir, sparkConf, hadoopConf, secMgr) > }.orNull > } > {code} > The above Spark2.3 SparkSubmit code does not download the corresponding file > of "spark.files". > I think it is possible to download remote files locally and add them to > classPath. > For example, can support --files configuration remote hive-site.xml -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org