HyukjinKwon commented on a change in pull request #30581:
URL: https://github.com/apache/spark/pull/30581#discussion_r535227299
##########
File path: core/src/main/scala/org/apache/spark/SparkContext.scala
##########
@@ -1639,7 +1639,9 @@ class SparkContext(config: SparkConf) extends Logging {
UriBuilder.fromUri(new
URI(key)).fragment(uri.getFragment).build().toString,
timestamp).isEmpty) {
logInfo(s"Added archive $path at $key with timestamp $timestamp")
- val uriToDownload = UriBuilder.fromUri(new
URI(key)).fragment(null).build()
+ // If the scheme is file, use URI to simply copy instead of downloading.
+ val uriToUse = if (!isLocal && scheme == "file") uri else new URI(key)
+ val uriToDownload = UriBuilder.fromUri(uriToUse).fragment(null).build()
Review comment:
This change is actually not related but I found it during debug. This
isn't a bug but improvement:
By using `file:/` instead of `spark:/` on the driver side, we can directly
copy the file in driver side instead of fetching through the Netty file server
layer. This is matched with `spark.files`.
##########
File path:
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
##########
@@ -1629,6 +1629,7 @@ private[spark] class YarnClusterApplication extends
SparkApplication {
// so remove them from sparkConf here for yarn mode.
conf.remove(JARS)
conf.remove(FILES)
+ conf.remove(ARCHIVES)
Review comment:
This is also unrelated changes but I just piggybacked here as it looks
pretty clear to me.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
