[
https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14936317#comment-14936317
]
Saisai Shao edited comment on SPARK-10858 at 9/30/15 5:49 AM:
--------------------------------------------------------------
So basically I think the problem is if we specify the scheme, do we need to
guarantee the preserved chars to be escaped by our own. If we rely on Spark to
handle this, I think we need to fix this issue.
Another interesting thing is that not sure why your result is different from
mine.
was (Author: jerryshao):
So basically I think the problem is do we need to treat this name "xx#xx" as a
legal name, if so we need to fix this behavior.
Another interesting thing is that not sure why your result is different from
mine.
> YARN: archives/jar/files rename with # doesn't work unless scheme given
> -----------------------------------------------------------------------
>
> Key: SPARK-10858
> URL: https://issues.apache.org/jira/browse/SPARK-10858
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.5.1
> Reporter: Thomas Graves
> Priority: Minor
>
> The YARN distributed cache feature with --jars, --archives, --files where you
> can rename the file/archive using a # symbol only works if you explicitly
> include the scheme in the path:
> works:
> --jars file:///home/foo/my.jar#renamed.jar
> doesn't work:
> --jars /home/foo/my.jar#renamed.jar
> Exception in thread "main" java.io.FileNotFoundException: File
> file:/home/foo/my.jar#renamed.jar does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:416)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:337)
> at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:289)
> at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:240)
> at
> org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:329)
> at
> org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:393)
> at
> org.apache.spark.deploy.yarn.Client$$anonfun$prepareLocalResources$6$$anonfun$apply$2.apply(Client.scala:392)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
