[
https://issues.apache.org/jira/browse/SPARK-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15065638#comment-15065638
]
Klaus Thul commented on SPARK-10596:
------------------------------------
sc.textFile wants to have real spaces and not %20 as part of the path in the
argument.
The argument to {{sc.textFile}} gets passed to
{{org.apache.hadoop.mapred.FileInputFormat.setInputPaths}}. Documentation at
https://hadoop.apache.org/docs/r2.6.2/api/org/apache/hadoop/fs/Path.html#Path(java.lang.String)
explicitly states ??Path strings are URIs, but with unescaped elements and some
additional normalisation??.
This is different from what you get from {{java.net.URI.toString method}}. I
could resolve a similar issue by using something like:
{quote}
val fileURI = new java.io.File(filename).toURI
sc.textFile(java.net.URLDecoder(fileURI.toString))
{quote}
This is likely not a bug, although documentation could be improved.
> SparkContext.textFile() cannot handle escape character for spaces
> -----------------------------------------------------------------
>
> Key: SPARK-10596
> URL: https://issues.apache.org/jira/browse/SPARK-10596
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Reporter: Jason Pohl
>
> Trying to use SparkContext.textFile() with an S3 URI, but there does not seem
> to be an accepted way to escape spaces in the path (which is a valid S3
> naming convention for folders)
> get an exception: "org.apache.hadoop.mapred.InvalidInputException: Input path
> does not exist:"
> Tried:
> +
> %20
> \%20
> \%20\
> None of these work
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
