[
https://issues.apache.org/jira/browse/SPARK-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624300#comment-14624300
]
Sun Rui commented on SPARK-8952:
--------------------------------
Currently normalizePath() is used in several places allowing users to specify
paths via the use of tilde expansion, or to normalize a relative path to an
absolute path. However, normalizePath() is used for paths which are actually
expected to be a URI. normalizePath() may display warning messages when it does
not recognize a URI as a local file path. So suppressWarnings() is used to
suppress the possible warnings.
Worse than warnings, call normalizePath() on a URI may cause error. Because it
may turn a user specified relative path to an absolute path using the local
current directory, but this may not be true because the path is actually
relative to the working directory of the default file system instead of the
local file system (depends on the Hadoop configuration of Spark).
So I would recommend to eliminate the usage of normalizePath() for URI in cases
for example, textFile, objectFile, jsonFile, parquetFile().
> JsonFile() of SQLContext display improper warning message for a S3 path
> -----------------------------------------------------------------------
>
> Key: SPARK-8952
> URL: https://issues.apache.org/jira/browse/SPARK-8952
> Project: Spark
> Issue Type: Bug
> Components: SparkR
> Affects Versions: 1.4.0
> Reporter: Sun Rui
>
> This is an issue reported by Ben Spark <[email protected]>.
> {quote}
> Spark 1.4 deployed on AWS EMR
> "jsonFile" is working though with some warning message
> Warning message:
> In normalizePath(path) :
>
> path[1]="s3://rea-consumer-data-dev/cbr/profiler/output/20150618/part-00000":
> No such file or directory
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
