[ https://issues.apache.org/jira/browse/SPARK-8952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624300#comment-14624300 ]
Sun Rui commented on SPARK-8952: -------------------------------- Currently normalizePath() is used in several places allowing users to specify paths via the use of tilde expansion, or to normalize a relative path to an absolute path. However, normalizePath() is used for paths which are actually expected to be a URI. normalizePath() may display warning messages when it does not recognize a URI as a local file path. So suppressWarnings() is used to suppress the possible warnings. Worse than warnings, call normalizePath() on a URI may cause error. Because it may turn a user specified relative path to an absolute path using the local current directory, but this may not be true because the path is actually relative to the working directory of the default file system instead of the local file system (depends on the Hadoop configuration of Spark). So I would recommend to eliminate the usage of normalizePath() for URI in cases for example, textFile, objectFile, jsonFile, parquetFile(). > JsonFile() of SQLContext display improper warning message for a S3 path > ----------------------------------------------------------------------- > > Key: SPARK-8952 > URL: https://issues.apache.org/jira/browse/SPARK-8952 > Project: Spark > Issue Type: Bug > Components: SparkR > Affects Versions: 1.4.0 > Reporter: Sun Rui > > This is an issue reported by Ben Spark <ben_spar...@yahoo.com.au>. > {quote} > Spark 1.4 deployed on AWS EMR > "jsonFile" is working though with some warning message > Warning message: > In normalizePath(path) : > > path[1]="s3://rea-consumer-data-dev/cbr/profiler/output/20150618/part-00000": > No such file or directory > {quote} -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org