[
https://issues.apache.org/jira/browse/SPARK-7155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yong Tang updated SPARK-7155:
-----------------------------
Description:
SparkContext's newAPIHadoopFile() does not support comma-separated list of
files. For example, the following:
sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt",
classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
will throw
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
not exist: file:/root/file1.txt,/root/file2.txt
However, the other API hadoopFile() is able to process comma-separated list of
files correctly.
In addition, since sc.textFile() uses hadoopFile(), it is also able to process
comma-separated list of files correctly.
The problem is that newAPIHadoopFile() use addInputPath() to add the file path
into NewHadoopRDD. See Ln 928-931, master branch:
val job = new NewHadoopJob(conf)
NewFileInputFormat.addInputPath(job, new Path(path))
val updatedConf = job.getConfiguration
new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)
Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will
resolve this issue.
was:
SparkContext's newAPIHadoopFile() does not support comma-separated list of
files. For example, the following:
sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt",
classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
will throw
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
not exist: file:/root/file1.txt,/root/file2.txt
However, the other API hadoopFile() is able to process comma-separated list of
files correctly. In addition, since sc.textFile() uses hadoopFile(), it is also
able to process comma-separated list of files correctly.
The problem is that newAPIHadoopFile() use addInputPath() to add the file path
into NewHadoopRDD. See Ln 928-931, master branch:
val job = new NewHadoopJob(conf)
NewFileInputFormat.addInputPath(job, new Path(path))
val updatedConf = job.getConfiguration
new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)
Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will
resolve this issue.
> SparkContext's newAPIHadoopFile does not support comma-separated list of
> files, but the other API hadoopFile does.
> ------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-7155
> URL: https://issues.apache.org/jira/browse/SPARK-7155
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.3.1
> Environment: Ubuntu 14.04
> Reporter: Yong Tang
>
> SparkContext's newAPIHadoopFile() does not support comma-separated list of
> files. For example, the following:
> sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt",
> classOf[TextInputFormat], classOf[LongWritable], classOf[Text])
> will throw
> org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does
> not exist: file:/root/file1.txt,/root/file2.txt
> However, the other API hadoopFile() is able to process comma-separated list
> of files correctly.
> In addition, since sc.textFile() uses hadoopFile(), it is also able to
> process comma-separated list of files correctly.
> The problem is that newAPIHadoopFile() use addInputPath() to add the file
> path into NewHadoopRDD. See Ln 928-931, master branch:
> val job = new NewHadoopJob(conf)
> NewFileInputFormat.addInputPath(job, new Path(path))
> val updatedConf = job.getConfiguration
> new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path)
> Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will
> resolve this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
