Yong Tang created SPARK-7155: -------------------------------- Summary: SparkContext's newAPIHadoopFile does not support comma-separated list of files, but the other API hadoopFile does. Key: SPARK-7155 URL: https://issues.apache.org/jira/browse/SPARK-7155 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 1.3.1 Environment: Ubuntu 14.04 Reporter: Yong Tang
SparkContext's newAPIHadoopFile() does not support comma-separated list of files. For example, the following: sc.newAPIHadoopFile("/root/file1.txt,/root/file2.txt", classOf[TextInputFormat], classOf[LongWritable], classOf[Text]) will throw org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/root/file1.txt,/root/file2.txt However, the other API hadoopFile() is able to process comma-separated list of files correctly. In addition, since sc.textFile() uses hadoopFile(), it is also able to process comma-separated list of files correctly. The problem is that newAPIHadoopFile() use addInputPath() to add the file path into NewHadoopRDD. See Ln 928-931, master branch: val job = new NewHadoopJob(conf) NewFileInputFormat.addInputPath(job, new Path(path)) val updatedConf = job.getConfiguration new NewHadoopRDD(this, fClass, kClass, vClass, updatedConf).setName(path) Change addInputPath(job, new Path(path)) to addInputPaths(job, path) will resolve this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org