Joseph Su created SPARK-2867:
--------------------------------
Summary: saveAsHadoopFile() in PairRDDFunction.scala should allow
use other OutputCommiter class
Key: SPARK-2867
URL: https://issues.apache.org/jira/browse/SPARK-2867
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0, 1.1.0
Reporter: Joseph Su
Priority: Minor
The saveAsHadoopFile() in PairRDDFunction.scala hard-coded the OutputCommitter
class as FileOutputCommitter because of the following code in the source:
hadoopConf.setOutputCommitter(classOf[FileOutputCommitter])
However, OutputCommitter is a changeable option in regular Hadoop MapReduce
program. Users can specify "mapred.output.committer.class" to change the
committer class used by other Hadoop programs.
The saveAsHadoopFile() function should remove this hard-coded assignment and
provide a way to specify the OutputCommitte used here.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
