Shaofeng SHI created HBASE-18885:
------------------------------------
Summary: HFileOutputFormat2 hardcodes default FileOutputCommitter
Key: HBASE-18885
URL: https://issues.apache.org/jira/browse/HBASE-18885
Project: HBase
Issue Type: Bug
Components: mapreduce
Reporter: Shaofeng SHI
Apache Kylin uses HBase's HFileOutputFormat2.java to configure the MR job. The
original reporting is in KYLIN-2788[1]. After some investigation, we found this
class always uses the default "FileOutputCommitter", see [2], regardless of the
job's configuration; so it always writing to "_temporary" folder. Since AWS EMR
configured to use DirectOutputCommitter for S3, then this problem occurs:
Hadoop expects to see the file directly under output path, while the
RecordWriter generates them in "_temporary" folder. This caused no data be
loaded to HTable.
Seems this problem exists in all versions so far.
[1] https://issues.apache.org/jira/browse/KYLIN-2788
[2]
https://github.com/apache/hbase/blob/master/hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L193
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
