HFileOutputFormat writes output to "unsafe" directory
-----------------------------------------------------
Key: HBASE-2280
URL: https://issues.apache.org/jira/browse/HBASE-2280
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.20.3
Reporter: Vladimir Klimontovich
Priority: Critical
Fix For: 0.20.5, 0.21.0, 0.22.0
HFileOutputFormat writes data direct to output folder. It's incorrect as failed
(or killed, or interrupted) reducers leaves inconsistent files in output folder.
The convinient way to ouput data from OutputFormat is to use "working
directory". The content of this directory is being moved to output directory at
the end of reducer process if only reducer succeeded (this process is called
"output commit" or "atomic commit").
If means that instead of
final Path outputdir = FileOutputFormat.getOutputPath(context);
hbase should use
final Path outputdir = FileOutputFormat.getWorkOutputPath(context);
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
