Compaction Ordering for Bulk Import Files
-----------------------------------------
Key: HBASE-3404
URL: https://issues.apache.org/jira/browse/HBASE-3404
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.0, 0.90.1, 0.92.0
Reporter: Nicolas Spiegelberg
Assignee: Nicolas Spiegelberg
We got into an issue today where we were using HFileOutputFormat to perform an
incremental load on an already-large cluster. Because bulk-loaded files don't
have a sequence ID, they are put in the front of the StoreFile list. This
resulted in the following StoreFile ordering
2GB (bulk) => 25GB => 2GB => ...
So this triggered a 30+GB major compaction for every single region. Optimally,
we would like bulk import files to be ordered in the compaction list at the
time of insertion so this can be a much smaller compaction and rely on
StoreFile age for major compaction trigger.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
