[
https://issues.apache.org/jira/browse/HBASE-3404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12976155#action_12976155
]
Nicolas Spiegelberg commented on HBASE-3404:
--------------------------------------------
@Todd : You're thinking of the original idea for max compaction size pruning in
HBASE-3209, where my first diff attempted to sort the files, then compact. The
problem is that we use StoreFile seqid during compaction to determine which
value to use when we encounter a duplicate ROW+COL+TS. The problem is this:
1. we could have 3 StoreFiles with seqid [1,2,3]
2. there is a duplicate in 1 & 2
3. we sort by size and 2 is small, so we compact [1,3]
4. the resulting StoreFile has seqid 3, so it would beat seqid 2 in compaction
This fix should be addressed in HBASE-2856, which associates each KV with a
seqid. However, I was hoping to get a quicker fix in for this case
> Compaction Ordering for Bulk Import Files
> -----------------------------------------
>
> Key: HBASE-3404
> URL: https://issues.apache.org/jira/browse/HBASE-3404
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.90.0, 0.90.1, 0.92.0
> Reporter: Nicolas Spiegelberg
> Assignee: Nicolas Spiegelberg
>
> We got into an issue today where we were using HFileOutputFormat to perform
> an incremental load on an already-large cluster. Because bulk-loaded files
> don't have a sequence ID, they are put in the front of the StoreFile list.
> This resulted in the following StoreFile ordering
> 2GB (bulk) => 25GB => 2GB => ...
> So this triggered a 30+GB major compaction for every single region.
> Optimally, we would like bulk import files to be ordered in the compaction
> list at the time of insertion so this can be a much smaller compaction and
> rely on StoreFile age for major compaction trigger.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
