Sergey Shelukhin created HBASE-8709:
---------------------------------------
Summary: consider a scheme to allow compating files in any
combination
Key: HBASE-8709
URL: https://issues.apache.org/jira/browse/HBASE-8709
Project: HBase
Issue Type: Brainstorming
Components: Compaction, HFile
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
We were discussing something and I came up with the following scheme. Consider
this. The main problem for choosing out-of-order files for compactions is full
key collisions (k, cf:c, ts are the same). We rely on file seqnum to resolve
these. There'd be no problem if we stored seqnum for each KV, but that is an
overkill. What can we do is this.
1) Store min seqnum for a file together with the max. Assume file seqnum ranges
don't overlap.
2) Store seqnum for each KV in memory of the memstore.
3) On flush, don't write out seqnums unless there's a full conflict inside this
memstore. We will have to change file format unfortunately to tuck on a bit
somewhere to indicate there's varint seqnum.
4) On compaction, when dropping versions we can drop these seqnums.
5) On compaction, if we see a full conflict with no seqnums (i.e. KVs coming
from different files), write out seqnums for the KVs involved as median of the
respective file ranges (or something like that). We only ever use these KVs to
resolve full conflicts so we don't care about relations between keys.
Both of the places where we write seqnums we will need to see the next KV
before writing previous KV, so there's some complexity, however the "buffering"
is never more than one KV long - if we see different k-c-t we know we don't
need a seqnum unless we are in conflict with previous KVs which we have already
written, if we see the same one we know we need a seqnum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
