[
https://issues.apache.org/jira/browse/HBASE-8709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678680#comment-13678680
]
stack commented on HBASE-8709:
------------------------------
bq. ...if we stored seqnum for each KV...
We have talked of doing this in the past a few times. We currently have in KV
an extra data member to hold the current mvcc number and when we flush an hfile
we write out the highest keys' mvcc number so we know where to slot in it's
content in say an outstanding scan.
Adding seqnum to some KVs and persisting out to storefile would take some work.
When we store the seqid in memstore, would we sort on it? Would we allow
multiple versions in memstore distingushed by their seqid only? Currently the
newer edit would overwrite the older in memstore.
> consider a scheme to allow compacting files in any combination
> --------------------------------------------------------------
>
> Key: HBASE-8709
> URL: https://issues.apache.org/jira/browse/HBASE-8709
> Project: HBase
> Issue Type: Brainstorming
> Components: Compaction, HFile
> Reporter: Sergey Shelukhin
>
> We were discussing something and I came up with the following scheme.
> Consider this. The main problem for choosing out-of-order files for
> compactions is full key collisions (k, cf:c, ts are the same). We rely on
> file seqnum to resolve these. There'd be no problem if we stored seqnum for
> each KV, but that is an overkill. What can we do is this.
> 1) Store min seqnum for a file together with the max. Assume file seqnum
> ranges don't overlap.
> 2) Store seqnum for each KV in memory of the memstore.
> 3) On flush, don't write out seqnums unless there's a full conflict inside
> this memstore. We will have to change file format unfortunately to tuck on a
> bit somewhere to indicate there's varint seqnum.
> 4) On compaction, when dropping versions we can drop these seqnums.
> 5) On compaction, if we see a full conflict with no seqnums (i.e. KVs coming
> from different files), write out seqnums for the KVs involved as median of
> the respective file ranges (or something like that). We only ever use these
> KVs to resolve full conflicts so we don't care about relations between keys.
> Both of the places where we write seqnums we will need to see the next KV
> before writing previous KV, so there's some complexity, however the
> "buffering" is never more than one KV long - if we see different k-c-t we
> know we don't need a seqnum unless we are in conflict with previous KVs which
> we have already written, if we see the same one we know we need a seqnum.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
