[
https://issues.apache.org/jira/browse/HBASE-6311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-6311:
--------------------------------
Description:
It is a big problem we found in 0.94, and you could reproduce the problem in
Trunk using the test case I uploaded.
When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC
for opened scanners;
However,It will make data mistake after majorCompaction because we will skip
delete type KV but keep the put type kv in the compacted storefile.
The following is the reason from code:
In StoreFileScanner, enforceMVCC is false when compaction, so we could read the
delete type KV,
However, we will skip this delete type KV in ScanQueryMatcher because following
code
{code}
if (kv.isDelete())
{
...
if (includeDeleteMarker
&& kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
System.out.println("add deletes,maxReadPointToTrackVersions="
+ maxReadPointToTrackVersions);
this.deletes.add(bytes, offset, qualLength, timestamp, type);
}
...
}
{code}
Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
and kv.getMemstoreTS() > maxReadPointToTrackVersions
So we won't add this to DeleteTracker.
Why test case passed if remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Because in the StoreFileScanner#skipKVsNewerThanReadpoint
{code}
if (cur.getMemstoreTS() <= readPoint) {
cur.setMemstoreTS(0);
}
{code}
So if we remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add
it to DeleteTracker in ScanQueryMatcher
Solution:
We use smallestReadPoint of region when compaction to keep MVCC for OPENED
scanner, So we should retain delete type kv in output in the case(Already
deleted KV is retained in output to make old opened scanner could read this KV)
even if it is a majorcompaction.
was:
It is a big problem we found in 0.94, and you could reproduce the problem in
Trunk using the test case I uploaded.
When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC
for opened scanners;
However,It will make data mistake after compaction because we will skip delete
type KV but keep the put type kv in the compacted storefile.
The following is the reason from code:
In StoreFileScanner, enforceMVCC is false when compaction, so we could read the
delete type KV,
However, we will skip this delete type KV in ScanQueryMatcher because following
code
{code}
if (kv.isDelete())
{
...
if (includeDeleteMarker
&& kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
System.out.println("add deletes,maxReadPointToTrackVersions="
+ maxReadPointToTrackVersions);
this.deletes.add(bytes, offset, qualLength, timestamp, type);
}
...
}
{code}
Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
and kv.getMemstoreTS() > maxReadPointToTrackVersions
So we won't add this to DeleteTracker.
Why test case passed if remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Because in the StoreFileScanner#skipKVsNewerThanReadpoint
{code}
if (cur.getMemstoreTS() <= readPoint) {
cur.setMemstoreTS(0);
}
{code}
So if we remove the line
MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will add
it to DeleteTracker in ScanQueryMatcher
Solution:
We use smallestReadPoint of region when compaction to keep MVCC for OPENED
scanner, So we should retain delete type kv in output in the case(Already
deleted KV is retained in output to make old opened scanner could read this KV)
even if it is a majorcompaction.
Summary: Data error after majorCompaction because keep MVCC for opened
scanners (was: Data error after compaction because keep MVCC for opened
scanners)
> Data error after majorCompaction because keep MVCC for opened scanners
> ----------------------------------------------------------------------
>
> Key: HBASE-6311
> URL: https://issues.apache.org/jira/browse/HBASE-6311
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.94.0
> Reporter: chunhui shen
> Assignee: chunhui shen
> Priority: Blocker
> Attachments: HBASE-6311-test.patch, HBASE-6311v1.patch
>
>
> It is a big problem we found in 0.94, and you could reproduce the problem in
> Trunk using the test case I uploaded.
> When we do compaction, we will use region.getSmallestReadPoint() to keep MVCC
> for opened scanners;
> However,It will make data mistake after majorCompaction because we will skip
> delete type KV but keep the put type kv in the compacted storefile.
> The following is the reason from code:
> In StoreFileScanner, enforceMVCC is false when compaction, so we could read
> the delete type KV,
> However, we will skip this delete type KV in ScanQueryMatcher because
> following code
> {code}
> if (kv.isDelete())
> {
> ...
> if (includeDeleteMarker
> && kv.getMemstoreTS() <= maxReadPointToTrackVersions) {
> System.out.println("add deletes,maxReadPointToTrackVersions="
> + maxReadPointToTrackVersions);
> this.deletes.add(bytes, offset, qualLength, timestamp, type);
> }
> ...
> }
> {code}
> Here maxReadPointToTrackVersions = region.getSmallestReadPoint();
> and kv.getMemstoreTS() > maxReadPointToTrackVersions
> So we won't add this to DeleteTracker.
> Why test case passed if remove the line
> MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Because in the StoreFileScanner#skipKVsNewerThanReadpoint
> {code}
> if (cur.getMemstoreTS() <= readPoint) {
> cur.setMemstoreTS(0);
> }
> {code}
> So if we remove the line
> MultiVersionConsistencyControl.setThreadReadPoint(smallestReadPoint);
> Here readPoint is LONG.MAX_VALUE, we will set memStore ts as 0, so we will
> add it to DeleteTracker in ScanQueryMatcher
> Solution:
> We use smallestReadPoint of region when compaction to keep MVCC for OPENED
> scanner, So we should retain delete type kv in output in the case(Already
> deleted KV is retained in output to make old opened scanner could read this
> KV) even if it is a majorcompaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
