[
https://issues.apache.org/jira/browse/HBASE-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932265#action_12932265
]
Gary Helmling commented on HBASE-3235:
--------------------------------------
Some further info on this. I added logging of the contents of
store.memstore.kvset when kvset.size() > 1 in
testIncrementColumnValue_UpdatingInPlace(), like so:
{code}
long value = 1L;
long amount = 3L;
Put put = new Put(row);
put.add(fam1, qual1, Bytes.toBytes(value));
region.put(put);
long result = region.incrementColumnValue(row, fam1, qual1, amount, true);
assertEquals(value+amount, result);
Store store = region.getStore(fam1);
// ICV removes any extra values floating around in there.
if (store.memstore.kvset.size() > 1) {
for (KeyValue kv : store.memstore.kvset) {
LOG.debug("memstore.kvset: row="+Bytes.toString(kv.getRow()) +
" fam="+Bytes.toString(kv.getFamily())+"
col="+Bytes.toString(kv.getQualifier())+
" val="+Bytes.toLong(kv.getValue())+" ts="+kv.getTimestamp()+"
memstore_ts="+kv.getMemstoreTS());
}
}
assertEquals(1, store.memstore.kvset.size());
assertTrue(store.memstore.snapshot.isEmpty());
{code}
With this in place, I get the following output for the failure case:
{noformat}
2010-11-15 15:36:24,242 DEBUG [main] regionserver.TestHRegion(1891):
memstore.kvset: row=rowA fam=colfamily1 col=qual1 val=1 ts=1289864184241
memstore_ts=1
2010-11-15 15:36:24,242 DEBUG [main] regionserver.TestHRegion(1891):
memstore.kvset: row=rowA fam=colfamily1 col=qual1 val=4 ts=1289864184241
memstore_ts=0
{noformat}
So it seems like it's only happening when the timestamps for the initial put
and the ICV are identical. In this case, the put gets memstoreTS=1 (from
RWCC.getWriteNumber()), but the ICV memstoreTS=0 in MemStore.upsert(List).
So, in MemStore.upsert(KeyValue), when we call kvset.tailSet(kv), we're missing
the initial put, because KeyValue.KV_COMPARATOR is evaluating it as less than
the ICV put, by inverting the memstoreTS comparison.
Given that that's the cause, what is the correct fix? Should the ICV put get
using RWCC.getWriteNumber() as well? Or should we be obtaining the tailset
without regards to the memstoreTS?
> Intermittent incrementColumnValue failure in TestHRegion
> --------------------------------------------------------
>
> Key: HBASE-3235
> URL: https://issues.apache.org/jira/browse/HBASE-3235
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.90.0
> Reporter: Gary Helmling
>
> I first saw this in a Hudson build, but can reproduce locally with enough
> test runs (5-10 times):
> {noformat}
> -------------------------------------------------------------------------------
> Test set: org.apache.hadoop.hbase.regionserver.TestHRegion
> -------------------------------------------------------------------------------
> Tests run: 51, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.413 sec
> <<< FAILURE!
> testIncrementColumnValue_UpdatingInPlace(org.apache.hadoop.hbase.regionserver.TestHRegion)
> Time elapsed: 0.079 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<1> but was:<2>
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:283)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:195)
> at junit.framework.Assert.assertEquals(Assert.java:201)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testIncrementColumnValue_UpdatingInPlace(TestHRegion.java:1889)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> Alternately, the failure can also show up in
> testIncrementColumnValue_UpdatingInPlace_Negative():
> {noformat}
> testIncrementColumnValue_UpdatingInPlace_Negative(org.apache.hadoop.hbase.regionserver.TestHRegion)
> Time elapsed: 0.03 sec <<< FAILURE!
> junit.framework.AssertionFailedError: expected:<2> but was:<3>
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.failNotEquals(Assert.java:283)
> at junit.framework.Assert.assertEquals(Assert.java:64)
> at junit.framework.Assert.assertEquals(Assert.java:130)
> at junit.framework.Assert.assertEquals(Assert.java:136)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.assertICV(TestHRegion.java:2081)
> at
> org.apache.hadoop.hbase.regionserver.TestHRegion.testIncrementColumnValue_UpdatingInPlace_Negative(TestHRegion.java:1990)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
