[
https://issues.apache.org/jira/browse/HBASE-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16948899#comment-16948899
]
Xu Cang commented on HBASE-23143:
---------------------------------
comes during flush I believe.
Related stacktrac: ( I have to remove some company confidential table names and
so on)
2019-10-03 09:44:11,600 FATAL [MemStoreFlusher.0] regionserver.HRegionServer -
ABORTING region server xxx: Replay of WAL required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region:
TABLEXXX,00D7F000000rGWE0D57F000025zuZT0057F000003biB1T\x000000000000000000057F000003biB1,1555980511237.d333ca887c6c78385dc44c0e7ddf97df.
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2621)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2297)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2259)
at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2143)
at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2068) at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:512)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:482)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:76)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:264)
at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: Added
a key not lexically larger than previous. Current cell =
00D7F000000xG5g0D52v00008UYAOg0F92v000000oOMnG\x000DB7F000000Cc3q000000000000000/0:COLUMNX/1570095189616/DeleteColumn/vlen=0/seqid=1981610,
lastCell =
00D7F000000xG5g0D52v00008UYAOg0F92v000000oOMnG\x000DB7F000000Cc3q000000000000000/0:COLUMNX/1570095164786/DeleteColumn/vlen=0/seqid=1981620
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:267)
at org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1184)
at
org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:138)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:991) at
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2506)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2567)
... 9 more
> Region Server Crash due to 2 cells out of order ( between 2 DELETEs)
> --------------------------------------------------------------------
>
> Key: HBASE-23143
> URL: https://issues.apache.org/jira/browse/HBASE-23143
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.3.2
> Reporter: Xu Cang
> Priority: Major
>
> Region Server Crash due to 2 cells out of order ( between 2 DELETEs)
>
> Caused by: java.io.IOException: Added a key not lexically larger than
> previous.
> Current cell =
> 00D7F000000xxQ10D52v00008UY6yV0057F000000bPaGT\x000000000000000000057F000000bPaG/0:TABLE1_ID/*1570095189597*/DeleteColumn/vlen=0/seqid=*2128373*,
>
> lastCell =
> 00D7F000000xxQ10D52v00008UY6yV0057F000000bPaGT\x000000000000000000057F000000bPaG/0:TABLE1_ID/*1570095165147*/DeleteColumn/vlen=0/seqid=*2128378*
>
>
> I am aware of this JIRA: https://issues.apache.org/jira/browse/HBASE-22862
> Though it's slightly different, HBASE-22862 issue was caused One Delete and
> One Put.
> This issue I am reporting is caused by 2 Deletes
>
> Has anyone seen this issue?
>
> After I read the code and debugged the test cases.
> In AbstractHFileWriter.java
> {code:java}
> int keyComp = comparator.compareOnlyKeyPortion(lastCell, cell);{code}
> This call will always ignore SequenceId. And time stamps are in the correct
> order (above case)
> And since these 2 cells have same KEY. The comparison result should be 0.
> *only possible issue I can think of is, in this code piece: in
> CellComparator.java:*
> {code:java}
> Bytes.compareTo(left.getRowArray(), left.getRowOffset(), left.getRowLength(),
> right.getRowArray(), right.getRowOffset(), right.getRowLength());{code}
> The getRowLength() returns a wrong value.
> Or the offset is messed up. (?)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
