[
https://issues.apache.org/jira/browse/HBASE-26763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17495872#comment-17495872
]
Xiaolin Ha commented on HBASE-26763:
------------------------------------
I think the key problem here is that some entries are repeatedly added to the
AsyncFSWAL#toWriteAppends, e.g. 1,2,3,4,5,6,7,8,9,4,5,6,7,8,9,10..., then the
IllegalArgumentException is very easy to occur. What's more, I think this
problem is relevant to https://issues.apache.org/jira/browse/HBASE-22761,
because once wal entry syncComplete may cause the direct byte buffer be
released and the next sync will write wrong data that the entry refers.
Here is a scenario that may reproduce the problem,
# toWriteAppends=1,2,3,4,5,6,7,8,9,10...
# sync entries 1,2,3, whose total size is up to the configured batch size.
toWriteAppends=4,5,6,7,8,9, unackedAppends=1,2,3;
# sync entries 4,5,6, toWriteAppends=7,8,9, unackedAppends=1,2,3,4,5,6;
# sync entries 7,8,9, toWriteAppends=10..., unackedAppends=1,2,3,4,5,6,7,8,9;
# 1,2,3 sync failed, then all the entries in the unackedAppends are added back
to toWriteAppends, toWriteAppends=1,2,3,4,5,6,7,8,9,10....,
unackedAppends=1,2,3,4,5,6,7,8,9;
# wal rolled success.
# sync entries 1,2,3, toWriteAppends=4,5,6,7,8,9,10....,
unackedAppends=1,2,3,4,5,6,7,8,9;
# 1,2,3 sync failed, again, all the entries in the unackedAppends are added
back to toWriteAppends, toWriteAppends=1,2,3,4,5,6,7,8,9,4,5,6,7,8,9,10....,
unackedAppends=1,2,3,4,5,6,7,8,9;
Now there are repeatedly entries in the toWriteAppends,
IllegalArgumentException can occur, e.g. sync(7,8,9,4).
And if 4,5,6,7,8,9 sync completed, their direct byte buffer will be release,
then when sync them again by the toWriteAppends, wrong data might be written to
wal.
> Setback occurs to AsyncFSWAL#highestProcessedAppendTxid and then
> IllegalArgumentException
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-26763
> URL: https://issues.apache.org/jira/browse/HBASE-26763
> Project: HBase
> Issue Type: Bug
> Components: wal
> Affects Versions: 3.0.0-alpha-2, 2.4.9
> Reporter: Xiaolin Ha
> Priority: Major
>
> The consumer thread in AsyncFSWAL crashed by IllegalArgumentException, stack
> trace is as follows,
> {code:java}
> Exception in thread "XXXX" java.lang.IllegalArgumentException: fromKey > toKey
> at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1368)
> at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1855)
> at java.util.TreeMap.subMap(TreeMap.java:913)
> at java.util.TreeSet.subSet(TreeSet.java:325)
> at java.util.TreeSet.subSet(TreeSet.java:361)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.isHsync(AsyncFSWAL.java:375)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:392)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:504)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:565)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Exception in thread "XXXX" java.lang.IllegalArgumentException: fromKey > toKey
> at java.util.TreeMap$NavigableSubMap.<init>(TreeMap.java:1368)
> at java.util.TreeMap$AscendingSubMap.<init>(TreeMap.java:1855)
> at java.util.TreeMap.subMap(TreeMap.java:913)
> at java.util.TreeSet.subSet(TreeSet.java:325)
> at java.util.TreeSet.subSet(TreeSet.java:361)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.isHsync(AsyncFSWAL.java:375)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.sync(AsyncFSWAL.java:392)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync(AsyncFSWAL.java:504)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume(AsyncFSWAL.java:565)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745) {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
