stack commented on HBASE-16698:

Committed below addendum to address this FindBugs complaint:

Code    Warning
 does not release lock on all paths
Bug type UL_UNRELEASED_LOCK (click for details) 
In class org.apache.hadoop.hbase.regionserver.HRegion
In method 
At HRegion.java:[line 3313]

stack-MBP:hbase stack$ git show -1
commit e1923b7c0c14b435ea0d9eb306d968f1927a0c6e
Author: Michael Stack <st...@apache.org>
Date:   Thu Oct 13 17:16:47 2016 -0700

    HBASE-16698 Performance issue: handlers stuck waiting for CountDownLatch 
inside WALKey#getWriteEntry under high writing workload; ADDENDUM. Fix findbugs

diff --git 
index 3715ca1..a486599 100644
@@ -3310,7 +3310,6 @@ public class HRegion implements HeapSize, 
PropagatingConfigurationObserver, Regi
       } else {
         // writeEntry won't be empty if not in replay mode
-        assert writeEntry != null;
         writeEntry = null;

@appy kicked me for committing w/ a FindBugs....  (Thanks @appy)

> Performance issue: handlers stuck waiting for CountDownLatch inside 
> WALKey#getWriteEntry under high writing workload
> --------------------------------------------------------------------------------------------------------------------
>                 Key: HBASE-16698
>                 URL: https://issues.apache.org/jira/browse/HBASE-16698
>             Project: HBase
>          Issue Type: Improvement
>          Components: Performance
>    Affects Versions: 1.1.6, 1.2.3
>            Reporter: Yu Li
>            Assignee: Yu Li
>             Fix For: 2.0.0
>         Attachments: HBASE-16698.branch-1.patch, HBASE-16698.patch, 
> HBASE-16698.v2.patch, hadoop0495.et2.jstack
> As titled, on our production environment we observed 98 out of 128 handlers 
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside 
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by 
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call 
> {{WALKey#getWriteEntry}} after appending edit to WAL, and 
> {{seqNumAssignedLatch}} is only released when the relative append call is 
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}). 
> Because currently we're using a single event handler for the ringbuffer, the 
> append calls are handled one by one (actually lot's of our current logic 
> depending on this sequential dealing logic), and this becomes a bottleneck 
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on 
> all regions are dealt with in sequential, which causes contention among 
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism, 
> that we could grab the WriteEntry before publishing append onto ringbuffer 
> and use it as sequence id, only that we need to add a lock to make "grab 
> WriteEntry" and "append edit" a transaction. This will still cause contention 
> inside a region but could avoid contention between different regions. This 
> solution is already verified in our online environment and proved to be 
> effective.
> Notice that for master (2.0) branch since we already change the write 
> pipeline to sync before writing memstore (HBASE-15158), this issue only 
> exists for the ASYNC_WAL writes scenario.

This message was sent by Atlassian JIRA

Reply via email to