high multiput latency due to checking global mem store size in a synchronized
function
--------------------------------------------------------------------------------------
Key: HBASE-3694
URL: https://issues.apache.org/jira/browse/HBASE-3694
Project: HBase
Issue Type: Improvement
Reporter: Liyin Tang
Assignee: Liyin Tang
The problem is we found the multiput latency is very high.
In our case, we have almost 22 Regions in each RS and there are no flush
happened during these puts.
After investigation, we believe that the root cause is the function
getGlobalMemStoreSize, which is to check the high water mark of mem store.
This function takes almost 40% of total execution time of multiput when
instrumenting some metrics in the code.
The actual percentage may be more higher. The execution time is spent on
synchronize contention.
One solution is to keep a static var in HRegion to keep the global MemStore
size instead of calculating them every time.
Why using static variable?
Since all the HRegion objects in the same JVM share the same memory heap, they
need to share fate as well.
The static variable, globalMemStroeSize, naturally shows the total mem usage in
this shared memory heap for this JVM.
If multiple RS need to run in the same JVM, they still need only one
globalMemStroeSize.
If multiple RS run on different JVMs, everything is fine.
After changing, in our cases, the avg multiput latency decrease from 60ms to
10ms.
I will submit a patch based on the current trunk.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
