[
https://issues.apache.org/jira/browse/HBASE-3504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990696#comment-12990696
]
Jonathan Gray commented on HBASE-3504:
--------------------------------------
We're measuring 20-40ms increment operations with all time being spent in WAL.
When jstacking, all threads are waiting on updateLock. At some point (and this
seems to be a common behavior under heavy concurrency) the lock becomes such a
point of contention that performance all of a sudden drops dramatically. We'll
see 0-2ms WAL on lower loaded servers and then 20-40ms on higher loaded
servers... seemingly there is a tipping point. This happened with other locks
as well that we have removed or moved to read/write locks.
Basically, the cluster is crawling because a couple RS have insanely long WAL
times with every thread piled on the updateLock. It is the only current
bottleneck we see (unless we turn off the WAL). The actual sync latency is
still low, it's just the updateLock contention.
The critical point of seqids and HLogs is that the range of seqids in each HLog
must never overlap (that's where the write lock comes in) and we must be able
to determine the highest seqid in an HLog. Also, that seqids within a row are
in order. Seqids across rows do not need to be in order; within a region, it's
only important to ensure we properly track the oldest seqid in the MemStore and
the latest seqid in each HLog.
> HLog performance improvement
> ----------------------------
>
> Key: HBASE-3504
> URL: https://issues.apache.org/jira/browse/HBASE-3504
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> The HLog.updateLock protects the rolling of logs with concurrent writes to
> the HDFS log file. This is a scalability bottleneck for a workload that
> comprises mostly of counter-increments.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
