[
https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15638184#comment-15638184
]
stack commented on HBASE-16890:
-------------------------------
I ran the tests a few times and results consistent. Looking in FSHLog run w/
JFR, I see more points of contention reported -- inside DFSClient. It uses
maybe 25% more CPU probably because of the upped throughput. Otherwise,
looking w/ JFR nothing jumps out. Let me put up pictures of the 'hot methods'
It is almost as though FSHLog is doing more work (The top consumers are the
WALPE random generation... we should fix that).
The FSHLog must have a better 'flow' going on. Here is histograms for FSHLog:
{code}
-- Histograms ------------------------------------------------------------------
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.latencyHistogram.nanos
count = 8461245
min = 838241
max = 115799121
mean = 2696785.63
stddev = 6486391.73
median = 2199081.00
75% <= 2571547.00
95% <= 3237948.00
98% <= 3621166.00
99% <= 5216818.00
99.9% <= 115799121.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncCountHistogram.countPerSync
count = 412764
min = 1
max = 86
mean = 21.04
stddev = 16.98
median = 17.00
75% <= 34.00
95% <= 53.00
98% <= 58.00
99% <= 62.00
99.9% <= 86.00
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncHistogram.nanos-between-syncs
count = 412764
min = 405379
max = 129879546
mean = 1680258.91
stddev = 7343616.88
median = 1127074.00
75% <= 1448611.00
95% <= 1812916.00
98% <= 1978098.00
99% <= 2150048.00
99.9% <= 122766311.00
-- Meters ----------------------------------------------------------------------
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.appendMeter.bytes
count = 59144801550
mean rate = 244727411.22 events/second
1-minute rate = 245882558.80 events/second
5-minute rate = 199668915.99 events/second
15-minute rate = 166822622.37 events/second
org.apache.hadoop.hbase.wal.WALPerformanceEvaluation.syncMeter.syncs
count = 412764
mean rate = 1707.90 events/second
1-minute rate = 1715.17 events/second
5-minute rate = 1342.77 events/second
15-minute rate = 1077.71 events/second
{code}
Let me get them for asyncwal...
> Analyze the performance of AsyncWAL and fix the same
> ----------------------------------------------------
>
> Key: HBASE-16890
> URL: https://issues.apache.org/jira/browse/HBASE-16890
> Project: HBase
> Issue Type: Sub-task
> Components: wal
> Affects Versions: 2.0.0
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: AsyncWAL_disruptor.patch, AsyncWAL_disruptor_1
> (2).patch, AsyncWAL_disruptor_3.patch, AsyncWAL_disruptor_3.patch,
> AsyncWAL_disruptor_4.patch, AsyncWAL_disruptor_6.patch,
> HBASE-16890-rc-v2.patch, HBASE-16890-rc-v3.patch,
> HBASE-16890-remove-contention-v1.patch, HBASE-16890-remove-contention.patch,
> Screen Shot 2016-10-25 at 7.34.47 PM.png, Screen Shot 2016-10-25 at 7.39.07
> PM.png, Screen Shot 2016-10-25 at 7.39.48 PM.png, async.svg, classic.svg,
> contention.png, contention_defaultWAL.png
>
>
> Tests reveal that AsyncWAL under load in single node cluster performs slower
> than the Default WAL. This task is to analyze and see if we could fix it.
> See some discussions in the tail of JIRA HBASE-15536.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
