[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13803932#comment-13803932
]
stack commented on HBASE-8755:
------------------------------
Five datanodes with fusionio. hbase tip of 0.96 branch and hadoop-2.1.0-beta.
HLogPE on master node.
||Threads||w/o patch time||w/o patch ops||w/ patch time||w/ patch ops||
|1|1048.033s|954.168ops/s|1100.423s|908.741ops/s|
|1|1042.126s|959.577ops/s|1156.557s|864.635ops/s|
|1|1052.601s|950.028ops/s|1143.271s|874.683ops/s|
|5|904.176s|5529.896ops/s|1916.229s|2609.292ops/s|
|5|910.469s|5491.675ops/s|1911.841s|2615.280ops/s|
|5|925.778s|5400.863ops/s|1970.565s|2537.344ops/s|
|50|2699.752s|18520.221ops/s|1889.877s|26456.748ops/s|
|50|2689.678s|18589.586ops/s|1922.716s|26004.881ops/s|
|50|2711.144s|18442.398ops/s|1893.439s|26406.977ops/s|
|75|4945.563s|15165.108ops/s|1997.553s|37545.938ops/s|
|75|4852.779s|15455.063ops/s|1992.425s|37642.570ops/s|
|75|4921.685s|15238.684ops/s|-|-|
|100|6224.527s|16065.479ops/s|2086.691s|47922.766ops/s|
|100|6195.727s|16140.156ops/s|2091.869s|47804.145ops/s|
Diffs are small when 1 thread only. Its bad at 5 threads but thereafter the
patch starts to shine. If we could make the 5 threads better, we could commit
this patch.
> A new write thread model for HLog to improve the overall HBase write
> throughput
> -------------------------------------------------------------------------------
>
> Key: HBASE-8755
> URL: https://issues.apache.org/jira/browse/HBASE-8755
> Project: HBase
> Issue Type: Improvement
> Components: Performance, wal
> Reporter: Feng Honghua
> Assignee: stack
> Priority: Critical
> Fix For: 0.96.1
>
> Attachments: 8755trunkV2.txt, HBASE-8755-0.94-V0.patch,
> HBASE-8755-0.94-V1.patch, HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch
>
>
> In current write model, each write handler thread (executing put()) will
> individually go through a full 'append (hlog local buffer) => HLog writer
> append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write,
> which incurs heavy race condition on updateLock and flushLock.
> The only optimization where checking if current syncTillHere > txid in
> expectation for other thread help write/sync its own txid to hdfs and
> omitting the write/sync actually help much less than expectation.
> Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
> proposed a new write thread model for writing hdfs sequence file and the
> prototype implementation shows a 4X improvement for throughput (from 17000 to
> 70000+).
> I apply this new write thread model in HLog and the performance test in our
> test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
> RS, from 22000 to 70000 for 5 RS), the 1 RS write throughput (1K row-size)
> even beats the one of BigTable (Precolator published in 2011 says Bigtable's
> write throughput then is 31002). I can provide the detailed performance test
> results if anyone is interested.
> The change for new write thread model is as below:
> 1> All put handler threads append the edits to HLog's local pending buffer;
> (it notifies AsyncWriter thread that there is new edits in local buffer)
> 2> All put handler threads wait in HLog.syncer() function for underlying
> threads to finish the sync that contains its txid;
> 3> An single AsyncWriter thread is responsible for retrieve all the buffered
> edits in HLog's local pending buffer and write to the hdfs
> (hlog.writer.append); (it notifies AsyncFlusher thread that there is new
> writes to hdfs that needs a sync)
> 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs
> to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread
> that sync watermark increases)
> 5> An single AsyncNotifier thread is responsible for notifying all pending
> put handler threads which are waiting in the HLog.syncer() function
> 6> No LogSyncer thread any more (since there is always
> AsyncWriter/AsyncFlusher threads do the same job it does)
--
This message was sent by Atlassian JIRA
(v6.1#6144)
