[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13685820#comment-13685820
]
stack commented on HBASE-8755:
------------------------------
[~fenghh] Your approach looks good. Thanks for working on this. I like the
numbers you are getting for throughput. Any idea of its effect on general
latencies? Does HLogPerformanceEvaluation help evaluating this approach? Did
you deploy this code to production?
Can I still (if only optionally) sync every write as it comes in? (For the
paranoid).
In general, remove rather than comment stuff out:
{code}
- assertTrue("Should have an outstanding WAL edit",
log.hasDeferredEntries());
+ //assertTrue("Should have an outstanding WAL edit",
log.hasDeferredEntries());
{code}
Regards the above, the test is no longer valid given the indirection around
sync/flush?
To be clear, when we call doWrite, we just append the log edit to a linked
list? (We call it a bufferLock but we just doing append to the linked list?)
Patch is looking good. Thanks [~fenghh]
How does deferred log flush still work when you remove stuff like
optionalFlushInterval? You say '...don't pend on HLog.syncer() waiting for its
txid to be sync-ed' but that is another behavior than what we had here
previously.
If first thread throws IE, then other threads will still be running (in below):
{code}
+ try {
+ asyncNotifier.interrupt();
+ asyncNotifier.join();
+
+ asyncFlusher.interrupt();
+ asyncFlusher.join();
+
+ asyncWriter.interrupt();
+ asyncWriter.join();
+ } catch (InterruptedException e) {
+ LOG.error("Exception while waiting for syncer thread to die", e);
}
{code}
Thread names usually follow a pattern where the prefix is the name of the host
(RS:SHORT_SERVERNAME-AsyncWriter).. we can fix around commit.
This new thread does not implement HasThread:
+ private class AsyncNotifier extends Thread {
> A new write thread model for HLog to improve the overall HBase write
> throughput
> -------------------------------------------------------------------------------
>
> Key: HBASE-8755
> URL: https://issues.apache.org/jira/browse/HBASE-8755
> Project: HBase
> Issue Type: Improvement
> Components: wal
> Reporter: Feng Honghua
> Attachments: HBASE-8755-0.94-V0.patch
>
>
> In current write model, each write handler thread (executing put()) will
> individually go through a full 'append (hlog local buffer) => HLog writer
> append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write,
> which incurs heavy race condition on updateLock and flushLock.
> The only optimization where checking if current syncTillHere > txid in
> expectation for other thread help write/sync its own txid to hdfs and
> omitting the write/sync actually help much less than expectation.
> Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
> proposed a new write thread model for writing hdfs sequence file and the
> prototype implementation shows a 4X improvement for throughput (from 17000 to
> 70000+).
> I apply this new write thread model in HLog and the performance test in our
> test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
> RS, from 22000 to 70000 for 5 RS), the 1 RS write throughput (1K row-size)
> even beats the one of BigTable (Precolator published in 2011 says Bigtable's
> write throughput then is 31002). I can provide the detailed performance test
> results if anyone is interested.
> The change for new write thread model is as below:
> 1> All put handler threads append the edits to HLog's local pending buffer;
> (it notifies AsyncWriter thread that there is new edits in local buffer)
> 2> All put handler threads wait in HLog.syncer() function for underlying
> threads to finish the sync that contains its txid;
> 3> An single AsyncWriter thread is responsible for retrieve all the buffered
> edits in HLog's local pending buffer and write to the hdfs
> (hlog.writer.append); (it notifies AsyncFlusher thread that there is new
> writes to hdfs that needs a sync)
> 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs
> to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread
> that sync watermark increases)
> 5> An single AsyncNotifier thread is responsible for notifying all pending
> put handler threads which are waiting in the HLog.syncer() function
> 6> No LogSyncer thread any more (since there is always
> AsyncWriter/AsyncFlusher threads do the same job it does)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
