[
https://issues.apache.org/jira/browse/HBASE-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919049#comment-13919049
]
stack commented on HBASE-10659:
-------------------------------
bq, IPC handler threads will prepare all Put requests, and append the WALEdit,
as one transaction, into a concurrent collection with a read lock. And then
just return;
Return to the client? The client then waits on notification back from server
on when the append completes? When you say 'prepare all Put requests', do you
mean the unmarshalling form RPC into a Put instance?
bq. Each HLogSyncer thread is corresponding to one HLog stream. It swaps the
concurrent collection with a write lock, and then iterate over all the elements
in the previous concurrent collection, generate the sequence id for each
transaction, and write to HLog. After the HLog sync is done, append these
transactions as a batch into a blocking queue.
You have multiple WALs per server? So one HLogSyncer per WAL? The concurrent
collections are kept by WAL or you have one collection and sort it by WALs
after taking it under write lock?
....
bq. Responder thread pool will return the RPC call in parallel.
In // because each MemStore Update Thread of which there may be many, each
checks out a Responder to reply to the client its mvcc/sequenceid?
Thanks Liyin. Just trying to understand and figuring it how it maps to trunk.
You like the asynchronous response back there in 0.89fb? We don't use it as
you do in trunk.
We keep talking about unifying seqid and mvcc....
> [89-fb] Optimize the threading model in HBase write path
> --------------------------------------------------------
>
> Key: HBASE-10659
> URL: https://issues.apache.org/jira/browse/HBASE-10659
> Project: HBase
> Issue Type: New Feature
> Reporter: Liyin Tang
>
> Recently, we have done multiple prototypes to optimize the HBase (0.89)write
> path. And based on the simulator results, the following model is able to
> achieve much higher overall throughput with less threads.
> IPC Writer Threads Pool:
> IPC handler threads will prepare all Put requests, and append the WALEdit, as
> one transaction, into a concurrent collection with a read lock. And then just
> return;
> HLogSyncer Thread:
> Each HLogSyncer thread is corresponding to one HLog stream. It swaps the
> concurrent collection with a write lock, and then iterate over all the
> elements in the previous concurrent collection, generate the sequence id for
> each transaction, and write to HLog. After the HLog sync is done, append
> these transactions as a batch into a blocking queue.
> Memstore Update Thread:
> The memstore update thread will poll the blocking queue and update the
> memstore for each transaction by using the sequence id as MVCC. Once the
> memstore update is done, dispatch to the responder thread pool to return to
> the client.
> Responder Thread Pool:
> Responder thread pool will return the RPC call in parallel.
> We are still evaluating this model and will share more results/numbers once
> it is ready. But really appreciate any comments in advance !
--
This message was sent by Atlassian JIRA
(v6.2#6252)
