[
https://issues.apache.org/jira/browse/ZOOKEEPER-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13539143#comment-13539143
]
Thawan Kooburat commented on ZOOKEEPER-1609:
--------------------------------------------
In ZOOKEEPER-1505, we can viewed it as there are multiple queues in
FinalRequestProcessor phase. The CommitProcessor determines how many requests
can enter these queues at once.
However, we currently have only one queue "entering" the CommitProcessor. In
both the existing and current logic, the CommitProcessor won't push any new
request down the pipeline at all if a write request is still waiting for a
commit from the leader. That logic is shown below.
{code}
/*
* Processing queuedRequests: Process the next requests until we
* find one for which we need to wait for a commit. We cannot
* process a read request while we are processing write request.
*/
while (!stopped && !isWaitingForCommit() &&
!isProcessingCommit() &&
(request = queuedRequests.poll()) != null) {
if (needCommit(request)) {
nextPending.set(request);
} else {
sendToNextProcessor(request);
}
}
{code}
In this proposal, the idea is to split queuedRequests into smaller queues. I
believe your suggestion can also be extended to address this problem but I am
hoping that this solution should require a few changes but give a lot of
performance improvement.
> Improve ZooKeeper performane under mixed workload
> -------------------------------------------------
>
> Key: ZOOKEEPER-1609
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-1609
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
> Affects Versions: 3.4.3
> Reporter: Thawan Kooburat
>
> ZOOKEEPER-1505 allows 1 write or N reads to pass through the CommitProcessor
> at any given time. I did performance experiment similar to
> http://wiki.apache.org/hadoop/ZooKeeper/Performance and found that read
> throughput drop dramatically when there are write requests. After a bit more
> investigation, I found that
> the biggest bottleneck is at the request queue entering the CommitProcessor.
> When the CommitProcessor see any write request, it will need to block the
> entire pipeline and wait until matching commit from the leader. This means
> that all read requests (including ping request) won't be able to go through.
> The time spent waiting for commit from the leader far exceed the time spent
> waiting for 1 write to goes through the CommitProcessor.
> The current plan is to create multiple request queues at the front of the
> CommitProcessor. Requests are hashed using sessionId and send to one of the
> queue. Whenever, the CommitProcessor saw a write request on one of the queue
> it moves on to process read requests. It will have to unblock the write
> requests in the same order that it sent to the leader, so it may need to
> maintain a separate list to keep track of that.
> The correctness is the same as having more learners in the ensemble. Sessions
> which are hashed onto a different queue is similar to sessions connecting to
> a different learners in the ensemble.
> I am hoping that this will improve read throughput and reduce disconnect rate
> on an ensemble with large number of clients
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
