[jira] [Commented] (ZOOKEEPER-3399) Remove logging in getGlobalOutstandingLimit for optimal performance.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848277#comment-16848277 ] Hudson commented on ZOOKEEPER-3399: --- SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #373 (See [https://builds.apache.org/job/Zookeeper-trunk-single-thread/373/]) ZOOKEEPER-3399: Remove logging in getGlobalOutstandingLimit for optimal (eolivelli: rev 968f5f365e53d0bcbbe0225cc382327badbd8380) * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/LeaderZooKeeperServer.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/FollowerZooKeeperServer.java > Remove logging in getGlobalOutstandingLimit for optimal performance. > > > Key: ZOOKEEPER-3399 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3399 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.6.0 >Reporter: Michael Han >Assignee: Michael Han >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Recently we have moved some of our production clusters to the top of the > trunk. One issue we found is a performance regression on read and write > latency on the clusters where the quorum is also serving traffic. The average > read latency increased by 50x, p99 read latency increased by 300x. > The root cause is a log statement introduced in ZOOKEEPER-3177 (PR711), where > we added a LOG.info statement in getGlobalOutstandingLimit. > getGlobalOutstandingLimit is on the critical code path for request processing > and for each request, it will be called twice (one at processing the packet, > one at finalizing the request response). This not only degrades performance > of the server, but also bloated the log file, when the QPS of a server is > high. > This only impacts clusters when the quorum (leader + follower) is serving > traffic. For clusters where only observers are serving traffic no impact is > observed. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3399) Remove logging in getGlobalOutstandingLimit for optimal performance.
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848273#comment-16848273 ] Hudson commented on ZOOKEEPER-3399: --- SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #538 (See [https://builds.apache.org/job/ZooKeeper-trunk/538/]) ZOOKEEPER-3399: Remove logging in getGlobalOutstandingLimit for optimal (eolivelli: rev 968f5f365e53d0bcbbe0225cc382327badbd8380) * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/LeaderZooKeeperServer.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/quorum/FollowerZooKeeperServer.java > Remove logging in getGlobalOutstandingLimit for optimal performance. > > > Key: ZOOKEEPER-3399 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3399 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.6.0 >Reporter: Michael Han >Assignee: Michael Han >Priority: Major > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Recently we have moved some of our production clusters to the top of the > trunk. One issue we found is a performance regression on read and write > latency on the clusters where the quorum is also serving traffic. The average > read latency increased by 50x, p99 read latency increased by 300x. > The root cause is a log statement introduced in ZOOKEEPER-3177 (PR711), where > we added a LOG.info statement in getGlobalOutstandingLimit. > getGlobalOutstandingLimit is on the critical code path for request processing > and for each request, it will be called twice (one at processing the packet, > one at finalizing the request response). This not only degrades performance > of the server, but also bloated the log file, when the QPS of a server is > high. > This only impacts clusters when the quorum (leader + follower) is serving > traffic. For clusters where only observers are serving traffic no impact is > observed. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3399) Remove logging in getGlobalOutstandingLimit for optimal performance.
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848259#comment-16848259
]
Enrico Olivelli commented on ZOOKEEPER-3399:
{quote}
Recently we have moved some of our production clusters to the top of the trunk
{quote}
This is encouraging !
I think it is time to think to cut a release for 3.6.0
> Remove logging in getGlobalOutstandingLimit for optimal performance.
>
>
> Key: ZOOKEEPER-3399
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3399
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.6.0
>Reporter: Michael Han
>Assignee: Michael Han
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Recently we have moved some of our production clusters to the top of the
> trunk. One issue we found is a performance regression on read and write
> latency on the clusters where the quorum is also serving traffic. The average
> read latency increased by 50x, p99 read latency increased by 300x.
> The root cause is a log statement introduced in ZOOKEEPER-3177 (PR711), where
> we added a LOG.info statement in getGlobalOutstandingLimit.
> getGlobalOutstandingLimit is on the critical code path for request processing
> and for each request, it will be called twice (one at processing the packet,
> one at finalizing the request response). This not only degrades performance
> of the server, but also bloated the log file, when the QPS of a server is
> high.
> This only impacts clusters when the quorum (leader + follower) is serving
> traffic. For clusters where only observers are serving traffic no impact is
> observed.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
