[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746687#comment-16746687 ] Brian Nixon commented on ZOOKEEPER-3180: Creating ZOOKEEPER-3252 as a follow up. > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Brian Nixon >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742597#comment-16742597 ] Hudson commented on ZOOKEEPER-3180: --- SUCCESS: Integrated in Jenkins build Zookeeper-trunk-single-thread #187 (See [https://builds.apache.org/job/Zookeeper-trunk-single-thread/187/]) ZOOKEEPER-3180: Add response cache to improve the throughput of read … (andor: rev 9828685d20bc99272ec0689a590d27b0f8f7ebb6) * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerMXBean.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/FinalRequestProcessor.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxn.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerCnxn.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerBean.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/DumbWatcher.java * (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md * (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/ResponseCache.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/MockServerCnxn.java * (add) zookeeper-server/src/test/java/org/apache/zookeeper/test/ResponseCacheTest.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/test/JMXEnv.java > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Brian Nixon >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742511#comment-16742511 ] Hudson commented on ZOOKEEPER-3180: --- SUCCESS: Integrated in Jenkins build ZooKeeper-trunk #350 (See [https://builds.apache.org/job/ZooKeeper-trunk/350/]) ZOOKEEPER-3180: Add response cache to improve the throughput of read … (andor: rev 9828685d20bc99272ec0689a590d27b0f8f7ebb6) * (add) zookeeper-server/src/main/java/org/apache/zookeeper/server/ResponseCache.java * (edit) zookeeper-docs/src/main/resources/markdown/zookeeperAdmin.md * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerMXBean.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/DumbWatcher.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerCnxn.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/test/JMXEnv.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NIOServerCnxn.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/NettyServerCnxn.java * (add) zookeeper-server/src/test/java/org/apache/zookeeper/test/ResponseCacheTest.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/FinalRequestProcessor.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServerBean.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ServerMetrics.java * (edit) zookeeper-server/src/main/java/org/apache/zookeeper/server/ZooKeeperServer.java * (edit) zookeeper-server/src/test/java/org/apache/zookeeper/server/MockServerCnxn.java > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Brian Nixon >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 4h 40m > Remaining Estimate: 0h > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719752#comment-16719752 ] Michael Han commented on ZOOKEEPER-3180: My experience with JVM GC and ZooKeeper is GC is rarely a real issue in production if tuned correctly (I ran fairly large ZK fleet which kind push ZK to its limit). Most GC issue I had is software bugs - such as leaking connections. For this cache case, the current implementation is good enough for my use case, though I do have interests on off heap solutions as well. My concern around off heap solution is it's probably going to be more complicated, and has overhead of serialization / deserialization between heap / off heap. I'd say we get this patch landed, have more people tested it out, then improve it with more options. And for caching in general, obviously it depends a lot on workload and actual use case, so it's kind hard to provide a cache solution that works for everyone in first place... > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Brian Nixon >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 3h 50m > Remaining Estimate: 0h > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[
https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698826#comment-16698826
]
Rakesh R commented on ZOOKEEPER-3180:
-
[~lvfangmin], Feature looks pretty interesting and thanks for the patch. I will
try to review it when I get a chance.
I'd like to understand scaling of this cache in the production env. Appreciate
if you could help me, adding more details about the memory usage:
# How much is the java heap size of ZooKeeper server and the possible number
of znodes (am assuming 4MB data per znode).
# Based on your experiment, what would be the ideal number of elements to be
kept into the cache for a better read performance gain.
Since you have mentioned that the intention of read cache feature is to reduce
the GC overhead, I would also like to explore the option of enabling the cache
backed by {{off heap}} memory. Again, you have {{4MB}} data in a single zNode
and it would occupy good amount of {{on heap}} memory area if the number of
zNode grows(in future). The off heap implementation uses {{DirectByteByffers}}
to manage cache outside of the JVM heap and provides scalability to a large
memory sizes without GC overhead.
One option could be to provide different cache modes (onheap or offheap) to the
users to efficiently utilize the heap area. Does this make sense to you?
> Add response cache to improve the throughput of read heavy traffic
> ---
>
> Key: ZOOKEEPER-3180
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180
> Project: ZooKeeper
> Issue Type: Improvement
> Components: server
>Reporter: Fangmin Lv
>Assignee: Brian Nixon
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.6.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> On read heavy use case with large response data size, the serialization of
> response takes time and added overhead to the GC.
> Add response cache helps improving the throughput we can support, which also
> reduces the latency in general.
> This Jira is going to implement a LRU cache for the response, which shows
> some performance gain on some of our production ensembles.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671158#comment-16671158 ] Fangmin Lv commented on ZOOKEEPER-3180: --- Currently, we only cache getData, but we can easily extend the code to cache getChildren in a separate Jira. > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Assignee: Brian Nixon >Priority: Minor > Labels: pull-request-available > Fix For: 3.6.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664668#comment-16664668 ] Michael Han commented on ZOOKEEPER-3180: This will be a very useful feature for my prod env as well, where some of our read heavy workloads require serialize large payload from (an almost immutable part of) the data tree - in our case it's not the data stored but the getChildren call with tens of thousands children under the zNode. I'll be glad to review and test the patch in our prod env. > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Priority: Minor > Fix For: 3.6.0 > > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664257#comment-16664257 ] Fangmin Lv commented on ZOOKEEPER-3180: --- Yes, it holds the serialized response, and re-uses that to serve all the read requests until there is change invalidated that cache. So it can avoid allocating and deserialize large response again and again, on our prod, we saw some users are storing more than 4MB data in a single znode. > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Priority: Minor > Fix For: 3.6.0 > > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ZOOKEEPER-3180) Add response cache to improve the throughput of read heavy traffic
[ https://issues.apache.org/jira/browse/ZOOKEEPER-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661452#comment-16661452 ] Michael Han commented on ZOOKEEPER-3180: What will be we caching here? Is it the byte buffers that holding the (serialized) response body that going to write out to socket? > Add response cache to improve the throughput of read heavy traffic > --- > > Key: ZOOKEEPER-3180 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3180 > Project: ZooKeeper > Issue Type: Improvement > Components: server >Reporter: Fangmin Lv >Priority: Minor > Fix For: 3.6.0 > > > On read heavy use case with large response data size, the serialization of > response takes time and added overhead to the GC. > Add response cache helps improving the throughput we can support, which also > reduces the latency in general. > This Jira is going to implement a LRU cache for the response, which shows > some performance gain on some of our production ensembles. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
