[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903618#action_12903618 ] Patrick Hunt commented on ZOOKEEPER-856: Btw, If it's possible for you to connect visualvm or jconsole to the JVMs in question you would also see some useful information (not always possible though, esp in production env). > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903617#action_12903617 ] Patrick Hunt commented on ZOOKEEPER-856: Agree, this has been an issue we've discussed before (connection balancing) and should address. The most pronounced is when you restart a server - all the connections shift to the other servers and the restarted server has 0 connections... > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903592#action_12903592 ] Travis Crawford commented on ZOOKEEPER-856: --- Thanks for the suggestions! I'm trying out the following options: -Xmx4000M -Xms4000M -Xloggc:/var/log/zookeeper/gc_20100827_201441.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+CMSIncrementalPacing I still think the connection imbalance issue is worth addressing though. Even with properly tuned ZK servers they can become overloaded as older processes acquire connections to the point of becoming overloaded. > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903486#action_12903486 ] Patrick Hunt commented on ZOOKEEPER-856: Well it could be that increased connection count correlates to increased activity, including longer gc pauses as a result. I believe incremental mode is what you're looking for (just one reference): http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.0.Incremental%20mode%7Coutline "The incremental mode is meant to lessen the impact of long concurrent phases by periodically stopping the concurrent phase to yield back the processor to the application." I've found this tool to be an excellent way to visualize GC activity: https://gchisto.dev.java.net/ Also, have you verified that you are not using a large % of the heap? Perhaps you could include some JVM stats (heap utilization say) in your monitoring framework. > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903170#action_12903170 ] Travis Crawford commented on ZOOKEEPER-856: --- @patrick - We're using these settings, which I believe are based on what's recommended in the troubleshooting guide. -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -XX:+UseConcMarkSweepGC Looking at the logs I do see lots of GC activity. For example: Total time for which application threads were stopped: 0.5599050 seconds Application time: 0.0056590 seconds I only see this on the hosts that became unresponsive after acquiring lots of connections. Any suggestions for the GC flags? If there's something better I can experiment, and update the wiki if we discover something interesting. http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903146#action_12903146 ] Patrick Hunt commented on ZOOKEEPER-856: Have you monitored the jvms for gc activity? Are you using CMS/incremental gc rather than the default GC setup? I'm all for adding balancing, but it would be good to rule GC/swap/IO out as an issue. > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903065#action_12903065 ] Travis Crawford commented on ZOOKEEPER-856: --- @mahadev - I would love to help test a patch :) I'm currently using 3.3.1 + ZOOKEEPER-744 + ZOOKEEPER-790, applied in that order. If there's a knob for how frequently to disconnect/reconnect I can try out different settings to see what a sensible default would be. Do you think this should be a client or server setting? I'm thinking a server setting because otherwise its not possible to enforce the policy. > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Fix For: 3.4.0 > > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances
[ https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903017#action_12903017 ] Mahadev konar commented on ZOOKEEPER-856: - travis, we have had a lot of discussion on load balancing. I'd really want to try and see how the disconnect and reconnect works for load balancing. I am also with you that it might be a good enough soln on load balancing. I can upload a simple patch for this. Would you have some bandwidth trying and it out and reporting how well it works? > Connection imbalance leads to overloaded ZK instances > - > > Key: ZOOKEEPER-856 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856 > Project: Zookeeper > Issue Type: Bug >Reporter: Travis Crawford > Attachments: zk_open_file_descriptor_count_members.gif, > zk_open_file_descriptor_count_total.gif > > > We've experienced a number of issues lately where "ruok" requests would take > upwards of 10 seconds to return, and ZooKeeper instances were extremely > sluggish. The sluggish instance requires a restart to make it responsive > again. > I believe the issue is connections are very imbalanced, leading to certain > instances having many thousands of connections, while other instances are > largely idle. > A potential solution is periodically disconnecting/reconnecting to balance > connections over time; this seems fine because sessions should not be > affected, and therefore ephemaral nodes and watches should not be affected. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
