[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-27 Thread Patrick Hunt (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903618#action_12903618
 ] 

Patrick Hunt commented on ZOOKEEPER-856:


Btw, If it's possible for you to connect visualvm or jconsole to the JVMs in 
question you would also see some useful information (not always possible 
though, esp in production env).

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-27 Thread Patrick Hunt (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903617#action_12903617
 ] 

Patrick Hunt commented on ZOOKEEPER-856:


Agree, this has been an issue we've discussed before (connection balancing) and 
should address. The most pronounced is when you restart a server - all the 
connections shift to the other servers and the restarted server has 0 
connections... 

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-27 Thread Travis Crawford (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903592#action_12903592
 ] 

Travis Crawford commented on ZOOKEEPER-856:
---

Thanks for the suggestions! I'm trying out the following options:

-Xmx4000M
-Xms4000M
-Xloggc:/var/log/zookeeper/gc_20100827_201441.log
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-XX:+UseConcMarkSweepGC
-XX:+CMSIncrementalMode
-XX:+CMSIncrementalPacing

I still think the connection imbalance issue is worth addressing though. Even 
with properly tuned ZK servers they can become overloaded as older processes 
acquire connections to the point of becoming overloaded.

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-27 Thread Patrick Hunt (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903486#action_12903486
 ] 

Patrick Hunt commented on ZOOKEEPER-856:


Well it could be that increased connection count correlates to increased 
activity, including longer gc pauses as a result.

I believe incremental mode is what you're looking for (just one reference):
http://www.oracle.com/technetwork/java/gc-tuning-5-138395.html#0.0.0.0.Incremental%20mode%7Coutline

"The incremental mode is meant to lessen the impact of long concurrent phases 
by periodically stopping the concurrent phase to yield back the processor to 
the application."

I've found this tool to be an excellent way to visualize GC activity: 
https://gchisto.dev.java.net/

Also, have you verified that you are not using a large % of the heap? Perhaps 
you could include some JVM stats (heap utilization say) in your monitoring 
framework.


> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-26 Thread Travis Crawford (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903170#action_12903170
 ] 

Travis Crawford commented on ZOOKEEPER-856:
---

@patrick - We're using these settings, which I believe are based on what's 
recommended in the troubleshooting guide.

-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCApplicationConcurrentTime
-XX:+UseConcMarkSweepGC

Looking at the logs I do see lots of GC activity. For example:

Total time for which application threads were stopped: 0.5599050 seconds
Application time: 0.0056590 seconds

I only see this on the hosts that became unresponsive after acquiring lots of 
connections.

Any suggestions for the GC flags? If there's something better I can experiment, 
and update the wiki if we discover something interesting.

http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-26 Thread Patrick Hunt (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903146#action_12903146
 ] 

Patrick Hunt commented on ZOOKEEPER-856:


Have you monitored the jvms for gc activity? Are you using CMS/incremental gc 
rather than the default GC setup? I'm all for adding balancing, but it would be 
good to rule GC/swap/IO out as an issue.

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-26 Thread Travis Crawford (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903065#action_12903065
 ] 

Travis Crawford commented on ZOOKEEPER-856:
---

@mahadev - I would love to help test a patch :) I'm currently using 3.3.1 + 
ZOOKEEPER-744 + ZOOKEEPER-790, applied in that order.

If there's a knob for how frequently to disconnect/reconnect I can try out 
different settings to see what a sensible default would be.

Do you think this should be a client or server setting? I'm thinking a server 
setting because otherwise its not possible to enforce the policy.

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Fix For: 3.4.0
>
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.



[jira] Commented: (ZOOKEEPER-856) Connection imbalance leads to overloaded ZK instances

2010-08-26 Thread Mahadev konar (JIRA)

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12903017#action_12903017
 ] 

Mahadev konar commented on ZOOKEEPER-856:
-

travis, 
 we have had a lot of discussion on load balancing. I'd really want to try and 
see how the disconnect and reconnect works for load balancing. I am also with 
you that it might be a good enough soln on load balancing. I can upload a 
simple patch for this. Would you have some bandwidth trying and it out and 
reporting how well it works?

> Connection imbalance leads to overloaded ZK instances
> -
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
>  Issue Type: Bug
>Reporter: Travis Crawford
> Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.