Travis Crawford updated ZOOKEEPER-856:

    Attachment: zk_open_file_descriptor_count_total.gif

Attached are two graphs showing:

- Total ZooKeeper connections to a 3 node cluster
- Connections per member in the cluster

In the totals graph, notice how its largely unchanged over time. This period 
represents a steady-state period of usage.

In the members graph, notice how the number of connections is significantly 
different between machines. This cluster allows the leader to service reads, so 
that's not something to factor in when interpreting number of  connections.

These graphs look very similar to an issue I had with another service (scribe) 
and we solved the issue by disconnecting every N+-K messages. We tried getting 
fancy by publishing load metrics and using a smart selection algorithm. Turns 
out in practice though the periodic disconnect/reconnect was easier to 
implement and worked better, so I'm tossing that idea out as a potential 
solution here.

> Connection imbalance leads to overloaded ZK instances
> -----------------------------------------------------
>                 Key: ZOOKEEPER-856
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
>             Project: Zookeeper
>          Issue Type: Bug
>            Reporter: Travis Crawford
>         Attachments: zk_open_file_descriptor_count_members.gif, 
> zk_open_file_descriptor_count_total.gif
> We've experienced a number of issues lately where "ruok" requests would take 
> upwards of 10 seconds to return, and ZooKeeper instances were extremely 
> sluggish. The sluggish instance requires a restart to make it responsive 
> again.
> I believe the issue is connections are very imbalanced, leading to certain 
> instances having many thousands of connections, while other instances are 
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance 
> connections over time; this seems fine because sessions should not be 
> affected, and therefore ephemaral nodes and watches should not be affected.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to