[
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Travis Crawford updated ZOOKEEPER-856:
--------------------------------------
Attachment: zk_open_file_descriptor_count_total.gif
zk_open_file_descriptor_count_members.gif
Attached are two graphs showing:
- Total ZooKeeper connections to a 3 node cluster
- Connections per member in the cluster
In the totals graph, notice how its largely unchanged over time. This period
represents a steady-state period of usage.
In the members graph, notice how the number of connections is significantly
different between machines. This cluster allows the leader to service reads, so
that's not something to factor in when interpreting number of connections.
These graphs look very similar to an issue I had with another service (scribe)
and we solved the issue by disconnecting every N+-K messages. We tried getting
fancy by publishing load metrics and using a smart selection algorithm. Turns
out in practice though the periodic disconnect/reconnect was easier to
implement and worked better, so I'm tossing that idea out as a potential
solution here.
> Connection imbalance leads to overloaded ZK instances
> -----------------------------------------------------
>
> Key: ZOOKEEPER-856
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
> Project: Zookeeper
> Issue Type: Bug
> Reporter: Travis Crawford
> Attachments: zk_open_file_descriptor_count_members.gif,
> zk_open_file_descriptor_count_total.gif
>
>
> We've experienced a number of issues lately where "ruok" requests would take
> upwards of 10 seconds to return, and ZooKeeper instances were extremely
> sluggish. The sluggish instance requires a restart to make it responsive
> again.
> I believe the issue is connections are very imbalanced, leading to certain
> instances having many thousands of connections, while other instances are
> largely idle.
> A potential solution is periodically disconnecting/reconnecting to balance
> connections over time; this seems fine because sessions should not be
> affected, and therefore ephemaral nodes and watches should not be affected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
