[
https://issues.apache.org/jira/browse/ZOOKEEPER-856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Travis Crawford updated ZOOKEEPER-856:
--
Attachment: zk_open_file_descriptor_count_total.gif
zk_open_file_descriptor_count_members.gif
Attached are two graphs showing:
- Total ZooKeeper connections to a 3 node cluster
- Connections per member in the cluster
In the totals graph, notice how its largely unchanged over time. This period
represents a steady-state period of usage.
In the members graph, notice how the number of connections is significantly
different between machines. This cluster allows the leader to service reads, so
that's not something to factor in when interpreting number of connections.
These graphs look very similar to an issue I had with another service (scribe)
and we solved the issue by disconnecting every N+-K messages. We tried getting
fancy by publishing load metrics and using a smart selection algorithm. Turns
out in practice though the periodic disconnect/reconnect was easier to
implement and worked better, so I'm tossing that idea out as a potential
solution here.
Connection imbalance leads to overloaded ZK instances
-
Key: ZOOKEEPER-856
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-856
Project: Zookeeper
Issue Type: Bug
Reporter: Travis Crawford
Attachments: zk_open_file_descriptor_count_members.gif,
zk_open_file_descriptor_count_total.gif
We've experienced a number of issues lately where ruok requests would take
upwards of 10 seconds to return, and ZooKeeper instances were extremely
sluggish. The sluggish instance requires a restart to make it responsive
again.
I believe the issue is connections are very imbalanced, leading to certain
instances having many thousands of connections, while other instances are
largely idle.
A potential solution is periodically disconnecting/reconnecting to balance
connections over time; this seems fine because sessions should not be
affected, and therefore ephemaral nodes and watches should not be affected.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
