Denis, Correct me in case I got it all wrong. Since metrics are scheduled, a possible situation is to have more than 1 TcpDiscoveryMetricsUpdateMessage inside the queue due to slow ... some reasons.
Proposed solution looks like a fix hides the real problem. My proposal is 1) Write a warning message in case second metrics added to the queue. BTW, looks like not only metrics messages should be checked. Do we have another scheduled messages? 2) Start new metrics journey as scheduled, but only in case previous journey finished. And, again, write a warning message in case journey was not started. On Thu, Jan 10, 2019 at 6:18 PM Denis Mekhanikov <dmekhani...@gmail.com> wrote: > A bit of background: > When TcpDiscoverySpi is used, TcpDiscoveryMetricsUpdateMessage is sent by a > coordinator once in metricsUpdateFrequency, which is 2 seconds by default. > It serves as a ping message, which ensures, that the ring is connected, and > all nodes are alive. These messages have a high priority, i.e., they are > put into the head of the RightMessageWorker's queue instead of its tail. > > Now consider a situation, when a single link between two nodes in the ring > works slowly. It may receive and deliver all messages, but with a certain > delay. This situation is possible, when network is unstable or one of the > nodes experiences a lot of short GC pauses. > It leads to a growing message queue on the node, that stands before the > slow link. The worst part is that if high priority messages are generated > faster than they are processed, then other messages won't even get a chance > to be processed. Thus, no nodes will be kicked out of the cluster, but no > useful progress will happen. Partition map exchange may hang for this > reason, and the reason won't be obvious from the logs. > > JIRA ticket: https://issues.apache.org/jira/browse/IGNITE-10808 > I made a draft of the fix for this problem: > https://github.com/apache/ignite/pull/5771 > The PR also contains a test, that reproduces this situation. > > The patch makes sure, that only a single TcpDiscoveryMetricsUpdateMessage > of one kind is stored in the queue, i.e., if a newer message comes to the > node, then the old one is discarded. It also checks, that regular messages > are also processed. If the last processed message had a high priority, then > new high-priority message won't be put to the head of the queue, but to the > tail instead. > The fix addresses the described problem, but increases the utilization of > the discovery threads a bit. > > What do you think of this fix? Do you have better ideas, how to improve the > heartbeat mechanism to avoid such situations? > > Denis >
