Re: MUTATION_* messages in log
Hello again! Bowen, your answer was right. After forcing clock synchronization, the count of MUTATION messages radically dropped down. After sync I made full repair on affected nodes and now I don't have any MUTATION drops in logs. Thanks for your help! вт, 12 апр. 2022 г. в 12:31, Bowen Song : > > First thing to check, do you have NTP client running on all Cassandra > servers? Are their clock in sync? If you answer "yes" to both, check the > server load, does any server have high CPU usage or disk utilization? > Any swapping activity? If not, check the GC logs, and looking for long > GC pauses. > > > On 12/04/2022 10:15, Azamat Hackimov wrote: > > Hello. > > > > I've got two DCs with 8 hardware servers on each. After migration to > > Cassandra 4 I have strange messages in system.log: > > > > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595 > > MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last > > 5000 ms: 0 internal and 241 cross node. Mean internal dropped latency: > > 0 ms and Mean cross-node dropped latency: 2395 ms > > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 > > MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last > > 5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0 > > ms and Mean cross-node dropped latency: 2395 ms > > INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 > > MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in > > last 5000 ms: 0 internal and 102 cross node. Mean internal dropped > > latency: 0 ms and Mean cross-node dropped latency: 2395 ms > > > > Can someone interpret these messages? I suspect that messages indicate > > that the cluster has some replication issues. -- >From Siberia with Love!
Re: MUTATION_* messages in log
First thing to check, do you have NTP client running on all Cassandra servers? Are their clock in sync? If you answer "yes" to both, check the server load, does any server have high CPU usage or disk utilization? Any swapping activity? If not, check the GC logs, and looking for long GC pauses. On 12/04/2022 10:15, Azamat Hackimov wrote: Hello. I've got two DCs with 8 hardware servers on each. After migration to Cassandra 4 I have strange messages in system.log: INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595 MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last 5000 ms: 0 internal and 241 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last 5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in last 5000 ms: 0 internal and 102 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms Can someone interpret these messages? I suspect that messages indicate that the cluster has some replication issues.
MUTATION_* messages in log
Hello. I've got two DCs with 8 hardware servers on each. After migration to Cassandra 4 I have strange messages in system.log: INFO [ScheduledTasks:1] 2022-04-12 12:05:58,595 MessagingMetrics.java:206 - MUTATION_RSP messages were dropped in last 5000 ms: 0 internal and 241 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 MessagingMetrics.java:206 - MUTATION_REQ messages were dropped in last 5000 ms: 0 internal and 8 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms INFO [ScheduledTasks:1] 2022-04-12 12:05:58,596 MessagingMetrics.java:206 - PAXOS_COMMIT_RSP messages were dropped in last 5000 ms: 0 internal and 102 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 2395 ms Can someone interpret these messages? I suspect that messages indicate that the cluster has some replication issues.