Hi, I've extended a cluster by 10% and after that each hour, on some of the nodes (which changes randomly each time), "dropped mutations cross node" appears on logs (each time 1 or 2 drops and some times some thousands with cross node latency from 3000ms to 90000ms or 90seconds!) and insert rate been decreased abour 50%:
on token ownership everything is OK (stdev.p of ownership percent even decreased with cluster extension) CPU usage on nodes is less than 30 percent and all well balanced disk usage is less than 10% watching through iostat and also no pending compaction on nodes there is no other log beside dropped reports (although a few GC about 200-300ms every 5 minuts) no sign of memory problem looking at jvisualVM honestly i do not monitor network equipments (Switches) but the network did not changed since the extend of cluster and not increase in packet discard counters at node side So to emphasize; there is mutation drop which i can not detect the root cause. Is there any workaround or monitoring metric that i missed here? Cluster Info: Cassandra 3.11.2 RF 3 30 Nodes Sent using https://www.zoho.com/mail/
