Hi Akhil, Thank you for your reply. Previously, I did ‘tune’ various timeouts – basically increased them a bit but none of those parameter listed in the link matches with that “were dropped in last 5000 ms”. I was wondering from where that [5000ms] number is coming from when, like I mentioned before, none of any timeout parameter settings matches that #!
Load is intermittently high but again cpu queue length never goes beyond medium depth. I wonder if there is some internal limit that I’m still not aware of. Thanks/Asad From: Akhil Mehra [mailto:akhilme...@gmail.com] Sent: Thursday, July 20, 2017 3:47 PM To: user@cassandra.apache.org Subject: Re: MUTATION messages were dropped in last 5000 ms for cross node timeout Hi Asad, http://cassandra.apache.org/doc/latest/faq/index.html#why-message-dropped<https://urldefense.proofpoint.com/v2/url?u=http-3A__cassandra.apache.org_doc_latest_faq_index.html-23why-2Dmessage-2Ddropped&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=FsmDztdsVuIKml8IDhdHdg&m=WcHuHKcjg2YCsAbw2NR_0-CiHr9JNxtCzYikia16mpo&s=0_0pQfoOZLuswpQ_lE-AU2bTMFLgRbR4k4Kh8vEOZSk&e=> As mentioned in the link above this is a load shedding mechanism used by Cassandra. Is you cluster under heavy load? Regards, Akhil On 21/07/2017, at 3:27 AM, ZAIDI, ASAD A <az1...@att.com<mailto:az1...@att.com>> wrote: Hello Folks – I’m using apache-cassandra 2.2.8. I see many messages like below in my system.log file. In Cassandra.yaml file [ cross_node_timeout: true] is set and NTP server is also running correcting clock drift on 16node cluster. I do not see pending or blocked HintedHandoff in tpstats output though there are bunch of MUTATIONS dropped observed. <start timeout message > INFO [ScheduledTasks:1] 2017-07-20 08:02:52,511 MessagingService.java:946 - MUTATION messages were dropped in last 5000 ms: 822 for internal timeout and 2152 for cross node timeout <end timeout message > I’m seeking help here if you please let me know what I need to check in order to address these cross node timeouts. Thank you, Asad
