Re: Problem with dropped mutations
Dropped mutations are load shedding - somethings not happy. Are you seeing GC pauses? What heap size and version? What memtable settings ? -- Jeff Jirsa > On Jul 2, 2018, at 12:48 AM, Hannu Kröger wrote: > > Yes, there are timeouts sometimes but more on the read side. And yes, there > are certain data modeling problems which will be soon addressed but we need > to keep things steady before we get there. > > I guess many write timeouts go unnoticed due to consistency level != ALL. > > Network looks to be working fine. > > Hannu > >> ZAIDI, ASAD A kirjoitti 26.6.2018 kello 21.42: >> >> Are you also seeing time-outs on certain Cassandra operations?? If yes, you >> may have to tweak *request_timeout parameter in order to get rid of dropped >> mutation messages if application data model is not upto mark! >> >> You can also check if network isn't dropping packets (ifconfig -a tool) + >> storage (dstat tool) isn't reporting too slow disks. >> >> Cheers/Asad >> >> >> -Original Message- >> From: Hannu Kröger [mailto:hkro...@gmail.com] >> Sent: Tuesday, June 26, 2018 9:49 AM >> To: user >> Subject: Problem with dropped mutations >> >> Hello, >> >> We have a cluster with somewhat heavy load and we are seeing dropped >> mutations (variable amount and not all nodes have those). >> >> Are there some clear trigger which cause those? What would be the best >> pragmatic approach to start debugging those? We have already added more >> memory which seemed to help somewhat but not completely. >> >> Cheers, >> Hannu >> >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Problem with dropped mutations
Yes, there are timeouts sometimes but more on the read side. And yes, there are certain data modeling problems which will be soon addressed but we need to keep things steady before we get there. I guess many write timeouts go unnoticed due to consistency level != ALL. Network looks to be working fine. Hannu > ZAIDI, ASAD A kirjoitti 26.6.2018 kello 21.42: > > Are you also seeing time-outs on certain Cassandra operations?? If yes, you > may have to tweak *request_timeout parameter in order to get rid of dropped > mutation messages if application data model is not upto mark! > > You can also check if network isn't dropping packets (ifconfig -a tool) + > storage (dstat tool) isn't reporting too slow disks. > > Cheers/Asad > > > -Original Message- > From: Hannu Kröger [mailto:hkro...@gmail.com] > Sent: Tuesday, June 26, 2018 9:49 AM > To: user > Subject: Problem with dropped mutations > > Hello, > > We have a cluster with somewhat heavy load and we are seeing dropped > mutations (variable amount and not all nodes have those). > > Are there some clear trigger which cause those? What would be the best > pragmatic approach to start debugging those? We have already added more > memory which seemed to help somewhat but not completely. > > Cheers, > Hannu > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
RE: Problem with dropped mutations
Are you also seeing time-outs on certain Cassandra operations?? If yes, you may have to tweak *request_timeout parameter in order to get rid of dropped mutation messages if application data model is not upto mark! You can also check if network isn't dropping packets (ifconfig -a tool) + storage (dstat tool) isn't reporting too slow disks. Cheers/Asad -Original Message- From: Hannu Kröger [mailto:hkro...@gmail.com] Sent: Tuesday, June 26, 2018 9:49 AM To: user Subject: Problem with dropped mutations Hello, We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those). Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely. Cheers, Hannu - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: Problem with dropped mutations
Hannu, Dropped mutations are often a sign of load-shedding due to an overloaded node or cluster. Are you seeing resource saturation like high CPU usage (because the write path is usually CPU-bound) on any of the nodes in your cluster? Some potential contributing factors that might be causing you to drop mutations are long garbage collection (GC) pauses or large partitions. Do the drops coincide with an increase in requests, a code change, or compaction activity? On Tue, Jun 26, 2018 at 7:48 AM, Hannu Kröger wrote: > Hello, > > We have a cluster with somewhat heavy load and we are seeing dropped > mutations (variable amount and not all nodes have those). > > Are there some clear trigger which cause those? What would be the best > pragmatic approach to start debugging those? We have already added more > memory which seemed to help somewhat but not completely. > > Cheers, > Hannu > > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > > -- *Joshua Galbraith *| Senior Software Engineer | New Relic
Problem with dropped mutations
Hello, We have a cluster with somewhat heavy load and we are seeing dropped mutations (variable amount and not all nodes have those). Are there some clear trigger which cause those? What would be the best pragmatic approach to start debugging those? We have already added more memory which seemed to help somewhat but not completely. Cheers, Hannu - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
