Re: Dropped Mutation and Read messages.
Indeed, sorry. Subscribed to both so missed which one this was. Sent from my iPhone > On 11 May 2017, at 19:56, Michael Kjellman > wrote: > > This discussion should be on the C* user mailing list. Thanks! > > best, > kjellman > >> On May 11, 2017, at 10:53 AM, Oskar Kjellin wrote: >> >> That seems way too low. Depending on what type of disk you have it should be >> closer to 1-200MB. >> That's probably causing your problems. It would still take a while for you >> to compact all your data tho >> >> Sent from my iPhone >> >>> On 11 May 2017, at 19:50, varun saluja wrote: >>> >>> nodetool getcompactionthrougput >>> >>> ./nodetool getcompactionthroughput >>> Current compaction throughput: 16 MB/s >>> >>> Regards, >>> Varun Saluja >>> On 11 May 2017 at 23:18, varun saluja wrote: Hi, PFB results for same. Numbers are scary here. [root@WA-CASSDB2 bin]# ./nodetool compactionstats pending tasks: 137 compaction type keyspace tablecompleted totalunit progress Compaction system hints 5762711108 837522028005 bytes 0.69% Compaction walletkeyspace user_txn_history_v2101477894 4722068388 bytes 2.15% Compaction walletkeyspace user_txn_history_v2 1511866634 753221762663 bytes 0.20% Compaction walletkeyspace user_txn_history_v2 3664734135 18605501268 bytes 19.70% Active compaction remaining time : 26h32m28s > On 11 May 2017 at 23:15, Oskar Kjellin wrote: > What does nodetool compactionstats show? > > I meant compaction throttling. nodetool getcompactionthrougput > > >> On 11 May 2017, at 19:41, varun saluja wrote: >> >> Hi Oskar, >> >> Thanks for response. >> >> Yes, could see lot of threads for compaction. Actually we are loading >> around 400GB data per node on 3 node cassandra cluster. >> Throttling was set to write around 7k TPS per node. Job ran fine for 2 >> days and then, we start getting Mutation drops , longer GC and very >> high load on system. >> >> System log reports: >> Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) >> off-heap >> >> The job was stopped 12 hours back. But, still these failures can be >> seen. Can you Please let me know how shall i proceed further. If >> possible, Please suggest some parameters for high write intensive jobs. >> >> >> Regards, >> Varun Saluja >> >> >>> On 11 May 2017 at 23:01, Oskar Kjellin wrote: >>> Do you have a lot of compactions going on? It sounds like you might've >>> built up a huge backlog. Is your throttling configured properly? >>> On 11 May 2017, at 18:50, varun saluja wrote: Hi Experts, Seeking your help on a production issue. We were running high write intensive job on our 3 node cassandra cluster V 2.1.7. TPS on nodes were high. Job ran for more than 2 days and thereafter, loadavg on 1 of the node increased to very high number like loadavg : 29. System log reports: INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 2 READ messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last 5000ms The job was stopped due to heavy load. But sill after 12 hours , we can see mutation drops messages and sudden increase on avgload Are these hintedhandoff mutations? Can we stop these. Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any load or any such activity. Due to heavy load and GC , there are intermittent gossip failures among node. Can you someone Please help. PS: Load job was stopped on cluster. Everything ran fine for few hours and and Later issue started again like mutation messages drops. Thanks and Regards, Varun Saluja - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org >> >>> > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Dropped Mutation and Read messages.
This discussion should be on the C* user mailing list. Thanks! best, kjellman > On May 11, 2017, at 10:53 AM, Oskar Kjellin wrote: > > That seems way too low. Depending on what type of disk you have it should be > closer to 1-200MB. > That's probably causing your problems. It would still take a while for you to > compact all your data tho > > Sent from my iPhone > >> On 11 May 2017, at 19:50, varun saluja wrote: >> >> nodetool getcompactionthrougput >> >> ./nodetool getcompactionthroughput >> Current compaction throughput: 16 MB/s >> >> Regards, >> Varun Saluja >> >>> On 11 May 2017 at 23:18, varun saluja wrote: >>> Hi, >>> >>> PFB results for same. Numbers are scary here. >>> >>> [root@WA-CASSDB2 bin]# ./nodetool compactionstats >>> pending tasks: 137 >>> compaction type keyspace tablecompleted >>>totalunit progress >>>Compaction system hints 5762711108 >>> 837522028005 bytes 0.69% >>>Compaction walletkeyspace user_txn_history_v2101477894 >>> 4722068388 bytes 2.15% >>>Compaction walletkeyspace user_txn_history_v2 1511866634 >>> 753221762663 bytes 0.20% >>>Compaction walletkeyspace user_txn_history_v2 3664734135 >>> 18605501268 bytes 19.70% >>> Active compaction remaining time : 26h32m28s >>> >>> >>> On 11 May 2017 at 23:15, Oskar Kjellin wrote: What does nodetool compactionstats show? I meant compaction throttling. nodetool getcompactionthrougput > On 11 May 2017, at 19:41, varun saluja wrote: > > Hi Oskar, > > Thanks for response. > > Yes, could see lot of threads for compaction. Actually we are loading > around 400GB data per node on 3 node cassandra cluster. > Throttling was set to write around 7k TPS per node. Job ran fine for 2 > days and then, we start getting Mutation drops , longer GC and very high > load on system. > > System log reports: > Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) > off-heap > > The job was stopped 12 hours back. But, still these failures can be seen. > Can you Please let me know how shall i proceed further. If possible, > Please suggest some parameters for high write intensive jobs. > > > Regards, > Varun Saluja > > >> On 11 May 2017 at 23:01, Oskar Kjellin wrote: >> Do you have a lot of compactions going on? It sounds like you might've >> built up a huge backlog. Is your throttling configured properly? >> >>> On 11 May 2017, at 18:50, varun saluja wrote: >>> >>> Hi Experts, >>> >>> Seeking your help on a production issue. We were running high write >>> intensive job on our 3 node cassandra cluster V 2.1.7. >>> >>> TPS on nodes were high. Job ran for more than 2 days and thereafter, >>> loadavg on 1 of the node increased to very high number like loadavg : >>> 29. >>> >>> System log reports: >>> >>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms >>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 2 READ messages dropped in last 5000ms >>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last >>> 5000ms >>> >>> The job was stopped due to heavy load. But sill after 12 hours , we can >>> see mutation drops messages and sudden increase on avgload >>> >>> Are these hintedhandoff mutations? Can we stop these. >>> Strangely this behaviour is seen only on 2 nodes. Node 1 does not show >>> any load or any such activity. >>> >>> Due to heavy load and GC , there are intermittent gossip failures among >>> node. Can you someone Please help. >>> >>> PS: Load job was stopped on cluster. Everything ran fine for few hours >>> and and Later issue started again like mutation messages drops. >>> >>> Thanks and Regards, >>> Varun Saluja >>> >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>> > >>> >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: Dropped Mutation and Read messages.
Hi Oskar, Thanks for response. Yes, could see lot of threads for compaction. Actually we are loading around 400GB data per node on 3 node cassandra cluster. Throttling was set to write around 7k TPS per node. Job ran fine for 2 days and then, we start getting Mutation drops , longer GC and very high load on system. System log reports: Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) off-heap The job was stopped 12 hours back. But, still these failures can be seen. Can you Please let me know how shall i proceed further. If possible, Please suggest some parameters for high write intensive jobs. Regards, Varun Saluja On 11 May 2017 at 23:01, Oskar Kjellin wrote: > Do you have a lot of compactions going on? It sounds like you might've > built up a huge backlog. Is your throttling configured properly? > > > On 11 May 2017, at 18:50, varun saluja wrote: > > > > Hi Experts, > > > > Seeking your help on a production issue. We were running high write > intensive job on our 3 node cassandra cluster V 2.1.7. > > > > TPS on nodes were high. Job ran for more than 2 days and thereafter, > loadavg on 1 of the node increased to very high number like loadavg : 29. > > > > System log reports: > > > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > MessagingService.java:888 - 2 READ messages dropped in last 5000ms > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last > 5000ms > > > > The job was stopped due to heavy load. But sill after 12 hours , we can > see mutation drops messages and sudden increase on avgload > > > > Are these hintedhandoff mutations? Can we stop these. > > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show > any load or any such activity. > > > > Due to heavy load and GC , there are intermittent gossip failures among > node. Can you someone Please help. > > > > PS: Load job was stopped on cluster. Everything ran fine for few hours > and and Later issue started again like mutation messages drops. > > > > Thanks and Regards, > > Varun Saluja > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > >
Re: Dropped Mutation and Read messages.
*nodetool getcompactionthrougput* ./nodetool getcompactionthroughput Current compaction throughput: 16 MB/s Regards, Varun Saluja On 11 May 2017 at 23:18, varun saluja wrote: > Hi, > > PFB results for same. Numbers are scary here. > > [root@WA-CASSDB2 bin]# ./nodetool compactionstats > pending tasks: 137 >compaction type keyspace tablecompleted > totalunit progress > Compaction system hints 5762711108 > 837522028005 bytes 0.69% > Compaction walletkeyspace user_txn_history_v2101477894 > 4722068388 bytes 2.15% > Compaction walletkeyspace user_txn_history_v2 1511866634 > 753221762663 bytes 0.20% > Compaction walletkeyspace user_txn_history_v2 3664734135 > 18605501268 bytes 19.70% > Active compaction remaining time : *26h32m28s* > > > > On 11 May 2017 at 23:15, Oskar Kjellin wrote: > >> What does nodetool compactionstats show? >> >> I meant compaction throttling. nodetool getcompactionthrougput >> >> >> On 11 May 2017, at 19:41, varun saluja wrote: >> >> Hi Oskar, >> >> Thanks for response. >> >> Yes, could see lot of threads for compaction. Actually we are loading >> around 400GB data per node on 3 node cassandra cluster. >> Throttling was set to write around 7k TPS per node. Job ran fine for 2 >> days and then, we start getting Mutation drops , longer GC and very high >> load on system. >> >> System log reports: >> Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) >> off-heap >> >> The job was stopped 12 hours back. But, still these failures can be >> seen. Can you Please let me know how shall i proceed further. If possible, >> Please suggest some parameters for high write intensive jobs. >> >> >> Regards, >> Varun Saluja >> >> >> On 11 May 2017 at 23:01, Oskar Kjellin wrote: >> >>> Do you have a lot of compactions going on? It sounds like you might've >>> built up a huge backlog. Is your throttling configured properly? >>> >>> > On 11 May 2017, at 18:50, varun saluja wrote: >>> > >>> > Hi Experts, >>> > >>> > Seeking your help on a production issue. We were running high write >>> intensive job on our 3 node cassandra cluster V 2.1.7. >>> > >>> > TPS on nodes were high. Job ran for more than 2 days and thereafter, >>> loadavg on 1 of the node increased to very high number like loadavg : 29. >>> > >>> > System log reports: >>> > >>> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms >>> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 2 READ messages dropped in last 5000ms >>> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>> MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last >>> 5000ms >>> > >>> > The job was stopped due to heavy load. But sill after 12 hours , we >>> can see mutation drops messages and sudden increase on avgload >>> > >>> > Are these hintedhandoff mutations? Can we stop these. >>> > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show >>> any load or any such activity. >>> > >>> > Due to heavy load and GC , there are intermittent gossip failures >>> among node. Can you someone Please help. >>> > >>> > PS: Load job was stopped on cluster. Everything ran fine for few hours >>> and and Later issue started again like mutation messages drops. >>> > >>> > Thanks and Regards, >>> > Varun Saluja >>> > >>> > - >>> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>> > For additional commands, e-mail: dev-h...@cassandra.apache.org >>> > >>> >> >> >
Re: Dropped Mutation and Read messages.
Hi, PFB results for same. Numbers are scary here. [root@WA-CASSDB2 bin]# ./nodetool compactionstats pending tasks: 137 compaction type keyspace tablecompleted totalunit progress Compaction system hints 5762711108 837522028005 bytes 0.69% Compaction walletkeyspace user_txn_history_v2101477894 4722068388 bytes 2.15% Compaction walletkeyspace user_txn_history_v2 1511866634 753221762663 bytes 0.20% Compaction walletkeyspace user_txn_history_v2 3664734135 18605501268 bytes 19.70% Active compaction remaining time : *26h32m28s* On 11 May 2017 at 23:15, Oskar Kjellin wrote: > What does nodetool compactionstats show? > > I meant compaction throttling. nodetool getcompactionthrougput > > > On 11 May 2017, at 19:41, varun saluja wrote: > > Hi Oskar, > > Thanks for response. > > Yes, could see lot of threads for compaction. Actually we are loading > around 400GB data per node on 3 node cassandra cluster. > Throttling was set to write around 7k TPS per node. Job ran fine for 2 > days and then, we start getting Mutation drops , longer GC and very high > load on system. > > System log reports: > Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) > off-heap > > The job was stopped 12 hours back. But, still these failures can be seen. > Can you Please let me know how shall i proceed further. If possible, Please > suggest some parameters for high write intensive jobs. > > > Regards, > Varun Saluja > > > On 11 May 2017 at 23:01, Oskar Kjellin wrote: > >> Do you have a lot of compactions going on? It sounds like you might've >> built up a huge backlog. Is your throttling configured properly? >> >> > On 11 May 2017, at 18:50, varun saluja wrote: >> > >> > Hi Experts, >> > >> > Seeking your help on a production issue. We were running high write >> intensive job on our 3 node cassandra cluster V 2.1.7. >> > >> > TPS on nodes were high. Job ran for more than 2 days and thereafter, >> loadavg on 1 of the node increased to very high number like loadavg : 29. >> > >> > System log reports: >> > >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >> MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >> MessagingService.java:888 - 2 READ messages dropped in last 5000ms >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >> MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last >> 5000ms >> > >> > The job was stopped due to heavy load. But sill after 12 hours , we can >> see mutation drops messages and sudden increase on avgload >> > >> > Are these hintedhandoff mutations? Can we stop these. >> > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show >> any load or any such activity. >> > >> > Due to heavy load and GC , there are intermittent gossip failures among >> node. Can you someone Please help. >> > >> > PS: Load job was stopped on cluster. Everything ran fine for few hours >> and and Later issue started again like mutation messages drops. >> > >> > Thanks and Regards, >> > Varun Saluja >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >> > >
Re: Dropped Mutation and Read messages.
That seems way too low. Depending on what type of disk you have it should be closer to 1-200MB. That's probably causing your problems. It would still take a while for you to compact all your data tho Sent from my iPhone > On 11 May 2017, at 19:50, varun saluja wrote: > > nodetool getcompactionthrougput > > ./nodetool getcompactionthroughput > Current compaction throughput: 16 MB/s > > Regards, > Varun Saluja > >> On 11 May 2017 at 23:18, varun saluja wrote: >> Hi, >> >> PFB results for same. Numbers are scary here. >> >> [root@WA-CASSDB2 bin]# ./nodetool compactionstats >> pending tasks: 137 >>compaction type keyspace tablecompleted >>totalunit progress >> Compaction system hints 5762711108 >> 837522028005 bytes 0.69% >> Compaction walletkeyspace user_txn_history_v2101477894 >> 4722068388 bytes 2.15% >> Compaction walletkeyspace user_txn_history_v2 1511866634 >> 753221762663 bytes 0.20% >> Compaction walletkeyspace user_txn_history_v2 3664734135 >> 18605501268 bytes 19.70% >> Active compaction remaining time : 26h32m28s >> >> >> >>> On 11 May 2017 at 23:15, Oskar Kjellin wrote: >>> What does nodetool compactionstats show? >>> >>> I meant compaction throttling. nodetool getcompactionthrougput >>> >>> On 11 May 2017, at 19:41, varun saluja wrote: Hi Oskar, Thanks for response. Yes, could see lot of threads for compaction. Actually we are loading around 400GB data per node on 3 node cassandra cluster. Throttling was set to write around 7k TPS per node. Job ran fine for 2 days and then, we start getting Mutation drops , longer GC and very high load on system. System log reports: Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) off-heap The job was stopped 12 hours back. But, still these failures can be seen. Can you Please let me know how shall i proceed further. If possible, Please suggest some parameters for high write intensive jobs. Regards, Varun Saluja > On 11 May 2017 at 23:01, Oskar Kjellin wrote: > Do you have a lot of compactions going on? It sounds like you might've > built up a huge backlog. Is your throttling configured properly? > > > On 11 May 2017, at 18:50, varun saluja wrote: > > > > Hi Experts, > > > > Seeking your help on a production issue. We were running high write > > intensive job on our 3 node cassandra cluster V 2.1.7. > > > > TPS on nodes were high. Job ran for more than 2 days and thereafter, > > loadavg on 1 of the node increased to very high number like loadavg : > > 29. > > > > System log reports: > > > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > > MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > > MessagingService.java:888 - 2 READ messages dropped in last 5000ms > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 > > MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last > > 5000ms > > > > The job was stopped due to heavy load. But sill after 12 hours , we can > > see mutation drops messages and sudden increase on avgload > > > > Are these hintedhandoff mutations? Can we stop these. > > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show > > any load or any such activity. > > > > Due to heavy load and GC , there are intermittent gossip failures among > > node. Can you someone Please help. > > > > PS: Load job was stopped on cluster. Everything ran fine for few hours > > and and Later issue started again like mutation messages drops. > > > > Thanks and Regards, > > Varun Saluja > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > >> >
Re: Dropped Mutation and Read messages.
What does nodetool compactionstats show? I meant compaction throttling. nodetool getcompactionthrougput > On 11 May 2017, at 19:41, varun saluja wrote: > > Hi Oskar, > > Thanks for response. > > Yes, could see lot of threads for compaction. Actually we are loading around > 400GB data per node on 3 node cassandra cluster. > Throttling was set to write around 7k TPS per node. Job ran fine for 2 days > and then, we start getting Mutation drops , longer GC and very high load on > system. > > System log reports: > Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) > off-heap > > The job was stopped 12 hours back. But, still these failures can be seen. > Can you Please let me know how shall i proceed further. If possible, Please > suggest some parameters for high write intensive jobs. > > > Regards, > Varun Saluja > > >> On 11 May 2017 at 23:01, Oskar Kjellin wrote: >> Do you have a lot of compactions going on? It sounds like you might've built >> up a huge backlog. Is your throttling configured properly? >> >> > On 11 May 2017, at 18:50, varun saluja wrote: >> > >> > Hi Experts, >> > >> > Seeking your help on a production issue. We were running high write >> > intensive job on our 3 node cassandra cluster V 2.1.7. >> > >> > TPS on nodes were high. Job ran for more than 2 days and thereafter, >> > loadavg on 1 of the node increased to very high number like loadavg : 29. >> > >> > System log reports: >> > >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 >> > - 839 MUTATION messages dropped in last 5000ms >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 >> > - 2 READ messages dropped in last 5000ms >> > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 >> > - 1 REQUEST_RESPONSE messages dropped in last 5000ms >> > >> > The job was stopped due to heavy load. But sill after 12 hours , we can >> > see mutation drops messages and sudden increase on avgload >> > >> > Are these hintedhandoff mutations? Can we stop these. >> > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any >> > load or any such activity. >> > >> > Due to heavy load and GC , there are intermittent gossip failures among >> > node. Can you someone Please help. >> > >> > PS: Load job was stopped on cluster. Everything ran fine for few hours and >> > and Later issue started again like mutation messages drops. >> > >> > Thanks and Regards, >> > Varun Saluja >> > >> > - >> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: dev-h...@cassandra.apache.org >> > >
Re: Dropped Mutation and Read messages.
Do you have a lot of compactions going on? It sounds like you might've built up a huge backlog. Is your throttling configured properly? > On 11 May 2017, at 18:50, varun saluja wrote: > > Hi Experts, > > Seeking your help on a production issue. We were running high write > intensive job on our 3 node cassandra cluster V 2.1.7. > > TPS on nodes were high. Job ran for more than 2 days and thereafter, loadavg > on 1 of the node increased to very high number like loadavg : 29. > > System log reports: > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 839 MUTATION messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 2 READ messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 1 REQUEST_RESPONSE messages dropped in last 5000ms > > The job was stopped due to heavy load. But sill after 12 hours , we can see > mutation drops messages and sudden increase on avgload > > Are these hintedhandoff mutations? Can we stop these. > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any > load or any such activity. > > Due to heavy load and GC , there are intermittent gossip failures among node. > Can you someone Please help. > > PS: Load job was stopped on cluster. Everything ran fine for few hours and > and Later issue started again like mutation messages drops. > > Thanks and Regards, > Varun Saluja > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Dropped Mutation and Read messages.
Hi Experts, Seeking your help on a production issue. We were running high write intensive job on our 3 node cassandra cluster V 2.1.7. TPS on nodes were high. Job ran for more than 2 days and thereafter, loadavg on 1 of the node increased to very high number like loadavg : 29. System log reports: INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 2 READ messages dropped in last 5000ms INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last 5000ms The job was stopped due to heavy load. But sill after 12 hours , we can see mutation drops messages and sudden increase on avgload Are these hintedhandoff mutations? Can we stop these. Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any load or any such activity. Due to heavy load and GC , there are intermittent gossip failures among node. Can you someone Please help. PS: Load job was stopped on cluster. Everything ran fine for few hours and and Later issue started again like mutation messages drops. Thanks and Regards, Varun Saluja - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
