This discussion should be on the C* user mailing list. Thanks! best, kjellman
> On May 11, 2017, at 10:53 AM, Oskar Kjellin <oskar.kjel...@gmail.com> wrote: > > That seems way too low. Depending on what type of disk you have it should be > closer to 1-200MB. > That's probably causing your problems. It would still take a while for you to > compact all your data tho > > Sent from my iPhone > >> On 11 May 2017, at 19:50, varun saluja <saluj...@gmail.com> wrote: >> >> nodetool getcompactionthrougput >> >> ./nodetool getcompactionthroughput >> Current compaction throughput: 16 MB/s >> >> Regards, >> Varun Saluja >> >>> On 11 May 2017 at 23:18, varun saluja <saluj...@gmail.com> wrote: >>> Hi, >>> >>> PFB results for same. Numbers are scary here. >>> >>> [root@WA-CASSDB2 bin]# ./nodetool compactionstats >>> pending tasks: 137 >>> compaction type keyspace table completed >>> total unit progress >>> Compaction system hints 5762711108 >>> 837522028005 bytes 0.69% >>> Compaction walletkeyspace user_txn_history_v2 101477894 >>> 4722068388 bytes 2.15% >>> Compaction walletkeyspace user_txn_history_v2 1511866634 >>> 753221762663 bytes 0.20% >>> Compaction walletkeyspace user_txn_history_v2 3664734135 >>> 18605501268 bytes 19.70% >>> Active compaction remaining time : 26h32m28s >>> >>> >>> >>>> On 11 May 2017 at 23:15, Oskar Kjellin <oskar.kjel...@gmail.com> wrote: >>>> What does nodetool compactionstats show? >>>> >>>> I meant compaction throttling. nodetool getcompactionthrougput >>>> >>>> >>>>> On 11 May 2017, at 19:41, varun saluja <saluj...@gmail.com> wrote: >>>>> >>>>> Hi Oskar, >>>>> >>>>> Thanks for response. >>>>> >>>>> Yes, could see lot of threads for compaction. Actually we are loading >>>>> around 400GB data per node on 3 node cassandra cluster. >>>>> Throttling was set to write around 7k TPS per node. Job ran fine for 2 >>>>> days and then, we start getting Mutation drops , longer GC and very high >>>>> load on system. >>>>> >>>>> System log reports: >>>>> Enqueuing flush of compactions_in_progress: 1156 (0%) on-heap, 1132 (0%) >>>>> off-heap >>>>> >>>>> The job was stopped 12 hours back. But, still these failures can be seen. >>>>> Can you Please let me know how shall i proceed further. If possible, >>>>> Please suggest some parameters for high write intensive jobs. >>>>> >>>>> >>>>> Regards, >>>>> Varun Saluja >>>>> >>>>> >>>>>> On 11 May 2017 at 23:01, Oskar Kjellin <oskar.kjel...@gmail.com> wrote: >>>>>> Do you have a lot of compactions going on? It sounds like you might've >>>>>> built up a huge backlog. Is your throttling configured properly? >>>>>> >>>>>>> On 11 May 2017, at 18:50, varun saluja <saluj...@gmail.com> wrote: >>>>>>> >>>>>>> Hi Experts, >>>>>>> >>>>>>> Seeking your help on a production issue. We were running high write >>>>>>> intensive job on our 3 node cassandra cluster V 2.1.7. >>>>>>> >>>>>>> TPS on nodes were high. Job ran for more than 2 days and thereafter, >>>>>>> loadavg on 1 of the node increased to very high number like loadavg : >>>>>>> 29. >>>>>>> >>>>>>> System log reports: >>>>>>> >>>>>>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>>>>>> MessagingService.java:888 - 839 MUTATION messages dropped in last 5000ms >>>>>>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>>>>>> MessagingService.java:888 - 2 READ messages dropped in last 5000ms >>>>>>> INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 >>>>>>> MessagingService.java:888 - 1 REQUEST_RESPONSE messages dropped in last >>>>>>> 5000ms >>>>>>> >>>>>>> The job was stopped due to heavy load. But sill after 12 hours , we can >>>>>>> see mutation drops messages and sudden increase on avgload >>>>>>> >>>>>>> Are these hintedhandoff mutations? Can we stop these. >>>>>>> Strangely this behaviour is seen only on 2 nodes. Node 1 does not show >>>>>>> any load or any such activity. >>>>>>> >>>>>>> Due to heavy load and GC , there are intermittent gossip failures among >>>>>>> node. Can you someone Please help. >>>>>>> >>>>>>> PS: Load job was stopped on cluster. Everything ran fine for few hours >>>>>>> and and Later issue started again like mutation messages drops. >>>>>>> >>>>>>> Thanks and Regards, >>>>>>> Varun Saluja >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >>>>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org >>>>>>> >>>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
