Thanks a lot Jeff. You have explaned very well here. We have consitency as local quorum. Will follow truncate hints and repair therafter.
I hope this brings cluster in stable state Thanks again. Regards, Varun Saluja Sent from my iPhone > On 16-May-2017, at 8:42 PM, Jeff Jirsa <jji...@apache.org> wrote: > > > In Cassandra versions up to 3.0, hints are stored within a table, where the > partition key is the host ID of the server for which the hints are stored. > > In such a data model, accumulating 800GB of hints is almost certain to cause > very wide rows, which will in turn cause GC pressure when you attempt to read > the hints for delivery. This will cause GC pauses, which will cause hints to > fail to be delivered, which will cause more hints to be stored. This is bad. > > In 3.0, hints were rewritten to work around this design flaw. In 2.1, your > most likely corrective course is to use 'nodetool truncatehints' on all > servers, followed by 'nodetool repair' to deliver the data you lost by > truncating the hints. > > NOTE: this is ONLY safe if you wrote with a consistency level stronger than > CL:ANY. If you wrote this data with CL:ANY, you may lose data if you truncate > hints. > > - Jeff > >> On 2017-05-16 06:50 (-0700), varun saluja <saluj...@gmail.com> wrote: >> Thanks for update. >> I could see lot of io waits. This causing Gc and mutation drops . >> But as i mentioned we do not have high load for now. Hint replays are >> creating such high disk I/O. >> compactionstats show very high hint bytes like 780gb around. Is this normal? >> >> Just mentioning we are using flash disks. >> >> In such case, if i run truncatehints , will it remove or decrease size of >> hints bytes in compaction stats. I can trigger repair therafter. >> Please let me know if any recommendation on same. >> >> Also , table which we dumped from kafka which created this much hints and >> compaction pendings is also dropped today. Because we have to redump table >> again once cluster is stable. >> >> Regards, >> Varun >> >> Sent from my iPhone >> >>> On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: >>> >>> Yes but it means data has to be replicated using repair. >>> >>> Hints are out come of unhealthy nodes, focus on finding why you have >>> mutation drops, is it node, io or network etc. ideally you shouldn't see >>> increasing hints all the time. >>> >>> Sent from my iPhone >>> >>>> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: >>>> >>>> Hi Nitan, >>>> >>>> Thanks for response. >>>> >>>> Yes, I could see mutation drops and increase count in system.hints. Is >>>> there any way , i can proceed to truncate hints like using nodetool >>>> truncatehints. >>>> >>>> >>>> Regards, >>>> Varun Saluja >>>> >>>>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: >>>>> Do you see mutation drops? >>>>> Select count from system.hints; is it increasing? >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: >>>>>> >>>>>> Hi Experts, >>>>>> >>>>>> We are facing issue on production cluster. Compaction on system.hint >>>>>> table is running from last 2 days. >>>>>> >>>>>> >>>>>> pending tasks: 1 >>>>>> compaction type keyspace table completed total >>>>>> unit progress >>>>>> Compaction system hints 20623021829 877874092407 >>>>>> bytes 2.35% >>>>>> Active compaction remaining time : 0h27m15s >>>>>> >>>>>> >>>>>> Active compaction remaining time shows in minutes. But, this is job is >>>>>> running like indefinitely. >>>>>> >>>>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last >>>>>> week on particular table. >>>>>> Compaction on this table finished but hint table size is growing >>>>>> continuously. >>>>>> >>>>>> Can someone Please help me. >>>>>> >>>>>> >>>>>> Thanks & Regards, >>>>>> Varun Saluja >>>>>> >>>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
