Thanks Nitan. Appreciate your help. Can anyone suggest parameter change or something which can help in this situation.
Regards, Varun Sent from my iPhone > On 16-May-2017, at 7:31 PM, Nitan Kainth <ni...@bamlabs.com> wrote: > > If target table is dropped then you can remove its hints but there could be > more hints from other table. If it has tables of your interest , then I won't > comment on truncating hints. > > Size of hints depends on Kafka load , looks like you had overloaded the > cluster during data load and not hints are just recovering from it. I would > say wait until cluster comes to normal state. May be some other expert can > suggest an alternate. > > Sent from my iPhone > >> On May 16, 2017, at 8:50 AM, varun saluja <saluj...@gmail.com> wrote: >> >> Thanks for update. >> I could see lot of io waits. This causing Gc and mutation drops . >> But as i mentioned we do not have high load for now. Hint replays are >> creating such high disk I/O. >> compactionstats show very high hint bytes like 780gb around. Is this normal? >> >> Just mentioning we are using flash disks. >> >> In such case, if i run truncatehints , will it remove or decrease size of >> hints bytes in compaction stats. I can trigger repair therafter. >> Please let me know if any recommendation on same. >> >> Also , table which we dumped from kafka which created this much hints and >> compaction pendings is also dropped today. Because we have to redump table >> again once cluster is stable. >> >> Regards, >> Varun >> >> Sent from my iPhone >> >>> On 16-May-2017, at 6:59 PM, Nitan Kainth <ni...@bamlabs.com> wrote: >>> >>> Yes but it means data has to be replicated using repair. >>> >>> Hints are out come of unhealthy nodes, focus on finding why you have >>> mutation drops, is it node, io or network etc. ideally you shouldn't see >>> increasing hints all the time. >>> >>> Sent from my iPhone >>> >>>> On May 16, 2017, at 7:58 AM, varun saluja <saluj...@gmail.com> wrote: >>>> >>>> Hi Nitan, >>>> >>>> Thanks for response. >>>> >>>> Yes, I could see mutation drops and increase count in system.hints. Is >>>> there any way , i can proceed to truncate hints like using nodetool >>>> truncatehints. >>>> >>>> >>>> Regards, >>>> Varun Saluja >>>> >>>>> On 16 May 2017 at 17:52, Nitan Kainth <ni...@bamlabs.com> wrote: >>>>> Do you see mutation drops? >>>>> Select count from system.hints; is it increasing? >>>>> >>>>> Sent from my iPhone >>>>> >>>>>> On May 16, 2017, at 5:52 AM, varun saluja <saluj...@gmail.com> wrote: >>>>>> >>>>>> Hi Experts, >>>>>> >>>>>> We are facing issue on production cluster. Compaction on system.hint >>>>>> table is running from last 2 days. >>>>>> >>>>>> >>>>>> pending tasks: 1 >>>>>> compaction type keyspace table completed total >>>>>> unit progress >>>>>> Compaction system hints 20623021829 877874092407 >>>>>> bytes 2.35% >>>>>> Active compaction remaining time : 0h27m15s >>>>>> >>>>>> >>>>>> Active compaction remaining time shows in minutes. But, this is job is >>>>>> running like indefinitely. >>>>>> >>>>>> We have 3 node cluster V 2.1.7. And we ran write intensive job last >>>>>> week on particular table. >>>>>> Compaction on this table finished but hint table size is growing >>>>>> continuously. >>>>>> >>>>>> Can someone Please help me. >>>>>> >>>>>> >>>>>> Thanks & Regards, >>>>>> Varun Saluja >>>>>> >>>>
