Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
finished but hint table size is growing continuously. Can someone Please help me. Thanks & Regards, Varun Saluja

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
Hi Nitan, Thanks for response. Yes, I could see mutation drops and increase count in system.hints. Is there any way , i can proceed to truncate hints like using nodetool truncatehints. Regards, Varun Saluja On 16 May 2017 at 17:52, Nitan Kainth wrote: > Do you see mutation drops? >

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
vice.java:888 - 228 MUTATION messages dropped in last 5000ms PS: As of now , there is no significant load on our cluster. The only load is of these hints been replayed. Can you Please help. Regards, Varun Saluja On 16 May 2017 at 18:28, varun saluja wrote: > Hi Nitan, > > Thanks for

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
finding why you have mutation > drops, is it node, io or network etc. ideally you shouldn't see increasing > hints all the time. > > Sent from my iPhone > >> On May 16, 2017, at 7:58 AM, varun saluja wrote: >> >> Hi Nitan, >> >> Thanks for re

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
cess hogging system? > > Sent from my iPhone > >> On May 16, 2017, at 7:58 AM, varun saluja wrote: >> >> Hi Nitan, >> >> Thanks for response. >> >> Yes, I could see mutation drops and increase count in system.hints. Is there >> any wa

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
say wait until cluster comes to normal state. May be some other expert can > suggest an alternate. > > Sent from my iPhone > >> On May 16, 2017, at 8:50 AM, varun saluja wrote: >> >> Thanks for update. >> I could see lot of io waits. This causing Gc and mut

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
Thanks a lot Jeff. You have explaned very well here. We have consitency as local quorum. Will follow truncate hints and repair therafter. I hope this brings cluster in stable state Thanks again. Regards, Varun Saluja Sent from my iPhone > On 16-May-2017, at 8:42 PM, Jeff Jirsa wr

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
851658989612 bytes 1.31% Active compaction remaining time : 0h26m43s Will truncatehints takes time for completion? Could not see anything related truncatehints in system logs. Please let me know if anything else can be checked here. Regards, Varun Saluja On 16 May 2017 at 20:58, varun saluja

Re: Long running compaction on huge hint table.

2017-05-16 Thread varun saluja
Compaction system hints 12152557998 869257869352 bytes 1.40% Active compaction remaining time : 0h27m14s Can anything else be checked here? Will manually deleting system.hint files and restart node fix this. Regards, Varun Saluja On 16 May 2017 at 23:29, varun saluja wrote: >

Re: Long running compaction on huge hint table.

2017-05-17 Thread varun saluja
; > > -- > Jeff Jirsa > > > On May 16, 2017, at 6:53 PM, varun saluja wrote: > > Hi, > > > Truncatehints on nodes is running for more than 7 hours now. Nothing > mentioned for same in sysemt logs even. > > And compaction stats reports increase in

Re: Long running compaction on huge hint table.

2017-05-21 Thread varun saluja
Hi All, Can someone Please suggest any recommendations for write intensive jobs Regards, Varun Saluja Sent from my iPhone > On 17-May-2017, at 3:52 PM, varun saluja wrote: > > Thanks Jeff. > > I have taken backup and did manual removal of hints with rolling restart. > Th

Re: Long running compaction on huge hint table.

2017-05-21 Thread varun saluja
Thanks a lot Ben. Really appreciate your suggestions here. Regards, Varun Saluja Sent from my iPhone > On 21-May-2017, at 5:40 PM, Ben Slater wrote: > > My main suggestion would be to monitor the compaction backlog (pending > compactions). If the backlog is growing you ne

memtable_allocation_type on Cassandra 2.1.x

2017-05-22 Thread varun saluja
should be use for below paramteres in 2.1.x memtable_allocation_type : Regards, Varun Saluja - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org

Re: memtable_allocation_type on Cassandra 2.1.x

2017-05-23 Thread varun saluja
parameters. memtable_flush_writers: memtable_cleanup_threshold: PS : We have high write intensive workload . 5Node cluster (12 Core , 62GB RAM and flash disk per node) Regards, Varun Saluja On 23 May 2017 at 03:26, Akhil Mehra wrote: > I believe off-heap storage was reintroduced in