HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread Allan C
Hi , I’m hitting a very odd issue with HintedHandoff on 1 node in my 12 node cluster running 1.2.13. Somehow it’s holding a large amount of hints for tokens that have never been part of the cluster. Pretty sure this is causing a bunch of memory pressure somehow that’s causing the node to go

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread sankalp kohli
Is this happening in one node or all. Did you try to delete the hints via JMX in other nodes? On Mon, Jan 20, 2014 at 12:18 PM, Allan C alla...@gmail.com wrote: Hi , I’m hitting a very odd issue with HintedHandoff on 1 node in my 12 node cluster running 1.2.13. Somehow it’s holding a large

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread Allan C
There are 3 other nodes that have a mild case. This is one node is worse by an order of magnitude. deleteHintsForEndpoint fails with the same error  on any of the affected nodes. -Allan On January 20, 2014 at 12:24:33 PM, sankalp kohli (kohlisank...@gmail.com) wrote: Is this happening in one

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread sankalp kohli
Yes as per code you cannot delete hints for endpoints which are not part of the ring. if (!StorageService.instance.getTokenMetadata().isMember(endpoint)) return; On Mon, Jan 20, 2014 at 12:34 PM, Allan C alla...@gmail.com wrote: There are 3 other nodes that have a mild case. This

Re: HintedHandoff Exception and node holding hints to random tokens

2014-01-20 Thread Allan C
 Certainly makes sense to not allow it. Any idea why the node would be holding hints for tokens that don’t exist? -Allan On January 20, 2014 at 1:09:51 PM, sankalp kohli (kohlisank...@gmail.com) wrote: Yes as per code you cannot delete hints for endpoints which are not part of the ring.   if