Re: Questions about cleaning up/purging Hinted Handoffs
Will try... Thank you Rahul Neelakantan On Sep 10, 2014, at 12:01 AM, Rahul Menon ra...@apigee.com wrote: I use jmxterm. http://wiki.cyclopsgroup.org/jmxterm/ attach it to your c* process and then use the org.apache.cassandra.db:HintedHandoffManager bean and run deleteHintsforEndpoint ip to drop hints for each ip. On Wed, Sep 10, 2014 at 3:37 AM, Rahul Neelakantan ra...@rahul.be wrote: RF=3, two DCs. (Going to 1.2.x in a few weeks) What's the procedure to drop via JMX? - Rahul 1-678-451-4545 (US) +91 99018-06625 (India) On Sep 9, 2014, at 9:23 AM, Rahul Menon ra...@apigee.com wrote: Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up Rahul On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote: The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . I've seen this behavior with HH in older versions, so probably. =Rob
Re: Questions about cleaning up/purging Hinted Handoffs
Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up Rahul On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote: The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . I've seen this behavior with HH in older versions, so probably. =Rob
Re: Questions about cleaning up/purging Hinted Handoffs
RF=3, two DCs. (Going to 1.2.x in a few weeks) What's the procedure to drop via JMX? - Rahul 1-678-451-4545 (US) +91 99018-06625 (India) On Sep 9, 2014, at 9:23 AM, Rahul Menon ra...@apigee.com wrote: Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up Rahul On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote: The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . I've seen this behavior with HH in older versions, so probably. =Rob
Re: Questions about cleaning up/purging Hinted Handoffs
I use jmxterm. http://wiki.cyclopsgroup.org/jmxterm/ attach it to your c* process and then use the org.apache.cassandra.db:HintedHandoffManager bean and run deleteHintsforEndpoint ip to drop hints for each ip. On Wed, Sep 10, 2014 at 3:37 AM, Rahul Neelakantan ra...@rahul.be wrote: RF=3, two DCs. (Going to 1.2.x in a few weeks) What's the procedure to drop via JMX? - Rahul 1-678-451-4545 (US) +91 99018-06625 (India) On Sep 9, 2014, at 9:23 AM, Rahul Menon ra...@apigee.com wrote: Yep, the hinted handoff in 1.0.8 is abysmal at best. What is your replication facter, i have had huge hints pile up, where i had to drop the entire coloumn family and then run a repair. Either that or you can use the JMX HintedHandoffManager and delete hints per endpoint. Also it maybe worthwhile to investigate why you have hints pile up Rahul On Mon, Sep 8, 2014 at 11:09 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote: The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . I've seen this behavior with HH in older versions, so probably. =Rob
Re: Questions about cleaning up/purging Hinted Handoffs
On Fri, Sep 5, 2014 at 3:20 PM, Rahul Neelakantan ra...@rahul.be wrote: The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . I've seen this behavior with HH in older versions, so probably. =Rob
Re: Questions about cleaning up/purging Hinted Handoffs
Yup... Upgrades are in the works.. Thanks for the response. The reason I asked about he hints is because I see hints being replayed but the large compacted hints stable still sticks around, perhaps it is a bug with that version . Rahul Neelakantan On Sep 5, 2014, at 1:03 PM, Robert Coli rc...@eventbrite.com wrote: On Fri, Sep 5, 2014 at 6:12 AM, Rahul Neelakantan ra...@rahul.be wrote: The questions are for Cassandra 1.0.8 but may apply to later versions too. Upgrade your very old version of Cassandra ASAP. I barely remember the implementation of HH in your old version. 1) When the coordinator notices that a node it is collecting hints for is down for more than the max hint window, it stops collecting hints for that node - Does it automatically purge the hints it collected so far, or does removing them require manual intervention? It tries to deliver the hints it has stored so far, when it can. When it delivers them, it deletes them. - What is the recommended way to remove hints if they are not automatically removed? Stop node and delete hints CF db files? or use truncate CF? Either way is fine. There's also a JMX method (maybe not in your old version) to purge hints. 2) When the coordinator replays hints to a node that has come back within the max hint window, does it purge those hints once they are replayed or does cleanup of those hints require manual intervention? The former. 3) If a node is flapping because of network issues and gossip keeps thinking the node is down, causing the coordinator to reset the max hint window for that node and start collecting hints again for the node - Does it automatically purge the hints collected in the previous windows? No. - Or Does it replay all hints collected in all windows that the node was down? Yes. =Rob
