Re: How to monitor the progress of a HintedHandoff task?

2013-12-06 Thread Rahul Menon
Tom, you should look at phi_convict_threshold and try and increase the value if you have too much chatter on your network. Also, rebuilding the entire node because of a OOM does not make sense, could you please post the C* version that you are using the head size you have configured? Thanks

How to monitor the progress of a HintedHandoff task?

2013-12-03 Thread Tom van den Berge
Hi, Is there a way to monitor the progress of a hinted handoff task? I found the following two mbeans providing some info: org.apache.cassandra.internal:type=HintedHandoff, which tells me that there is 1 active task, and org.apache.cassandra.db:type=HintedHandoffManager#countPendingHints(),

Re: How to monitor the progress of a HintedHandoff task?

2013-12-03 Thread Rahul Menon
Tom, You should check the size of the hints column family to determine how much are present. The hints are a super column family and its keys are destination tokens. You could look at it if you would like. Hints send and timedouts are logged, you should be seeing something like Timed out

Re: How to monitor the progress of a HintedHandoff task?

2013-12-03 Thread Tom van den Berge
Hi Rahul, Thanks for your reply. I have never seen message like Timed out replaying hints to..., which is a good thing then, I suppose ;) Normally, I do see the Finished hinted handoff... log message. However, every now and then this message is not logged, not even after several hours. This is

Re: How to monitor the progress of a HintedHandoff task?

2013-12-03 Thread Rahul Menon
Tom, Do you know why these hints are piling up? What is the size of the hints cf? Thanks Rahul On Tue, Dec 3, 2013 at 6:41 PM, Tom van den Berge t...@drillster.com wrote: Hi Rahul, Thanks for your reply. I have never seen message like Timed out replaying hints to..., which is a good

Re: How to monitor the progress of a HintedHandoff task?

2013-12-03 Thread Tom van den Berge
Rahul, This problem occurs every now and then, and currently everything is ok, so there are no hints. But whenever it happens, the hints are quickly piling up. This results in heap problems on the node (Heap is 0.813462 full... appears many times). This in turn results in the flushing of the