You can check which nodes hints are being held for using the JMX api. Look for the org.apache.cassandra.db:type=HintedHandoffManager MBean and call the listEndpointsPendingHints() function.
There are two points where hints may be stored, if the node is down when the request started or if the node timed out and did not return before rpc_timeout. To check for the first, look for log lines about a node being "dead" on the coordinator. To check for the second look for dropped messages on the other nodes. This will be logged, or you can use nodetool tpstats to look for them. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 15/03/2013, at 2:30 AM, Andras Szerdahelyi <andras.szerdahe...@ignitionone.com> wrote: > ( The previous letter was sent prematurely, sorry. ) > > This node is the only node being written to, but the Cfs being written > replicate to almost all of the other nodes > My understanding is that hinted handoff is mutations kept around on the > coordinator node, to be replayed when the target node re-appears on the ring. > All my nodes are up and again, no hinted handoff is logged on the node itself > > Thanks! > Andras > > From: Andras Szerdahelyi <andras.szerdahe...@ignitionone.com> > Date: Thursday 14 March 2013 14:25 > To: "user@cassandra.apache.org" <user@cassandra.apache.org> > Subject: 33million hinted handoffs from nowhere > > Hi list, > > I am experiencing seemingly uncontrollable and unexplained growth of my > HintedHandoff CF on a single node. Unexplained because there are no hinted > handoffs being logged on the node, uncontrollable because I see 33 million > inserts in cfstats and the size of the stables is over 10 gigs all in an hour > of uptime. > > > I have done the following to try and reproduce this: > > - shut down my cluster > - on all nodes: remove sstables from the HintsColumnFamily data dir > - on all nodes: remove commit logs > - start all nodes but the one that’s showing this problem > - nothing is writing to any of the nodes. There are no hinted handoff going > on anywhere > - bring back the node in question last > - few seconds after boot: > > Column Family: HintsColumnFamily > SSTable count: 1 > Space used (live): 44946532 > Space used (total): 44946532 > Number of Keys (estimate): 256 > Memtable Columns Count: 17840 > Memtable Data Size: 17569909 > Memtable Switch Count: 2 > Read Count: 0 > Read Latency: NaN ms. > Write Count: 184836 > Write Latency: 0.668 ms. > Pending Tasks: 0 > Bloom Filter False Postives: 0 > Bloom Filter False Ratio: 0.00000 > Bloom Filter Space Used: 16 > Compacted row minimum size: 20924301 > Compacted row maximum size: 25109160 > Compacted row mean size: 25109160 > > > >
