I'm assuming the two nodes are the ones receiving the HH after they were down.
Are there a lot of hints collected while they are down ? you can check the HintedHandOffManager MBean in JConsole What does the TPStats look like on the nodes under pressure ? And how many nodes are delivering hints to the nodes when they restart? Finally hinted_handoff_throttle_delay_in_ms in conf/cassandra.yaml will let you slow down the delivery rate if HH is indeed the problem. Hope that helps. ----------------- Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 12 May 2011, at 06:55, Gabriel Tataranu wrote: > Greetings, > > I'm experiencing some issues with 2 nodes (out of more than 10). Right > after startup (Listening for thrift clients...) the nodes will create > objects at high rate using all available CPU cores: > > INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed > leaving 2024909864 used; max is 6658457600 > INFO 18:13:20,393 GC for PS Scavenge: 252 ms, 478691280 reclaimed > leaving 2184252600 used; max is 6658457600 > .... > INFO 18:15:23,909 GC for PS Scavenge: 283 ms, 452943472 reclaimed > leaving 5523891120 used; max is 6658457600 > INFO 18:15:24,912 GC for PS Scavenge: 273 ms, 466157568 reclaimed > leaving 5594606128 used; max is 6658457600 > > This will eventually trigger old-gen GC and then the process repeats > until hinted handoff finishes. > > The build version was updated from 0.7.2 to 0.7.5 but the behavior was > exactly the same. > > Thank you. >
