The number of Completed HH tasks is interesting. AFAIK a task is started when
the node detects another in the cluster has returned. Were you doing some other
restarts around the cluster ?
I don't want to divert from the GC issue, just wondering if something else is
going on as well. Like the
The number of Completed HH tasks is interesting. AFAIK a task is started when
the node detects another in the cluster has returned. Were you doing some
other restarts around the cluster ?
Not at all. The restarts seem to happen as normal operation.
I don't want to divert from the GC
Just out of curiosity is this on the receiver or sender side?
I have been wondering a bit if the hint playback could need some
adjustment.
There is potentially quite big differences on how much is sent per throttle
delay time depending on what your data looks like.
Early 0.7 releases also built
An if you have 10 nodes, do all of them happen to send hints to the two with
GC?
Terje
On Thu, May 12, 2011 at 6:10 PM, Terje Marthinussen tmarthinus...@gmail.com
wrote:
Just out of curiosity is this on the receiver or sender side?
I have been wondering a bit if the hint playback could need
Greetings,
Just out of curiosity is this on the receiver or sender side?
Looks like sender side, although the 2 nodes were replicating to each
other so it's hard to tell.
I have been wondering a bit if the hint playback could need some
adjustment.
There is potentially quite big
An if you have 10 nodes, do all of them happen to send hints to the two
with GC?
The 2 nodes are adjacent in token range. They are replicating to each other.
Other nodes have no data to replicate so there's no proof one way or
another.
Best,
Gabriel
Greetings,
Doesn't really look abnormal to me for a heavy write load situation
which is what receiving hints is.
I would agree with you but this raises some questions about write
performance. Plus I've only seen this kind of behavior recently and only
on 2 adjacent nodes. So I have good reason
I'm assuming the two nodes are the ones receiving the HH after they were
down.
Adjacent, so yes.
Are there a lot of hints collected while they are down ? you can check the
HintedHandOffManager MBean in JConsole
There wasn't any downtime - that's something else that's weird.
What
What does the TPStats look like on the nodes under pressure ? And how many
nodes are delivering hints to the nodes when they restart?
$nodetool -h 127.0.0.1 tpstats
Pool NameActive Pending Completed
ReadStage 1 11992475
Greetings,
I'm experiencing some issues with 2 nodes (out of more than 10). Right
after startup (Listening for thrift clients...) the nodes will create
objects at high rate using all available CPU cores:
INFO 18:13:15,350 GC for PS Scavenge: 292 ms, 494902976 reclaimed
leaving 2024909864 used;
I'm assuming the two nodes are the ones receiving the HH after they were down.
Are there a lot of hints collected while they are down ? you can check the
HintedHandOffManager MBean in JConsole
What does the TPStats look like on the nodes under pressure ? And how many
nodes are delivering
Doesn't really look abnormal to me for a heavy write load situation
which is what receiving hints is.
On Wed, May 11, 2011 at 1:55 PM, Gabriel Tataranu gabr...@wajam.com wrote:
Greetings,
I'm experiencing some issues with 2 nodes (out of more than 10). Right
after startup (Listening for
12 matches
Mail list logo
