[
https://issues.apache.org/jira/browse/CASSANDRA-2357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Harvey updated CASSANDRA-2357:
------------------------------------
Attachment: thread_dump.txt
> Load spikes on coordinators since upgrade from 0.6.8 to 0.7
> -----------------------------------------------------------
>
> Key: CASSANDRA-2357
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2357
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.4
> Reporter: Jason Harvey
> Attachments: thread_dump.txt
>
>
> Since our move from 0.6.8 to 0.7, all of the nodes which speak with clients
> have been having periodic, abrupt load spikes going into the hundreds. We
> have been seeing these load spikes 1 to 2 times per hour on every node which
> clients are speaking with. The load graph for a typical spike:
> http://i.imgur.com/jY8AV.png
> I have verified that client connections are not spiking at the same time via
> TCP statistics. I have also verified that we aren't seeing any spikes in
> reads/mutations/etc.
> We were using the DynamicSnitch, but I turned that off as a troubleshooting
> step. The issue was unchanged.
> When the spikes occur, the box's responsiveness slows to a crawl so I am
> unable to do much in terms of real-time diagnostics. I was able to get a
> thread dump a few seconds after a spike, which I have attached to this
> ticket. Not sure if it will show anything since I couldn't capture it
> immediately during the spike.
> I should note that David King noticed a similar problem (#2058) when he tried
> moving us from 0.6.8 to 0.6.10. The main issue at the time was a long-lasting
> load spike, but he also saw occasional abrupt load spikes like we are seeing
> now. When we moved back to 0.6.8, we didn't see the problem again, until the
> move to 0.7.
> I realize this information is somewhat nebulous. If there is any further info
> I can provide, please let me know. The spikes are causing quite a bit of
> instability, so we are considering retreating back to 0.6.8. I'd like to
> investigate every possible solution before we resort to that.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
