[
https://issues.apache.org/jira/browse/CASSANDRA-2394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13029630#comment-13029630
]
Thibaut edited comment on CASSANDRA-2394 at 5/5/11 10:45 PM:
-------------------------------------------------------------
I did grep the cassandra log for errors and no error showed up.
There were errors in kern.log. (reading errors) On another server before, after
seeing the reading errors, I did try to copy the cassandra data directory and
it caused input/output errors. I didn't try this on the latest server where
this error occured, but I suppose it would have caused input/output errors as
well.
Yes, it will still repond from any node (our application connects locally
first, and these nodes are also not replying), but very very very slowly (e.g.
maybe 1% of normal cluster performance) . Before it did show up on the entire
cluster a massive commands/responds pending queue (> 1000, version 0.7.4 on all
20 nodes as far as I can remember). Normally the queue has about 0-5 entries at
most.
I didn't output it when it occured on the latest server. Iterating over a table
with hector will be continoulsy paused by timeout exceptions. Even if our
application is turned off and this is the only application running. I'm also
using the dynamic snitch with 0.8 badness threshold.
Replication level is set to 3 (EDIT 3 that ist). So if one node goes down, it
should still work as 2 nodes are still available.
And as I said before, stopping cassandra on the faulty node will nearly
instantly return proper cluster performance.
What can help you? Shall I do a jstack trace next time the error occurs? But
this can take some time until a hd dies. Do you know of a tool where I can
simulate a hd error (or very slow read)?
was (Author: tbritz):
I did grep the cassandra log for errors and no error showed up.
There were errors in kern.log. (reading errors) On another server before, after
seeing the reading errors, I did try to copy the cassandra data directory and
it caused input/output errors. I didn't try this on the latest server where
this error occured, but I suppose it would have caused input/output errors as
well.
Yes, it will still repond from any node (our application connects locally
first, and these nodes are also not replying), but very very very slowly (e.g.
maybe 1% of normal cluster performance) . Before it did show up on the entire
cluster a massive commands/responds pending queue (> 1000, version 0.7.4 on all
20 nodes as far as I can remember). Normally the queue has about 0-5 entries at
most.
I didn't output it when it occured on the latest server. Iterating over a table
with hector will be continoulsy paused by timeout exceptions. Even if our
application is turned off and this is the only application running. I'm also
using the dynamic snitch with 0.8 badness threshold.
Replication level is set to 4. So if one node goes down, it should still work
as 2 nodes are still available.
And as I said before, stopping cassandra on the faulty node will nearly
instantly return proper cluster performance.
What can help you? Shall I do a jstack trace next time the error occurs? But
this can take some time until a hd dies. Do you know of a tool where I can
simulate a hd error (or very slow read)?
> Faulty hd kills cluster performance
> -----------------------------------
>
> Key: CASSANDRA-2394
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2394
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.7.4
> Reporter: Thibaut
> Priority: Minor
> Fix For: 0.7.6
>
>
> Hi,
> About every week, a node from our main cluster (>100 nodes) has a faulty hd
> (Listing the cassandra data storage directoy triggers an input/output error).
> Whenever this occurs, I see many timeoutexceptions in our application on
> various nodes which cause everything to run very very slowly. Keyrange scans
> just timeout and will sometimes never succeed. If I stop cassandra on the
> faulty node, everything runs normal again.
> It would be great to have some kind of monitoring thread in cassandra which
> marks a node as "down" if there are multiple read/write errors to the data
> directories. A single faulty hd on 1 node shouldn't affect global cluster
> performance.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
