[
https://issues.apache.org/jira/browse/CASSANDRA-8870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14353202#comment-14353202
]
Shawn Kumar commented on CASSANDRA-8870:
----------------------------------------
I created a few tests to try and reproduce this problem, and check more
specifically whether 1) HintedHandoff exhibited any abnormal tombstone
behaviour, 2) A TombstoneOverwhelmingException in system.hints would cause any
other issues (NoHostAvailableException). I was not able to reproduce problems
in either aspect. For aspect 2, I was able to artificially cause the
TombstoneOverwhelmingException by having more hints than the
tombstone_failure_threshold (and flushing) - but this would seem to be expected
behaviour and I was still able to connect to the cluster. Jeff if you have any
other information about the context of the error that would be useful ie -
queries, schema's, usage, node status; please feel free to share them and I can
give it another shot.
> Tombstone overwhelming issue aborts client queries
> --------------------------------------------------
>
> Key: CASSANDRA-8870
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8870
> Project: Cassandra
> Issue Type: Bug
> Environment: cassandra 2.1.2 ubunbtu 12.04
> Reporter: Jeff Liu
>
> We are getting client queries timeout issues on the clients who are trying to
> query data from cassandra cluster.
> Nodetool status shows that all nodes are still up regardless.
> Logs from client side:
> {noformat}
> com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s)
> tried for query failed (tried:
> cass-chisel01.abc01.abc02.abc.abc.com/10.66.182.113:9042
> (com.datastax.driver.core.TransportException:
> [cass-chisel01.tgr01.iad02.testd.nestlabs.com/10.66.182.113:9042] Connection
> has been closed))
> at
> com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
> ~[com.datastax.cassandra.cassandra-driver-core-2.1.3.jar:na]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
> Logs from cassandra/system.log
> {noformat}
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,410 SliceQueryFilter.java:212 -
> Scanned over 100000 tombstones in system.hints; query aborted (see
> tombstone_failure_threshold)
> ERROR [HintedHandoff:2] 2015-02-23 23:46:28,417 CassandraDaemon.java:153 -
> Exception in thread Thread[HintedHandoff:2,1,main]
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException: null
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:69)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:310)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:60)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1858)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1666)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:385)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:344)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager.access$400(HintedHandOffManager.java:94)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:555)
> ~[apache-cassandra-2.1.2.jar:2.1.2]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ~[na:1.7.0_55]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ~[na:1.7.0_55]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_55]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
