[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Scooletz updated CASSANDRA-6998:
--------------------------------
Description:
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is run
only after successful handoff. This leaves the HH practically disabled till
administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
was:
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is run
only after successful handoff. This leaves the HH practically disabled till
administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
> HintedHandoff - expired hints may block future hints deliveries
> ---------------------------------------------------------------
>
> Key: CASSANDRA-6998
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: - cluster of two DCs: DC1, DC2
> - keyspace using NetworkTopologyStrategy (replication factors for both DCs)
> - heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
> with DC awareness, writing to DC1
> Reporter: Scooletz
> Labels: HintedHandoff, TTL
> Fix For: 2.0.3
>
>
> For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
> with millions of rows. Now, when _HintedHandOffManager_ tries to
> _doDeliverHintsToEndpoint_ it queries the store with
> QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
> Throwing this exception stops the manager from running compaction as it is
> run only after successful handoff. This leaves the HH practically disabled
> till administrator runs truncateAllHints.
> Wouldn't it be nicer if on
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
> That would remove TTLed hints leaving whole HH mechanism in a healthy state.
> The stacktrace is:
> {quote}
> org.apache.cassandra.db.filter.TombstoneOverwhelmingException
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
> at
> org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
> at
> org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
> at
> org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
> at
> org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
> at
> org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
> at
> org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
> at
> org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
