[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172514#comment-14172514
]
Jonathan Ellis commented on CASSANDRA-6998:
---
LGTM, +1
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Assignee: Aleksey Yeschenko
Labels: HintedHandoff, TTL
Fix For: 2.0.11, 2.1.1
Attachments: 6998, 6998-v2.txt
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13995547#comment-13995547
]
Jonathan Ellis commented on CASSANDRA-6998:
---
I see.
That would fix the problem, but I'm not really a fan of adding extra layers of
complexity to mask symptoms of a deeper underlying problem (CASSANDRA-).
Let's first figure out *why* these tombstones aren't going away the way we
expect them to; it's likely that we can then solve the problem without breaking
the SQF contract (and then trying to make up for that at a higher level).
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Attachments: 6998
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994200#comment-13994200
]
Muhammad Adel commented on CASSANDRA-6998:
--
Sorry if I didn't make myself clear enough. I will try to explain what this
patch does:
1-Normally, the page size passed to SliceQueryFilter represents number of live
columns returned. The SliceQueryFilter returns a mix of live columns and
tombstones which are not eligible for garbage collection yet. If the number of
tombstones seen during constructing the query result exceeds the threshold, an
exception is thrown.
2-When retrieving hinted handoff data, use the SliceQueryFilter in a different
way. The page size will represent the total number of columns returned, live
columns count + tombstones count. Even tombstones that are eligible for garbage
collection are returned. The check for the tombstones threshold is removed. No
memory overload will take place since the size of data processed and returned
is controlled and predictable (exactly the page size).
3-This will send all data in the hinted handoff table to the recovering node
allowing for higher data consistency even if the node was down for a time
larger than the gc_grace period. On the other hand it can cause a higher
network load.
This is a suggested way to deal with the issue of consistency of deleted hinted
handoff data. It is not just an addressing for the exception problem, because I
think the exception problem is a symptom for the more delicate issue of deleted
hinted handoff.
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Fix For: 2.0.3
Attachments: 6998
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979480#comment-13979480
]
Muhammad Adel commented on CASSANDRA-6998:
--
Instead of ignoring tombstones when counting returned page size, and instead
of ignoring irrelevant data, deliver both relevant and Irrelevant columns. This
will increase data consistency when the recovering node was down for a time
longer than the gc grace period.
To avoid performance problems, when choosing to return both relevant and
irrelevant data, the slice query filter will count the tombstones and
irrelevant columns, compared to counting live columns only in the ordinary
case. The check for tombstone threshold is ignored in this specific case.
Patch submitted.
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Fix For: 2.0.3
Attachments: 6998
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980393#comment-13980393
]
Muhammad Adel commented on CASSANDRA-6998:
--
Sorry, changed status to TESTING by error. Can someone of the moderators return
the status back to Patch Available? there is no undo functionality
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Fix For: 2.0.3
Attachments: 6998
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13968457#comment-13968457
]
Muhammad Adel commented on CASSANDRA-6998:
--
Deleting undelivered tombstones can introduce problems with consistency. I
don't think a hinted handoff row should ever be deleted before delivery even if
it was a tombstone.
The exception is thrown when the number of deleted columns in the query result
exceed the tombstonefailurethreshold.
This can be solved using in two ways:
1-Instead of just removing the tombstones by compacting the table, we can
decrease the page size when catching the exception and then try to perform the
hinted handoff again.This will decrease the number of deleted columns below the
tombstonefailurethreshold.
2-a better design choice in my opinion is to use another type of queries, one
that gets all rows with a given size regardless of being a live row or
not.SliceQueries are not the suitable type for this task since hinted handoff
deliver doesn't care for the status of the given row.
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Fix For: 2.0.3
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
The stacktrace is:
{quote}
org.apache.cassandra.db.filter.TombstoneOverwhelmingException
at
org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:201)
at
org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
at
org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
at
org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1487)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1306)
at
org.apache.cassandra.db.HintedHandOffManager.doDeliverHintsToEndpoint(HintedHandOffManager.java:351)
at
org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(HintedHandOffManager.java:309)
at
org.apache.cassandra.db.HintedHandOffManager.access$300(HintedHandOffManager.java:92)
at
org.apache.cassandra.db.HintedHandOffManager$4.run(HintedHandOffManager.java:530)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[ https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962935#comment-13962935 ] Aleksey Yeschenko commented on CASSANDRA-6998: -- See CASSANDRA- HintedHandoff - expired hints may block future hints deliveries --- Key: CASSANDRA-6998 URL: https://issues.apache.org/jira/browse/CASSANDRA-6998 Project: Cassandra Issue Type: Bug Components: Core Environment: - cluster of two DCs: DC1, DC2 - keyspace using NetworkTopologyStrategy (replication factors for both DCs) - heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup with DC awareness, writing to DC1 Reporter: Scooletz Labels: HintedHandoff, TTL Fix For: 2.0.3 For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled with millions of rows. Now, when _HintedHandOffManager_ tries to _doDeliverHintsToEndpoint_ it queries the store with QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws org.apache.cassandra.db.filter.TombstoneOverwhelmingException. Throwing this exception stops the manager from running compaction as it is run only after successful handoff. This leaves the HH practically disabled till administrator runs truncateAllHints. Wouldn't it be nicer if on org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction? That would remove TTLed hints leaving whole HH mechanism in a healthy state. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-6998) HintedHandoff - expired hints may block future hints deliveries
[
https://issues.apache.org/jira/browse/CASSANDRA-6998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963007#comment-13963007
]
Scooletz commented on CASSANDRA-6998:
-
Thx [~iamaleksey] for your quick comment, I think this problem is not solved
yet though. In the comments section of the issue referenced by you
https://issues.apache.org/jira/browse/CASSANDRA- [~gsanderson] asks:
{quote}
One quick question... this patch will hopefully prevent us getting into this
state in many cases, unless you have a huge number of hints for a node that is
down for a very long time: since there is no auto-compaction, a large number of
hints may have expired from TTL, thus preventing any further hint delivery.
{quote}
That's the case I'm referring to. I got this Is it covered by ? If not, is
it considered as a bug? I added stacktrace to the issue.
HintedHandoff - expired hints may block future hints deliveries
---
Key: CASSANDRA-6998
URL: https://issues.apache.org/jira/browse/CASSANDRA-6998
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: - cluster of two DCs: DC1, DC2
- keyspace using NetworkTopologyStrategy (replication factors for both DCs)
- heavy load (write:read, 100:1) with LOCAL_QUORUM using Java driver setup
with DC awareness, writing to DC1
Reporter: Scooletz
Labels: HintedHandoff, TTL
Fix For: 2.0.3
For tests purposes, DC2 was shut down for 1 day. The _hints_ table was filled
with millions of rows. Now, when _HintedHandOffManager_ tries to
_doDeliverHintsToEndpoint_ it queries the store with
QueryFilter.getSliceFilter which counts deleted (TTLed) cells and throws
org.apache.cassandra.db.filter.TombstoneOverwhelmingException.
Throwing this exception stops the manager from running compaction as it is
run only after successful handoff. This leaves the HH practically disabled
till administrator runs truncateAllHints.
Wouldn't it be nicer if on
org.apache.cassandra.db.filter.TombstoneOverwhelmingException run compaction?
That would remove TTLed hints leaving whole HH mechanism in a healthy state.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
