[jira] [Comment Edited] (CASSANDRA-14085) Excessive update of ReadLatency metric in digest calculation
[ https://issues.apache.org/jira/browse/CASSANDRA-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273593#comment-16273593 ] Andrew Whang edited comment on CASSANDRA-14085 at 12/11/17 9:27 PM: https://github.com/whangsf/cassandra/commit/d6ec955da577da614de0c093625ae175158362c3 was (Author: whangsf): https://github.com/whangsf/cassandra/commit/2ae3589ce9eefd8699bbd4e29bf1c61a486d394e > Excessive update of ReadLatency metric in digest calculation > > > Key: CASSANDRA-14085 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14085 > Project: Cassandra > Issue Type: Bug > Components: Core, Metrics >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 3.0.x, 3.11.x, 4.x > > > We noticed an increase in read latency after upgrading to 3.x, specifically > for requests with CL>ONE. It turns out the read latency metric is being > doubly updated for digest calculations. This code > (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java#L243) > makes an improper copy of an iterator that's wrapped by MetricRecording, > whose onClose() records the latency of the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-14085) Excessive update of ReadLatency metric in digest calculation
[ https://issues.apache.org/jira/browse/CASSANDRA-14085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-14085: - Fix Version/s: 4.0 3.0.16 Status: Patch Available (was: Open) https://github.com/whangsf/cassandra/commit/2ae3589ce9eefd8699bbd4e29bf1c61a486d394e > Excessive update of ReadLatency metric in digest calculation > > > Key: CASSANDRA-14085 > URL: https://issues.apache.org/jira/browse/CASSANDRA-14085 > Project: Cassandra > Issue Type: Bug > Components: Metrics >Reporter: Andrew Whang >Priority: Minor > Fix For: 3.0.16, 4.0 > > > We noticed an increase in read latency after upgrading to 3.x, specifically > for requests with CL>ONE. It turns out the read latency metric is being > doubly updated for digest calculations. This code > (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java#L243) > makes an improper copy of an iterator that's wrapped by MetricRecording, > whose onClose() records the latency of the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-14085) Excessive update of ReadLatency metric in digest calculation
Andrew Whang created CASSANDRA-14085: Summary: Excessive update of ReadLatency metric in digest calculation Key: CASSANDRA-14085 URL: https://issues.apache.org/jira/browse/CASSANDRA-14085 Project: Cassandra Issue Type: Bug Components: Metrics Reporter: Andrew Whang Priority: Minor We noticed an increase in read latency after upgrading to 3.x, specifically for requests with CL>ONE. It turns out the read latency metric is being doubly updated for digest calculations. This code (https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/partitions/UnfilteredPartitionIterators.java#L243) makes an improper copy of an iterator that's wrapped by MetricRecording, whose onClose() records the latency of the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13561: - Resolution: Duplicate Status: Resolved (was: Patch Available) > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246911#comment-16246911 ] Andrew Whang edited comment on CASSANDRA-13561 at 11/10/17 2:16 AM: CASSANDRA-13643 seems to do something similar, but safer. I think we can close this out. was (Author: whangsf): CASSANDRA-13643 seems to do something similar, but safer. Closing this out. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246911#comment-16246911 ] Andrew Whang commented on CASSANDRA-13561: -- CASSANDRA-13643 seems to do something similar, but safer. Closing this out. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13967) Hints for hosts no longer part of the ring remain in the hints directory
[ https://issues.apache.org/jira/browse/CASSANDRA-13967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16210081#comment-16210081 ] Andrew Whang commented on CASSANDRA-13967: -- It seems nodetool removenode and assassinate both successfully removes the hints for the node being removed. I do see accumulated hints for invalid hosts, just not sure how to reproduce it. Will close this ticket if unable to repro. > Hints for hosts no longer part of the ring remain in the hints directory > > > Key: CASSANDRA-13967 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13967 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Andrew Whang >Priority: Minor > > I find the hints directory grows over time as hosts are decommissioned > because the hints for the old hosts remain undelivered/unremoved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13967) Hints for hosts no longer part of the ring remain in the hints directory
[ https://issues.apache.org/jira/browse/CASSANDRA-13967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209873#comment-16209873 ] Andrew Whang commented on CASSANDRA-13967: -- nodetool truncatehints is one way to remove these stale hints, but it would be nice for the system to automatically detect and remove stale hints (for target hosts no longer part of the ring). > Hints for hosts no longer part of the ring remain in the hints directory > > > Key: CASSANDRA-13967 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13967 > Project: Cassandra > Issue Type: Bug > Components: Coordination >Reporter: Andrew Whang >Priority: Minor > > I find the hints directory grows over time as hosts are decommissioned > because the hints for the old hosts remain undelivered/unremoved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13967) Hints for hosts no longer part of the ring remain in the hints directory
Andrew Whang created CASSANDRA-13967: Summary: Hints for hosts no longer part of the ring remain in the hints directory Key: CASSANDRA-13967 URL: https://issues.apache.org/jira/browse/CASSANDRA-13967 Project: Cassandra Issue Type: Bug Components: Coordination Reporter: Andrew Whang Priority: Minor I find the hints directory grows over time as hosts are decommissioned because the hints for the old hosts remain undelivered/unremoved. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13632) Digest mismatch if row is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16170271#comment-16170271 ] Andrew Whang commented on CASSANDRA-13632: -- [~jasobrown] We're using old-school thrift via pycassa. I can repro in our shadow environment, but unable to in my dev using ccm + pycassa. Will let you know once I can get this error reproducible. > Digest mismatch if row is empty > --- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Andrew Whang >Assignee: Andrew Whang > Fix For: 3.0.x > > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the row is > empty (key does not exist). It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > The mismatch on the read path ends up doubling our p99 read latency. We > discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13752) Corrupted SSTables created in 3.11
[
https://issues.apache.org/jira/browse/CASSANDRA-13752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138959#comment-16138959
]
Andrew Whang commented on CASSANDRA-13752:
--
Can we get this patched to 3.0 as well?
> Corrupted SSTables created in 3.11
> --
>
> Key: CASSANDRA-13752
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13752
> Project: Cassandra
> Issue Type: Bug
>Reporter: Hannu Kröger
>Priority: Blocker
>
> We have discovered issues with corrupted SSTables.
> {code}
> ERROR [SSTableBatchOpen:22] 2017-08-03 20:19:53,195 SSTableReader.java:577 -
> Cannot read sstable
> /cassandra/data/mykeyspace/mytable-7a4992800d5611e7b782cb90016f2d17/mc-35556-big=[Data.db,
> Statistics.db, Summary.db, Digest.crc32, CompressionInfo.db, TOC.txt,
> Index.db, Filter.db]; other IO error, skipping table
> java.io.EOFException: EOF after 1898 bytes out of 21093
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:68)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.util.RebufferingInputStream.readFully(RebufferingInputStream.java:60)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:402)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:377)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:325)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.metadata.StatsMetadata$StatsMetadataSerializer.deserialize(StatsMetadata.java:231)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:122)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:93)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:488)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:396)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$5.run(SSTableReader.java:561)
> ~[apache-cassandra-3.11.0.jar:3.11.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_111]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_111]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_111]
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> [apache-cassandra-3.11.0.jar:3.11.0]
> {code}
> Files look like this:
> {code}
> -rw-r--r--. 1 cassandra cassandra 3899251 Aug 7 08:37
> mc-6166-big-CompressionInfo.db
> -rw-r--r--. 1 cassandra cassandra 16874421686 Aug 7 08:37 mc-6166-big-Data.db
> -rw-r--r--. 1 cassandra cassandra 10 Aug 7 08:37
> mc-6166-big-Digest.crc32
> -rw-r--r--. 1 cassandra cassandra 2930904 Aug 7 08:37
> mc-6166-big-Filter.db
> -rw-r--r--. 1 cassandra cassandra 75880 Aug 7 08:37
> mc-6166-big-Index.db
> -rw-r--r--. 1 cassandra cassandra 13762 Aug 7 08:37
> mc-6166-big-Statistics.db
> -rw-r--r--. 1 cassandra cassandra 882008 Aug 7 08:37
> mc-6166-big-Summary.db
> -rw-r--r--. 1 cassandra cassandra 92 Aug 7 08:37 mc-6166-big-TOC.txt
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093343#comment-16093343 ] Andrew Whang commented on CASSANDRA-13561: -- Correct, this implementation would still require a compaction to purge the expired cells. Also correct, in the scenario you described there is a risk of the expired cell returning from the dead. The scenario does disregard HH. If hints are properly delivered, the risk is mitigated. To be clear, the scenario is similar to setting GCGS=0. The user has to understand the risk of using these settings. The risk is also mitigated in use cases where there is a default TTL on the table or client mutations use a default TTL. These are the scenarios for which we use this feature in our environment. In these use cases, we noticed the table suffered from high droppable tombstone ratio and high read latency. Using this feature to purge TTL as they expired addressed both droppable tombstone ratio and read latency. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077291#comment-16077291 ] Andrew Whang commented on CASSANDRA-13561: -- Correct me if I'm wrong, but the limitations I see with CASSANDRA-13418 are: 1. It's tied to a compaction strategy, and currently only supported in TWCS. 2. It helps cleanup sstables, but only when all items in the sstable have expired. The approach I'm proposing would support dropping individual (expired) cells within the sstable, without having to wait for all other items to expire. > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Assignee: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13632) Digest mismatch if row is empty
[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13632: - Summary: Digest mismatch if row is empty (was: Digest mismatch if (non-static) column is null) > Digest mismatch if row is empty > --- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Andrew Whang > Fix For: 3.0.14 > > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the row is > empty (key does not exist). It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > The mismatch on the read path ends up doubling our p99 read latency. We > discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13632) Digest mismatch if (non-static) column is null
[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13632: - Description: This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the row is empty (key does not exist). It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same. The mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. One thing to note is that we're using thrift, which ends up handling the ColumnFilter differently than the CQL path. As with CASSANDRA-12090, fixing the digest seems sensible. was: This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the column value is null. It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same. Our use case involves frequent deletion of partition columns, so the mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. One thing to note is that we're using thrift, which ends up handling the ColumnFilter differently than the CQL path. As with CASSANDRA-12090, fixing the digest seems sensible. > Digest mismatch if (non-static) column is null > -- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Andrew Whang > Fix For: 3.0.14 > > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the row is > empty (key does not exist). It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > The mismatch on the read path ends up doubling our p99 read latency. We > discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13632) Digest mismatch if (non-static) column is null
[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13632: - Fix Version/s: 3.0.14 Status: Patch Available (was: Open) https://github.com/whangsf/cassandra/commit/0efb32db3eaf76229c53451f2fc3ab85693a76bd > Digest mismatch if (non-static) column is null > -- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Andrew Whang > Fix For: 3.0.14 > > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the column > value is null. It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > Our use case involves frequent deletion of partition columns, so the mismatch > on the read path ends up doubling our p99 read latency. We discovered this > issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13632) Digest mismatch if (non-static) column is null
[ https://issues.apache.org/jira/browse/CASSANDRA-13632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13632: - Description: This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the column value is null. It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same. Our use case involves frequent deletion of partition columns, so the mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. One thing to note is that we're using thrift, which ends up handling the ColumnFilter differently than the CQL path. As with CASSANDRA-12090, fixing the digest seems sensible. was: This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the column value is null. It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same. Our use case involves frequent deletion of partition columns, so the mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. As with CASSANDRA-12090, fixing the digest seems sensible. > Digest mismatch if (non-static) column is null > -- > > Key: CASSANDRA-13632 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 > Project: Cassandra > Issue Type: Bug > Components: Local Write-Read Paths >Reporter: Andrew Whang > > This issue is similar to CASSANDRA-12090. Quorum read queries that include a > column selector (non-wildcard) result in digest mismatch when the column > value is null. It seems the data serialization path checks if > rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY > flag). However, the digest serialization path does not perform this check and > includes column names. The digest comparison results in a mismatch. The > mismatch does not end up issuing a read repair mutation since the underlying > data is the same. > Our use case involves frequent deletion of partition columns, so the mismatch > on the read path ends up doubling our p99 read latency. We discovered this > issue while testing a 2.2.5 to 3.0.13 upgrade. > One thing to note is that we're using thrift, which ends up handling the > ColumnFilter differently than the CQL path. > As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13632) Digest mismatch if (non-static) column is null
Andrew Whang created CASSANDRA-13632: Summary: Digest mismatch if (non-static) column is null Key: CASSANDRA-13632 URL: https://issues.apache.org/jira/browse/CASSANDRA-13632 Project: Cassandra Issue Type: Bug Components: Local Write-Read Paths Reporter: Andrew Whang This issue is similar to CASSANDRA-12090. Quorum read queries that include a column selector (non-wildcard) result in digest mismatch when the column value is null. It seems the data serialization path checks if rowIterator.isEmpty() and if so ignores column names (by setting IS_EMPTY flag). However, the digest serialization path does not perform this check and includes column names. The digest comparison results in a mismatch. The mismatch does not end up issuing a read repair mutation since the underlying data is the same. Our use case involves frequent deletion of partition columns, so the mismatch on the read path ends up doubling our p99 read latency. We discovered this issue while testing a 2.2.5 to 3.0.13 upgrade. As with CASSANDRA-12090, fixing the digest seems sensible. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-13561) Purge TTL on expiration
[ https://issues.apache.org/jira/browse/CASSANDRA-13561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Whang updated CASSANDRA-13561: - Fix Version/s: 4.0 Status: Patch Available (was: Open) Patch here https://github.com/whangsf/cassandra/commit/6f46e18988122f80608b1f5ba4a3d5c5dbbe1c61 > Purge TTL on expiration > --- > > Key: CASSANDRA-13561 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 > Project: Cassandra > Issue Type: New Feature >Reporter: Andrew Whang >Priority: Minor > Fix For: 4.0 > > > Tables with mostly TTL columns tend to suffer from high droppable tombstone > ratio, which results in higher read latency, cpu utilization, and disk usage. > Expired TTL data become tombstones, and the nature of purging tombstones > during compaction (due to checking for overlapping SSTables) make them > susceptible to surviving much longer than expected. A table option to purge > TTL on expiration would address this issue, by preventing them from becoming > tombstones. A boolean purge_ttl_on_expiration table setting would allow users > to easily turn the feature on or off. > Being more aggressive with gc_grace could also address the problem of long > lasting tombstones, but that would affect tombstones from deletes as well. > Even if a purged [expired] cell is revived via repair from a node that hasn't > yet compacted away the cell, it would be revived as an expiring cell with the > same localDeletionTime, so reads should properly handle them. As well, it > would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-13561) Purge TTL on expiration
Andrew Whang created CASSANDRA-13561: Summary: Purge TTL on expiration Key: CASSANDRA-13561 URL: https://issues.apache.org/jira/browse/CASSANDRA-13561 Project: Cassandra Issue Type: New Feature Reporter: Andrew Whang Priority: Minor Tables with mostly TTL columns tend to suffer from high droppable tombstone ratio, which results in higher read latency, cpu utilization, and disk usage. Expired TTL data become tombstones, and the nature of purging tombstones during compaction (due to checking for overlapping SSTables) make them susceptible to surviving much longer than expected. A table option to purge TTL on expiration would address this issue, by preventing them from becoming tombstones. A boolean purge_ttl_on_expiration table setting would allow users to easily turn the feature on or off. Being more aggressive with gc_grace could also address the problem of long lasting tombstones, but that would affect tombstones from deletes as well. Even if a purged [expired] cell is revived via repair from a node that hasn't yet compacted away the cell, it would be revived as an expiring cell with the same localDeletionTime, so reads should properly handle them. As well, it would be purged in the next compaction. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
