[
https://issues.apache.org/jira/browse/CASSANDRA-8140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davide updated CASSANDRA-8140:
------------------------------
Description:
Hi there,
I'm on cassandra 2.1 since then I figured out that in some circumstances (I
can't find a way to reproduce them constantly) minor compactions and full
compactions had no effects.
We are on a cluster composed of 5 nodes with around 500gb of data, no deletions
around 1.5k updates/s and same on reads.
After a repair I saw that a couple of nodes were `slow`, I investigate further
and I found that on these two nodes the number of sstable were around 20.000+ !
We use STC.
So with node tool I triggered a full compaction, It took less than I minute
(with nothing in the logs) and of course the number of sstable didn't go down.
Then I drained the node, and I ran again with `nodetool compact`, at that point
the number of sstables went down to less than 10.
I tough was a strange spot problem. However after a week I noticed that one
node had ~100 sstabels where others just 8-10.
I ran again the compaction (It last less than a minute with nothing in logs)
and didn't change anything. I drained it and restarted then compacted and took
several hours to get it back close to 2/3 sstables.
What could be? We never incurred this behavior before.
Here informations about the table:
{code}
CREATE TABLE xyz (
ppk text PRIMARY KEY,
.. ten more columns...
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'cold_reads_to_omit': '0.0',
'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}
Here the current cf stats:
{code}
SSTable count: 11
Space used (live), bytes: 118007220865
Space used (total), bytes: 118007220865
Space used by snapshots (total), bytes: 170591332257
SSTable Compression Ratio: 0.3643916626015517
Memtable cell count: 920306
Memtable data size, bytes: 70034097
Memtable switch count: 25
Local read count: 5358772
Local read latency: 54.621 ms
Local write count: 4715106
Local write latency: 0.069 ms
Pending flushes: 0
Bloom filter false positives: 53757
Bloom filter false ratio: 0.04103
Bloom filter space used, bytes: 220634056
Compacted partition minimum bytes: 18
Compacted partition maximum bytes: 61214
Compacted partition mean bytes: 1935
Average live cells per slice (last five minutes):
0.8139232271871242
Average tombstones per slice (last five minutes):
0.5493417148555677
{code}
Is there anything else that I can provide?
Thanks!
DD
was:
Hi there,
I'm on cassandra 2.1 since then I figure out that in some circumstances (I
can't find a way to reproduce them constantly) minor compactions and full
compactions takes no effects.
So we are on a cluster composed of 5 nodes with around 500gb of data, no
deletions around 1.5k updates/s and same on reads.
After a repair I saw that a couple of nodes were `slow`, I investigate further
and I found that on these two nodes the number of sstable were around 20.000+ !
We use STC.
So with node tool I triggered a full compaction, It took less than I minute
(with noting in the logs) and of course the number of sstable didn't go down.
Then I drained the node, and I ran again the `nodetool compact`, at that point
the number of sstables went down to less than 10.
I tough was a strange spot problem. However after a week I noticed that one
node had ~100 sstabels where others just 8-10.
I ran again the compaction (It last less than a minute with nothing in logs)
and didn't change anything. I drained it and restarted then compacted and took
several hours to get it back close to 2/3 sstables.
What could be? We never incurred this behavior before.
Here informations about the table:
{code}
CREATE TABLE xyz (
ppk text PRIMARY KEY,
.. ten more columns...
) WITH bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'min_threshold': '4', 'cold_reads_to_omit': '0.0',
'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
{code}
Here the current cf stats:
{code}
SSTable count: 11
Space used (live), bytes: 118007220865
Space used (total), bytes: 118007220865
Space used by snapshots (total), bytes: 170591332257
SSTable Compression Ratio: 0.3643916626015517
Memtable cell count: 920306
Memtable data size, bytes: 70034097
Memtable switch count: 25
Local read count: 5358772
Local read latency: 54.621 ms
Local write count: 4715106
Local write latency: 0.069 ms
Pending flushes: 0
Bloom filter false positives: 53757
Bloom filter false ratio: 0.04103
Bloom filter space used, bytes: 220634056
Compacted partition minimum bytes: 18
Compacted partition maximum bytes: 61214
Compacted partition mean bytes: 1935
Average live cells per slice (last five minutes):
0.8139232271871242
Average tombstones per slice (last five minutes):
0.5493417148555677
{code}
Is there anything else that I can provide?
Thanks!
DD
> Compaction has no effects
> -------------------------
>
> Key: CASSANDRA-8140
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8140
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Reporter: Davide
>
> Hi there,
> I'm on cassandra 2.1 since then I figured out that in some circumstances (I
> can't find a way to reproduce them constantly) minor compactions and full
> compactions had no effects.
> We are on a cluster composed of 5 nodes with around 500gb of data, no
> deletions around 1.5k updates/s and same on reads.
> After a repair I saw that a couple of nodes were `slow`, I investigate
> further and I found that on these two nodes the number of sstable were around
> 20.000+ ! We use STC.
> So with node tool I triggered a full compaction, It took less than I minute
> (with nothing in the logs) and of course the number of sstable didn't go down.
> Then I drained the node, and I ran again with `nodetool compact`, at that
> point the number of sstables went down to less than 10.
> I tough was a strange spot problem. However after a week I noticed that one
> node had ~100 sstabels where others just 8-10.
> I ran again the compaction (It last less than a minute with nothing in logs)
> and didn't change anything. I drained it and restarted then compacted and
> took several hours to get it back close to 2/3 sstables.
> What could be? We never incurred this behavior before.
> Here informations about the table:
> {code}
> CREATE TABLE xyz (
> ppk text PRIMARY KEY,
> .. ten more columns...
> ) WITH bloom_filter_fp_chance = 0.01
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = ''
> AND compaction = {'min_threshold': '4', 'cold_reads_to_omit': '0.0',
> 'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
> 'max_threshold': '32'}
> AND compression = {'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Here the current cf stats:
> {code}
> SSTable count: 11
> Space used (live), bytes: 118007220865
> Space used (total), bytes: 118007220865
> Space used by snapshots (total), bytes: 170591332257
> SSTable Compression Ratio: 0.3643916626015517
> Memtable cell count: 920306
> Memtable data size, bytes: 70034097
> Memtable switch count: 25
> Local read count: 5358772
> Local read latency: 54.621 ms
> Local write count: 4715106
> Local write latency: 0.069 ms
> Pending flushes: 0
> Bloom filter false positives: 53757
> Bloom filter false ratio: 0.04103
> Bloom filter space used, bytes: 220634056
> Compacted partition minimum bytes: 18
> Compacted partition maximum bytes: 61214
> Compacted partition mean bytes: 1935
> Average live cells per slice (last five minutes):
> 0.8139232271871242
> Average tombstones per slice (last five minutes):
> 0.5493417148555677
> {code}
> Is there anything else that I can provide?
> Thanks!
> DD
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
