[
https://issues.apache.org/jira/browse/CASSANDRA-14647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dinesh Joshi updated CASSANDRA-14647:
-------------------------------------
Description:
There is some issue with sstable metadata which is visible in system.log, the
messages says:
{noformat}
WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
cardinality from Statistics.db failed for
/opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
Although there is no such file.
The message has appeared after i've changed the compaction strategy from
SizeTiered to Leveled. Compaction strategy has been changed region by region
(total 3 regions) and it has coincided with the double client write traffic
increase.
I have tried to run nodetool scrub to rebuilt the sstable, but that does not
fix the issue.
So very hard to define the steps to reproduce, probably it will be:
# run stress tool with write traffic
# under load change compaction strategy from SireTiered to Leveled for the
bunch of hosts
# add more write traffic
Reading the code it is said that if this metadata is broken, then "estimating
the keys will be done using index summary".
[https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
was:
There is some issue with sstable metadata which is visible in system.log, the
messages says:
WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
cardinality from Statistics.db failed for
/opt/data/disk5/data/keyspace/table/mc-big-Data.db. Although there is no such
file.
The message has appeared after i've changed the compaction strategy from
SizeTiered to Leveled. Compaction strategy has been changed region by region
(total 3 regions) and it has coincided with the double client write traffic
increase.
I have tried to run nodetool scrub to rebuilt the sstable, but that does not
fix the issue.
So very hard to define the steps to reproduce, probably it will be:
1) run stress tool with write traffic
2) under load change compaction strategy from SireTiered to Leveled for the
bunch of hosts
3) add more write traffic
Reading the code it is said that if this metadata is broken, then "estimating
the keys will be done using index summary".
[https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
> Reading cardinality from Statistics.db failed
> ---------------------------------------------
>
> Key: CASSANDRA-14647
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14647
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: Clients are doing only writes with Local One, cluster
> consist of 3 regions with RF3.
> Storage is configured wth jbod/XFS on 10 x 1Tb disks
> IOPS limit for each disk 500 (total 5000 iops)
> Bandwith for each disk 60mb/s (600 total)
> OS is Debian linux.
> Reporter: Vitali Djatsuk
> Priority: Major
> Fix For: 3.0.x
>
> Attachments: cassandra_compaction_pending_tasks_7days.png
>
>
> There is some issue with sstable metadata which is visible in system.log, the
> messages says:
> {noformat}
> WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading
> cardinality from Statistics.db failed for
> /opt/data/disk5/data/keyspace/table/mc-big-Data.db.{noformat}
> Although there is no such file.
> The message has appeared after i've changed the compaction strategy from
> SizeTiered to Leveled. Compaction strategy has been changed region by region
> (total 3 regions) and it has coincided with the double client write traffic
> increase.
> I have tried to run nodetool scrub to rebuilt the sstable, but that does not
> fix the issue.
> So very hard to define the steps to reproduce, probably it will be:
> # run stress tool with write traffic
> # under load change compaction strategy from SireTiered to Leveled for the
> bunch of hosts
> # add more write traffic
> Reading the code it is said that if this metadata is broken, then "estimating
> the keys will be done using index summary".
>
> [https://github.com/apache/cassandra/blob/cassandra-3.0.17/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L247]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
