[
https://issues.apache.org/jira/browse/CASSANDRA-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom van der Woerdt resolved CASSANDRA-9728.
-------------------------------------------
Resolution: Invalid
Turns out that waiting for compaction to catch up fixed the problem.
> Very bad read performance after 2.1.5->2.1.7 upgrade
> ----------------------------------------------------
>
> Key: CASSANDRA-9728
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9728
> Project: Cassandra
> Issue Type: Bug
> Reporter: Tom van der Woerdt
>
> After upgrading from 2.1.5 to 2.1.7, read performance has significantly
> slowed down for queries that use indexes. A query that used to take under a
> second to execute now takes 8 seconds or longer.
> The table I am noticing this on is a fairly large table (possibly pushing
> Cassandra's limits a bit, with around 1TB per node) with a single index.
> {noformat}
> CREATE TABLE the_table (
> id bigint PRIMARY KEY,
> id_2 bigint,
> field_1 ascii,
> field_2 blob, -- "Wide" column in that it's on average 15KB
> field_3 ascii,
> field_4 ascii,
> field_5 text,
> field_6 ascii
> ) WITH compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '1024', 'sstable_compression':
> 'org.apache.cassandra.io.compress.DeflateCompressor'};
> CREATE INDEX the_table_id_2_idx ON the_table (id_2);
> {noformat}
> Compaction status, according to nodetool's cfstats :
> index: SSTables in each level: [1714/4, 7, 0, 0, 0, 0, 0, 0, 0]
> main: SSTables in each level: [79/4, 1, 105/100, 1049/1000,
> 172, 0, 0, 0, 0]
> It's had a few days to catch up, but the indexes don't seem to be catching up
> well. Do note that the performance issue I'm reporting was also experienced
> when the main table had not yet caught up. Queries fetching by the primary
> key have been equally fast before and after the compaction catch-up.
> One thing I have noticed is that cfstats is reporting a *LOT* of reads on the
> index, while there's no way I'm firing that many queries:
> Local read count: 6670463
> Local read latency: 4.134 ms
> Based on the application's logs, I would say there shouldn't have been more
> than 5000 queries against the index since Cassandra was started on this node.
> What may also be relevant is that during the upgrade to 2.1.7, nodes that
> were still on 2.1.5 were responding quickly (tested in cqlsh), while nodes
> that had already upgraded were slow. Even when only one node was left to be
> upgraded, that node was fast with responding.
> For completeness, the query that's slow is {{select * from the_table where
> id_2=?}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
