Re: High read Latency

To add onto this message: Queries are all on the partition key  (select origvalue,ingestdate,mediatype from doc.origdoc where uuid=?). Queries were very fast when the table was <10 million rows. Table description: describe doc.origdoc; CREATE TABLE doc.origdoc (     uuid text,    

Re: High read latency

Read requires avg. 6 sstables and my read latency is 42 ms. so on avg. we can say Cassandra is taking 7ms to process data from one sstable *which is entirely in memory*. I think there is something wrong here. If we go with this math then we can say Cassandra latency would be always > 7ms for most

Re: High read latency

1. Is it consistently taking that long? 2. Have you traced the requests? 3. Are you watching your GC history? 4. What's the load on the machine? Does dstat show high CPU or disk utilization? I did a webinar about a year ago on how to dig into these issues, you may find it useful:

Re: High read latency

Since you have most of your reads hitting 5-8 SSTables, it's probably related to that increasing your latency. That makes this look like your write workload is either overwrite-heavy or append-heavy. Data for a single partition key is being written to repeatedly over long time periods, and this

Re: High read latency

Maybe compaction not keeping up - since you are hitting so many sstables? Read heavy... are you using LCS? Plenty of resources... tune to increase memtable size? On Sat, Sep 26, 2015 at 9:19 AM, Eric Stevens wrote: > Since you have most of your reads hitting 5-8 SSTables,

Re: High read latency

if you run: nodetool cfhistograms On the given table and that will tell you how wide your rows are getting. At some point you can get wide enough rows that just the physics of retrieving them all take some time. > On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi

Re: High read latency

Please find histogram attached. On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla wrote: > if everything is in ram there could be a number of issues unrelated to > Cassandra and there could be hardware limitations or contention problems. > Otherwise cell count can really deeply

Re: High read latency

I understand that but everything is in RAM (my data dir is tmpfs) and my row is not that wide approx. less than 5MB in size. So my question is if everything is in RAM then why does it take 43ms latency? On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla wrote: > if you run: > >

Re: High read latency

if everything is in ram there could be a number of issues unrelated to Cassandra and there could be hardware limitations or contention problems. Otherwise cell count can really deeply impact reads, all ram or not, and some of this is because of the nature of GC and some of it is the age of the

Re: High read latency

Jaydeep; since your primary key involves a clustering column, you may be having pretty wide rows. The read would be sequential. The latency could be acceptable, if the read were to involve really wide rows. If your primary key was like ((a,b)) without the clustering column, it's like reading a

RE: High read latency

] Envoyé : mardi 22 septembre 2015 19:50 À : user@cassandra.apache.org Objet : Re: High read latency select * from test where a = ? and b = ? On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi <pskraj...@gmail.com<mailto:pskraj...@gmail.com>> wrote: thanks for the informat

Re: High read latency

select * from test where a = ? and b = ? On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi < pskraj...@gmail.com> wrote: > thanks for the information. Posting the query too would be of help. > > On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia < > chovatia.jayd...@gmail.com> wrote: >

Re: High read latency

thanks for the information. Posting the query too would be of help. On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia < chovatia.jayd...@gmail.com> wrote: > Please find required details here: > > - Number of req/s > > 2k reads/s > > - Schema details > > create table test { > >

Re: High read latency

Please find required details here: - Number of req/s 2k reads/s - Schema details create table test { a timeuuid, b bigint, c int, d int static, e int static, f int static, g int static, h int, i text, j text, k text, l text, m set n bigint o bigint p bigint q

RE: High read latency

Hi, Before speaking about tuning, can you provide some additional information ? - Number of req/s - Schema details - JVM settings about the heap - Execution time of the GC 43ms for a read latency may be acceptable according to the number of request per

Re: High read latency after data volume increased

There's likely 2 things occurring 1) the cfhistograms error is due to https://issues.apache.org/jira/browse/CASSANDRA-8028 Which is resolved in 2.1.3. Looks like voting is under way for 2.1.3. As rcoli mentioned, you are running the latest open source of C* which should be treated as beta until

Re: High read latency after data volume increased

Hi there, The compaction remains running with our workload. We are using SATA HDDs RAIDs. When trying to run cfhistograms on our user_data table, we are getting this message: nodetool: Unable to compute when histogram overflowed Please see what happens when running some queries on this cf:

Re: High read latency after data volume increased

C* seems to have more than its share of version x doesn't work, use version y type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli rc...@eventbrite.com wrote: On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar ronibaltha...@gmail.com wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1

Re: High read latency after data volume increased

Hi Robert, We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Please see the graphs below... ​ Trying another keyspace with 600+ reads/sec, we are getting the acceptable ~30ms read latency. Let

Re: High read latency after data volume increased

On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar ronibaltha...@gmail.com wrote: We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Note that downgrading is officially unsupported, but is

Re: High read latency after data volume increased

On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar ronibaltha...@gmail.com wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better

Re: High-read latency for non-existing rows with LCS and 1.2.5

We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node, we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744. The current default bloom_filter_fp_chance is 0.1 for levelled compaction. Reducing this

Re: High-read latency for non-existing rows with LCS and 1.2.5

Try doing request tracing. http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2 On Thu, Jun 27, 2013 at 2:40 PM, Bao Le l...@yahoo.com wrote: Hi, We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node, we have anywhere from 700+ to 800+

Re: High read latency cluster

300 GB is a lot of data for cloud machines (especially with their weaker performance in general). If you are unhappy with performance why not scale the cluster out to more servers, with that much data you are usually contending with the physics of spinning disks. Three nodes + replication factor 3

Re: High read latency cluster

FlushWriter 0 0 8252 0 299 If you are not suffering from gc pressure/pauses (possibly not, because you don't seem to have a lot of read failures in tpstats or outlier latency on the histograms), then the flush writer errors are

Re: High read latency

As written in the third point of http://wiki.apache.org/cassandra/CassandraLimitations, right now, super columns are not indexed and deserialized fully when you access them. Another way to put it is, you'll want to user super columns with only a relatively small number of columns in them. Because