RE: Questions about high read latency and related metrics
Hello C* community, I have been experimenting a bit with my lab node. I am assuming the following as observing the progress of metrics over time: 1. EstimatedPartitionSizeHistogram metric derives from READ operations. Cassandra reports values to this metric as it serves Read queries. 2. PartitionSize derives from the Compaction activities. Cassandra reports values to this metric as it performs the compaction of sstables. I am not sure whether those assumptions are valid but at least provide a good explanation to the progress of the stats observed. Thanks a lot and CU on the next topic. BR MK From: Michail Kotsiouros via user Sent: Thursday, May 11, 2023 14:08 To: user@cassandra.apache.org Subject: RE: Questions about high read latency and related metrics Hello Erick, No Max/Min/Mean vs Histogram difference is clear. What confuses me is the description of those metrics: Size of the compacted partition (in bytes). Vs estimated partition size. I am after what is measured by each metric. To be more specific: What metric should be consider when we want to see the partition size over time? Does this “compacted partition” means that only the partitions which have undergone a compaction in the respective sstables are taken into account for PartitionSize metrics? What “estimated” means in the EstimatedPartitionSizeHistogram metric? Excuse me if those questions sound trivial. BR MK From: Erick Ramirez mailto:erickramire...@apache.org>> Sent: Thursday, May 11, 2023 13:16 To: user@cassandra.apache.org<mailto:user@cassandra.apache.org>; Michail Kotsiouros mailto:michail.kotsiou...@ericsson.com>> Subject: Re: Questions about high read latency and related metrics Is it the concept of histograms that's not clear? Something else?
RE: Questions about high read latency and related metrics
Hello Erick, No Max/Min/Mean vs Histogram difference is clear. What confuses me is the description of those metrics: Size of the compacted partition (in bytes). Vs estimated partition size. I am after what is measured by each metric. To be more specific: What metric should be consider when we want to see the partition size over time? Does this “compacted partition” means that only the partitions which have undergone a compaction in the respective sstables are taken into account for PartitionSize metrics? What “estimated” means in the EstimatedPartitionSizeHistogram metric? Excuse me if those questions sound trivial. BR MK From: Erick Ramirez Sent: Thursday, May 11, 2023 13:16 To: user@cassandra.apache.org; Michail Kotsiouros Subject: Re: Questions about high read latency and related metrics Is it the concept of histograms that's not clear? Something else?
Re: Questions about high read latency and related metrics
Is it the concept of histograms that's not clear? Something else? >
RE: Questions about high read latency and related metrics
Hello Erick, Thanks a lot for the immediate reply but still the difference between those 2 metrics is not clear to me. BR MK From: Erick Ramirez Sent: Thursday, May 11, 2023 13:04 To: user@cassandra.apache.org Subject: Re: Questions about high read latency and related metrics The min/max/mean partition sizes are the sizes in bytes which are the same statistics reported by nodetool tablestats. EstimatedPartitionSizeHistogram is the distribution of partition sizes within specified ranges (percentiles) and is the same histogram reported by nodetool tablehistograms (in the Partition Size column). Cheers!
Re: Questions about high read latency and related metrics
The min/max/mean partition sizes are the sizes in bytes which are the same statistics reported by nodetool tablestats. EstimatedPartitionSizeHistogram is the distribution of partition sizes within specified ranges (percentiles) and is the same histogram reported by nodetool tablehistograms (in the Partition Size column). Cheers! >
Questions about high read latency and related metrics
Hello Cassandra community, I see the following metrics in JMX Metric Name org.apache.cassandra.metrics.Table... MinPartitionSize Gauge Size of the smallest compacted partition (in bytes). MaxPartitionSize Gauge Size of the largest compacted partition (in bytes). MeanPartitionSize Gauge Size of the average compacted partition (in bytes). And EstimatedPartitionSizeHistogram Gauge Histogram of estimated partition size (in bytes). Could you, please, help me clarify the difference of those 2 metrics. We suspect that the increasing partition size by the application data model has an impact on Read latency. What would be the appropriate metric to monitor from PartitionSize and EstimatedPartitionSizeHistogram. BR Michail Kotisouros
Re: High read Latency
To add onto this message:
Queries are all on the partition key (select
origvalue,ingestdate,mediatype from doc.origdoc where uuid=?). Queries
were very fast when the table was <10 million rows.
Table description:
describe doc.origdoc;
CREATE TABLE doc.origdoc (
uuid text,
ingestdate timestamp,
markings text,
mediatype text,
origvalue text,
source text,
PRIMARY KEY (uuid, ingestdate)
) WITH CLUSTERING ORDER BY (ingestdate ASC)
AND additional_write_policy = '99p'
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND cdc = false
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '16', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND default_time_to_live = 0
AND extensions = {}
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair = 'BLOCKING'
AND speculative_retry = '99p';
-Joe
On 11/29/2021 11:22 AM, Joe Obernberger wrote:
I have an 11 node cluster and am experiencing high read latency on one
table. This table has ~112 million rows:
nodetool tablehistograms doc.origdoc
doc/origdoc histograms
Percentile Read Latency Write Latency SSTables Partition
Size Cell Count
(micros) (micros) (bytes)
50% 36157.19 0.00 1.00 310 4
75% 74975.55 0.00 1.00 372 4
95% 155469.30 0.00 1.00 642 4
98% 223875.79 0.00 1.00 924 4
99% 268650.95 0.00 1.00 924 4
Min 152.32 0.00 1.00 180 4
Max 464228.84 0.00 1.00
9887 17
What should I look for to debug?
Thank you!
-Joe
High read Latency
I have an 11 node cluster and am experiencing high read latency on one table. This table has ~112 million rows: nodetool tablehistograms doc.origdoc doc/origdoc histograms Percentile Read Latency Write Latency SSTables Partition Size Cell Count (micros) (micros) (bytes) 50% 36157.19 0.00 1.00 310 4 75% 74975.55 0.00 1.00 372 4 95% 155469.30 0.00 1.00 642 4 98% 223875.79 0.00 1.00 924 4 99% 268650.95 0.00 1.00 924 4 Min 152.32 0.00 1.00 180 4 Max 464228.84 0.00 1.00 9887 17 What should I look for to debug? Thank you! -Joe
Re: High read latency
gt;>>>
>>>>>>> - Number of req/s
>>>>>>>
>>>>>>> 2k reads/s
>>>>>>>
>>>>>>> - Schema details
>>>>>>>
>>>>>>> create table test {
>>>>>>>
>>>>>>> a timeuuid,
>>>>>>>
>>>>>>> b bigint,
>>>>>>>
>>>>>>> c int,
>>>>>>>
>>>>>>> d int static,
>>>>>>>
>>>>>>> e int static,
>>>>>>>
>>>>>>> f int static,
>>>>>>>
>>>>>>> g int static,
>>>>>>>
>>>>>>> h int,
>>>>>>>
>>>>>>> i text,
>>>>>>>
>>>>>>> j text,
>>>>>>>
>>>>>>> k text,
>>>>>>>
>>>>>>> l text,
>>>>>>>
>>>>>>> m set
>>>>>>>
>>>>>>> n bigint
>>>>>>>
>>>>>>> o bigint
>>>>>>>
>>>>>>> p bigint
>>>>>>>
>>>>>>> q bigint
>>>>>>>
>>>>>>> r int
>>>>>>>
>>>>>>> s text
>>>>>>>
>>>>>>> t bigint
>>>>>>>
>>>>>>> u text
>>>>>>>
>>>>>>> v text
>>>>>>>
>>>>>>> w text
>>>>>>>
>>>>>>> x bigint
>>>>>>>
>>>>>>> y bigint
>>>>>>>
>>>>>>> z bigint,
>>>>>>>
>>>>>>> primary key ((a, b), c)
>>>>>>>
>>>>>>> };
>>>>>>>
>>>>>>> - JVM settings about the heap
>>>>>>>
>>>>>>> Default settings
>>>>>>>
>>>>>>> - Execution time of the GC
>>>>>>>
>>>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>>>>
>>>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <
>>>>>>> eric.le...@worldline.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Before speaking about tuning, can you provide some additional
>>>>>>>> information ?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> - Number of req/s
>>>>>>>>
>>>>>>>> - Schema details
>>>>>>>>
>>>>>>>> - JVM settings about the heap
>>>>>>>>
>>>>>>>> - Execution time of the GC
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> 43ms for a read latency may be acceptable according to the number
>>>>>>>> of request per second.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Eric
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>>>>> *À :* user@cassandra.apache.org
>>>>>>>> *Objet :* High read latency
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> My application issues more read requests than write, I do see that
>>>>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Local read count: 114479357
>>>>>>>>
>>>>>>>> Local read latency: 43.442 ms
>>>>>>>>
>>>>>>>> Local write count: 22288868
>>>>>>>>
>>>>>>>> Local write latency: 0.609 ms
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Here is my node configuration:
>>>>>>>>
>>>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5
>>>>>>>> GB of data on each node (and for experiment purpose I stored data in
>>>>>>>> tmpfs)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any idea what should I tune?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Jaydeep
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé
>>>>>>>> par
>>>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci
>>>>>>>> d'en
>>>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message.
>>>>>>>> Bien
>>>>>>>> que les meilleurs efforts soient faits pour maintenir cette
>>>>>>>> transmission
>>>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet
>>>>>>>> égard et
>>>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage
>>>>>>>> résultant
>>>>>>>> d'un virus transmis.
>>>>>>>>
>>>>>>>> This e-mail and the documents attached are confidential and
>>>>>>>> intended solely for the addressee; it may also be privileged. If you
>>>>>>>> receive this e-mail in error, please notify the sender immediately and
>>>>>>>> destroy it. As its integrity cannot be secured on the Internet, the
>>>>>>>> Worldline liability cannot be triggered for the message content.
>>>>>>>> Although
>>>>>>>> the sender endeavours to maintain a computer virus-free network, the
>>>>>>>> sender
>>>>>>>> does not warrant that this transmission is virus-free and will not be
>>>>>>>> liable for any damages resulting from any virus transmitted.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>> Ryan Svihla
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
Re: High read latency
;>>> h int, >>>>>> >>>>>> i text, >>>>>> >>>>>> j text, >>>>>> >>>>>> k text, >>>>>> >>>>>> l text, >>>>>> >>>>>> m set >>>>>> >>>>>> n bigint >>>>>> >>>>>> o bigint >>>>>> >>>>>> p bigint >>>>>> >>>>>> q bigint >>>>>> >>>>>> r int >>>>>> >>>>>> s text >>>>>> >>>>>> t bigint >>>>>> >>>>>> u text >>>>>> >>>>>> v text >>>>>> >>>>>> w text >>>>>> >>>>>> x bigint >>>>>> >>>>>> y bigint >>>>>> >>>>>> z bigint, >>>>>> >>>>>> primary key ((a, b), c) >>>>>> >>>>>> }; >>>>>> >>>>>> - JVM settings about the heap >>>>>> >>>>>> Default settings >>>>>> >>>>>> - Execution time of the GC >>>>>> >>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file. >>>>>> >>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric >>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Before speaking about tuning, can you provide some additional >>>>>>> information ? >>>>>>> >>>>>>> >>>>>>> >>>>>>> - Number of req/s >>>>>>> >>>>>>> - Schema details >>>>>>> >>>>>>> - JVM settings about the heap >>>>>>> >>>>>>> - Execution time of the GC >>>>>>> >>>>>>> >>>>>>> >>>>>>> 43ms for a read latency may be acceptable according to the number of >>>>>>> request per second. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> >>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com] >>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07 >>>>>>> *À :* user@cassandra.apache.org >>>>>>> *Objet :* High read latency >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> My application issues more read requests than write, I do see that >>>>>>> under load cfstats for one of the table is quite high around 43ms >>>>>>> >>>>>>> >>>>>>> >>>>>>> Local read count: 114479357 >>>>>>> >>>>>>> Local read latency: 43.442 ms >>>>>>> >>>>>>> Local write count: 22288868 >>>>>>> >>>>>>> Local write latency: 0.609 ms >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is my node configuration: >>>>>>> >>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 >>>>>>> GB of data on each node (and for experiment purpose I stored data in >>>>>>> tmpfs) >>>>>>> >>>>>>> >>>>>>> >>>>>>> I've tried increasing concurrent_read count upto 512 but no help in >>>>>>> read latency. CPU/Memory/IO looks fine on system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any idea what should I tune? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jaydeep >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à >>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé >>>>>>> par >>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci >>>>>>> d'en >>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du >>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de >>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien >>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission >>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet >>>>>>> égard et >>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant >>>>>>> d'un virus transmis. >>>>>>> >>>>>>> This e-mail and the documents attached are confidential and intended >>>>>>> solely for the addressee; it may also be privileged. If you receive this >>>>>>> e-mail in error, please notify the sender immediately and destroy it. As >>>>>>> its integrity cannot be secured on the Internet, the Worldline liability >>>>>>> cannot be triggered for the message content. Although the sender >>>>>>> endeavours >>>>>>> to maintain a computer virus-free network, the sender does not warrant >>>>>>> that >>>>>>> this transmission is virus-free and will not be liable for any damages >>>>>>> resulting from any virus transmitted. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> Regards, >>>> >>>> Ryan Svihla >>>> >>>> >>> >>> >> >
Re: High read latency
gt;>>>> >>>>>> l text, >>>>>> >>>>>> m set >>>>>> >>>>>> n bigint >>>>>> >>>>>> o bigint >>>>>> >>>>>> p bigint >>>>>> >>>>>> q bigint >>>>>> >>>>>> r int >>>>>> >>>>>> s text >>>>>> >>>>>> t bigint >>>>>> >>>>>> u text >>>>>> >>>>>> v text >>>>>> >>>>>> w text >>>>>> >>>>>> x bigint >>>>>> >>>>>> y bigint >>>>>> >>>>>> z bigint, >>>>>> >>>>>> primary key ((a, b), c) >>>>>> >>>>>> }; >>>>>> >>>>>> - JVM settings about the heap >>>>>> >>>>>> Default settings >>>>>> >>>>>> - Execution time of the GC >>>>>> >>>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file. >>>>>> >>>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric >>>>> > wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Before speaking about tuning, can you provide some additional >>>>>>> information ? >>>>>>> >>>>>>> >>>>>>> >>>>>>> - Number of req/s >>>>>>> >>>>>>> - Schema details >>>>>>> >>>>>>> - JVM settings about the heap >>>>>>> >>>>>>> - Execution time of the GC >>>>>>> >>>>>>> >>>>>>> >>>>>>> 43ms for a read latency may be acceptable according to the number of >>>>>>> request per second. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Eric >>>>>>> >>>>>>> >>>>>>> >>>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com] >>>>>>> *Envoyé :* mardi 22 septembre 2015 00:07 >>>>>>> *À :* user@cassandra.apache.org >>>>>>> *Objet :* High read latency >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> My application issues more read requests than write, I do see that >>>>>>> under load cfstats for one of the table is quite high around 43ms >>>>>>> >>>>>>> >>>>>>> >>>>>>> Local read count: 114479357 >>>>>>> >>>>>>> Local read latency: 43.442 ms >>>>>>> >>>>>>> Local write count: 22288868 >>>>>>> >>>>>>> Local write latency: 0.609 ms >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Here is my node configuration: >>>>>>> >>>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 >>>>>>> GB of data on each node (and for experiment purpose I stored data in >>>>>>> tmpfs) >>>>>>> >>>>>>> >>>>>>> >>>>>>> I've tried increasing concurrent_read count upto 512 but no help in >>>>>>> read latency. CPU/Memory/IO looks fine on system. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any idea what should I tune? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Jaydeep >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Ce message et les pièces jointes sont confidentiels et réservés à >>>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé >>>>>>> par >>>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci >>>>>>> d'en >>>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du >>>>>>> message ne pouvant être assurée sur Internet, la responsabilité de >>>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien >>>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission >>>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet >>>>>>> égard et >>>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant >>>>>>> d'un virus transmis. >>>>>>> >>>>>>> This e-mail and the documents attached are confidential and intended >>>>>>> solely for the addressee; it may also be privileged. If you receive this >>>>>>> e-mail in error, please notify the sender immediately and destroy it. As >>>>>>> its integrity cannot be secured on the Internet, the Worldline liability >>>>>>> cannot be triggered for the message content. Although the sender >>>>>>> endeavours >>>>>>> to maintain a computer virus-free network, the sender does not warrant >>>>>>> that >>>>>>> this transmission is virus-free and will not be liable for any damages >>>>>>> resulting from any virus transmitted. >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> Regards, >>>> >>>> Ryan Svihla >>>> >>>> >>> >>> >> >
Re: High read latency
Since you have most of your reads hitting 5-8 SSTables, it's probably
related to that increasing your latency. That makes this look like your
write workload is either overwrite-heavy or append-heavy. Data for a
single partition key is being written to repeatedly over long time periods,
and this will definitely impact read performance.
You can enable tracing in cqlsh and run your select to see where the time
is going.
On Fri, Sep 25, 2015 at 3:07 PM, Jaydeep Chovatia <
chovatia.jayd...@gmail.com> wrote:
> Please find histogram attached.
>
> On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla wrote:
>
>> if everything is in ram there could be a number of issues unrelated to
>> Cassandra and there could be hardware limitations or contention problems.
>> Otherwise cell count can really deeply impact reads, all ram or not, and
>> some of this is because of the nature of GC and some of it is the age of
>> the sstable format (which is due to be revamped in 3.0). Also partition
>> size can matter just because of physics, if one of those is a 1gb
>> partition, the network interface can only move that back across the wire so
>> quickly not to mention the GC issues you’d run into.
>>
>> Anyway this is why I asked for the histograms, I wanted to get cell count
>> and partition size. I’ve seen otherwise very stout hardware get slow on
>> reads of large results because either a bottleneck was hit somewhere, or
>> the CPU got slammed with GC, or other processes running on the machine were
>> contending with Cassandra.
>>
>>
>> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <
>> chovatia.jayd...@gmail.com> wrote:
>>
>> I understand that but everything is in RAM (my data dir is tmpfs) and my
>> row is not that wide approx. less than 5MB in size. So my question is if
>> everything is in RAM then why does it take 43ms latency?
>>
>> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla wrote:
>>
>>> if you run:
>>>
>>> nodetool cfhistograms
>>>
>>> On the given table and that will tell you how wide your rows are
>>> getting. At some point you can get wide enough rows that just the physics
>>> of retrieving them all take some time.
>>>
>>>
>>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>>> pskraj...@gmail.com> wrote:
>>>
>>> Jaydeep; since your primary key involves a clustering column, you may be
>>> having pretty wide rows. The read would be sequential. The latency could be
>>> acceptable, if the read were to involve really wide rows.
>>>
>>> If your primary key was like ((a,b)) without the clustering column, it's
>>> like reading a key value pair, and 40ms latency may have been a concern.
>>>
>>> Bottom Line : The latency depends on how wide the row is.
>>>
>>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>>> pskraj...@gmail.com> wrote:
>>>
>>>> thanks for the information. Posting the query too would be of help.
>>>>
>>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>>> chovatia.jayd...@gmail.com> wrote:
>>>>
>>>>> Please find required details here:
>>>>>
>>>>> - Number of req/s
>>>>>
>>>>> 2k reads/s
>>>>>
>>>>> - Schema details
>>>>>
>>>>> create table test {
>>>>>
>>>>> a timeuuid,
>>>>>
>>>>> b bigint,
>>>>>
>>>>> c int,
>>>>>
>>>>> d int static,
>>>>>
>>>>> e int static,
>>>>>
>>>>> f int static,
>>>>>
>>>>> g int static,
>>>>>
>>>>> h int,
>>>>>
>>>>> i text,
>>>>>
>>>>> j text,
>>>>>
>>>>> k text,
>>>>>
>>>>> l text,
>>>>>
>>>>> m set
>>>>>
>>>>> n bigint
>>>>>
>>>>> o bigint
>>>>>
>>>>> p bigint
>>>>>
>>>>> q bigint
>>>>>
>>>>> r int
>>>>>
>>>>> s text
>>>>>
>>>>> t bigint
>>>>>
>>>>> u text
>>>>>
>>>>> v text
>>>>>
>>>>> w text
>>>
Re: High read latency
Please find histogram attached.
On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla wrote:
> if everything is in ram there could be a number of issues unrelated to
> Cassandra and there could be hardware limitations or contention problems.
> Otherwise cell count can really deeply impact reads, all ram or not, and
> some of this is because of the nature of GC and some of it is the age of
> the sstable format (which is due to be revamped in 3.0). Also partition
> size can matter just because of physics, if one of those is a 1gb
> partition, the network interface can only move that back across the wire so
> quickly not to mention the GC issues you’d run into.
>
> Anyway this is why I asked for the histograms, I wanted to get cell count
> and partition size. I’ve seen otherwise very stout hardware get slow on
> reads of large results because either a bottleneck was hit somewhere, or
> the CPU got slammed with GC, or other processes running on the machine were
> contending with Cassandra.
>
>
> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia
> wrote:
>
> I understand that but everything is in RAM (my data dir is tmpfs) and my
> row is not that wide approx. less than 5MB in size. So my question is if
> everything is in RAM then why does it take 43ms latency?
>
> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla wrote:
>
>> if you run:
>>
>> nodetool cfhistograms
>>
>> On the given table and that will tell you how wide your rows are getting.
>> At some point you can get wide enough rows that just the physics of
>> retrieving them all take some time.
>>
>>
>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>> Jaydeep; since your primary key involves a clustering column, you may be
>> having pretty wide rows. The read would be sequential. The latency could be
>> acceptable, if the read were to involve really wide rows.
>>
>> If your primary key was like ((a,b)) without the clustering column, it's
>> like reading a key value pair, and 40ms latency may have been a concern.
>>
>> Bottom Line : The latency depends on how wide the row is.
>>
>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>>> thanks for the information. Posting the query too would be of help.
>>>
>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>> chovatia.jayd...@gmail.com> wrote:
>>>
>>>> Please find required details here:
>>>>
>>>> - Number of req/s
>>>>
>>>> 2k reads/s
>>>>
>>>> - Schema details
>>>>
>>>> create table test {
>>>>
>>>> a timeuuid,
>>>>
>>>> b bigint,
>>>>
>>>> c int,
>>>>
>>>> d int static,
>>>>
>>>> e int static,
>>>>
>>>> f int static,
>>>>
>>>> g int static,
>>>>
>>>> h int,
>>>>
>>>> i text,
>>>>
>>>> j text,
>>>>
>>>> k text,
>>>>
>>>> l text,
>>>>
>>>> m set
>>>>
>>>> n bigint
>>>>
>>>> o bigint
>>>>
>>>> p bigint
>>>>
>>>> q bigint
>>>>
>>>> r int
>>>>
>>>> s text
>>>>
>>>> t bigint
>>>>
>>>> u text
>>>>
>>>> v text
>>>>
>>>> w text
>>>>
>>>> x bigint
>>>>
>>>> y bigint
>>>>
>>>> z bigint,
>>>>
>>>> primary key ((a, b), c)
>>>>
>>>> };
>>>>
>>>> - JVM settings about the heap
>>>>
>>>> Default settings
>>>>
>>>> - Execution time of the GC
>>>>
>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>
>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Before speaking about tuning, can you provide some additional
>>>>> information ?
>>>>>
>>>>>
>>>>>
>>>>> - Number of req/s
>>>>>
>>>>> - Schema details
&
Re: High read latency
if everything is in ram there could be a number of issues unrelated to
Cassandra and there could be hardware limitations or contention problems.
Otherwise cell count can really deeply impact reads, all ram or not, and some
of this is because of the nature of GC and some of it is the age of the sstable
format (which is due to be revamped in 3.0). Also partition size can matter
just because of physics, if one of those is a 1gb partition, the network
interface can only move that back across the wire so quickly not to mention the
GC issues you’d run into.
Anyway this is why I asked for the histograms, I wanted to get cell count and
partition size. I’ve seen otherwise very stout hardware get slow on reads of
large results because either a bottleneck was hit somewhere, or the CPU got
slammed with GC, or other processes running on the machine were contending with
Cassandra.
> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia
> wrote:
>
> I understand that but everything is in RAM (my data dir is tmpfs) and my row
> is not that wide approx. less than 5MB in size. So my question is if
> everything is in RAM then why does it take 43ms latency?
>
> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <mailto:r...@foundev.pro>> wrote:
> if you run:
>
> nodetool cfhistograms
>
> On the given table and that will tell you how wide your rows are getting. At
> some point you can get wide enough rows that just the physics of retrieving
> them all take some time.
>
>
>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi > <mailto:pskraj...@gmail.com>> wrote:
>>
>> Jaydeep; since your primary key involves a clustering column, you may be
>> having pretty wide rows. The read would be sequential. The latency could be
>> acceptable, if the read were to involve really wide rows.
>>
>> If your primary key was like ((a,b)) without the clustering column, it's
>> like reading a key value pair, and 40ms latency may have been a concern.
>>
>> Bottom Line : The latency depends on how wide the row is.
>>
>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi
>> mailto:pskraj...@gmail.com>> wrote:
>> thanks for the information. Posting the query too would be of help.
>>
>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia
>> mailto:chovatia.jayd...@gmail.com>> wrote:
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric > <mailto:eric.le...@worldline.com>> wrote:
>> Hi,
>>
>>
>>
>>
>>
>> Before speaking about tuning, can you provide some additional information ?
>>
>>
>>
>> - Number of req/s
>>
>> - Schema details
>>
>> - JVM settings about the heap
>>
>> - Execution time of the GC
>>
>>
>>
>> 43ms for a read latency may be acceptable according to the number of request
>> per second.
>>
>>
>>
>>
>>
>> Eric
>>
>>
>>
>> De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com
>> <mailto:chovatia.jayd...@gmail.com>]
>> Envoyé : mardi 22 septembre 2015 00:07
>> À : user@cassandra.apache.org <mailto:user@cassandra.apache.org>
>> Objet : High read latency
>>
>>
>>
>> Hi,
>>
>>
>>
>> My application issues more read requests than write, I do see that under
>> load cfstats for one of the table is quite high around 43ms
>
Re: High read latency
I understand that but everything is in RAM (my data dir is tmpfs) and my
row is not that wide approx. less than 5MB in size. So my question is if
everything is in RAM then why does it take 43ms latency?
On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla wrote:
> if you run:
>
> nodetool cfhistograms
>
> On the given table and that will tell you how wide your rows are getting.
> At some point you can get wide enough rows that just the physics of
> retrieving them all take some time.
>
>
> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
> pskraj...@gmail.com> wrote:
>
> Jaydeep; since your primary key involves a clustering column, you may be
> having pretty wide rows. The read would be sequential. The latency could be
> acceptable, if the read were to involve really wide rows.
>
> If your primary key was like ((a,b)) without the clustering column, it's
> like reading a key value pair, and 40ms latency may have been a concern.
>
> Bottom Line : The latency depends on how wide the row is.
>
> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
> pskraj...@gmail.com> wrote:
>
>> thanks for the information. Posting the query too would be of help.
>>
>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>> chovatia.jayd...@gmail.com> wrote:
>>
>>> Please find required details here:
>>>
>>> - Number of req/s
>>>
>>> 2k reads/s
>>>
>>> - Schema details
>>>
>>> create table test {
>>>
>>> a timeuuid,
>>>
>>> b bigint,
>>>
>>> c int,
>>>
>>> d int static,
>>>
>>> e int static,
>>>
>>> f int static,
>>>
>>> g int static,
>>>
>>> h int,
>>>
>>> i text,
>>>
>>> j text,
>>>
>>> k text,
>>>
>>> l text,
>>>
>>> m set
>>>
>>> n bigint
>>>
>>> o bigint
>>>
>>> p bigint
>>>
>>> q bigint
>>>
>>> r int
>>>
>>> s text
>>>
>>> t bigint
>>>
>>> u text
>>>
>>> v text
>>>
>>> w text
>>>
>>> x bigint
>>>
>>> y bigint
>>>
>>> z bigint,
>>>
>>> primary key ((a, b), c)
>>>
>>> };
>>>
>>> - JVM settings about the heap
>>>
>>> Default settings
>>>
>>> - Execution time of the GC
>>>
>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>
>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Before speaking about tuning, can you provide some additional
>>>> information ?
>>>>
>>>>
>>>>
>>>> - Number of req/s
>>>>
>>>> - Schema details
>>>>
>>>> - JVM settings about the heap
>>>>
>>>> - Execution time of the GC
>>>>
>>>>
>>>>
>>>> 43ms for a read latency may be acceptable according to the number of
>>>> request per second.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Eric
>>>>
>>>>
>>>>
>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>> *À :* user@cassandra.apache.org
>>>> *Objet :* High read latency
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> My application issues more read requests than write, I do see that
>>>> under load cfstats for one of the table is quite high around 43ms
>>>>
>>>>
>>>>
>>>> Local read count: 114479357
>>>>
>>>> Local read latency: 43.442 ms
>>>>
>>>> Local write count: 22288868
>>>>
>>>> Local write latency: 0.609 ms
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Here is my node configuration:
>>>>
>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB
>>>> of data on each node
Re: High read latency
if you run:
nodetool cfhistograms
On the given table and that will tell you how wide your rows are getting. At
some point you can get wide enough rows that just the physics of retrieving
them all take some time.
> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi
> wrote:
>
> Jaydeep; since your primary key involves a clustering column, you may be
> having pretty wide rows. The read would be sequential. The latency could be
> acceptable, if the read were to involve really wide rows.
>
> If your primary key was like ((a,b)) without the clustering column, it's like
> reading a key value pair, and 40ms latency may have been a concern.
>
> Bottom Line : The latency depends on how wide the row is.
>
> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi
> mailto:pskraj...@gmail.com>> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia
> mailto:chovatia.jayd...@gmail.com>> wrote:
> Please find required details here:
>
> - Number of req/s
>
> 2k reads/s
>
> - Schema details
>
> create table test {
>
> a timeuuid,
>
> b bigint,
>
> c int,
>
> d int static,
>
> e int static,
>
> f int static,
>
> g int static,
>
> h int,
>
> i text,
>
> j text,
>
> k text,
>
> l text,
>
> m set
>
> n bigint
>
> o bigint
>
> p bigint
>
> q bigint
>
> r int
>
> s text
>
> t bigint
>
> u text
>
> v text
>
> w text
>
> x bigint
>
> y bigint
>
> z bigint,
>
> primary key ((a, b), c)
>
> };
>
> - JVM settings about the heap
>
> Default settings
>
> - Execution time of the GC
>
> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>
>
> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <mailto:eric.le...@worldline.com>> wrote:
> Hi,
>
>
>
>
>
> Before speaking about tuning, can you provide some additional information ?
>
>
>
> - Number of req/s
>
> - Schema details
>
> - JVM settings about the heap
>
> - Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of request
> per second.
>
>
>
>
>
> Eric
>
>
>
> De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com
> <mailto:chovatia.jayd...@gmail.com>]
> Envoyé : mardi 22 septembre 2015 00:07
> À : user@cassandra.apache.org <mailto:user@cassandra.apache.org>
> Objet : High read latency
>
>
>
> Hi,
>
>
>
> My application issues more read requests than write, I do see that under load
> cfstats for one of the table is quite high around 43ms
>
>
>
> Local read count: 114479357
>
> Local read latency: 43.442 ms
>
> Local write count: 22288868
>
> Local write latency: 0.609 ms
>
>
>
>
>
> Here is my node configuration:
>
> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data
> on each node (and for experiment purpose I stored data in tmpfs)
>
>
>
> I've tried increasing concurrent_read count upto 512 but no help in read
> latency. CPU/Memory/IO looks fine on system.
>
>
>
> Any idea what should I tune?
>
>
>
> Jaydeep
>
>
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout virus,
> l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne
> saurait être recherchée pour tout dommage résultant d'un virus transmis.
>
> This e-mail and the documents attached are confidential and intended solely
> for the addressee; it may also be privileged. If you receive this e-mail in
> error, please notify the sender immediately and destroy it. As its integrity
> cannot be secured on the Internet, the Worldline liability cannot be
> triggered for the message content. Although the sender endeavours to maintain
> a computer virus-free network, the sender does not warrant that this
> transmission is virus-free and will not be liable for any damages resulting
> from any virus transmitted.
>
>
>
Regards,
Ryan Svihla
Re: High read latency
Jaydeep; since your primary key involves a clustering column, you may be
having pretty wide rows. The read would be sequential. The latency could be
acceptable, if the read were to involve really wide rows.
If your primary key was like ((a,b)) without the clustering column, it's
like reading a key value pair, and 40ms latency may have been a concern.
Bottom Line : The latency depends on how wide the row is.
On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
> chovatia.jayd...@gmail.com> wrote:
>
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before speaking about tuning, can you provide some additional
>>> information ?
>>>
>>>
>>>
>>> - Number of req/s
>>>
>>> - Schema details
>>>
>>> - JVM settings about the heap
>>>
>>> - Execution time of the GC
>>>
>>>
>>>
>>> 43ms for a read latency may be acceptable according to the number of
>>> request per second.
>>>
>>>
>>>
>>>
>>>
>>> Eric
>>>
>>>
>>>
>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>> *À :* user@cassandra.apache.org
>>> *Objet :* High read latency
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> My application issues more read requests than write, I do see that under
>>> load cfstats for one of the table is quite high around 43ms
>>>
>>>
>>>
>>> Local read count: 114479357
>>>
>>> Local read latency: 43.442 ms
>>>
>>> Local write count: 22288868
>>>
>>> Local write latency: 0.609 ms
>>>
>>>
>>>
>>>
>>>
>>> Here is my node configuration:
>>>
>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>>> data on each node (and for experiment purpose I stored data in tmpfs)
>>>
>>>
>>>
>>> I've tried increasing concurrent_read count upto 512 but no help in read
>>> latency. CPU/Memory/IO looks fine on system.
>>>
>>>
>>>
>>> Any idea what should I tune?
>>>
>>>
>>>
>>> Jaydeep
>>>
>>> --
>>>
>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>> d'un virus transmis.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive this
>>> e-mail in error, please notify the sender immediately and destroy it. As
>>> its integrity cannot be secured on the Internet, the Worldline liability
>>> cannot be triggered for the message content. Although the sender endeavours
>>> to maintain a computer virus-free network, the sender does not warrant that
>>> this transmission is virus-free and will not be liable for any damages
>>> resulting from any virus transmitted.
>>>
>>
>>
>
RE: High read latency
For read heavy workload, JVM GC can cause latency issue. (see
http://tech.shift.com/post/74311817513/cassandra-tuning-the-jvm-for-read-heavy-workloads)
If you have frequent minor GC taking 400ms, it may increase your read latency.
Eric
De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
Envoyé : mardi 22 septembre 2015 19:50
À : user@cassandra.apache.org
Objet : Re: High read latency
select * from test where a = ? and b = ?
On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi
mailto:pskraj...@gmail.com>> wrote:
thanks for the information. Posting the query too would be of help.
On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia
mailto:chovatia.jayd...@gmail.com>> wrote:
Please find required details here:
- Number of req/s
2k reads/s
- Schema details
create table test {
a timeuuid,
b bigint,
c int,
d int static,
e int static,
f int static,
g int static,
h int,
i text,
j text,
k text,
l text,
m set
n bigint
o bigint
p bigint
q bigint
r int
s text
t bigint
u text
v text
w text
x bigint
y bigint
z bigint,
primary key ((a, b), c)
};
- JVM settings about the heap
Default settings
- Execution time of the GC
Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
mailto:eric.le...@worldline.com>> wrote:
Hi,
Before speaking about tuning, can you provide some additional information ?
- Number of req/s
- Schema details
- JVM settings about the heap
- Execution time of the GC
43ms for a read latency may be acceptable according to the number of request
per second.
Eric
De : Jaydeep Chovatia
[mailto:chovatia.jayd...@gmail.com<mailto:chovatia.jayd...@gmail.com>]
Envoyé : mardi 22 septembre 2015 00:07
À : user@cassandra.apache.org<mailto:user@cassandra.apache.org>
Objet : High read latency
Hi,
My application issues more read requests than write, I do see that under load
cfstats for one of the table is quite high around 43ms
Local read count: 114479357
Local read latency: 43.442 ms
Local write count: 22288868
Local write latency: 0.609 ms
Here is my node configuration:
RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data
on each node (and for experiment purpose I stored data in tmpfs)
I've tried increasing concurrent_read count upto 512 but no help in read
latency. CPU/Memory/IO looks fine on system.
Any idea what should I tune?
Jaydeep
Ce message et les pièces jointes sont confidentiels et réservés à l'usage
exclusif de ses destinataires. Il peut également être protégé par le secret
professionnel. Si vous recevez ce message par erreur, merci d'en avertir
immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant
être assurée sur Internet, la responsabilité de Worldline ne pourra être
recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus,
l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne
saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for
the addressee; it may also be privileged. If you receive this e-mail in error,
please notify the sender immediately and destroy it. As its integrity cannot be
secured on the Internet, the Worldline liability cannot be triggered for the
message content. Although the sender endeavours to maintain a computer
virus-free network, the sender does not warrant that this transmission is
virus-free and will not be liable for any damages resulting from any virus
transmitted.
Ce message et les pièces jointes sont confidentiels et réservés à l'usage
exclusif de ses destinataires. Il peut également être protégé par le secret
professionnel. Si vous recevez ce message par erreur, merci d'en avertir
immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant
être assurée sur Internet, la responsabilité de Worldline ne pourra être
recherchée quant au contenu de ce message. Bien que les meilleurs efforts
soient faits pour maintenir cette transmission exempte de tout virus,
l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne
saurait être recherchée pour tout dommage résultant d'un virus transmis.
This e-mail and the documents attached are confidential and intended solely for
the addressee; it may also be privileged. If you receive this e-mail in error,
please notify the sender immediately and destroy it. As its integrity cannot be
secured on the Internet, the Worldline liability cannot be triggered for the
message content. Although the send
Re: High read latency
select * from test where a = ? and b = ?
On Tue, Sep 22, 2015 at 10:27 AM, sai krishnam raju potturi <
pskraj...@gmail.com> wrote:
> thanks for the information. Posting the query too would be of help.
>
> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
> chovatia.jayd...@gmail.com> wrote:
>
>> Please find required details here:
>>
>> - Number of req/s
>>
>> 2k reads/s
>>
>> - Schema details
>>
>> create table test {
>>
>> a timeuuid,
>>
>> b bigint,
>>
>> c int,
>>
>> d int static,
>>
>> e int static,
>>
>> f int static,
>>
>> g int static,
>>
>> h int,
>>
>> i text,
>>
>> j text,
>>
>> k text,
>>
>> l text,
>>
>> m set
>>
>> n bigint
>>
>> o bigint
>>
>> p bigint
>>
>> q bigint
>>
>> r int
>>
>> s text
>>
>> t bigint
>>
>> u text
>>
>> v text
>>
>> w text
>>
>> x bigint
>>
>> y bigint
>>
>> z bigint,
>>
>> primary key ((a, b), c)
>>
>> };
>>
>> - JVM settings about the heap
>>
>> Default settings
>>
>> - Execution time of the GC
>>
>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>
>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>>
>>>
>>> Before speaking about tuning, can you provide some additional
>>> information ?
>>>
>>>
>>>
>>> - Number of req/s
>>>
>>> - Schema details
>>>
>>> - JVM settings about the heap
>>>
>>> - Execution time of the GC
>>>
>>>
>>>
>>> 43ms for a read latency may be acceptable according to the number of
>>> request per second.
>>>
>>>
>>>
>>>
>>>
>>> Eric
>>>
>>>
>>>
>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>> *À :* user@cassandra.apache.org
>>> *Objet :* High read latency
>>>
>>>
>>>
>>> Hi,
>>>
>>>
>>>
>>> My application issues more read requests than write, I do see that under
>>> load cfstats for one of the table is quite high around 43ms
>>>
>>>
>>>
>>> Local read count: 114479357
>>>
>>> Local read latency: 43.442 ms
>>>
>>> Local write count: 22288868
>>>
>>> Local write latency: 0.609 ms
>>>
>>>
>>>
>>>
>>>
>>> Here is my node configuration:
>>>
>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>>> data on each node (and for experiment purpose I stored data in tmpfs)
>>>
>>>
>>>
>>> I've tried increasing concurrent_read count upto 512 but no help in read
>>> latency. CPU/Memory/IO looks fine on system.
>>>
>>>
>>>
>>> Any idea what should I tune?
>>>
>>>
>>>
>>> Jaydeep
>>>
>>> --
>>>
>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et
>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>> d'un virus transmis.
>>>
>>> This e-mail and the documents attached are confidential and intended
>>> solely for the addressee; it may also be privileged. If you receive this
>>> e-mail in error, please notify the sender immediately and destroy it. As
>>> its integrity cannot be secured on the Internet, the Worldline liability
>>> cannot be triggered for the message content. Although the sender endeavours
>>> to maintain a computer virus-free network, the sender does not warrant that
>>> this transmission is virus-free and will not be liable for any damages
>>> resulting from any virus transmitted.
>>>
>>
>>
>
Re: High read latency
thanks for the information. Posting the query too would be of help.
On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
chovatia.jayd...@gmail.com> wrote:
> Please find required details here:
>
> - Number of req/s
>
> 2k reads/s
>
> - Schema details
>
> create table test {
>
> a timeuuid,
>
> b bigint,
>
> c int,
>
> d int static,
>
> e int static,
>
> f int static,
>
> g int static,
>
> h int,
>
> i text,
>
> j text,
>
> k text,
>
> l text,
>
> m set
>
> n bigint
>
> o bigint
>
> p bigint
>
> q bigint
>
> r int
>
> s text
>
> t bigint
>
> u text
>
> v text
>
> w text
>
> x bigint
>
> y bigint
>
> z bigint,
>
> primary key ((a, b), c)
>
> };
>
> - JVM settings about the heap
>
> Default settings
>
> - Execution time of the GC
>
> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>
> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
> wrote:
>
>> Hi,
>>
>>
>>
>>
>>
>> Before speaking about tuning, can you provide some additional information
>> ?
>>
>>
>>
>> - Number of req/s
>>
>> - Schema details
>>
>> - JVM settings about the heap
>>
>> - Execution time of the GC
>>
>>
>>
>> 43ms for a read latency may be acceptable according to the number of
>> request per second.
>>
>>
>>
>>
>>
>> Eric
>>
>>
>>
>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>> *Envoyé :* mardi 22 septembre 2015 00:07
>> *À :* user@cassandra.apache.org
>> *Objet :* High read latency
>>
>>
>>
>> Hi,
>>
>>
>>
>> My application issues more read requests than write, I do see that under
>> load cfstats for one of the table is quite high around 43ms
>>
>>
>>
>> Local read count: 114479357
>>
>> Local read latency: 43.442 ms
>>
>> Local write count: 22288868
>>
>> Local write latency: 0.609 ms
>>
>>
>>
>>
>>
>> Here is my node configuration:
>>
>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
>> data on each node (and for experiment purpose I stored data in tmpfs)
>>
>>
>>
>> I've tried increasing concurrent_read count upto 512 but no help in read
>> latency. CPU/Memory/IO looks fine on system.
>>
>>
>>
>> Any idea what should I tune?
>>
>>
>>
>> Jaydeep
>>
>> --
>>
>> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
>> exclusif de ses destinataires. Il peut également être protégé par le secret
>> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
>> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
>> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
>> être recherchée quant au contenu de ce message. Bien que les meilleurs
>> efforts soient faits pour maintenir cette transmission exempte de tout
>> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
>> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
>> virus transmis.
>>
>> This e-mail and the documents attached are confidential and intended
>> solely for the addressee; it may also be privileged. If you receive this
>> e-mail in error, please notify the sender immediately and destroy it. As
>> its integrity cannot be secured on the Internet, the Worldline liability
>> cannot be triggered for the message content. Although the sender endeavours
>> to maintain a computer virus-free network, the sender does not warrant that
>> this transmission is virus-free and will not be liable for any damages
>> resulting from any virus transmitted.
>>
>
>
Re: High read latency
Please find required details here:
- Number of req/s
2k reads/s
- Schema details
create table test {
a timeuuid,
b bigint,
c int,
d int static,
e int static,
f int static,
g int static,
h int,
i text,
j text,
k text,
l text,
m set
n bigint
o bigint
p bigint
q bigint
r int
s text
t bigint
u text
v text
w text
x bigint
y bigint
z bigint,
primary key ((a, b), c)
};
- JVM settings about the heap
Default settings
- Execution time of the GC
Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric
wrote:
> Hi,
>
>
>
>
>
> Before speaking about tuning, can you provide some additional information ?
>
>
>
> - Number of req/s
>
> - Schema details
>
> - JVM settings about the heap
>
> - Execution time of the GC
>
>
>
> 43ms for a read latency may be acceptable according to the number of
> request per second.
>
>
>
>
>
> Eric
>
>
>
> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
> *Envoyé :* mardi 22 septembre 2015 00:07
> *À :* user@cassandra.apache.org
> *Objet :* High read latency
>
>
>
> Hi,
>
>
>
> My application issues more read requests than write, I do see that under
> load cfstats for one of the table is quite high around 43ms
>
>
>
> Local read count: 114479357
>
> Local read latency: 43.442 ms
>
> Local write count: 22288868
>
> Local write latency: 0.609 ms
>
>
>
>
>
> Here is my node configuration:
>
> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of
> data on each node (and for experiment purpose I stored data in tmpfs)
>
>
>
> I've tried increasing concurrent_read count upto 512 but no help in read
> latency. CPU/Memory/IO looks fine on system.
>
>
>
> Any idea what should I tune?
>
>
>
> Jaydeep
>
> --
>
> Ce message et les pièces jointes sont confidentiels et réservés à l'usage
> exclusif de ses destinataires. Il peut également être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de le détruire. L'intégrité du message ne
> pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra
> être recherchée quant au contenu de ce message. Bien que les meilleurs
> efforts soient faits pour maintenir cette transmission exempte de tout
> virus, l'expéditeur ne donne aucune garantie à cet égard et sa
> responsabilité ne saurait être recherchée pour tout dommage résultant d'un
> virus transmis.
>
> This e-mail and the documents attached are confidential and intended
> solely for the addressee; it may also be privileged. If you receive this
> e-mail in error, please notify the sender immediately and destroy it. As
> its integrity cannot be secured on the Internet, the Worldline liability
> cannot be triggered for the message content. Although the sender endeavours
> to maintain a computer virus-free network, the sender does not warrant that
> this transmission is virus-free and will not be liable for any damages
> resulting from any virus transmitted.
>
RE: High read latency
Hi, Before speaking about tuning, can you provide some additional information ? - Number of req/s - Schema details - JVM settings about the heap - Execution time of the GC 43ms for a read latency may be acceptable according to the number of request per second. Eric De : Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com] Envoyé : mardi 22 septembre 2015 00:07 À : user@cassandra.apache.org Objet : High read latency Hi, My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms Local read count: 114479357 Local read latency: 43.442 ms Local write count: 22288868 Local write latency: 0.609 ms Here is my node configuration: RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs) I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system. Any idea what should I tune? Jaydeep Ce message et les pièces jointes sont confidentiels et réservés à l'usage exclusif de ses destinataires. Il peut également être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de le détruire. L'intégrité du message ne pouvant être assurée sur Internet, la responsabilité de Worldline ne pourra être recherchée quant au contenu de ce message. Bien que les meilleurs efforts soient faits pour maintenir cette transmission exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard et sa responsabilité ne saurait être recherchée pour tout dommage résultant d'un virus transmis. This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Worldline liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted.
High read latency
Hi, My application issues more read requests than write, I do see that under load cfstats for one of the table is quite high around 43ms Local read count: 114479357 Local read latency: 43.442 ms Local write count: 22288868 Local write latency: 0.609 ms Here is my node configuration: RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB of data on each node (and for experiment purpose I stored data in tmpfs) I've tried increasing concurrent_read count upto 512 but no help in read latency. CPU/Memory/IO looks fine on system. Any idea what should I tune? Jaydeep
Re: Really high read latency
nodetool cfhistograms is also very helpful in diagnosing these kinds of data modelling issues. On 23 March 2015 at 14:43, Chris Lohfink wrote: > > >> Compacted partition maximum bytes: 36904729268 > > thats huge... 36gb rows are gonna cause a lot of problems, even when you > specify a precise cell under this it still is going to have an enormous > column index to deserialize on every read for the partition. As mentioned > above, you should include your attribute name in the partition key ((row_time, > attrs)) > to spread this out... Id call that critical > > Chris > > On Mon, Mar 23, 2015 at 4:13 PM, Dave Galbraith < > david92galbra...@gmail.com> wrote: > >> I haven't deleted anything. Here's output from a traced cqlsh query (I >> tried to make the spaces line up, hope it's legible): >> >> Execute CQL3 >> query >> | 2015-03-23 21:04:37.422000 | 172.31.32.211 | 0 >> Parsing select * from default.metrics where row_time = 16511 and attrs = >> '[redacted]' limit 100; [SharedPool-Worker-2] | 2015-03-23 21:04:37.423000 >> | 172.31.32.211 | 93 >> Preparing statement >> [SharedPool-Worker-2] >> | 2015-03-23 21:04:37.423000 | 172.31.32.211 |696 >> Executing single-partition query on metrics [SharedPool-Worker-1] >> >> | 2015-03-23 >> 21:04:37.425000 | 172.31.32.211 | 2807 >> Acquiring sstable references [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:37.425000 | >> 172.31.32.211 | 2993 >> Merging memtable tombstones [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:37.426000 | >> 172.31.32.211 | 3049 >> Partition index with 484338 entries found for sstable 15966 >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 >> | 202304 >> Seeking to partition indexed section in data file >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202354 >> Bloom filter allows skipping sstable 5613 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 | >> 202445 >> Bloom filter allows skipping sstable 5582 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 | >> 202478 >> Bloom filter allows skipping sstable 5611 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 | >> 202508 >> Bloom filter allows skipping sstable 5610 >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202539 >> Bloom filter allows skipping sstable 5549 >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202678 >> Bloom filter allows skipping sstable 5544 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | >> 202720 >> Bloom filter allows skipping sstable 5237 >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202752 >> Bloom filter allows skipping sstable 2516 >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202782 >> Bloom filter allows skipping sstable 2632 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | >> 202812 >> Bloom filter allows skipping sstable 3015 [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | >> 202852 >> Skipped 0/11 non-slice-intersecting sstables, included 0 due to >> tombstones [SharedPool-Worker-1] | 2015-03-23 >> 21:04:38.625001 | 172.31.32.211 | 202882 >> Merging data from memtables and 1 sstables [SharedPool-Worker-1] >> >> | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202902 >> Read 101 live and 0 tombstoned cells >> [SharedPool-Worker-1] >> | 2015-03-23 21:04:38.626000 | 172.31.32.211 | 203752 >> Request complete >> >> | 2015-03-23 >> 21:04:38.628253 | 172.31.32.211 | 206253 >> >> On Mon, Mar 23, 2015 at 11:53 AM, Eric Stevens wrote: >> >>> Enable tracing in cqlsh and see how many sstables are being lifted to >>> satisfy the query (are you repeatedly writing to the same partition >>> [row_time]) over time?). >>> >>> Also watch for whether you're hitting a lot of tombstones (are you >>> deleting lots of values in the same partition over time?). >>> >>> On Mon, Mar 23, 2015 at 4:01 AM, Dave Galbraith < >>> david92galbra...@gmail.com> wrote: >>> Duncan: I'm thinking it might be something like that. I'm also seeing just a ton of garbage collection on the box, could it be pulling rows for all 100k attrs for a given row_time into memory since only row_time is the partition key? Jens: I'm not using EBS (although I used to until I read up on how useless it is). I'm not sure what constitutes proper paging but my client
Re: Really high read latency
>> Compacted partition maximum bytes: 36904729268 thats huge... 36gb rows are gonna cause a lot of problems, even when you specify a precise cell under this it still is going to have an enormous column index to deserialize on every read for the partition. As mentioned above, you should include your attribute name in the partition key ((row_time, attrs)) to spread this out... Id call that critical Chris On Mon, Mar 23, 2015 at 4:13 PM, Dave Galbraith wrote: > I haven't deleted anything. Here's output from a traced cqlsh query (I > tried to make the spaces line up, hope it's legible): > > Execute CQL3 > query > | 2015-03-23 21:04:37.422000 | 172.31.32.211 | 0 > Parsing select * from default.metrics where row_time = 16511 and attrs = > '[redacted]' limit 100; [SharedPool-Worker-2] | 2015-03-23 21:04:37.423000 > | 172.31.32.211 | 93 > Preparing statement > [SharedPool-Worker-2] > | 2015-03-23 21:04:37.423000 | 172.31.32.211 |696 > Executing single-partition query on metrics [SharedPool-Worker-1] > > | 2015-03-23 > 21:04:37.425000 | 172.31.32.211 | 2807 > Acquiring sstable references [SharedPool-Worker-1] > > | 2015-03-23 21:04:37.425000 | > 172.31.32.211 | 2993 > Merging memtable tombstones [SharedPool-Worker-1] > > | 2015-03-23 21:04:37.426000 | > 172.31.32.211 | 3049 > Partition index with 484338 entries found for sstable 15966 > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625000 | 172.31.32.211 > | 202304 > Seeking to partition indexed section in data file > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202354 > Bloom filter allows skipping sstable 5613 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625000 | 172.31.32.211 | > 202445 > Bloom filter allows skipping sstable 5582 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625000 | 172.31.32.211 | > 202478 > Bloom filter allows skipping sstable 5611 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625000 | 172.31.32.211 | > 202508 > Bloom filter allows skipping sstable 5610 > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202539 > Bloom filter allows skipping sstable 5549 > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202678 > Bloom filter allows skipping sstable 5544 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | > 202720 > Bloom filter allows skipping sstable 5237 > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202752 > Bloom filter allows skipping sstable 2516 > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202782 > Bloom filter allows skipping sstable 2632 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | > 202812 > Bloom filter allows skipping sstable 3015 [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | > 202852 > Skipped 0/11 non-slice-intersecting sstables, included 0 due to tombstones > [SharedPool-Worker-1] | 2015-03-23 > 21:04:38.625001 | 172.31.32.211 | 202882 > Merging data from memtables and 1 sstables [SharedPool-Worker-1] > > | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202902 > Read 101 live and 0 tombstoned cells > [SharedPool-Worker-1] > | 2015-03-23 21:04:38.626000 | 172.31.32.211 | 203752 > Request complete > > | 2015-03-23 > 21:04:38.628253 | 172.31.32.211 | 206253 > > On Mon, Mar 23, 2015 at 11:53 AM, Eric Stevens wrote: > >> Enable tracing in cqlsh and see how many sstables are being lifted to >> satisfy the query (are you repeatedly writing to the same partition >> [row_time]) over time?). >> >> Also watch for whether you're hitting a lot of tombstones (are you >> deleting lots of values in the same partition over time?). >> >> On Mon, Mar 23, 2015 at 4:01 AM, Dave Galbraith < >> david92galbra...@gmail.com> wrote: >> >>> Duncan: I'm thinking it might be something like that. I'm also seeing >>> just a ton of garbage collection on the box, could it be pulling rows for >>> all 100k attrs for a given row_time into memory since only row_time is the >>> partition key? >>> >>> Jens: I'm not using EBS (although I used to until I read up on how >>> useless it is). I'm not sure what constitutes proper paging but my client >>> has a pretty small amount of available memory so I'm doing pages of size 5k >>> using the C++ Datastax driver. >>> >>> Thanks for the replies! >>> >>> -Dave >>> >>> On Mon, Mar 23, 2015 at 2:00 AM, Jens Rantil >>> wrote: >>> Also, two control questions: - Are you using EBS for data
Re: Really high read latency
I haven't deleted anything. Here's output from a traced cqlsh query (I tried to make the spaces line up, hope it's legible): Execute CQL3 query | 2015-03-23 21:04:37.422000 | 172.31.32.211 | 0 Parsing select * from default.metrics where row_time = 16511 and attrs = '[redacted]' limit 100; [SharedPool-Worker-2] | 2015-03-23 21:04:37.423000 | 172.31.32.211 | 93 Preparing statement [SharedPool-Worker-2] | 2015-03-23 21:04:37.423000 | 172.31.32.211 |696 Executing single-partition query on metrics [SharedPool-Worker-1] | 2015-03-23 21:04:37.425000 | 172.31.32.211 | 2807 Acquiring sstable references [SharedPool-Worker-1] | 2015-03-23 21:04:37.425000 | 172.31.32.211 | 2993 Merging memtable tombstones [SharedPool-Worker-1] | 2015-03-23 21:04:37.426000 | 172.31.32.211 | 3049 Partition index with 484338 entries found for sstable 15966 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202304 Seeking to partition indexed section in data file [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202354 Bloom filter allows skipping sstable 5613 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202445 Bloom filter allows skipping sstable 5582 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202478 Bloom filter allows skipping sstable 5611 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202508 Bloom filter allows skipping sstable 5610 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625000 | 172.31.32.211 | 202539 Bloom filter allows skipping sstable 5549 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202678 Bloom filter allows skipping sstable 5544 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202720 Bloom filter allows skipping sstable 5237 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202752 Bloom filter allows skipping sstable 2516 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202782 Bloom filter allows skipping sstable 2632 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202812 Bloom filter allows skipping sstable 3015 [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202852 Skipped 0/11 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202882 Merging data from memtables and 1 sstables [SharedPool-Worker-1] | 2015-03-23 21:04:38.625001 | 172.31.32.211 | 202902 Read 101 live and 0 tombstoned cells [SharedPool-Worker-1] | 2015-03-23 21:04:38.626000 | 172.31.32.211 | 203752 Request complete | 2015-03-23 21:04:38.628253 | 172.31.32.211 | 206253 On Mon, Mar 23, 2015 at 11:53 AM, Eric Stevens wrote: > Enable tracing in cqlsh and see how many sstables are being lifted to > satisfy the query (are you repeatedly writing to the same partition > [row_time]) over time?). > > Also watch for whether you're hitting a lot of tombstones (are you > deleting lots of values in the same partition over time?). > > On Mon, Mar 23, 2015 at 4:01 AM, Dave Galbraith < > david92galbra...@gmail.com> wrote: > >> Duncan: I'm thinking it might be something like that. I'm also seeing >> just a ton of garbage collection on the box, could it be pulling rows for >> all 100k attrs for a given row_time into memory since only row_time is the >> partition key? >> >> Jens: I'm not using EBS (although I used to until I read up on how >> useless it is). I'm not sure what constitutes proper paging but my client >> has a pretty small amount of available memory so I'm doing pages of size 5k >> using the C++ Datastax driver. >> >> Thanks for the replies! >> >> -Dave >> >> On Mon, Mar 23, 2015 at 2:00 AM, Jens Rantil wrote: >> >>> Also, two control questions: >>> >>>- Are you using EBS for data storage? It might introduce additional >>>latencies. >>>- Are you doing proper paging when querying the keyspace? >>> >>> Cheers, >>> Jens >>> >>> On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith < >>> david92galbra...@gmail.com> wrote: >>> Hi! So I've got a table like this: CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND index_interval=128 AND read_repair_chance=1 AND replicat
Re: Really high read latency
Enable tracing in cqlsh and see how many sstables are being lifted to
satisfy the query (are you repeatedly writing to the same partition
[row_time]) over time?).
Also watch for whether you're hitting a lot of tombstones (are you deleting
lots of values in the same partition over time?).
On Mon, Mar 23, 2015 at 4:01 AM, Dave Galbraith
wrote:
> Duncan: I'm thinking it might be something like that. I'm also seeing just
> a ton of garbage collection on the box, could it be pulling rows for all
> 100k attrs for a given row_time into memory since only row_time is the
> partition key?
>
> Jens: I'm not using EBS (although I used to until I read up on how useless
> it is). I'm not sure what constitutes proper paging but my client has a
> pretty small amount of available memory so I'm doing pages of size 5k using
> the C++ Datastax driver.
>
> Thanks for the replies!
>
> -Dave
>
> On Mon, Mar 23, 2015 at 2:00 AM, Jens Rantil wrote:
>
>> Also, two control questions:
>>
>>- Are you using EBS for data storage? It might introduce additional
>>latencies.
>>- Are you doing proper paging when querying the keyspace?
>>
>> Cheers,
>> Jens
>>
>> On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith <
>> david92galbra...@gmail.com> wrote:
>>
>>> Hi! So I've got a table like this:
>>>
>>> CREATE TABLE "default".metrics (row_time int,attrs varchar,offset
>>> int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT
>>> STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND
>>> comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND
>>> index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
>>> AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
>>> speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
>>> compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
>>> AND compression={'sstable_compression':'LZ4Compressor'};
>>>
>>> and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with 4
>>> GB of heap space. So it's timeseries data that I'm doing so I increment
>>> "row_time" each day, "attrs" is additional identifying information about
>>> each series, and "offset" is the number of milliseconds into the day for
>>> each data point. So for the past 5 days, I've been inserting 3k
>>> points/second distributed across 100k distinct "attrs"es. And now when I
>>> try to run queries on this data that look like
>>>
>>> "SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
>>> 'potatoes_and_jam'"
>>>
>>> it takes an absurdly long time and sometimes just times out. I did
>>> "nodetool cftsats default" and here's what I get:
>>>
>>> Keyspace: default
>>> Read Count: 59
>>> Read Latency: 397.12523728813557 ms.
>>> Write Count: 155128
>>> Write Latency: 0.3675690719921613 ms.
>>> Pending Flushes: 0
>>> Table: metrics
>>> SSTable count: 26
>>> Space used (live): 35146349027
>>> Space used (total): 35146349027
>>> Space used by snapshots (total): 0
>>> SSTable Compression Ratio: 0.10386468749216264
>>> Memtable cell count: 141800
>>> Memtable data size: 31071290
>>> Memtable switch count: 41
>>> Local read count: 59
>>> Local read latency: 397.126 ms
>>> Local write count: 155128
>>> Local write latency: 0.368 ms
>>> Pending flushes: 0
>>> Bloom filter false positives: 0
>>> Bloom filter false ratio: 0.0
>>> Bloom filter space used: 2856
>>> Compacted partition minimum bytes: 104
>>> Compacted partition maximum bytes: 36904729268
>>> Compacted partition mean bytes: 986530969
>>> Average live cells per slice (last five minutes):
>>> 501.66101694915255
>>> Maximum live cells per slice (last five minutes): 502.0
>>> Average tombstones per slice (last five minutes): 0.0
>>> Maximum tombstones per slice (last five minutes): 0.0
>>>
>>> Ouch! 400ms of read latency, orders of magnitude higher than it has any
>>> right to be. How could this have happened? Is there something fundamentally
>>> broken about my data model? Thanks!
>>>
>>>
>>
>>
>> --
>> Jens Rantil
>> Backend engineer
>> Tink AB
>>
>> Email: jens.ran...@tink.se
>> Phone: +46 708 84 18 32
>> Web: www.tink.se
>>
>> Facebook Linkedin
>>
>> Twitter
>>
>
>
Re: Really high read latency
Duncan: I'm thinking it might be something like that. I'm also seeing just
a ton of garbage collection on the box, could it be pulling rows for all
100k attrs for a given row_time into memory since only row_time is the
partition key?
Jens: I'm not using EBS (although I used to until I read up on how useless
it is). I'm not sure what constitutes proper paging but my client has a
pretty small amount of available memory so I'm doing pages of size 5k using
the C++ Datastax driver.
Thanks for the replies!
-Dave
On Mon, Mar 23, 2015 at 2:00 AM, Jens Rantil wrote:
> Also, two control questions:
>
>- Are you using EBS for data storage? It might introduce additional
>latencies.
>- Are you doing proper paging when querying the keyspace?
>
> Cheers,
> Jens
>
> On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith <
> david92galbra...@gmail.com> wrote:
>
>> Hi! So I've got a table like this:
>>
>> CREATE TABLE "default".metrics (row_time int,attrs varchar,offset
>> int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT
>> STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND
>> comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND
>> index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
>> AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
>> speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
>> compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
>> AND compression={'sstable_compression':'LZ4Compressor'};
>>
>> and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with 4
>> GB of heap space. So it's timeseries data that I'm doing so I increment
>> "row_time" each day, "attrs" is additional identifying information about
>> each series, and "offset" is the number of milliseconds into the day for
>> each data point. So for the past 5 days, I've been inserting 3k
>> points/second distributed across 100k distinct "attrs"es. And now when I
>> try to run queries on this data that look like
>>
>> "SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
>> 'potatoes_and_jam'"
>>
>> it takes an absurdly long time and sometimes just times out. I did
>> "nodetool cftsats default" and here's what I get:
>>
>> Keyspace: default
>> Read Count: 59
>> Read Latency: 397.12523728813557 ms.
>> Write Count: 155128
>> Write Latency: 0.3675690719921613 ms.
>> Pending Flushes: 0
>> Table: metrics
>> SSTable count: 26
>> Space used (live): 35146349027
>> Space used (total): 35146349027
>> Space used by snapshots (total): 0
>> SSTable Compression Ratio: 0.10386468749216264
>> Memtable cell count: 141800
>> Memtable data size: 31071290
>> Memtable switch count: 41
>> Local read count: 59
>> Local read latency: 397.126 ms
>> Local write count: 155128
>> Local write latency: 0.368 ms
>> Pending flushes: 0
>> Bloom filter false positives: 0
>> Bloom filter false ratio: 0.0
>> Bloom filter space used: 2856
>> Compacted partition minimum bytes: 104
>> Compacted partition maximum bytes: 36904729268
>> Compacted partition mean bytes: 986530969
>> Average live cells per slice (last five minutes):
>> 501.66101694915255
>> Maximum live cells per slice (last five minutes): 502.0
>> Average tombstones per slice (last five minutes): 0.0
>> Maximum tombstones per slice (last five minutes): 0.0
>>
>> Ouch! 400ms of read latency, orders of magnitude higher than it has any
>> right to be. How could this have happened? Is there something fundamentally
>> broken about my data model? Thanks!
>>
>>
>
>
> --
> Jens Rantil
> Backend engineer
> Tink AB
>
> Email: jens.ran...@tink.se
> Phone: +46 708 84 18 32
> Web: www.tink.se
>
> Facebook Linkedin
>
> Twitter
>
Re: Really high read latency
Also, two control questions:
- Are you using EBS for data storage? It might introduce additional
latencies.
- Are you doing proper paging when querying the keyspace?
Cheers,
Jens
On Mon, Mar 23, 2015 at 5:56 AM, Dave Galbraith
wrote:
> Hi! So I've got a table like this:
>
> CREATE TABLE "default".metrics (row_time int,attrs varchar,offset
> int,value double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT
> STORAGE AND bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND
> comment='' AND dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND
> index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
> AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
> speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
> compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
> AND compression={'sstable_compression':'LZ4Compressor'};
>
> and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with 4 GB
> of heap space. So it's timeseries data that I'm doing so I increment
> "row_time" each day, "attrs" is additional identifying information about
> each series, and "offset" is the number of milliseconds into the day for
> each data point. So for the past 5 days, I've been inserting 3k
> points/second distributed across 100k distinct "attrs"es. And now when I
> try to run queries on this data that look like
>
> "SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
> 'potatoes_and_jam'"
>
> it takes an absurdly long time and sometimes just times out. I did
> "nodetool cftsats default" and here's what I get:
>
> Keyspace: default
> Read Count: 59
> Read Latency: 397.12523728813557 ms.
> Write Count: 155128
> Write Latency: 0.3675690719921613 ms.
> Pending Flushes: 0
> Table: metrics
> SSTable count: 26
> Space used (live): 35146349027
> Space used (total): 35146349027
> Space used by snapshots (total): 0
> SSTable Compression Ratio: 0.10386468749216264
> Memtable cell count: 141800
> Memtable data size: 31071290
> Memtable switch count: 41
> Local read count: 59
> Local read latency: 397.126 ms
> Local write count: 155128
> Local write latency: 0.368 ms
> Pending flushes: 0
> Bloom filter false positives: 0
> Bloom filter false ratio: 0.0
> Bloom filter space used: 2856
> Compacted partition minimum bytes: 104
> Compacted partition maximum bytes: 36904729268
> Compacted partition mean bytes: 986530969
> Average live cells per slice (last five minutes):
> 501.66101694915255
> Maximum live cells per slice (last five minutes): 502.0
> Average tombstones per slice (last five minutes): 0.0
> Maximum tombstones per slice (last five minutes): 0.0
>
> Ouch! 400ms of read latency, orders of magnitude higher than it has any
> right to be. How could this have happened? Is there something fundamentally
> broken about my data model? Thanks!
>
>
--
Jens Rantil
Backend engineer
Tink AB
Email: jens.ran...@tink.se
Phone: +46 708 84 18 32
Web: www.tink.se
Facebook Linkedin
Twitter
Re: Really high read latency
Hi Dave,
On 23/03/15 05:56, Dave Galbraith wrote:
Hi! So I've got a table like this:
CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value
double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND
dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND index_interval=128
AND read_repair_chance=1 AND replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
AND compression={'sstable_compression':'LZ4Compressor'};
does it work better with
PRIMARY KEY((row_time, attrs), offset)
?
Ciao, Duncan.
and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with 4 GB of
heap space. So it's timeseries data that I'm doing so I increment "row_time"
each day, "attrs" is additional identifying information about each series, and
"offset" is the number of milliseconds into the day for each data point. So for
the past 5 days, I've been inserting 3k points/second distributed across 100k
distinct "attrs"es. And now when I try to run queries on this data that look
like
"SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
'potatoes_and_jam'"
it takes an absurdly long time and sometimes just times out. I did "nodetool
cftsats default" and here's what I get:
Keyspace: default
Read Count: 59
Read Latency: 397.12523728813557 ms.
Write Count: 155128
Write Latency: 0.3675690719921613 ms.
Pending Flushes: 0
Table: metrics
SSTable count: 26
Space used (live): 35146349027
Space used (total): 35146349027
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.10386468749216264
Memtable cell count: 141800
Memtable data size: 31071290
Memtable switch count: 41
Local read count: 59
Local read latency: 397.126 ms
Local write count: 155128
Local write latency: 0.368 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 2856
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 36904729268
Compacted partition mean bytes: 986530969
Average live cells per slice (last five minutes): 501.66101694915255
Maximum live cells per slice (last five minutes): 502.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Ouch! 400ms of read latency, orders of magnitude higher than it has any right to
be. How could this have happened? Is there something fundamentally broken about
my data model? Thanks!
Really high read latency
Hi! So I've got a table like this:
CREATE TABLE "default".metrics (row_time int,attrs varchar,offset int,value
double, PRIMARY KEY(row_time, attrs, offset)) WITH COMPACT STORAGE AND
bloom_filter_fp_chance=0.01 AND caching='KEYS_ONLY' AND comment='' AND
dclocal_read_repair_chance=0 AND gc_grace_seconds=864000 AND
index_interval=128 AND read_repair_chance=1 AND replicate_on_write='true'
AND populate_io_cache_on_flush='false' AND default_time_to_live=0 AND
speculative_retry='NONE' AND memtable_flush_period_in_ms=0 AND
compaction={'class':'DateTieredCompactionStrategy','timestamp_resolution':'MILLISECONDS'}
AND compression={'sstable_compression':'LZ4Compressor'};
and I'm running Cassandra on an EC2 m3.2xlarge out in the cloud, with 4 GB
of heap space. So it's timeseries data that I'm doing so I increment
"row_time" each day, "attrs" is additional identifying information about
each series, and "offset" is the number of milliseconds into the day for
each data point. So for the past 5 days, I've been inserting 3k
points/second distributed across 100k distinct "attrs"es. And now when I
try to run queries on this data that look like
"SELECT * FROM "default".metrics WHERE row_time = 5 AND attrs =
'potatoes_and_jam'"
it takes an absurdly long time and sometimes just times out. I did
"nodetool cftsats default" and here's what I get:
Keyspace: default
Read Count: 59
Read Latency: 397.12523728813557 ms.
Write Count: 155128
Write Latency: 0.3675690719921613 ms.
Pending Flushes: 0
Table: metrics
SSTable count: 26
Space used (live): 35146349027
Space used (total): 35146349027
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.10386468749216264
Memtable cell count: 141800
Memtable data size: 31071290
Memtable switch count: 41
Local read count: 59
Local read latency: 397.126 ms
Local write count: 155128
Local write latency: 0.368 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 2856
Compacted partition minimum bytes: 104
Compacted partition maximum bytes: 36904729268
Compacted partition mean bytes: 986530969
Average live cells per slice (last five minutes): 501.66101694915255
Maximum live cells per slice (last five minutes): 502.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
Ouch! 400ms of read latency, orders of magnitude higher than it has any
right to be. How could this have happened? Is there something fundamentally
broken about my data model? Thanks!
Re: High read latency after data volume increased
There's likely 2 things occurring 1) the cfhistograms error is due to https://issues.apache.org/jira/browse/CASSANDRA-8028 Which is resolved in 2.1.3. Looks like voting is under way for 2.1.3. As rcoli mentioned, you are running the latest open source of C* which should be treated as beta until a few dot releases are published. 2) compaction running all the time doesn't mean that compaction is "caught up". It's possible that the nodes are behind in compaction which will cause slow reads. C* read performance is typically associated with disk system performance, both to service reads from disk as well as to enable fast background processing, like compaction. You mentioned raided hdds. What type of raid is configured? How fast are your disks responding? You may want to check iostat to see how large your queues and awaits are. If the await is high, then you could be experiencing disk perf issues impacting reads. Hope this helps On Jan 9, 2015, at 9:29 AM, Roni Balthazar wrote: Hi there, The compaction remains running with our workload. We are using SATA HDDs RAIDs. When trying to run cfhistograms on our user_data table, we are getting this message: nodetool: Unable to compute when histogram overflowed Please see what happens when running some queries on this cf: http://pastebin.com/jbAgDzVK Thanks, Roni Balthazar On Fri, Jan 9, 2015 at 12:03 PM, datastax wrote: Hello You may not be experiencing versioning issues. Do you know if compaction is keeping up with your workload? The behavior described in the subject is typically associated with compaction falling behind or having a suboptimal compaction strategy configured. What does the output of nodetool cfhistograms look like for a table that is experiencing this issue? Also, what type of disks are you using on the nodes? Sent from my iPad On Jan 9, 2015, at 8:55 AM, Brian Tarbox wrote: C* seems to have more than its share of "version x doesn't work, use version y " type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote: On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better off running 2.1.1 ... =Rob -- http://about.me/BrianTarbox
Re: High read latency after data volume increased
Hi there, The compaction remains running with our workload. We are using SATA HDDs RAIDs. When trying to run cfhistograms on our user_data table, we are getting this message: nodetool: Unable to compute when histogram overflowed Please see what happens when running some queries on this cf: http://pastebin.com/jbAgDzVK Thanks, Roni Balthazar On Fri, Jan 9, 2015 at 12:03 PM, datastax wrote: > Hello > > You may not be experiencing versioning issues. Do you know if compaction > is keeping up with your workload? The behavior described in the subject is > typically associated with compaction falling behind or having a suboptimal > compaction strategy configured. What does the output of nodetool > cfhistograms look like for a table that is experiencing > this issue? Also, what type of disks are you using on the nodes? > > Sent from my iPad > > On Jan 9, 2015, at 8:55 AM, Brian Tarbox wrote: > > C* seems to have more than its share of "version x doesn't work, use version > y " type issues > > On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote: >> >> On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar >> wrote: >>> >>> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. >> >> >> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ >> >> 2.1.2 in particular is known to have significant issues. You'd be better >> off running 2.1.1 ... >> >> =Rob >> > > > > > -- > http://about.me/BrianTarbox
Re: High read latency after data volume increased
Hello You may not be experiencing versioning issues. Do you know if compaction is keeping up with your workload? The behavior described in the subject is typically associated with compaction falling behind or having a suboptimal compaction strategy configured. What does the output of nodetool cfhistograms look like for a table that is experiencing this issue? Also, what type of disks are you using on the nodes? Sent from my iPad > On Jan 9, 2015, at 8:55 AM, Brian Tarbox wrote: > > C* seems to have more than its share of "version x doesn't work, use version > y " type issues > >> On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote: >>> On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar >>> wrote: >>> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. >> >> https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ >> >> 2.1.2 in particular is known to have significant issues. You'd be better off >> running 2.1.1 ... >> >> =Rob > > > > -- > http://about.me/BrianTarbox
RE: High read latency after data volume increased
I was about to say I thought 2.1 was a development version, but when I went to prove that to myself: http://cassandra.apache.org/download/ “ The latest stable release of Apache Cassandra is 2.1.2 (released on 2014-11-10). If you're just starting out, download this one.” But then, after visiting planet Cassandra (this is what I was thinking of, I had just read it) http://planetcassandra.org/cassandra/ “ v2.0.11(Stable & Recommended) v2.1.2(Latest Development Release) v1.2.19 (Archive) “ Seems to be a mixed message of what is stable between the two sites… Jason From: Brian Tarbox [mailto:briantar...@gmail.com] Sent: Friday, January 9, 2015 8:56 AM To: user@cassandra.apache.org Subject: Re: High read latency after data volume increased C* seems to have more than its share of "version x doesn't work, use version y " type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli mailto:rc...@eventbrite.com>> wrote: On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar mailto:ronibaltha...@gmail.com>> wrote: We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better off running 2.1.1 ... =Rob -- http://about.me/BrianTarbox
Re: High read latency after data volume increased
C* seems to have more than its share of "version x doesn't work, use version y " type issues On Thu, Jan 8, 2015 at 2:23 PM, Robert Coli wrote: > On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar > wrote: > >> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. >> > > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > > 2.1.2 in particular is known to have significant issues. You'd be better > off running 2.1.1 ... > > =Rob > > -- http://about.me/BrianTarbox
Re: High read latency after data volume increased
On Thu, Jan 8, 2015 at 6:38 PM, Roni Balthazar wrote: > We downgraded to 2.1.1, but got the very same result. The read latency is > still high, but we figured out that it happens only using a specific > keyspace. > Note that downgrading is officially unsupported, but is probably safe between those two versions. Enable tracing and paste results for a high latency query. Also, how much RAM is used for heap? =Rob
Re: High read latency after data volume increased
Hi Robert, We downgraded to 2.1.1, but got the very same result. The read latency is still high, but we figured out that it happens only using a specific keyspace. Please see the graphs below... Trying another keyspace with 600+ reads/sec, we are getting the acceptable ~30ms read latency. Let me know if I need to provide more information. Thanks, Roni Balthazar On Thu, Jan 8, 2015 at 5:23 PM, Robert Coli wrote: > On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar > wrote: > >> We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. >> > > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ > > 2.1.2 in particular is known to have significant issues. You'd be better > off running 2.1.1 ... > > =Rob > >
Re: High read latency after data volume increased
On Thu, Jan 8, 2015 at 11:14 AM, Roni Balthazar wrote: > We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. > https://engineering.eventbrite.com/what-version-of-cassandra-should-i-run/ 2.1.2 in particular is known to have significant issues. You'd be better off running 2.1.1 ... =Rob
High read latency after data volume increased
Hi there, We are using C* 2.1.2 with 2 DCs. 30 nodes DC1 and 10 nodes DC2. While our data volume is increasing (34 TB now), we are running into some problems: 1) Read latency is around 1000 ms when running 600 reads/sec (DC1 CL.LOCAL_ONE). At the same time the load average is about 20-30 on all DC1 nodes(8 cores CPU - 32 GB RAM). C* starts timing out connections. Still in this scenario OpsCenter has some issues as well. Opscenter resets all Graphs layout and backs to the default layout on every refresh. It doesn't back to normal after the load decrease. I only managed to put OpsCenter to it's normal behavior after reinstalling it. Just for reference, we are using SATA HDDs on all nodes and running hdparm to check disk performance under this load, some nodes are reporting very low read rates (under 10 MB/sec), while others above 100 MB/sec. Under low load average this rate is above 250 MB/sec. 2) Repair takes at least 4-5 days to complete. Last repair was 20 days ago. Running repair under high loads is bringing some nodes down with the exception: "JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to: java.lang.OutOfMemoryError: Java heap space" Any hints? Regards, Roni Balthazar
Re: Noticing really high read latency
Disregard... heh. Was reading the latency as SECONDS. Sorry, it's been one
of those weeks.
On Wed, Mar 5, 2014 at 1:44 AM, Eric Plowe wrote:
> Background info:
>
> 6 node cluster.
> 24 gigs of ram per machine
> 8 gigs of ram dedicated to c*
> 4 4 core cpu's
> 2 250 gig SSD's raid 0
> Running c* 1.2.6
>
> The CF is configured as followed
>
> CREATE TABLE behaviors (
> uid text,
> buid int,
> name text,
> expires text,
> value text,
> PRIMARY KEY (uid, buid, name)
> ) WITH
> bloom_filter_fp_chance=0.01 AND
> caching='KEYS_ONLY' AND
> comment='' AND
> dclocal_read_repair_chance=0.00 AND
> gc_grace_seconds=864000 AND
> read_repair_chance=0.10 AND
> replicate_on_write='true' AND
> populate_io_cache_on_flush='false' AND
> compaction={'sstable_size_in_mb': '160', 'class':
> 'LeveledCompactionStrategy'} AND
> compression={'sstable_compression': 'SnappyCompressor'};
>
> I am noticing that the read latency is very high considering when I look
> at the output of nodetool cfstats.
>
> This is the example output of one of the nodes:
>
> Column Family: behaviors
> SSTable count: 2
> SSTables in each level: [1, 1, 0, 0, 0, 0, 0, 0, 0]
> Space used (live): 171496198
> Space used (total): 171496591
> Number of Keys (estimate): 1153664
> Memtable Columns Count: 14445
> Memtable Data Size: 1048576
> Memtable Switch Count: 1
> Read Count: 1894
> Read Latency: 0.497 ms.
> Write Count: 7169
> Write Latency: 0.041 ms.
> Pending Tasks: 0
> Bloom Filter False Positives: 4
> Bloom Filter False Ratio: 0.00862
> Bloom Filter Space Used: 3533152
> Compacted row minimum size: 125
> Compacted row maximum size: 9887
> Compacted row mean size: 365
>
> The write latency is awesome, but the read latency, not so much. The
> output of iostat doesn't show anything out of the ordinary. The cpu
> utilization is between 1% to 5%.
>
> All read queries are issued with a CL of ONE. We always include "WHERE uid
> = ''" for the queries.
>
> If there is any more info I can provide, please let me know. At this point
> in time, I am a bit stumped.
>
> Regards,
>
> Eric Plowe
>
>
>
>
Noticing really high read latency
Background info:
6 node cluster.
24 gigs of ram per machine
8 gigs of ram dedicated to c*
4 4 core cpu's
2 250 gig SSD's raid 0
Running c* 1.2.6
The CF is configured as followed
CREATE TABLE behaviors (
uid text,
buid int,
name text,
expires text,
value text,
PRIMARY KEY (uid, buid, name)
) WITH
bloom_filter_fp_chance=0.01 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.00 AND
gc_grace_seconds=864000 AND
read_repair_chance=0.10 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
compaction={'sstable_size_in_mb': '160', 'class':
'LeveledCompactionStrategy'} AND
compression={'sstable_compression': 'SnappyCompressor'};
I am noticing that the read latency is very high considering when I look at
the output of nodetool cfstats.
This is the example output of one of the nodes:
Column Family: behaviors
SSTable count: 2
SSTables in each level: [1, 1, 0, 0, 0, 0, 0, 0, 0]
Space used (live): 171496198
Space used (total): 171496591
Number of Keys (estimate): 1153664
Memtable Columns Count: 14445
Memtable Data Size: 1048576
Memtable Switch Count: 1
Read Count: 1894
Read Latency: 0.497 ms.
Write Count: 7169
Write Latency: 0.041 ms.
Pending Tasks: 0
Bloom Filter False Positives: 4
Bloom Filter False Ratio: 0.00862
Bloom Filter Space Used: 3533152
Compacted row minimum size: 125
Compacted row maximum size: 9887
Compacted row mean size: 365
The write latency is awesome, but the read latency, not so much. The output
of iostat doesn't show anything out of the ordinary. The cpu utilization is
between 1% to 5%.
All read queries are issued with a CL of ONE. We always include "WHERE uid
= ''" for the queries.
If there is any more info I can provide, please let me know. At this point
in time, I am a bit stumped.
Regards,
Eric Plowe
Re: High-read latency for non-existing rows with LCS and 1.2.5
> We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is > 100M. On each node, > we have anywhere from 700+ to 800+ sstables (for all levels). The > bloom_filter_fp_chance is set at 0.000744. The current default bloom_filter_fp_chance is 0.1 for levelled compaction. Reducing this (and running nodetool upgradesstables) will reduce the bloom filter size significantly. > the latency is running into hundreds of milliseconds and sometimes seconds. Check the number of SSTables per read using nodetool cfhistograms. With levelled compaction you should not see above 3 http://www.datastax.com/dev/blog/when-to-use-leveled-compaction Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 28/06/2013, at 7:44 AM, sankalp kohli wrote: > Try doing request tracing. > http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2 > > > On Thu, Jun 27, 2013 at 2:40 PM, Bao Le wrote: > Hi, > > We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is > 100M. On each node, > we have anywhere from 700+ to 800+ sstables (for all levels). The > bloom_filter_fp_chance is set at 0.000744. > > For read requests that ask for existing row records, the latency is great, > mostly under 20 milliseconds with key cache and row cache set. For read > requests that ask for non-existing row records (not because of deletes, but > rather, have never been in the system to start with), the latency is running > into hundreds of milliseconds and sometimes seconds. > > Just wonder if anyone has come across this before and has some pointers on > how to reduce the latency in this case. > > Thanks > Bao > > > >
Re: High-read latency for non-existing rows with LCS and 1.2.5
Try doing request tracing. http://www.datastax.com/dev/blog/tracing-in-cassandra-1-2 On Thu, Jun 27, 2013 at 2:40 PM, Bao Le wrote: > Hi, > > We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size > is 100M. On each node, > we have anywhere from 700+ to 800+ sstables (for all levels). The > bloom_filter_fp_chance is set at 0.000744. > > For read requests that ask for existing row records, the latency is > great, mostly under 20 milliseconds with key cache and row cache set. For > read requests that ask for non-existing row records (not because of > deletes, but rather, have never been in the system to start with), the > latency is running into hundreds of milliseconds and sometimes seconds. > > Just wonder if anyone has come across this before and has some pointers > on how to reduce the latency in this case. > > Thanks > Bao > > > >
High-read latency for non-existing rows with LCS and 1.2.5
Hi, We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node, we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744. For read requests that ask for existing row records, the latency is great, mostly under 20 milliseconds with key cache and row cache set. For read requests that ask for non-existing row records (not because of deletes, but rather, have never been in the system to start with), the latency is running into hundreds of milliseconds and sometimes seconds. Just wonder if anyone has come across this before and has some pointers on how to reduce the latency in this case. Thanks Bao
Re: High read latency cluster
> FlushWriter 0 0 8252 0
>299
If you are not suffering from gc pressure/pauses (possibly not, because you
don't seem to have a lot of read failures in tpstats or outlier latency on the
histograms), then the flush writer errors are suggestive of memtable pressure,
which may be followed by compactions that grind the disk.
> maybe because of bloomfilters
If you think bloom filters or indexes are occupying heap on startup, then you
could alleviate things for a while with memtable/cache tuning, resampling the
index interval, or increasing the heap to 10G (yes, not generally recommended).
Not enough working ram can also impact the key cache which then puts more
pressure on disk - check info to see if your caches are being resized. If then
disk I/O is simply be falling behind in conjunction with a server that doesn't
have much memory headroom, then you'll want to expand the cluster at some point
to spread out the load.
Bill
On 8 Feb 2013, at 13:03, Alain RODRIGUEZ wrote:
> Hi,
>
> I have some big latencies (OpsCenter homepage shows an average about 30-60
> ms), inducing instability in my front servers, stacking queries, waiting for
> C* to answer, in the following 1.1.6 C* cluster:
>
> 10.208.45.173 eu-west 1b Up Normal 297.02 GB
> 100.00% 0
> 10.208.40.6 eu-west 1b Up Normal 292.91 GB
> 100.00% 56713727820156407428984779325531226112
> 10.208.47.135 eu-west 1b Up Normal 307.96 GB
> 100.00% 113427455640312814857969558651062452224
>
> I run on 3 AWS m1.xLarge with mostly the Datastax AMI default node
> configuration (But with the following options MAX_HEAP_SIZE="8G"
> HEAP_NEWSIZE="400M", I was under regular memory pressure with the default 4GB
> heap, maybe because of bloomfilters). RF = 3, CL = QUORUM r/w.
>
> I have a high load from 4 to 15 with an average of 8 (mainly because of
> iowait which can reach up to 40-60%).
>
> extract from "iostat -mx 5 10" :
> avg-cpu: %user %nice %system %iowait %steal %idle
> 16.660.004.82 35.47 0.21 42.85
>
>
> I use compression and Size Tiered Compaction Strategy for any of my CF.
>
> A typical CF :
>
> create column family active_product
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> And there is a typical counter CF:
>
> create column family algo_product_view
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'CounterColumnType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> I attach my cfhistograms, proxyhistograms, cfstats and tpstats hopping a clue
> is somewhere in there, even if I was unable to learn something there by
> myself.
>
> cfstats: http://pastebin.com/z3sAshjP
> tpstats: http://pastebin.com/LETPqfLV
> proxyhistograms: http://pastebin.com/FqwMFrxG
> cfhistograms (from the 2 most read / highest latencies):
> http://pastebin.com/BCsdc50z & http://pastebin.com/CGZZpydL
>
> These latencies are quite annoying, hope you'll help me figuring out what I
> am doing wrong or how I can tune Cassandra better.
>
> Alain
Re: High read latency cluster
300 GB is a lot of data for cloud machines (especially with their
weaker performance in general). If you are unhappy with performance
why not scale the cluster out to more servers, with that much data you
are usually contending with the physics of spinning disks. Three nodes
+ replication factor 3 also means all your data is replicated to every
node and your not getting any scale out benefits.
On Fri, Feb 8, 2013 at 8:03 AM, Alain RODRIGUEZ wrote:
> Hi,
>
> I have some big latencies (OpsCenter homepage shows an average about 30-60
> ms), inducing instability in my front servers, stacking queries, waiting for
> C* to answer, in the following 1.1.6 C* cluster:
>
> 10.208.45.173 eu-west 1b Up Normal 297.02 GB
> 100.00% 0
> 10.208.40.6 eu-west 1b Up Normal 292.91 GB
> 100.00% 56713727820156407428984779325531226112
> 10.208.47.135 eu-west 1b Up Normal 307.96 GB
> 100.00% 113427455640312814857969558651062452224
>
> I run on 3 AWS m1.xLarge with mostly the Datastax AMI default node
> configuration (But with the following options MAX_HEAP_SIZE="8G"
> HEAP_NEWSIZE="400M", I was under regular memory pressure with the default
> 4GB heap, maybe because of bloomfilters). RF = 3, CL = QUORUM r/w.
>
> I have a high load from 4 to 15 with an average of 8 (mainly because of
> iowait which can reach up to 40-60%).
>
> extract from "iostat -mx 5 10" :
> avg-cpu: %user %nice %system %iowait %steal %idle
> 16.660.004.82 35.47 0.21 42.85
>
>
> I use compression and Size Tiered Compaction Strategy for any of my CF.
>
> A typical CF :
>
> create column family active_product
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'UTF8Type'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> And there is a typical counter CF:
>
> create column family algo_product_view
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'CounterColumnType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 12
> and replicate_on_write = true
> and compaction_strategy =
> 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and bloom_filter_fp_chance = 0.01
> and compression_options = {'sstable_compression' :
> 'org.apache.cassandra.io.compress.SnappyCompressor'};
>
> I attach my cfhistograms, proxyhistograms, cfstats and tpstats hopping a
> clue is somewhere in there, even if I was unable to learn something there by
> myself.
>
> cfstats: http://pastebin.com/z3sAshjP
> tpstats: http://pastebin.com/LETPqfLV
> proxyhistograms: http://pastebin.com/FqwMFrxG
> cfhistograms (from the 2 most read / highest latencies):
> http://pastebin.com/BCsdc50z & http://pastebin.com/CGZZpydL
>
> These latencies are quite annoying, hope you'll help me figuring out what I
> am doing wrong or how I can tune Cassandra better.
>
> Alain
Re: High read latency
It's really a pain to modify the data model, the problem is how to handle "one-to-many" relation in cassandra? The limitation of the row size will lead to impossible to store them with columns. On Fri, Jun 4, 2010 at 4:13 PM, Sylvain Lebresne wrote: > As written in the third point of > http://wiki.apache.org/cassandra/CassandraLimitations, > right now, super columns are not indexed and deserialized fully when you > access > them. Another way to put it is, you'll want to user super columns with > only a relatively > small number of columns in them. > Because in you example, even if you read 1 column in 1 row, the full > supercolumn > where this column is is read from disk. Now, 50ms to read 5 records isn't > necessarily too bad. > > ColumnIndexSizeInKB will not help here (as superColumns are not indexed > anyway) > and your only way out is to change you model so that you don't have > super columns > with so many columns. > > On Fri, Jun 4, 2010 at 8:26 AM, Ma Xiao wrote: >> we have a SupperCF which may have up to 1000 supper columns and 5 >> clumns for each supper column, the read latency may go up to 50ms >> (even higher), I think it's a long time to response, how to tune the >> storage config to optimize the performace? I read the wiki, >> may help to do this, supose that by asign a big >> value to this( 2 ex. ), no row and reach this limit so it never >> generate a index for a row. In our production scenario, we only access >> 1 row at a time and with up to 1000 columns slice retuned. Any >> suggestion? >> >
Re: High read latency
As written in the third point of http://wiki.apache.org/cassandra/CassandraLimitations, right now, super columns are not indexed and deserialized fully when you access them. Another way to put it is, you'll want to user super columns with only a relatively small number of columns in them. Because in you example, even if you read 1 column in 1 row, the full supercolumn where this column is is read from disk. Now, 50ms to read 5 records isn't necessarily too bad. ColumnIndexSizeInKB will not help here (as superColumns are not indexed anyway) and your only way out is to change you model so that you don't have super columns with so many columns. On Fri, Jun 4, 2010 at 8:26 AM, Ma Xiao wrote: > we have a SupperCF which may have up to 1000 supper columns and 5 > clumns for each supper column, the read latency may go up to 50ms > (even higher), I think it's a long time to response, how to tune the > storage config to optimize the performace? I read the wiki, > may help to do this, supose that by asign a big > value to this( 2 ex. ), no row and reach this limit so it never > generate a index for a row. In our production scenario, we only access > 1 row at a time and with up to 1000 columns slice retuned. Any > suggestion? >
High read latency
we have a SupperCF which may have up to 1000 supper columns and 5 clumns for each supper column, the read latency may go up to 50ms (even higher), I think it's a long time to response, how to tune the storage config to optimize the performace? I read the wiki, may help to do this, supose that by asign a big value to this( 2 ex. ), no row and reach this limit so it never generate a index for a row. In our production scenario, we only access 1 row at a time and with up to 1000 columns slice retuned. Any suggestion?
