Please find histogram attached. On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla <r...@foundev.pro> wrote:
> if everything is in ram there could be a number of issues unrelated to
> Cassandra and there could be hardware limitations or contention problems.
> Otherwise cell count can really deeply impact reads, all ram or not, and
> some of this is because of the nature of GC and some of it is the age of
> the sstable format (which is due to be revamped in 3.0). Also partition
> size can matter just because of physics, if one of those is a 1gb
> partition, the network interface can only move that back across the wire so
> quickly not to mention the GC issues you’d run into.
>
> Anyway this is why I asked for the histograms, I wanted to get cell count
> and partition size. I’ve seen otherwise very stout hardware get slow on
> reads of large results because either a bottleneck was hit somewhere, or
> the CPU got slammed with GC, or other processes running on the machine were
> contending with Cassandra.
>
>
> On Sep 25, 2015, at 12:45 PM, Jaydeep Chovatia <chovatia.jayd...@gmail.com>
> wrote:
>
> I understand that but everything is in RAM (my data dir is tmpfs) and my
> row is not that wide approx. less than 5MB in size. So my question is if
> everything is in RAM then why does it take 43ms latency?
>
> On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla <r...@foundev.pro> wrote:
>
>> if you run:
>>
>> nodetool cfhistograms <keyspace> <table name>
>>
>> On the given table and that will tell you how wide your rows are getting.
>> At some point you can get wide enough rows that just the physics of
>> retrieving them all take some time.
>>
>>
>> On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>> Jaydeep; since your primary key involves a clustering column, you may be
>> having pretty wide rows. The read would be sequential. The latency could be
>> acceptable, if the read were to involve really wide rows.
>>
>> If your primary key was like ((a,b)) without the clustering column, it's
>> like reading a key value pair, and 40ms latency may have been a concern.
>>
>> Bottom Line : The latency depends on how wide the row is.
>>
>> On Tue, Sep 22, 2015 at 1:27 PM, sai krishnam raju potturi <
>> pskraj...@gmail.com> wrote:
>>
>>> thanks for the information. Posting the query too would be of help.
>>>
>>> On Tue, Sep 22, 2015 at 11:56 AM, Jaydeep Chovatia <
>>> chovatia.jayd...@gmail.com> wrote:
>>>
>>>> Please find required details here:
>>>>
>>>> - Number of req/s
>>>>
>>>> 2k reads/s
>>>>
>>>> - Schema details
>>>>
>>>> create table test {
>>>>
>>>> a timeuuid,
>>>>
>>>> b bigint,
>>>>
>>>> c int,
>>>>
>>>> d int static,
>>>>
>>>> e int static,
>>>>
>>>> f int static,
>>>>
>>>> g int static,
>>>>
>>>> h int,
>>>>
>>>> i text,
>>>>
>>>> j text,
>>>>
>>>> k text,
>>>>
>>>> l text,
>>>>
>>>> m set<text>
>>>>
>>>> n bigint
>>>>
>>>> o bigint
>>>>
>>>> p bigint
>>>>
>>>> q bigint
>>>>
>>>> r int
>>>>
>>>> s text
>>>>
>>>> t bigint
>>>>
>>>> u text
>>>>
>>>> v text
>>>>
>>>> w text
>>>>
>>>> x bigint
>>>>
>>>> y bigint
>>>>
>>>> z bigint,
>>>>
>>>> primary key ((a, b), c)
>>>>
>>>> };
>>>>
>>>> - JVM settings about the heap
>>>>
>>>> Default settings
>>>>
>>>> - Execution time of the GC
>>>>
>>>> Avg. 400ms. I do not see long pauses of GC anywhere in the log file.
>>>>
>>>> On Tue, Sep 22, 2015 at 5:34 AM, Leleu Eric <eric.le...@worldline.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Before speaking about tuning, can you provide some additional
>>>>> information ?
>>>>>
>>>>>
>>>>>
>>>>> - Number of req/s
>>>>>
>>>>> - Schema details
>>>>>
>>>>> - JVM settings about the heap
>>>>>
>>>>> - Execution time of the GC
>>>>>
>>>>>
>>>>>
>>>>> 43ms for a read latency may be acceptable according to the number of
>>>>> request per second.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Eric
>>>>>
>>>>>
>>>>>
>>>>> *De :* Jaydeep Chovatia [mailto:chovatia.jayd...@gmail.com]
>>>>> *Envoyé :* mardi 22 septembre 2015 00:07
>>>>> *À :* user@cassandra.apache.org
>>>>> *Objet :* High read latency
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>>
>>>>>
>>>>> My application issues more read requests than write, I do see that
>>>>> under load cfstats for one of the table is quite high around 43ms
>>>>>
>>>>>
>>>>>
>>>>> Local read count: 114479357
>>>>>
>>>>> Local read latency: 43.442 ms
>>>>>
>>>>> Local write count: 22288868
>>>>>
>>>>> Local write latency: 0.609 ms
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Here is my node configuration:
>>>>>
>>>>> RF=3, Read/Write with QUORUM, 64GB RAM, 48 CPU core. I have only 5 GB
>>>>> of data on each node (and for experiment purpose I stored data in tmpfs)
>>>>>
>>>>>
>>>>>
>>>>> I've tried increasing concurrent_read count upto 512 but no help in
>>>>> read latency. CPU/Memory/IO looks fine on system.
>>>>>
>>>>>
>>>>>
>>>>> Any idea what should I tune?
>>>>>
>>>>>
>>>>>
>>>>> Jaydeep
>>>>>
>>>>> ------------------------------
>>>>>
>>>>> Ce message et les pièces jointes sont confidentiels et réservés à
>>>>> l'usage exclusif de ses destinataires. Il peut également être protégé par
>>>>> le secret professionnel. Si vous recevez ce message par erreur, merci d'en
>>>>> avertir immédiatement l'expéditeur et de le détruire. L'intégrité du
>>>>> message ne pouvant être assurée sur Internet, la responsabilité de
>>>>> Worldline ne pourra être recherchée quant au contenu de ce message. Bien
>>>>> que les meilleurs efforts soient faits pour maintenir cette transmission
>>>>> exempte de tout virus, l'expéditeur ne donne aucune garantie à cet égard
>>>>> et
>>>>> sa responsabilité ne saurait être recherchée pour tout dommage résultant
>>>>> d'un virus transmis.
>>>>>
>>>>> This e-mail and the documents attached are confidential and intended
>>>>> solely for the addressee; it may also be privileged. If you receive this
>>>>> e-mail in error, please notify the sender immediately and destroy it. As
>>>>> its integrity cannot be secured on the Internet, the Worldline liability
>>>>> cannot be triggered for the message content. Although the sender
>>>>> endeavours
>>>>> to maintain a computer virus-free network, the sender does not warrant
>>>>> that
>>>>> this transmission is virus-free and will not be liable for any damages
>>>>> resulting from any virus transmitted.
>>>>>
>>>>
>>>>
>>>
>>
>> Regards,
>>
>> Ryan Svihla
>>
>>
>
>
histogram
Description: Binary data
