Antwort: Re: Re: Increased read latency with Cassandra >= 3.11.7

2020-09-23 Thread Nicolai Lune Vest 


Hi Maxim,

unfortunately no. We decided to stay at 3.11.6 for production until we
found a solution because our system is latency sensitive too.

>From what I can see the most impact on read-latency is on the tables using
time windowed compaction strategy. Tables with e.g. leveled compaction
strategy are not affected that much.
I'm suspecting some change in the behavior of the bloom filter. Bloom
filter false positive rate hast increased too with 3.11.7, probably causing
more SSTables to be read on request unnecessarily. Reading more SSTables
probably causes a higher read-latency. However, the way leveled compaction
strategy is organizing the SSTables bloom filter changes might not have
such a big impact on the number of SSTables to be read on request.

In case you have a test environment you could try the version update and
compare bloom filter and SSTable reads on your tables. Would be interested
to see your results too.

Regards
Nico






-"Maxim Parkachov"  schrieb: -
An: user@cassandra.apache.org
Von: "Maxim Parkachov" 
Datum: 23.09.2020 08:15
Betreff: Re: Re: Increased read latency with Cassandra >= 3.11.7


Hi Nico,


we wanted to upgrade to 3.11.8 (from .6), but now I'm very concerned as our
load is mostly read-only and very latency sensitive.
Did you figure out the reason for such behaviour with the new version ?


Regards,
Maxim.







LANCOM Systems GmbH
Adenauerstr. 20 / B2
52146 Würselen
Deutschland

Tel: +49 2405 49936-0
Fax: +49 2405 49936-99

Web: https://www.lancom-systems.de



Geschäftsführer: Ralf Koenzen, Stefan Herrlich
Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976

Re: Re: Increased read latency with Cassandra >= 3.11.7

2020-09-23 Thread Maxim Parkachov 
Hi Nico,

we wanted to upgrade to 3.11.8 (from .6), but now I'm very concerned as our
load is mostly read-only and very latency sensitive.
Did you figure out the reason for such behaviour with the new version ?

Regards,
Maxim.

On Tue, Sep 15, 2020 at 5:24 AM Sagar Jambhulkar 
wrote:

> That is odd. I thought maybe lower version had a bigger custom cache or
> some other setting missed in the upgrade.
>
> On Mon, 14 Sep 2020, 23:00 Nicolai Lune Vest, 
> wrote:
>
>> While cache configuration stays the same, cache hit rate for both, the
>> chunk cache and the key cache are worse (but still ok) after the update.
>> You can see the change in the attached screenshot (update was performed at
>> September the 7th ~4pm).
>>
>> @Sagar Any specific reason you are suspecting the cache?
>>
>>
>>
>> -"Sagar Jambhulkar"  schrieb: -
>> An: user@cassandra.apache.org
>> Von: "Sagar Jambhulkar" 
>> Datum: 14.09.2020 16:25
>> Betreff: Re: Increased read latency with Cassandra >= 3.11.7
>>
>>
>> Maybe compare the cache size see if anything different in two versions?
>>
>>
>> On Mon, 14 Sep 2020, 19:21 Nicolai Lune Vest, 
>> wrote:
>>
>>
>> Dear Cassandra community,
>>
>> after upgrading my Cassandra nodes from version 3.11.6 to either 3.11.7
>> or 3.11.8 I experience a significant increase in read latency.
>> With 3.11.6 average read latency is ~0.35 ms. With 3.11.7 average read
>> latency increases to ~2.9 ms, almost 10 times worse! This behavior applies
>> to all my environments, testing and productive.
>>
>> I do also observe that the "bloom-filter false positive rate" and
>> "SSTable reads" increases significantly with version >= 3.11.7
>>
>> I'm wondering if someone experienced the same problem when updating
>> Cassandra from 3.11.6 to either 3.11.7 or 3.11.8 and can help me out.
>>
>> I already tried nodetool scrub, sstablescrub, manual compaction, and
>> repair on selected tables without impact. When downgrading back to 3.11.6
>> read latency recovers immediately.
>>
>> My update routine is very simple. I'm just stopping the service, install
>> the new version (using apt), and start the service again. One node after
>> the other. As coming from 3.11.6 updating to 3.11.7 (or 3.11.8) is just a
>> patch update and I expect this routine should do well, but maybe I'm
>> missing some important steps here.
>>
>> I can provide more information about my system or configuration if
>> required.
>>
>> Appreciate your support!
>>
>> Kind regards
>> Nico
>>
>>
>>
>>
>>
>>
>>
>> LANCOM Systems GmbH
>> Adenauerstr. 20 / B2
>> 52146 Würselen
>> Deutschland
>>
>> Tel: +49 2405 49936-0
>> Fax: +49 2405 49936-99
>>
>> Web: https://www.lancom-systems.de
>>
>>
>>
>> Geschäftsführer: Ralf Koenzen, Stefan Herrlich
>> Sitz der Gesellschaft: Aachen, Amtsgericht Aachen, HRB 16976
>>
>>
>>
>>
>>
>>
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>