Re: Cassandra p95 latencies

[Shaurya Gupta]

The queries are rightly designed as I already explained. 40 ms is way too
high as compared to what I seen with other DBs and many a times with
Cassandra 3.x versions.
CPU consumed as I mentioned is not high, it is around 20%.

On Thu, Aug 10, 2023 at 5:14 PM MyWorld  wrote:

> Hi,
> P95 should not be a problem if rightly designed. Levelled compaction
> strategy further reduces this, however it consume some resources. For read,
> caching is also helpful.
> Can you check your cpu iowait as it could be the reason for delay
>
> Regards,
> Ashish
>
> On Fri, 11 Aug, 2023, 04:58 Shaurya Gupta,  wrote:
>
>> Hi community
>>
>> What is the expected P95 latency for Cassandra Read and Write queries
>> executed with Local_Quorum over a table with 3 replicas ? The queries are
>> done using the partition + clustering key and row size in bytes is not too
>> much, maybe 1-2 KB maximum.
>> Assuming CPU is not a crunch ?
>>
>> We observe those to be 40 ms P95 Reads and same for Writes. This looks
>> very high as compared to what we expected. We are using Cassandra 4.0.
>>
>> Any documentation / numbers will be helpful.
>>
>> Thanks
>> --
>> Shaurya Gupta
>>
>>
>>

-- 
Shaurya Gupta

Re: Cassandra p95 latencies

[Abe Ratnofsky]

40ms is definitely higher than expected. Have you run your queries with TRACING enabled to see where the latency is coming from?https://docs.datastax.com/en/cql-oss/3.3/cql/cql_reference/cqlshTracing.html40ms is also a fairly specific duration: https://eklitzke.org/the-caveats-of-tcp-nodelay> On Linux this can cause up to a 40 ms delay when acking packets—AbeOn Aug 10, 2023, at 17:13, MyWorld  wrote:Hi,P95 should not be a problem if rightly designed. Levelled compaction strategy further reduces this, however it consume some resources. For read, caching is also helpful. Can you check your cpu iowait as it could be the reason for delay Regards,AshishOn Fri, 11 Aug, 2023, 04:58 Shaurya Gupta,  wrote:Hi communityWhat is the expected P95 latency for Cassandra Read and Write queries executed with Local_Quorum over a table with 3 replicas ? The queries are done using the partition + clustering key and row size in bytes is not too much, maybe 1-2 KB maximum.Assuming CPU is not a crunch ?We observe those to be 40 ms P95 Reads and same for Writes. This looks very high as compared to what we expected. We are using Cassandra 4.0.Any documentation / numbers will be helpful.Thanks-- Shaurya Gupta

Re: Upgrade from 3.11.5 to 4.1.x

[MyWorld]

You can check in your lower environment.

On Fri, 11 Aug, 2023, 06:25 Surbhi Gupta,  wrote:

> Thanks,
>
> I am looking to to upgrade to 4.1.x .
> Please advise.
>
> Thanks
> Surbhi
>
> On Thu, Aug 10, 2023 at 5:39 PM MyWorld  wrote:
>
>> Though it's recommended to upgrade to latest version of 3.11.x and then
>> to ver 4  but even upgrading directly won't be a problem. Just check the
>> release notes.
>>
>> However for production, I would recommend to go for 4.0.x latest stable
>> version.
>>
>> Regards
>> Ashish
>>
>> On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta, 
>> wrote:
>>
>>> Hi,
>>>
>>> We have to upgrade from 3.11.5 to 4.1.x .
>>> Can we do it in one go ?
>>> Or do we have to go to an intermediate version first?
>>>
>>> Thanks
>>> Surbhi
>>>
>>

Re: Upgrade from 3.11.5 to 4.1.x

[Surbhi Gupta]

Thanks,

I am looking to to upgrade to 4.1.x .
Please advise.

Thanks
Surbhi

On Thu, Aug 10, 2023 at 5:39 PM MyWorld  wrote:

> Though it's recommended to upgrade to latest version of 3.11.x and then to
> ver 4  but even upgrading directly won't be a problem. Just check the
> release notes.
>
> However for production, I would recommend to go for 4.0.x latest stable
> version.
>
> Regards
> Ashish
>
> On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta,  wrote:
>
>> Hi,
>>
>> We have to upgrade from 3.11.5 to 4.1.x .
>> Can we do it in one go ?
>> Or do we have to go to an intermediate version first?
>>
>> Thanks
>> Surbhi
>>
>

Re: Upgrade from 3.11.5 to 4.1.x

[MyWorld]

Though it's recommended to upgrade to latest version of 3.11.x and then to
ver 4  but even upgrading directly won't be a problem. Just check the
release notes.

However for production, I would recommend to go for 4.0.x latest stable
version.

Regards
Ashish

On Sat, 8 Jul, 2023, 05:44 Surbhi Gupta,  wrote:

> Hi,
>
> We have to upgrade from 3.11.5 to 4.1.x .
> Can we do it in one go ?
> Or do we have to go to an intermediate version first?
>
> Thanks
> Surbhi
>

Re: Materialized View inconsistency issue

[MyWorld]

Hi surbhi ,
There are 2 drawbacks associated with MV.
1. Inconsistent view
2. The lock it takes on the base table. This gets worse when you have huge
number of clustering keys in a specific partition.

It's better you re-design a seperate table and let your API do a parallel
write on both.

Regards,
Ashish

On Fri, 11 Aug, 2023, 02:03 Surbhi Gupta,  wrote:

> Thanks everyone.
>
>
> On Wed, 9 Aug 2023 at 01:00, Regis Le Bretonnic
>  wrote:
> >
> > Hi Surbhi
> >
> > We do use cassandra materialized views even if not recommended.
> > There are known issues you have to make with. Despite of them, we still
> use VM.
> > What we observe is :
> > * there are  inconsistency issues but few. Most of them are rows that
> should not exist in the MV...
> > * we made a spark script downloading the master table and the MV, and
> comparing them and fixing data (as said previously we have very few errors
> and we run it maybe once a year)
> >
> > * Things go very very very bad when you add or remove a node ! Limit
> this operation if possible and do it knowing what can happen (we isolate
> the ring/datacenter and fix data before putting it back to production. We
> did this only once in the last 4 years).
> >
> > PS : all proposals avoiding MV failed for our project. Basically
> managing a table like a MV (by deleting and inserting rows from code) is
> worse and more corrupted than what MV does...
> > The worse issue is adding and removing nodes. Maybe cassandra 4 improves
> this point (not tested yet).
> >
> > Have fun...
> >
> > Le mar. 8 août 2023 à 22:36, Surbhi Gupta  a
> écrit :
> >>
> >> Hi,
> >>
> >> We get complaints about Materialized View inconsistency issues.
> >> We are on 3.11.5 and on 3.11.5 Materialized Views were not production
> ready.
> >> We are ok to upgrade.
> >>
> >> On which version of cassandra MVs doesnt have inconsistency issues?
> >>
> >> Thanks
> >> Surbhi
>

Re: Cassandra p95 latencies

[MyWorld]

Hi,
P95 should not be a problem if rightly designed. Levelled compaction
strategy further reduces this, however it consume some resources. For read,
caching is also helpful.
Can you check your cpu iowait as it could be the reason for delay

Regards,
Ashish

On Fri, 11 Aug, 2023, 04:58 Shaurya Gupta,  wrote:

> Hi community
>
> What is the expected P95 latency for Cassandra Read and Write queries
> executed with Local_Quorum over a table with 3 replicas ? The queries are
> done using the partition + clustering key and row size in bytes is not too
> much, maybe 1-2 KB maximum.
> Assuming CPU is not a crunch ?
>
> We observe those to be 40 ms P95 Reads and same for Writes. This looks
> very high as compared to what we expected. We are using Cassandra 4.0.
>
> Any documentation / numbers will be helpful.
>
> Thanks
> --
> Shaurya Gupta
>
>
>

Cassandra p95 latencies

[Shaurya Gupta]

Hi community

What is the expected P95 latency for Cassandra Read and Write queries
executed with Local_Quorum over a table with 3 replicas ? The queries are
done using the partition + clustering key and row size in bytes is not too
much, maybe 1-2 KB maximum.
Assuming CPU is not a crunch ?

We observe those to be 40 ms P95 Reads and same for Writes. This looks very
high as compared to what we expected. We are using Cassandra 4.0.

Any documentation / numbers will be helpful.

Thanks
-- 
Shaurya Gupta

Re: Materialized View inconsistency issue

[Surbhi Gupta]

Thanks everyone.


On Wed, 9 Aug 2023 at 01:00, Regis Le Bretonnic
 wrote:
>
> Hi Surbhi
>
> We do use cassandra materialized views even if not recommended.
> There are known issues you have to make with. Despite of them, we still use 
> VM.
> What we observe is :
> * there are  inconsistency issues but few. Most of them are rows that should 
> not exist in the MV...
> * we made a spark script downloading the master table and the MV, and 
> comparing them and fixing data (as said previously we have very few errors 
> and we run it maybe once a year)
>
> * Things go very very very bad when you add or remove a node ! Limit this 
> operation if possible and do it knowing what can happen (we isolate the 
> ring/datacenter and fix data before putting it back to production. We did 
> this only once in the last 4 years).
>
> PS : all proposals avoiding MV failed for our project. Basically managing a 
> table like a MV (by deleting and inserting rows from code) is worse and more 
> corrupted than what MV does...
> The worse issue is adding and removing nodes. Maybe cassandra 4 improves this 
> point (not tested yet).
>
> Have fun...
>
> Le mar. 8 août 2023 à 22:36, Surbhi Gupta  a écrit :
>>
>> Hi,
>>
>> We get complaints about Materialized View inconsistency issues.
>> We are on 3.11.5 and on 3.11.5 Materialized Views were not production ready.
>> We are ok to upgrade.
>>
>> On which version of cassandra MVs doesnt have inconsistency issues?
>>
>> Thanks
>> Surbhi