Re: Understanding multi region read query and latency

[Bowen Song via user]

Adding sleep to solve racing conditions is a bad practice, and should be 
avoided if possible. Instead, use read and write CL that guarantees 
strong consistency when it is required/needed.


On 09/08/2022 23:49, Jim Shaw wrote:

Raphael:
   Have you found  root cause ? If not, here are a few tips, based on 
what I experienced before, but may not  be same as your case, just 
hope it is helpful.

1) app side called wrong code module

get the cql from system.prepared_statements

cql statement is helpful to developers to search their code and find 
issue parts. In my case,  was function disabled but actually not, when 
they see cql statement, they realized.


2) app side code query immediately after write

from the trace, you have read time,  get this row write time by

select writetime ("any non-key column here") from "table_name_here" 
where ...;


if read time is too close to write time,  ask developers to add a 
sleep in code.


while earlier phase of projects using cassandra, developers still get 
used to rdbms style, forget cassandra is distributed database (i.e. in 
code, 10 cql statements in a logic order, they assume they will be 
executed in order, but actually in distributed system, no order, last 
line in code may execute 1st in cassandra cluster).


3) duplicate the case
use copy tables, testing data, by comparing the traces, duplicate the 
case, so know your debug direction right or not right.



Regards,


Jim

On Sun, Aug 7, 2022 at 5:14 PM Stéphane Alleaume 
 wrote:


You're right too, this option is not new, sorry.

Is this option can be useful ?


Le dim. 7 août 2022, 22:18, Bowen Song via user
 a écrit :

Do you mean "nodetool settraceprobability"? This is not
exactly new, I remember it was available on Cassandra 2.x.

On 07/08/2022 20:43, Stéphane Alleaume wrote:

I think perhaps you already know but i read you can now trace
only a % of all queries, i will look to retrieve the name of
this fonctionnality (in new Cassandra release).

Hope it will help
Kind regards
Stéphane


Le dim. 7 août 2022, 20:26, Raphael Mazelier
 a écrit :

> "Read repair is in the blocking read path for the
query, yep"

OK interesting. This is not what I understood from the
documentation. And I use localOne level consistency.

I enabled tracing (see in the attachment of my first
msg)/ but I didn't see read repair in the trace (and btw
I tried to completely disable it on my table setting both
read_repair_chance and local_dc_read_repair_chance to 0).

The problem when enabling trace in cqlsh is that I only
get slow result. For having fast answer I need to iterate
faster on my queries.

I can provide again trace for analysis. I got something
more readable in python.

Best,

--

Raphael


On 07/08/2022 19:30, C. Scott Andreas wrote:

> but still as I understand the documentation the read
repair should not be in the blocking path of a query ?

Read repair is in the blocking read path for the query,
yep. At quorum consistency levels, the read repair must
complete before returning a result to the client to
ensure the data returned would be visible on subsequent
reads that address the remainder of the quorum.

If you enable tracing - either for a single CQL
statement that is expected to be slow, or probabilistic
from the server side to catch a slow query in the act -
that will help identify what’s happening.

- Scott


On Aug 7, 2022, at 10:25 AM, Raphael Mazelier
  wrote:



Nope. And what really puzzle me is in the trace we
really show the difference between queries. The fast
queries only request read from one replicas, while slow
queries request from multiple replicas (and not only
local to the dc).

On 07/08/2022 14:02, Stéphane Alleaume wrote:

Hi

Is there some GC which could affect coordinarir node ?

Kind regards
Stéphane

Le dim. 7 août 2022, 13:41, Raphael Mazelier
 a écrit :

Thanks for the answer but I was well aware of
this. I use localOne as consistency level.

My client connect to a local seeds, then choose a
local coordinator (as far I can understand the
trace log).

Then for a batch of request I got approximately
98% of request treated in 2/3ms in local DC with
one read request, and 2% treated by many nodes
(according 

Re: Understanding multi region read query and latency

[Jim Shaw]

Raphael:
   Have you found  root cause ? If not, here are a few tips, based on what
I experienced before, but may not  be same as your case, just hope it is
helpful.
1) app side called wrong code module

get the cql from system.prepared_statements

cql statement is helpful to developers to search their code and find issue
parts. In my case,  was function disabled but actually not, when they see
cql statement, they realized.

2) app side code query immediately after write

from the trace, you have read time,  get this row write time by

select writetime ("any non-key column here") from "table_name_here" where
...;

if read time is too close to write time,  ask developers to add a sleep in
code.

while earlier phase of projects using cassandra, developers still get used
to rdbms style, forget cassandra is distributed database (i.e. in code, 10
cql statements in a logic order, they assume they will be executed in
order, but actually in distributed system, no order, last line in code may
execute 1st in cassandra cluster).

3) duplicate the case
use copy tables, testing data, by comparing the traces, duplicate the case,
so know your debug direction right or not right.


Regards,


Jim

On Sun, Aug 7, 2022 at 5:14 PM Stéphane Alleaume 
wrote:

> You're right too, this option is not new, sorry.
>
> Is this option can be useful ?
>
>
> Le dim. 7 août 2022, 22:18, Bowen Song via user 
> a écrit :
>
>> Do you mean "nodetool settraceprobability"? This is not exactly new, I
>> remember it was available on Cassandra 2.x.
>> On 07/08/2022 20:43, Stéphane Alleaume wrote:
>>
>> I think perhaps you already know but i read you can now trace only a % of
>> all queries, i will look to retrieve the name of this fonctionnality (in
>> new Cassandra release).
>>
>> Hope it will help
>> Kind regards
>> Stéphane
>>
>>
>> Le dim. 7 août 2022, 20:26, Raphael Mazelier  a
>> écrit :
>>
>>> > "Read repair is in the blocking read path for the query, yep"
>>>
>>> OK interesting. This is not what I understood from the documentation.
>>> And I use localOne level consistency.
>>>
>>> I enabled tracing (see in the attachment of my first msg)/ but I didn't
>>> see read repair in the trace (and btw I tried to completely disable it on
>>> my table setting both read_repair_chance and local_dc_read_repair_chance to
>>> 0).
>>>
>>> The problem when enabling trace in cqlsh is that I only get slow result.
>>> For having fast answer I need to iterate faster on my queries.
>>>
>>> I can provide again trace for analysis. I got something more readable in
>>> python.
>>>
>>> Best,
>>>
>>> --
>>>
>>> Raphael
>>>
>>>
>>> On 07/08/2022 19:30, C. Scott Andreas wrote:
>>>
>>> > but still as I understand the documentation the read repair should
>>> not be in the blocking path of a query ?
>>>
>>> Read repair is in the blocking read path for the query, yep. At quorum
>>> consistency levels, the read repair must complete before returning a result
>>> to the client to ensure the data returned would be visible on subsequent
>>> reads that address the remainder of the quorum.
>>>
>>> If you enable tracing - either for a single CQL statement that is
>>> expected to be slow, or probabilistic from the server side to catch a slow
>>> query in the act - that will help identify what’s happening.
>>>
>>> - Scott
>>>
>>> On Aug 7, 2022, at 10:25 AM, Raphael Mazelier 
>>>  wrote:
>>>
>>> 
>>>
>>> Nope. And what really puzzle me is in the trace we really show the
>>> difference between queries. The fast queries only request read from one
>>> replicas, while slow queries request from multiple replicas (and not only
>>> local to the dc).
>>> On 07/08/2022 14:02, Stéphane Alleaume wrote:
>>>
>>> Hi
>>>
>>> Is there some GC which could affect coordinarir node ?
>>>
>>> Kind regards
>>> Stéphane
>>>
>>> Le dim. 7 août 2022, 13:41, Raphael Mazelier  a
>>> écrit :
>>>
 Thanks for the answer but I was well aware of this. I use localOne as
 consistency level.

 My client connect to a local seeds, then choose a local coordinator (as
 far I can understand the trace log).

 Then for a batch of request I got approximately 98% of request treated
 in 2/3ms in local DC with one read request, and 2% treated by many nodes
 (according to the trace) and then way longer (250ms).

 ?
 On 06/08/2022 14:30, Bowen Song via user wrote:

 See the diagram below. Your problem almost certainly arises from step
 4, in which an incorrect consistency level set by the client caused the
 coordinator node to send the READ command to nodes in other DCs.

 The load balancing policy only affects step 2 and 3, not step 1 or 4.

 You should change the consistency level to LOCAL_ONE/LOCAL_QUORUM/etc.
 to fix the problem.

 On 05/08/2022 22:54, Bowen Song wrote:

 The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
 Cassandra coordinator node the client sends queries to, not the nodes it
 

Re: Understanding multi region read query and latency

[Stéphane Alleaume]

You're right too, this option is not new, sorry.

Is this option can be useful ?


Le dim. 7 août 2022, 22:18, Bowen Song via user 
a écrit :

> Do you mean "nodetool settraceprobability"? This is not exactly new, I
> remember it was available on Cassandra 2.x.
> On 07/08/2022 20:43, Stéphane Alleaume wrote:
>
> I think perhaps you already know but i read you can now trace only a % of
> all queries, i will look to retrieve the name of this fonctionnality (in
> new Cassandra release).
>
> Hope it will help
> Kind regards
> Stéphane
>
>
> Le dim. 7 août 2022, 20:26, Raphael Mazelier  a écrit :
>
>> > "Read repair is in the blocking read path for the query, yep"
>>
>> OK interesting. This is not what I understood from the documentation. And
>> I use localOne level consistency.
>>
>> I enabled tracing (see in the attachment of my first msg)/ but I didn't
>> see read repair in the trace (and btw I tried to completely disable it on
>> my table setting both read_repair_chance and local_dc_read_repair_chance to
>> 0).
>>
>> The problem when enabling trace in cqlsh is that I only get slow result.
>> For having fast answer I need to iterate faster on my queries.
>>
>> I can provide again trace for analysis. I got something more readable in
>> python.
>>
>> Best,
>>
>> --
>>
>> Raphael
>>
>>
>> On 07/08/2022 19:30, C. Scott Andreas wrote:
>>
>> > but still as I understand the documentation the read repair should not
>> be in the blocking path of a query ?
>>
>> Read repair is in the blocking read path for the query, yep. At quorum
>> consistency levels, the read repair must complete before returning a result
>> to the client to ensure the data returned would be visible on subsequent
>> reads that address the remainder of the quorum.
>>
>> If you enable tracing - either for a single CQL statement that is
>> expected to be slow, or probabilistic from the server side to catch a slow
>> query in the act - that will help identify what’s happening.
>>
>> - Scott
>>
>> On Aug 7, 2022, at 10:25 AM, Raphael Mazelier 
>>  wrote:
>>
>> 
>>
>> Nope. And what really puzzle me is in the trace we really show the
>> difference between queries. The fast queries only request read from one
>> replicas, while slow queries request from multiple replicas (and not only
>> local to the dc).
>> On 07/08/2022 14:02, Stéphane Alleaume wrote:
>>
>> Hi
>>
>> Is there some GC which could affect coordinarir node ?
>>
>> Kind regards
>> Stéphane
>>
>> Le dim. 7 août 2022, 13:41, Raphael Mazelier  a
>> écrit :
>>
>>> Thanks for the answer but I was well aware of this. I use localOne as
>>> consistency level.
>>>
>>> My client connect to a local seeds, then choose a local coordinator (as
>>> far I can understand the trace log).
>>>
>>> Then for a batch of request I got approximately 98% of request treated
>>> in 2/3ms in local DC with one read request, and 2% treated by many nodes
>>> (according to the trace) and then way longer (250ms).
>>>
>>> ?
>>> On 06/08/2022 14:30, Bowen Song via user wrote:
>>>
>>> See the diagram below. Your problem almost certainly arises from step 4,
>>> in which an incorrect consistency level set by the client caused the
>>> coordinator node to send the READ command to nodes in other DCs.
>>>
>>> The load balancing policy only affects step 2 and 3, not step 1 or 4.
>>>
>>> You should change the consistency level to LOCAL_ONE/LOCAL_QUORUM/etc.
>>> to fix the problem.
>>>
>>> On 05/08/2022 22:54, Bowen Song wrote:
>>>
>>> The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
>>> Cassandra coordinator node the client sends queries to, not the nodes it
>>> connects to, nor the nodes that performs the actual read.
>>>
>>> A client sends a CQL read query to a coordinator node, and the
>>> coordinator node parses the CQL query, and send READ requests to other
>>> nodes in the cluster based on the consistency level.
>>>
>>> Have you checked the consistency level of the session (and the query if
>>> applicable)? Is it prefixed with "LOCAL_"? If not, the coordinator will
>>> send the READ requests to non-local DCs.
>>>
>>>
>>> On 05/08/2022 19:40, Raphael Mazelier wrote:
>>>
>>>
>>> Hi Cassandra Users,
>>>
>>> I'm relatively new to Cassandra and first I have to say I'm really
>>> impressed by the technology.
>>>
>>> Good design and a lot of stuff to understand the underlying (the Oreilly
>>> book help a lot as well as thelastpickle blog post).
>>>
>>> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
>>> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
>>> US. Everything deployed in AWS.
>>>
>>> We have a keyspace configured with network topology and two replicas on
>>> every region like this: {'class': 'NetworkTopologyStrategy',
>>> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>>>
>>>
>>> Investigating some performance issue I noticed strange things in my
>>> experiment:
>>>
>>> What we expect is very slow latency 3/5ms max for this specific select
>>> 

Re: Understanding multi region read query and latency

[Bowen Song via user]

Do you mean "nodetool settraceprobability"? This is not exactly new, I 
remember it was available on Cassandra 2.x.


On 07/08/2022 20:43, Stéphane Alleaume wrote:
I think perhaps you already know but i read you can now trace only a % 
of all queries, i will look to retrieve the name of this 
fonctionnality (in new Cassandra release).


Hope it will help
Kind regards
Stéphane


Le dim. 7 août 2022, 20:26, Raphael Mazelier  a écrit :

> "Read repair is in the blocking read path for the query, yep"

OK interesting. This is not what I understood from the
documentation. And I use localOne level consistency.

I enabled tracing (see in the attachment of my first msg)/ but I
didn't see read repair in the trace (and btw I tried to completely
disable it on my table setting both read_repair_chance and
local_dc_read_repair_chance to 0).

The problem when enabling trace in cqlsh is that I only get slow
result. For having fast answer I need to iterate faster on my
queries.

I can provide again trace for analysis. I got something more
readable in python.

Best,

--

Raphael


On 07/08/2022 19:30, C. Scott Andreas wrote:

> but still as I understand the documentation the read repair
should not be in the blocking path of a query ?

Read repair is in the blocking read path for the query, yep. At
quorum consistency levels, the read repair must complete before
returning a result to the client to ensure the data returned
would be visible on subsequent reads that address the remainder
of the quorum.

If you enable tracing - either for a single CQL statement that is
expected to be slow, or probabilistic from the server side to
catch a slow query in the act - that will help identify what’s
happening.

- Scott


On Aug 7, 2022, at 10:25 AM, Raphael Mazelier
  wrote:



Nope. And what really puzzle me is in the trace we really show
the difference between queries. The fast queries only request
read from one replicas, while slow queries request from multiple
replicas (and not only local to the dc).

On 07/08/2022 14:02, Stéphane Alleaume wrote:

Hi

Is there some GC which could affect coordinarir node ?

Kind regards
Stéphane

Le dim. 7 août 2022, 13:41, Raphael Mazelier
 a écrit :

Thanks for the answer but I was well aware of this. I use
localOne as consistency level.

My client connect to a local seeds, then choose a local
coordinator (as far I can understand the trace log).

Then for a batch of request I got approximately 98% of
request treated in 2/3ms in local DC with one read request,
and 2% treated by many nodes (according to the trace) and
then way longer (250ms).

?

On 06/08/2022 14:30, Bowen Song via user wrote:


See the diagram below. Your problem almost certainly
arises from step 4, in which an incorrect consistency
level set by the client caused the coordinator node to
send the READ command to nodes in other DCs.

The load balancing policy only affects step 2 and 3, not
step 1 or 4.

You should change the consistency level to
LOCAL_ONE/LOCAL_QUORUM/etc. to fix the problem.

On 05/08/2022 22:54, Bowen Song wrote:

The DCAwareRoundRobinPolicy/TokenAwareHostPolicy
controlls which Cassandra coordinator node the client
sends queries to, not the nodes it connects to, nor the
nodes that performs the actual read.

A client sends a CQL read query to a coordinator node,
and the coordinator node parses the CQL query, and send
READ requests to other nodes in the cluster based on the
consistency level.

Have you checked the consistency level of the session
(and the query if applicable)? Is it prefixed with
"LOCAL_"? If not, the coordinator will send the READ
requests to non-local DCs.


On 05/08/2022 19:40, Raphael Mazelier wrote:


Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say
I'm really impressed by the technology.

Good design and a lot of stuff to understand the
underlying (the Oreilly book help a lot as well as
thelastpickle blog post).

I have an muli-datacenter c* cluster (US, Europe,
Singapore) with eight node on each (two seeds on each
region), two racks on Eu, Singapore, 3 on US. Everything
deployed in AWS.

We have a keyspace configured with network topology and
two replicas on every region like this: {'class':
'NetworkTopologyStrategy', 'ap-southeast-1': '2',
'eu-west-1': '2', 'us-east-1': '2'}


Investigating some performance issue I noticed strange
things in my experiment:

What we expect is very slow latency 

Re: Understanding multi region read query and latency

[Stéphane Alleaume]

Thanks a lot Scott, i didn't knew this fact.

Kind regards
Stéphane

Le dim. 7 août 2022, 19:31, C. Scott Andreas  a
écrit :

> > but still as I understand the documentation the read repair should not
> be in the blocking path of a query ?
>
> Read repair is in the blocking read path for the query, yep. At quorum
> consistency levels, the read repair must complete before returning a result
> to the client to ensure the data returned would be visible on subsequent
> reads that address the remainder of the quorum.
>
> If you enable tracing - either for a single CQL statement that is expected
> to be slow, or probabilistic from the server side to catch a slow query in
> the act - that will help identify what’s happening.
>
> - Scott
>
> On Aug 7, 2022, at 10:25 AM, Raphael Mazelier  wrote:
>
> 
>
> Nope. And what really puzzle me is in the trace we really show the
> difference between queries. The fast queries only request read from one
> replicas, while slow queries request from multiple replicas (and not only
> local to the dc).
> On 07/08/2022 14:02, Stéphane Alleaume wrote:
>
> Hi
>
> Is there some GC which could affect coordinarir node ?
>
> Kind regards
> Stéphane
>
> Le dim. 7 août 2022, 13:41, Raphael Mazelier  a écrit :
>
>> Thanks for the answer but I was well aware of this. I use localOne as
>> consistency level.
>>
>> My client connect to a local seeds, then choose a local coordinator (as
>> far I can understand the trace log).
>>
>> Then for a batch of request I got approximately 98% of request treated in
>> 2/3ms in local DC with one read request, and 2% treated by many nodes
>> (according to the trace) and then way longer (250ms).
>>
>> ?
>> On 06/08/2022 14:30, Bowen Song via user wrote:
>>
>> See the diagram below. Your problem almost certainly arises from step 4,
>> in which an incorrect consistency level set by the client caused the
>> coordinator node to send the READ command to nodes in other DCs.
>>
>> The load balancing policy only affects step 2 and 3, not step 1 or 4.
>>
>> You should change the consistency level to LOCAL_ONE/LOCAL_QUORUM/etc. to
>> fix the problem.
>>
>> On 05/08/2022 22:54, Bowen Song wrote:
>>
>> The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
>> Cassandra coordinator node the client sends queries to, not the nodes it
>> connects to, nor the nodes that performs the actual read.
>>
>> A client sends a CQL read query to a coordinator node, and the
>> coordinator node parses the CQL query, and send READ requests to other
>> nodes in the cluster based on the consistency level.
>>
>> Have you checked the consistency level of the session (and the query if
>> applicable)? Is it prefixed with "LOCAL_"? If not, the coordinator will
>> send the READ requests to non-local DCs.
>>
>>
>> On 05/08/2022 19:40, Raphael Mazelier wrote:
>>
>>
>> Hi Cassandra Users,
>>
>> I'm relatively new to Cassandra and first I have to say I'm really
>> impressed by the technology.
>>
>> Good design and a lot of stuff to understand the underlying (the Oreilly
>> book help a lot as well as thelastpickle blog post).
>>
>> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
>> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
>> US. Everything deployed in AWS.
>>
>> We have a keyspace configured with network topology and two replicas on
>> every region like this: {'class': 'NetworkTopologyStrategy',
>> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>>
>>
>> Investigating some performance issue I noticed strange things in my
>> experiment:
>>
>> What we expect is very slow latency 3/5ms max for this specific select
>> query. So we want every read to be local the each datacenter.
>>
>> We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the same
>> in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
>>
>> Testing a bit with two short program (I can provide them) in go and
>> python I notice very strange result. Basically I do the same query over and
>> over with a very limited dataset of id.
>>
>> The first result were surprising cause the very first query were always
>> more than 250ms and after with stressing c* (playing with sleep between
>> query) I can achieve a good ratio of query at 3/4 ms (what I expected).
>>
>> My guess was that long query were somewhat executed not locally (or at
>> least imply multi datacenter queries) and short one no.
>>
>> Activating tracing in my program (like enalbing trace in cqlsh) kindla
>> confirm my suspicion.
>>
>> (I will provide trace in attachment).
>>
>> My question is why sometime C* try to read not localy? how we can disable
>> it? what is the criteria for this?
>>
>> (btw I'm very not fan of this multi region design for theses very
>> specific kind of issues...)
>>
>> Also side question: why C* is so slow at connection? it's like it's
>> trying to reach every nodes in each DC? (we only provide locals seeds
>> however). Sometimes it take more than 

Re: Understanding multi region read query and latency

[Stéphane Alleaume]

I think perhaps you already know but i read you can now trace only a % of
all queries, i will look to retrieve the name of this fonctionnality (in
new Cassandra release).

Hope it will help
Kind regards
Stéphane


Le dim. 7 août 2022, 20:26, Raphael Mazelier  a écrit :

> > "Read repair is in the blocking read path for the query, yep"
>
> OK interesting. This is not what I understood from the documentation. And
> I use localOne level consistency.
>
> I enabled tracing (see in the attachment of my first msg)/ but I didn't
> see read repair in the trace (and btw I tried to completely disable it on
> my table setting both read_repair_chance and local_dc_read_repair_chance to
> 0).
>
> The problem when enabling trace in cqlsh is that I only get slow result.
> For having fast answer I need to iterate faster on my queries.
>
> I can provide again trace for analysis. I got something more readable in
> python.
>
> Best,
>
> --
>
> Raphael
>
>
> On 07/08/2022 19:30, C. Scott Andreas wrote:
>
> > but still as I understand the documentation the read repair should not
> be in the blocking path of a query ?
>
> Read repair is in the blocking read path for the query, yep. At quorum
> consistency levels, the read repair must complete before returning a result
> to the client to ensure the data returned would be visible on subsequent
> reads that address the remainder of the quorum.
>
> If you enable tracing - either for a single CQL statement that is expected
> to be slow, or probabilistic from the server side to catch a slow query in
> the act - that will help identify what’s happening.
>
> - Scott
>
> On Aug 7, 2022, at 10:25 AM, Raphael Mazelier 
>  wrote:
>
> 
>
> Nope. And what really puzzle me is in the trace we really show the
> difference between queries. The fast queries only request read from one
> replicas, while slow queries request from multiple replicas (and not only
> local to the dc).
> On 07/08/2022 14:02, Stéphane Alleaume wrote:
>
> Hi
>
> Is there some GC which could affect coordinarir node ?
>
> Kind regards
> Stéphane
>
> Le dim. 7 août 2022, 13:41, Raphael Mazelier  a écrit :
>
>> Thanks for the answer but I was well aware of this. I use localOne as
>> consistency level.
>>
>> My client connect to a local seeds, then choose a local coordinator (as
>> far I can understand the trace log).
>>
>> Then for a batch of request I got approximately 98% of request treated in
>> 2/3ms in local DC with one read request, and 2% treated by many nodes
>> (according to the trace) and then way longer (250ms).
>>
>> ?
>> On 06/08/2022 14:30, Bowen Song via user wrote:
>>
>> See the diagram below. Your problem almost certainly arises from step 4,
>> in which an incorrect consistency level set by the client caused the
>> coordinator node to send the READ command to nodes in other DCs.
>>
>> The load balancing policy only affects step 2 and 3, not step 1 or 4.
>>
>> You should change the consistency level to LOCAL_ONE/LOCAL_QUORUM/etc. to
>> fix the problem.
>>
>> On 05/08/2022 22:54, Bowen Song wrote:
>>
>> The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
>> Cassandra coordinator node the client sends queries to, not the nodes it
>> connects to, nor the nodes that performs the actual read.
>>
>> A client sends a CQL read query to a coordinator node, and the
>> coordinator node parses the CQL query, and send READ requests to other
>> nodes in the cluster based on the consistency level.
>>
>> Have you checked the consistency level of the session (and the query if
>> applicable)? Is it prefixed with "LOCAL_"? If not, the coordinator will
>> send the READ requests to non-local DCs.
>>
>>
>> On 05/08/2022 19:40, Raphael Mazelier wrote:
>>
>>
>> Hi Cassandra Users,
>>
>> I'm relatively new to Cassandra and first I have to say I'm really
>> impressed by the technology.
>>
>> Good design and a lot of stuff to understand the underlying (the Oreilly
>> book help a lot as well as thelastpickle blog post).
>>
>> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
>> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
>> US. Everything deployed in AWS.
>>
>> We have a keyspace configured with network topology and two replicas on
>> every region like this: {'class': 'NetworkTopologyStrategy',
>> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>>
>>
>> Investigating some performance issue I noticed strange things in my
>> experiment:
>>
>> What we expect is very slow latency 3/5ms max for this specific select
>> query. So we want every read to be local the each datacenter.
>>
>> We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the same
>> in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
>>
>> Testing a bit with two short program (I can provide them) in go and
>> python I notice very strange result. Basically I do the same query over and
>> over with a very limited dataset of id.
>>
>> The first result were surprising cause 

Re: Understanding multi region read query and latency

[Raphael Mazelier]

> "Read repair is in the blocking read path for the query, yep"

OK interesting. This is not what I understood from the documentation. 
And I use localOne level consistency.


I enabled tracing (see in the attachment of my first msg)/ but I didn't 
see read repair in the trace (and btw I tried to completely disable it 
on my table setting both read_repair_chance and 
local_dc_read_repair_chance to 0).


The problem when enabling trace in cqlsh is that I only get slow result. 
For having fast answer I need to iterate faster on my queries.


I can provide again trace for analysis. I got something more readable in 
python.


Best,

--

Raphael


On 07/08/2022 19:30, C. Scott Andreas wrote:
> but still as I understand the documentation the read repair should 
not be in the blocking path of a query ?


Read repair is in the blocking read path for the query, yep. At quorum 
consistency levels, the read repair must complete before returning a 
result to the client to ensure the data returned would be visible on 
subsequent reads that address the remainder of the quorum.


If you enable tracing - either for a single CQL statement that is 
expected to be slow, or probabilistic from the server side to catch a 
slow query in the act - that will help identify what’s happening.


- Scott


On Aug 7, 2022, at 10:25 AM, Raphael Mazelier  wrote:



Nope. And what really puzzle me is in the trace we really show the 
difference between queries. The fast queries only request read from 
one replicas, while slow queries request from multiple replicas (and 
not only local to the dc).


On 07/08/2022 14:02, Stéphane Alleaume wrote:

Hi

Is there some GC which could affect coordinarir node ?

Kind regards
Stéphane

Le dim. 7 août 2022, 13:41, Raphael Mazelier  a 
écrit :


Thanks for the answer but I was well aware of this. I use
localOne as consistency level.

My client connect to a local seeds, then choose a local
coordinator (as far I can understand the trace log).

Then for a batch of request I got approximately 98% of request
treated in 2/3ms in local DC with one read request, and 2%
treated by many nodes (according to the trace) and then way
longer (250ms).

?

On 06/08/2022 14:30, Bowen Song via user wrote:


See the diagram below. Your problem almost certainly arises
from step 4, in which an incorrect consistency level set by the
client caused the coordinator node to send the READ command to
nodes in other DCs.

The load balancing policy only affects step 2 and 3, not step 1
or 4.

You should change the consistency level to
LOCAL_ONE/LOCAL_QUORUM/etc. to fix the problem.

On 05/08/2022 22:54, Bowen Song wrote:

The DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls
which Cassandra coordinator node the client sends queries to,
not the nodes it connects to, nor the nodes that performs the
actual read.

A client sends a CQL read query to a coordinator node, and the
coordinator node parses the CQL query, and send READ requests
to other nodes in the cluster based on the consistency level.

Have you checked the consistency level of the session (and the
query if applicable)? Is it prefixed with "LOCAL_"? If not,
the coordinator will send the READ requests to non-local DCs.


On 05/08/2022 19:40, Raphael Mazelier wrote:


Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say I'm
really impressed by the technology.

Good design and a lot of stuff to understand the underlying
(the Oreilly book help a lot as well as thelastpickle blog
post).

I have an muli-datacenter c* cluster (US, Europe, Singapore)
with eight node on each (two seeds on each region), two racks
on Eu, Singapore, 3 on US. Everything deployed in AWS.

We have a keyspace configured with network topology and two
replicas on every region like this: {'class':
'NetworkTopologyStrategy', 'ap-southeast-1': '2',
'eu-west-1': '2', 'us-east-1': '2'}


Investigating some performance issue I noticed strange things
in my experiment:

What we expect is very slow latency 3/5ms max for this
specific select query. So we want every read to be local the
each datacenter.

We configure DCAwareRoundRobinPolicy(local_dc=DC) in python,
and the same in Go
gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))

Testing a bit with two short program (I can provide them) in
go and python I notice very strange result. Basically I do
the same query over and over with a very limited dataset of id.

The first result were surprising cause the very first query
were always more than 250ms and after with stressing c*
(playing with sleep between query) I can achieve a good ratio
of query at 3/4 ms (what I expected).

My guess was that long query were somewhat executed not
locally (or at least imply multi datacenter queries) and
  

Re: Understanding multi region read query and latency

[C. Scott Andreas]

> but still as I understand the documentation the read repair should not be in the blocking path of a query ?Read repair is in the blocking read path for the query, yep. At quorum consistency levels, the read repair must complete before returning a result to the client to ensure the data returned would be visible on subsequent reads that address the remainder of the quorum.If you enable tracing - either for a single CQL statement that is expected to be slow, or probabilistic from the server side to catch a slow query in the act - that will help identify what’s happening.- ScottOn Aug 7, 2022, at 10:25 AM, Raphael Mazelier  wrote:
  

  
  
Nope. And what really puzzle me is in the
trace we really show the difference between queries. The fast
queries only request read from one replicas, while slow queries
request from multiple replicas (and not only local to the dc).

On 07/08/2022 14:02, Stéphane Alleaume
  wrote:


  
  Hi


Is there some GC which could affect coordinarir
  node ?


Kind regards
Stéphane 
  
  
  
Le dim. 7 août 2022, 13:41,
  Raphael Mazelier 
  a écrit :


  
Thanks for the answer but I was
well aware of this. I use localOne as consistency level.
  
My client connect to a local seeds, then choose a local
  coordinator (as far I can understand the trace log).
Then for a batch of request I got approximately 98% of
  request treated in 2/3ms in local DC with one read
  request, and 2% treated by many nodes (according to the
  trace) and then way longer (250ms).
?

On 06/08/2022 14:30, Bowen Song via user wrote:


  See the diagram below. Your problem almost certainly
arises from step 4, in which an incorrect consistency
level set by the client caused the coordinator node to
send the READ command to nodes in other DCs.
  The load balancing policy only affects step 2 and 3,
not step 1 or 4.
  You should change the consistency level to
LOCAL_ONE/LOCAL_QUORUM/etc. to fix the problem.
  
  On 05/08/2022 22:54, Bowen Song wrote:
  
  The 
DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls
which Cassandra coordinator node the client sends
queries to, not the nodes it connects to, nor the nodes
that performs the actual read. 

A client sends a CQL read query to a coordinator node,
and the coordinator node parses the CQL query, and send
READ requests to other nodes in the cluster based on the
consistency level. 

Have you checked the consistency level of the session
(and the query if applicable)? Is it prefixed with
"LOCAL_"? If not, the coordinator will send the READ
requests to non-local DCs. 


On 05/08/2022 19:40, Raphael Mazelier wrote: 
 
  Hi Cassandra Users, 
  
  I'm relatively new to Cassandra and first I have to
  say I'm really impressed by the technology. 
  
  Good design and a lot of stuff to understand the
  underlying (the Oreilly book help a lot as well as
  thelastpickle blog post). 
  
  I have an muli-datacenter c* cluster (US, Europe,
  Singapore) with eight node on each (two seeds on each
  region), two racks on Eu, Singapore, 3 on US.
  Everything deployed in AWS. 
  
  We have a keyspace configured with network topology
  and two replicas on every region like this: {'class':
  'NetworkTopologyStrategy', 'ap-southeast-1': '2',
  'eu-west-1': '2', 'us-east-1': '2'} 
  
  
  Investigating some performance issue I noticed strange
  things in my experiment: 
  
  What we expect is very slow latency 3/5ms max for this
  specific select query. So we want every read to be
  local the each datacenter. 
  
  We configure DCAwareRoundRobinPolicy(local_dc=DC) in
  python, and the same in Go
  gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
  
  
  

Re: Understanding multi region read query and latency

[Stéphane Alleaume]

Read repair chance ?

Le dim. 7 août 2022, 19:25, Raphael Mazelier  a écrit :

> Nope. And what really puzzle me is in the trace we really show the
> difference between queries. The fast queries only request read from one
> replicas, while slow queries request from multiple replicas (and not only
> local to the dc).
> On 07/08/2022 14:02, Stéphane Alleaume wrote:
>
> Hi
>
> Is there some GC which could affect coordinarir node ?
>
> Kind regards
> Stéphane
>
> Le dim. 7 août 2022, 13:41, Raphael Mazelier  a écrit :
>
>> Thanks for the answer but I was well aware of this. I use localOne as
>> consistency level.
>>
>> My client connect to a local seeds, then choose a local coordinator (as
>> far I can understand the trace log).
>>
>> Then for a batch of request I got approximately 98% of request treated in
>> 2/3ms in local DC with one read request, and 2% treated by many nodes
>> (according to the trace) and then way longer (250ms).
>>
>> ?
>> On 06/08/2022 14:30, Bowen Song via user wrote:
>>
>> See the diagram below. Your problem almost certainly arises from step 4,
>> in which an incorrect consistency level set by the client caused the
>> coordinator node to send the READ command to nodes in other DCs.
>>
>> The load balancing policy only affects step 2 and 3, not step 1 or 4.
>>
>> You should change the consistency level to LOCAL_ONE/LOCAL_QUORUM/etc. to
>> fix the problem.
>>
>> On 05/08/2022 22:54, Bowen Song wrote:
>>
>> The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
>> Cassandra coordinator node the client sends queries to, not the nodes it
>> connects to, nor the nodes that performs the actual read.
>>
>> A client sends a CQL read query to a coordinator node, and the
>> coordinator node parses the CQL query, and send READ requests to other
>> nodes in the cluster based on the consistency level.
>>
>> Have you checked the consistency level of the session (and the query if
>> applicable)? Is it prefixed with "LOCAL_"? If not, the coordinator will
>> send the READ requests to non-local DCs.
>>
>>
>> On 05/08/2022 19:40, Raphael Mazelier wrote:
>>
>>
>> Hi Cassandra Users,
>>
>> I'm relatively new to Cassandra and first I have to say I'm really
>> impressed by the technology.
>>
>> Good design and a lot of stuff to understand the underlying (the Oreilly
>> book help a lot as well as thelastpickle blog post).
>>
>> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
>> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
>> US. Everything deployed in AWS.
>>
>> We have a keyspace configured with network topology and two replicas on
>> every region like this: {'class': 'NetworkTopologyStrategy',
>> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>>
>>
>> Investigating some performance issue I noticed strange things in my
>> experiment:
>>
>> What we expect is very slow latency 3/5ms max for this specific select
>> query. So we want every read to be local the each datacenter.
>>
>> We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the same
>> in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
>>
>> Testing a bit with two short program (I can provide them) in go and
>> python I notice very strange result. Basically I do the same query over and
>> over with a very limited dataset of id.
>>
>> The first result were surprising cause the very first query were always
>> more than 250ms and after with stressing c* (playing with sleep between
>> query) I can achieve a good ratio of query at 3/4 ms (what I expected).
>>
>> My guess was that long query were somewhat executed not locally (or at
>> least imply multi datacenter queries) and short one no.
>>
>> Activating tracing in my program (like enalbing trace in cqlsh) kindla
>> confirm my suspicion.
>>
>> (I will provide trace in attachment).
>>
>> My question is why sometime C* try to read not localy? how we can disable
>> it? what is the criteria for this?
>>
>> (btw I'm very not fan of this multi region design for theses very
>> specific kind of issues...)
>>
>> Also side question: why C* is so slow at connection? it's like it's
>> trying to reach every nodes in each DC? (we only provide locals seeds
>> however). Sometimes it take more than 20s...
>>
>> Any help appreciated.
>>
>> Best,
>>
>> --
>>
>> Raphael Mazelier
>>
>>

Re: Understanding multi region read query and latency

[Raphael Mazelier]

Nope. And what really puzzle me is in the trace we really show the 
difference between queries. The fast queries only request read from one 
replicas, while slow queries request from multiple replicas (and not 
only local to the dc).


On 07/08/2022 14:02, Stéphane Alleaume wrote:

Hi

Is there some GC which could affect coordinarir node ?

Kind regards
Stéphane

Le dim. 7 août 2022, 13:41, Raphael Mazelier  a écrit :

Thanks for the answer but I was well aware of this. I use localOne
as consistency level.

My client connect to a local seeds, then choose a local
coordinator (as far I can understand the trace log).

Then for a batch of request I got approximately 98% of request
treated in 2/3ms in local DC with one read request, and 2% treated
by many nodes (according to the trace) and then way longer (250ms).

?

On 06/08/2022 14:30, Bowen Song via user wrote:


See the diagram below. Your problem almost certainly arises from
step 4, in which an incorrect consistency level set by the client
caused the coordinator node to send the READ command to nodes in
other DCs.

The load balancing policy only affects step 2 and 3, not step 1 or 4.

You should change the consistency level to
LOCAL_ONE/LOCAL_QUORUM/etc. to fix the problem.

On 05/08/2022 22:54, Bowen Song wrote:

The DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which
Cassandra coordinator node the client sends queries to, not the
nodes it connects to, nor the nodes that performs the actual read.

A client sends a CQL read query to a coordinator node, and the
coordinator node parses the CQL query, and send READ requests to
other nodes in the cluster based on the consistency level.

Have you checked the consistency level of the session (and the
query if applicable)? Is it prefixed with "LOCAL_"? If not, the
coordinator will send the READ requests to non-local DCs.


On 05/08/2022 19:40, Raphael Mazelier wrote:


Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say I'm
really impressed by the technology.

Good design and a lot of stuff to understand the underlying
(the Oreilly book help a lot as well as thelastpickle blog post).

I have an muli-datacenter c* cluster (US, Europe, Singapore)
with eight node on each (two seeds on each region), two racks
on Eu, Singapore, 3 on US. Everything deployed in AWS.

We have a keyspace configured with network topology and two
replicas on every region like this: {'class':
'NetworkTopologyStrategy', 'ap-southeast-1': '2', 'eu-west-1':
'2', 'us-east-1': '2'}


Investigating some performance issue I noticed strange things
in my experiment:

What we expect is very slow latency 3/5ms max for this specific
select query. So we want every read to be local the each
datacenter.

We configure DCAwareRoundRobinPolicy(local_dc=DC) in python,
and the same in Go
gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))

Testing a bit with two short program (I can provide them) in go
and python I notice very strange result. Basically I do the
same query over and over with a very limited dataset of id.

The first result were surprising cause the very first query
were always more than 250ms and after with stressing c*
(playing with sleep between query) I can achieve a good ratio
of query at 3/4 ms (what I expected).

My guess was that long query were somewhat executed not locally
(or at least imply multi datacenter queries) and short one no.

Activating tracing in my program (like enalbing trace in cqlsh)
kindla confirm my suspicion.

(I will provide trace in attachment).

My question is why sometime C* try to read not localy? how we
can disable it? what is the criteria for this?

(btw I'm very not fan of this multi region design for theses
very specific kind of issues...)

Also side question: why C* is so slow at connection? it's like
it's trying to reach every nodes in each DC? (we only provide
locals seeds however). Sometimes it take more than 20s...

Any help appreciated.

Best,

-- 


Raphael Mazelier

Re: Understanding multi region read query and latency

[Raphael Mazelier]

Well answering to myself this is not related to read_repair chance. 
Settings them to 0 change also nothing. So the question remains: why 
from time to time C* want to make multiple read on a non local dc ?


On 06/08/2022 12:31, Raphael Mazelier wrote:


Well I tried (but already have some whiteListFilter) it changed 
nothing but it's more convenient that using whiteListFilter (speeding 
up the connection time).


So still from time to time (dedanding of the frequency of my requests) 
I got slow request when I notice in the trace that c* try to read on 
other DC.


Btw it's not limited to gocql (I got the pretty same result in python).

I wonder if it's related to read_repair_chance parameter and 
dclocal_read_repair_chance.


but still as I understand the documentation the read repair should not 
be in the blocking path of a query ?


--

Raphael Mazelier

On 05/08/2022 23:13, Jim Shaw wrote:
I remember gocql.DataCentreHostFilter was used.  try add it to see 
whether will read local DC only in your case ?


Thanks,

James

On Fri, Aug 5, 2022 at 2:40 PM Raphael Mazelier  
wrote:


Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say I'm
really impressed by the technology.

Good design and a lot of stuff to understand the underlying (the
Oreilly book help a lot as well as thelastpickle blog post).

I have an muli-datacenter c* cluster (US, Europe, Singapore) with
eight node on each (two seeds on each region), two racks on Eu,
Singapore, 3 on US. Everything deployed in AWS.

We have a keyspace configured with network topology and two
replicas on every region like this: {'class':
'NetworkTopologyStrategy', 'ap-southeast-1': '2', 'eu-west-1':
'2', 'us-east-1': '2'}


Investigating some performance issue I noticed strange things in
my experiment:

What we expect is very slow latency 3/5ms max for this specific
select query. So we want every read to be local the each datacenter.

We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and
the same in Go
gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))

Testing a bit with two short program (I can provide them) in go
and python I notice very strange result. Basically I do the same
query over and over with a very limited dataset of id.

The first result were surprising cause the very first query were
always more than 250ms and after with stressing c* (playing with
sleep between query) I can achieve a good ratio of query at 3/4
ms (what I expected).

My guess was that long query were somewhat executed not locally
(or at least imply multi datacenter queries) and short one no.

Activating tracing in my program (like enalbing trace in cqlsh)
kindla confirm my suspicion.

(I will provide trace in attachment).

My question is why sometime C* try to read not localy? how we can
disable it? what is the criteria for this?

(btw I'm very not fan of this multi region design for theses very
specific kind of issues...)

Also side question: why C* is so slow at connection? it's like
it's trying to reach every nodes in each DC? (we only provide
locals seeds however). Sometimes it take more than 20s...

Any help appreciated.

Best,

--

Raphael Mazelier

Re: Understanding multi region read query and latency

[Raphael Mazelier]

Well I tried (but already have some whiteListFilter) it changed nothing 
but it's more convenient that using whiteListFilter (speeding up the 
connection time).


So still from time to time (dedanding of the frequency of my requests) I 
got slow request when I notice in the trace that c* try to read on other DC.


Btw it's not limited to gocql (I got the pretty same result in python).

I wonder if it's related to read_repair_chance parameter and 
dclocal_read_repair_chance.


but still as I understand the documentation the read repair should not 
be in the blocking path of a query ?


--

Raphael Mazelier

On 05/08/2022 23:13, Jim Shaw wrote:
I remember gocql.DataCentreHostFilter was used.  try add it to see 
whether will read local DC only in your case ?


Thanks,

James

On Fri, Aug 5, 2022 at 2:40 PM Raphael Mazelier  wrote:

Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say I'm really
impressed by the technology.

Good design and a lot of stuff to understand the underlying (the
Oreilly book help a lot as well as thelastpickle blog post).

I have an muli-datacenter c* cluster (US, Europe, Singapore) with
eight node on each (two seeds on each region), two racks on Eu,
Singapore, 3 on US. Everything deployed in AWS.

We have a keyspace configured with network topology and two
replicas on every region like this: {'class':
'NetworkTopologyStrategy', 'ap-southeast-1': '2', 'eu-west-1':
'2', 'us-east-1': '2'}


Investigating some performance issue I noticed strange things in
my experiment:

What we expect is very slow latency 3/5ms max for this specific
select query. So we want every read to be local the each datacenter.

We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and
the same in Go
gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))

Testing a bit with two short program (I can provide them) in go
and python I notice very strange result. Basically I do the same
query over and over with a very limited dataset of id.

The first result were surprising cause the very first query were
always more than 250ms and after with stressing c* (playing with
sleep between query) I can achieve a good ratio of query at 3/4 ms
(what I expected).

My guess was that long query were somewhat executed not locally
(or at least imply multi datacenter queries) and short one no.

Activating tracing in my program (like enalbing trace in cqlsh)
kindla confirm my suspicion.

(I will provide trace in attachment).

My question is why sometime C* try to read not localy? how we can
disable it? what is the criteria for this?

(btw I'm very not fan of this multi region design for theses very
specific kind of issues...)

Also side question: why C* is so slow at connection? it's like
it's trying to reach every nodes in each DC? (we only provide
locals seeds however). Sometimes it take more than 20s...

Any help appreciated.

Best,

--

Raphael Mazelier

Re: Understanding multi region read query and latency

[Bowen Song via user]

The  DCAwareRoundRobinPolicy/TokenAwareHostPolicy controlls which 
Cassandra coordinator node the client sends queries to, not the nodes it 
connects to, nor the nodes that performs the actual read.


A client sends a CQL read query to a coordinator node, and the 
coordinator node parses the CQL query, and send READ requests to other 
nodes in the cluster based on the consistency level.


Have you checked the consistency level of the session (and the query if 
applicable)? Is it prefixed with "LOCAL_"? If not, the coordinator will 
send the READ requests to non-local DCs.



On 05/08/2022 19:40, Raphael Mazelier wrote:


Hi Cassandra Users,

I'm relatively new to Cassandra and first I have to say I'm really 
impressed by the technology.


Good design and a lot of stuff to understand the underlying (the 
Oreilly book help a lot as well as thelastpickle blog post).


I have an muli-datacenter c* cluster (US, Europe, Singapore) with 
eight node on each (two seeds on each region), two racks on Eu, 
Singapore, 3 on US. Everything deployed in AWS.


We have a keyspace configured with network topology and two replicas 
on every region like this: {'class': 'NetworkTopologyStrategy', 
'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}



Investigating some performance issue I noticed strange things in my 
experiment:


What we expect is very slow latency 3/5ms max for this specific select 
query. So we want every read to be local the each datacenter.


We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the 
same in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))


Testing a bit with two short program (I can provide them) in go and 
python I notice very strange result. Basically I do the same query 
over and over with a very limited dataset of id.


The first result were surprising cause the very first query were 
always more than 250ms and after with stressing c* (playing with sleep 
between query) I can achieve a good ratio of query at 3/4 ms (what I 
expected).


My guess was that long query were somewhat executed not locally (or at 
least imply multi datacenter queries) and short one no.


Activating tracing in my program (like enalbing trace in cqlsh) kindla 
confirm my suspicion.


(I will provide trace in attachment).

My question is why sometime C* try to read not localy? how we can 
disable it? what is the criteria for this?


(btw I'm very not fan of this multi region design for theses very 
specific kind of issues...)


Also side question: why C* is so slow at connection? it's like it's 
trying to reach every nodes in each DC? (we only provide locals seeds 
however). Sometimes it take more than 20s...


Any help appreciated.

Best,

--

Raphael Mazelier

Re: Understanding multi region read query and latency

[Jim Shaw]

I remember gocql.DataCentreHostFilter was used.  try add it to see whether
will read local DC only in your case ?

Thanks,

James

On Fri, Aug 5, 2022 at 2:40 PM Raphael Mazelier  wrote:

> Hi Cassandra Users,
>
> I'm relatively new to Cassandra and first I have to say I'm really
> impressed by the technology.
>
> Good design and a lot of stuff to understand the underlying (the Oreilly
> book help a lot as well as thelastpickle blog post).
>
> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
> US. Everything deployed in AWS.
>
> We have a keyspace configured with network topology and two replicas on
> every region like this: {'class': 'NetworkTopologyStrategy',
> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>
>
> Investigating some performance issue I noticed strange things in my
> experiment:
>
> What we expect is very slow latency 3/5ms max for this specific select
> query. So we want every read to be local the each datacenter.
>
> We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the same
> in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
>
> Testing a bit with two short program (I can provide them) in go and python
> I notice very strange result. Basically I do the same query over and over
> with a very limited dataset of id.
>
> The first result were surprising cause the very first query were always
> more than 250ms and after with stressing c* (playing with sleep between
> query) I can achieve a good ratio of query at 3/4 ms (what I expected).
>
> My guess was that long query were somewhat executed not locally (or at
> least imply multi datacenter queries) and short one no.
>
> Activating tracing in my program (like enalbing trace in cqlsh) kindla
> confirm my suspicion.
>
> (I will provide trace in attachment).
>
> My question is why sometime C* try to read not localy? how we can disable
> it? what is the criteria for this?
>
> (btw I'm very not fan of this multi region design for theses very specific
> kind of issues...)
>
> Also side question: why C* is so slow at connection? it's like it's trying
> to reach every nodes in each DC? (we only provide locals seeds however).
> Sometimes it take more than 20s...
>
> Any help appreciated.
>
> Best,
>
> --
>
> Raphael Mazelier
>