I remember gocql.DataCentreHostFilter was used. try add it to see whether
will read local DC only in your case ?
Thanks,
James
On Fri, Aug 5, 2022 at 2:40 PM Raphael Mazelier <r...@futomaki.net> wrote:
> Hi Cassandra Users,
>
> I'm relatively new to Cassandra and first I have to say I'm really
> impressed by the technology.
>
> Good design and a lot of stuff to understand the underlying (the Oreilly
> book help a lot as well as thelastpickle blog post).
>
> I have an muli-datacenter c* cluster (US, Europe, Singapore) with eight
> node on each (two seeds on each region), two racks on Eu, Singapore, 3 on
> US. Everything deployed in AWS.
>
> We have a keyspace configured with network topology and two replicas on
> every region like this: {'class': 'NetworkTopologyStrategy',
> 'ap-southeast-1': '2', 'eu-west-1': '2', 'us-east-1': '2'}
>
>
> Investigating some performance issue I noticed strange things in my
> experiment:
>
> What we expect is very slow latency 3/5ms max for this specific select
> query. So we want every read to be local the each datacenter.
>
> We configure DCAwareRoundRobinPolicy(local_dc=DC) in python, and the same
> in Go gocql.TokenAwareHostPolicy(gocql.DCAwareRoundRobinPolicy("DC"))
>
> Testing a bit with two short program (I can provide them) in go and python
> I notice very strange result. Basically I do the same query over and over
> with a very limited dataset of id.
>
> The first result were surprising cause the very first query were always
> more than 250ms and after with stressing c* (playing with sleep between
> query) I can achieve a good ratio of query at 3/4 ms (what I expected).
>
> My guess was that long query were somewhat executed not locally (or at
> least imply multi datacenter queries) and short one no.
>
> Activating tracing in my program (like enalbing trace in cqlsh) kindla
> confirm my suspicion.
>
> (I will provide trace in attachment).
>
> My question is why sometime C* try to read not localy? how we can disable
> it? what is the criteria for this?
>
> (btw I'm very not fan of this multi region design for theses very specific
> kind of issues...)
>
> Also side question: why C* is so slow at connection? it's like it's trying
> to reach every nodes in each DC? (we only provide locals seeds however).
> Sometimes it take more than 20s...
>
> Any help appreciated.
>
> Best,
>
> --
>
> Raphael Mazelier
>
