Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-11 Thread Jeff Jirsa
For tracking delay in very recent versions, this exists: 
https://issues.apache.org/jira/browse/CASSANDRA-13289



-- 
Jeff Jirsa


> On Jul 11, 2018, at 1:05 AM, Simon Fontana Oscarsson 
>  wrote:
> 
> I thought you just wanted to test how big delay you have, that's why I 
> suggested trace.
> 
> Your best option is to write with EACH_QUORUM as Alain said.
> That way you will get a response when the write is successful on all dcs.
> The downside is that the request will fail if one dc is down.
> As usual it's up to you to decide if that's acceptable or not.
> You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and 
> rely on hints and repair to get consistency eventually.
> 
> -- 
> SIMON FONTANA OSCARSSON
> Software Developer
> 
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, Sweden
> simon.fontana.oscars...@ericsson.com
> www.ericsson.com
> 
>> On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote:
>> Simon,
>> Trace would be significant burden on the cluster and it has to be on all the 
>> time. I am trying to find a way to know when a row is written on demand 
>> basis, is there a way to determine that?
>>  
>> Naidu Saladi 
>> 
>> 
>> On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson 
>>  wrote:
>> 
>> 
>> Have you tried trace?
>> -- 
>> SIMON FONTANA OSCARSSON
>> Software Developer
>> 
>> Ericsson
>> Ölandsgatan 1
>> 37133 Karlskrona, Sweden
>> simon.fontana.oscars...@ericsson.com
>> www.ericsson.com
>> 
>>> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
>>> Cassandra is an eventual consistent DB, how to find when a row is actually 
>>> written in multi DC environment? Here is the problem I am trying to solve 
>>>  
>>> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
>>> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried 
>>> to read same row from DC2 and could not find
>> the
>>> row. Our both DC's have sub milli second latency at network level, usually 
>>> <2 ms. We promised 20 ms consistency. In this case Application could not 
>>> find the row in DC2 in 50 ms
>>>  
>>> I tried to use "select WRITETIME(authorizations_json) from 
>>> token_authorizations where " to find  when the Row is written in each 
>>> DC, but both DC's returned same Timestamp. After further
>> research
>>> I found that Client V3 onwards Timestamp is supplied at Client level so 
>>> WRITETIME does not help 
>>> "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
>>>  
>>> So how to determine when the row is actually written in each DC?
>>>  
>>>  
>>> Naidu Saladi 
>> 


Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-11 Thread Simon Fontana Oscarsson
I thought you just wanted to test how big delay you have, that's why I 
suggested trace.

Your best option is to write with EACH_QUORUM as Alain said.
That way you will get a response when the write is successful on all dcs.
The downside is that the request will fail if one dc is down.
As usual it's up to you to decide if that's acceptable or not.
You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and 
rely on hints and repair to get consistency eventually.

-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com

On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote:
> Simon,
> Trace would be significant burden on the cluster and it has to be on all the 
> time. I am trying to find a way to know when a row is written on demand 
> basis, is there a way to determine that?
>  
> Naidu Saladi 
> 
> 
> On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson 
>  wrote:
> 
> 
> Have you tried trace?
> -- 
> SIMON FONTANA OSCARSSON
> Software Developer
> 
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, Sweden
> simon.fontana.oscars...@ericsson.com
> www.ericsson.com
> 
> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> > Cassandra is an eventual consistent DB, how to find when a row is actually 
> > written in multi DC environment? Here is the problem I am trying to solve 
> > 
> > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
> > wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried 
> > to read same row from DC2 and could not find
> the
> > row. Our both DC's have sub milli second latency at network level, usually 
> > <2 ms. We promised 20 ms consistency. In this case Application could not 
> > find the row in DC2 in 50 ms
> > 
> > I tried to use "select WRITETIME(authorizations_json) from 
> > token_authorizations where " to find  when the Row is written in each 
> > DC, but both DC's returned same Timestamp. After further
> research
> > I found that Client V3 onwards Timestamp is supplied at Client level so 
> > WRITETIME does not help 
> > "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
> > 
> > So how to determine when the row is actually written in each DC?
> > 
> >  
> > Naidu Saladi 
> 
> 

smime.p7s
Description: S/MIME cryptographic signature


Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-10 Thread Saladi Naidu
Simon,Trace would be significant burden on the cluster and it has to be on all 
the time. I am trying to find a way to know when a row is written on demand 
basis, is there a way to determine that? Naidu Saladi 
 

On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson 
 wrote:
 

 Have you tried trace?
-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com

On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> Cassandra is an eventual consistent DB, how to find when a row is actually 
> written in multi DC environment? Here is the problem I am trying to solve 
> 
> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried to 
> read same row from DC2 and could not find the
> row. Our both DC's have sub milli second latency at network level, usually <2 
> ms. We promised 20 ms consistency. In this case Application could not find 
> the row in DC2 in 50 ms
> 
> I tried to use "select WRITETIME(authorizations_json) from 
> token_authorizations where " to find  when the Row is written in each DC, 
> but both DC's returned same Timestamp. After further research
> I found that Client V3 onwards Timestamp is supplied at Client level so 
> WRITETIME does not help 
> "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
> 
> So how to determine when the row is actually written in each DC?
> 
>  
> Naidu Saladi 

   

Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-10 Thread Saladi Naidu
Alain,Thanks for the response and I completely agree with you about your 
approach but there is a small caveat, we have another DC in Europe, right now 
this keyspace is not replicating there but eventually will be added. EU DC has 
significant latency of 200 ms RTT, so going with EACH_QUORUM would not be 
feasible. We can reset the SLA's for consistency but my question is how to 
determine when was the row written to remote DC? Is there anyway to determine 
that Naidu Saladi 
 

On Tuesday, July 10, 2018 8:56 AM, Alain RODRIGUEZ  
wrote:
 

 Hello,

 I have multi DC (3 DC's) Cassandra cluster/ring - One of the application wrote 
a row to DC1(using Local Quorum)  and within span of 50 ms, it tried to read 
same row from DC2 and could not find the row.

 [...]

So how to determine when the row is actually written in each DC? 

To me, this guarantee you try to achieve could obtained using 'EACH_QUORUM' for 
writes (ie 'local_quorum' on each DC), and 'LOCAL_QUORUM' for reads for 
example. You would then have a strong consistency, as long as the same client 
application is running write then read or that it sends a trigger for the 
second call sequentially, after validating the write, in some way.

Our both DC's have sub milli second latency at network level, usually <2 ms. We 
promised 20 ms consistency. In this case Application could not find the row in 
DC2 in 50 ms


In these conditions, using 'EACH_QUORUM' might not be too much of a burden for 
the coordinator and the client. The writes are already being processed, this 
would increase the latency at the coordinator level (and thus at the client 
level), but you would be sure that all the clusters have the row in a majority 
of the replicas before triggering the read.
C*heers,---Alain Rodriguez - @arodream - 
alain@thelastpickle.comFrance / Spain
The Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com

2018-07-10 8:24 GMT+01:00 Simon Fontana Oscarsson 
:

Have you tried trace?
-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscarsson@ ericsson.com
www.ericsson.com

On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> Cassandra is an eventual consistent DB, how to find when a row is actually 
> written in multi DC environment? Here is the problem I am trying to solve 
> 
> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried to 
> read same row from DC2 and could not find the
> row. Our both DC's have sub milli second latency at network level, usually <2 
> ms. We promised 20 ms consistency. In this case Application could not find 
> the row in DC2 in 50 ms
> 
> I tried to use "select WRITETIME(authorizations_json) from 
> token_authorizations where " to find  when the Row is written in each DC, 
> but both DC's returned same Timestamp. After further research
> I found that Client V3 onwards Timestamp is supplied at Client level so 
> WRITETIME does not help "https://docs.datastax.com/en/ 
> developer/java-driver/3.4/ manual/query_timestamps/"
> 
> So how to determine when the row is actually written in each DC?
> 
>  
> Naidu Saladi 



   

Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-10 Thread Alain RODRIGUEZ
Hello,

 I have multi DC (3 DC's) Cassandra cluster/ring - One of the application
> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried
> to read same row from DC2 and could not find the row.

 [...]

So how to determine when the row is actually written in each DC?


To me, this guarantee you try to achieve could obtained using 'EACH_QUORUM'
for writes (ie 'local_quorum' on each DC), and 'LOCAL_QUORUM' for reads for
example. You would then have a strong consistency, as long as the same
client application is running write then read or that it sends a trigger
for the second call sequentially, after validating the write, in some way.

Our both DC's have sub milli second latency at network level, usually <2
> ms. We promised 20 ms consistency. In this case Application could not find
> the row in DC2 in 50 ms
>

In these conditions, using 'EACH_QUORUM' might not be too much of a burden
for the coordinator and the client. The writes are already being processed,
this would increase the latency at the coordinator level (and thus at the
client level), but you would be sure that all the clusters have the row in
a majority of the replicas before triggering the read.

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com


2018-07-10 8:24 GMT+01:00 Simon Fontana Oscarsson <
simon.fontana.oscars...@ericsson.com>:

> Have you tried trace?
> --
> SIMON FONTANA OSCARSSON
> Software Developer
>
> Ericsson
> Ölandsgatan 1
> 37133 Karlskrona, Sweden
> simon.fontana.oscars...@ericsson.com
> www.ericsson.com
>
> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> > Cassandra is an eventual consistent DB, how to find when a row is
> actually written in multi DC environment? Here is the problem I am trying
> to solve
> >
> > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the
> application wrote a row to DC1(using Local Quorum)  and within span of 50
> ms, it tried to read same row from DC2 and could not find the
> > row. Our both DC's have sub milli second latency at network level,
> usually <2 ms. We promised 20 ms consistency. In this case Application
> could not find the row in DC2 in 50 ms
> >
> > I tried to use "select WRITETIME(authorizations_json) from
> token_authorizations where " to find  when the Row is written in each
> DC, but both DC's returned same Timestamp. After further research
> > I found that Client V3 onwards Timestamp is supplied at Client level so
> WRITETIME does not help "https://docs.datastax.com/en/
> developer/java-driver/3.4/manual/query_timestamps/"
> >
> > So how to determine when the row is actually written in each DC?
> >
> >
> > Naidu Saladi
>


Re: Write Time of a Row in Multi DC Cassandra Cluster

2018-07-10 Thread Simon Fontana Oscarsson
Have you tried trace?
-- 
SIMON FONTANA OSCARSSON
Software Developer

Ericsson
Ölandsgatan 1
37133 Karlskrona, Sweden
simon.fontana.oscars...@ericsson.com
www.ericsson.com

On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote:
> Cassandra is an eventual consistent DB, how to find when a row is actually 
> written in multi DC environment? Here is the problem I am trying to solve 
> 
> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application 
> wrote a row to DC1(using Local Quorum)  and within span of 50 ms, it tried to 
> read same row from DC2 and could not find the
> row. Our both DC's have sub milli second latency at network level, usually <2 
> ms. We promised 20 ms consistency. In this case Application could not find 
> the row in DC2 in 50 ms
> 
> I tried to use "select WRITETIME(authorizations_json) from 
> token_authorizations where " to find  when the Row is written in each DC, 
> but both DC's returned same Timestamp. After further research
> I found that Client V3 onwards Timestamp is supplied at Client level so 
> WRITETIME does not help 
> "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/;
> 
> So how to determine when the row is actually written in each DC?
> 
>  
> Naidu Saladi 

smime.p7s
Description: S/MIME cryptographic signature