Re: Write Time of a Row in Multi DC Cassandra Cluster
For tracking delay in very recent versions, this exists: https://issues.apache.org/jira/browse/CASSANDRA-13289 -- Jeff Jirsa > On Jul 11, 2018, at 1:05 AM, Simon Fontana Oscarsson > wrote: > > I thought you just wanted to test how big delay you have, that's why I > suggested trace. > > Your best option is to write with EACH_QUORUM as Alain said. > That way you will get a response when the write is successful on all dcs. > The downside is that the request will fail if one dc is down. > As usual it's up to you to decide if that's acceptable or not. > You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and > rely on hints and repair to get consistency eventually. > > -- > SIMON FONTANA OSCARSSON > Software Developer > > Ericsson > Ölandsgatan 1 > 37133 Karlskrona, Sweden > simon.fontana.oscars...@ericsson.com > www.ericsson.com > >> On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote: >> Simon, >> Trace would be significant burden on the cluster and it has to be on all the >> time. I am trying to find a way to know when a row is written on demand >> basis, is there a way to determine that? >> >> Naidu Saladi >> >> >> On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson >> wrote: >> >> >> Have you tried trace? >> -- >> SIMON FONTANA OSCARSSON >> Software Developer >> >> Ericsson >> Ölandsgatan 1 >> 37133 Karlskrona, Sweden >> simon.fontana.oscars...@ericsson.com >> www.ericsson.com >> >>> On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: >>> Cassandra is an eventual consistent DB, how to find when a row is actually >>> written in multi DC environment? Here is the problem I am trying to solve >>> >>> - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application >>> wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried >>> to read same row from DC2 and could not find >> the >>> row. Our both DC's have sub milli second latency at network level, usually >>> <2 ms. We promised 20 ms consistency. In this case Application could not >>> find the row in DC2 in 50 ms >>> >>> I tried to use "select WRITETIME(authorizations_json) from >>> token_authorizations where " to find when the Row is written in each >>> DC, but both DC's returned same Timestamp. After further >> research >>> I found that Client V3 onwards Timestamp is supplied at Client level so >>> WRITETIME does not help >>> "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/; >>> >>> So how to determine when the row is actually written in each DC? >>> >>> >>> Naidu Saladi >>
Re: Write Time of a Row in Multi DC Cassandra Cluster
I thought you just wanted to test how big delay you have, that's why I suggested trace. Your best option is to write with EACH_QUORUM as Alain said. That way you will get a response when the write is successful on all dcs. The downside is that the request will fail if one dc is down. As usual it's up to you to decide if that's acceptable or not. You could follow up with a LOCAL_QUORUM request if the EACH_QUORUM fails and rely on hints and repair to get consistency eventually. -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden simon.fontana.oscars...@ericsson.com www.ericsson.com On tis, 2018-07-10 at 16:32 +, Saladi Naidu wrote: > Simon, > Trace would be significant burden on the cluster and it has to be on all the > time. I am trying to find a way to know when a row is written on demand > basis, is there a way to determine that? > > Naidu Saladi > > > On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson > wrote: > > > Have you tried trace? > -- > SIMON FONTANA OSCARSSON > Software Developer > > Ericsson > Ölandsgatan 1 > 37133 Karlskrona, Sweden > simon.fontana.oscars...@ericsson.com > www.ericsson.com > > On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: > > Cassandra is an eventual consistent DB, how to find when a row is actually > > written in multi DC environment? Here is the problem I am trying to solve > > > > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application > > wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried > > to read same row from DC2 and could not find > the > > row. Our both DC's have sub milli second latency at network level, usually > > <2 ms. We promised 20 ms consistency. In this case Application could not > > find the row in DC2 in 50 ms > > > > I tried to use "select WRITETIME(authorizations_json) from > > token_authorizations where " to find when the Row is written in each > > DC, but both DC's returned same Timestamp. After further > research > > I found that Client V3 onwards Timestamp is supplied at Client level so > > WRITETIME does not help > > "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/; > > > > So how to determine when the row is actually written in each DC? > > > > > > Naidu Saladi > > smime.p7s Description: S/MIME cryptographic signature
Re: Write Time of a Row in Multi DC Cassandra Cluster
Simon,Trace would be significant burden on the cluster and it has to be on all the time. I am trying to find a way to know when a row is written on demand basis, is there a way to determine that? Naidu Saladi On Tuesday, July 10, 2018 2:24 AM, Simon Fontana Oscarsson wrote: Have you tried trace? -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden simon.fontana.oscars...@ericsson.com www.ericsson.com On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: > Cassandra is an eventual consistent DB, how to find when a row is actually > written in multi DC environment? Here is the problem I am trying to solve > > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application > wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried to > read same row from DC2 and could not find the > row. Our both DC's have sub milli second latency at network level, usually <2 > ms. We promised 20 ms consistency. In this case Application could not find > the row in DC2 in 50 ms > > I tried to use "select WRITETIME(authorizations_json) from > token_authorizations where " to find when the Row is written in each DC, > but both DC's returned same Timestamp. After further research > I found that Client V3 onwards Timestamp is supplied at Client level so > WRITETIME does not help > "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/; > > So how to determine when the row is actually written in each DC? > > > Naidu Saladi
Re: Write Time of a Row in Multi DC Cassandra Cluster
Alain,Thanks for the response and I completely agree with you about your approach but there is a small caveat, we have another DC in Europe, right now this keyspace is not replicating there but eventually will be added. EU DC has significant latency of 200 ms RTT, so going with EACH_QUORUM would not be feasible. We can reset the SLA's for consistency but my question is how to determine when was the row written to remote DC? Is there anyway to determine that Naidu Saladi On Tuesday, July 10, 2018 8:56 AM, Alain RODRIGUEZ wrote: Hello, I have multi DC (3 DC's) Cassandra cluster/ring - One of the application wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried to read same row from DC2 and could not find the row. [...] So how to determine when the row is actually written in each DC? To me, this guarantee you try to achieve could obtained using 'EACH_QUORUM' for writes (ie 'local_quorum' on each DC), and 'LOCAL_QUORUM' for reads for example. You would then have a strong consistency, as long as the same client application is running write then read or that it sends a trigger for the second call sequentially, after validating the write, in some way. Our both DC's have sub milli second latency at network level, usually <2 ms. We promised 20 ms consistency. In this case Application could not find the row in DC2 in 50 ms In these conditions, using 'EACH_QUORUM' might not be too much of a burden for the coordinator and the client. The writes are already being processed, this would increase the latency at the coordinator level (and thus at the client level), but you would be sure that all the clusters have the row in a majority of the replicas before triggering the read. C*heers,---Alain Rodriguez - @arodream - alain@thelastpickle.comFrance / Spain The Last Pickle - Apache Cassandra Consultinghttp://www.thelastpickle.com 2018-07-10 8:24 GMT+01:00 Simon Fontana Oscarsson : Have you tried trace? -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden simon.fontana.oscarsson@ ericsson.com www.ericsson.com On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: > Cassandra is an eventual consistent DB, how to find when a row is actually > written in multi DC environment? Here is the problem I am trying to solve > > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application > wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried to > read same row from DC2 and could not find the > row. Our both DC's have sub milli second latency at network level, usually <2 > ms. We promised 20 ms consistency. In this case Application could not find > the row in DC2 in 50 ms > > I tried to use "select WRITETIME(authorizations_json) from > token_authorizations where " to find when the Row is written in each DC, > but both DC's returned same Timestamp. After further research > I found that Client V3 onwards Timestamp is supplied at Client level so > WRITETIME does not help "https://docs.datastax.com/en/ > developer/java-driver/3.4/ manual/query_timestamps/" > > So how to determine when the row is actually written in each DC? > > > Naidu Saladi
Re: Write Time of a Row in Multi DC Cassandra Cluster
Hello, I have multi DC (3 DC's) Cassandra cluster/ring - One of the application > wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried > to read same row from DC2 and could not find the row. [...] So how to determine when the row is actually written in each DC? To me, this guarantee you try to achieve could obtained using 'EACH_QUORUM' for writes (ie 'local_quorum' on each DC), and 'LOCAL_QUORUM' for reads for example. You would then have a strong consistency, as long as the same client application is running write then read or that it sends a trigger for the second call sequentially, after validating the write, in some way. Our both DC's have sub milli second latency at network level, usually <2 > ms. We promised 20 ms consistency. In this case Application could not find > the row in DC2 in 50 ms > In these conditions, using 'EACH_QUORUM' might not be too much of a burden for the coordinator and the client. The writes are already being processed, this would increase the latency at the coordinator level (and thus at the client level), but you would be sure that all the clusters have the row in a majority of the replicas before triggering the read. C*heers, --- Alain Rodriguez - @arodream - al...@thelastpickle.com France / Spain The Last Pickle - Apache Cassandra Consulting http://www.thelastpickle.com 2018-07-10 8:24 GMT+01:00 Simon Fontana Oscarsson < simon.fontana.oscars...@ericsson.com>: > Have you tried trace? > -- > SIMON FONTANA OSCARSSON > Software Developer > > Ericsson > Ölandsgatan 1 > 37133 Karlskrona, Sweden > simon.fontana.oscars...@ericsson.com > www.ericsson.com > > On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: > > Cassandra is an eventual consistent DB, how to find when a row is > actually written in multi DC environment? Here is the problem I am trying > to solve > > > > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the > application wrote a row to DC1(using Local Quorum) and within span of 50 > ms, it tried to read same row from DC2 and could not find the > > row. Our both DC's have sub milli second latency at network level, > usually <2 ms. We promised 20 ms consistency. In this case Application > could not find the row in DC2 in 50 ms > > > > I tried to use "select WRITETIME(authorizations_json) from > token_authorizations where " to find when the Row is written in each > DC, but both DC's returned same Timestamp. After further research > > I found that Client V3 onwards Timestamp is supplied at Client level so > WRITETIME does not help "https://docs.datastax.com/en/ > developer/java-driver/3.4/manual/query_timestamps/" > > > > So how to determine when the row is actually written in each DC? > > > > > > Naidu Saladi >
Re: Write Time of a Row in Multi DC Cassandra Cluster
Have you tried trace? -- SIMON FONTANA OSCARSSON Software Developer Ericsson Ölandsgatan 1 37133 Karlskrona, Sweden simon.fontana.oscars...@ericsson.com www.ericsson.com On mån, 2018-07-09 at 19:30 +, Saladi Naidu wrote: > Cassandra is an eventual consistent DB, how to find when a row is actually > written in multi DC environment? Here is the problem I am trying to solve > > - I have multi DC (3 DC's) Cassandra cluster/ring - One of the application > wrote a row to DC1(using Local Quorum) and within span of 50 ms, it tried to > read same row from DC2 and could not find the > row. Our both DC's have sub milli second latency at network level, usually <2 > ms. We promised 20 ms consistency. In this case Application could not find > the row in DC2 in 50 ms > > I tried to use "select WRITETIME(authorizations_json) from > token_authorizations where " to find when the Row is written in each DC, > but both DC's returned same Timestamp. After further research > I found that Client V3 onwards Timestamp is supplied at Client level so > WRITETIME does not help > "https://docs.datastax.com/en/developer/java-driver/3.4/manual/query_timestamps/; > > So how to determine when the row is actually written in each DC? > > > Naidu Saladi smime.p7s Description: S/MIME cryptographic signature