Ok,
With LWT, I am completely understand that it will achieve total order/linearizability, therefore above scenario cannot occur. However, when you said "the scenario will occur if your clocks are not sync’d". This is ambiguous statement. Because both client and server sides are likely to have different wall-clock for many reasons: 1. Clients and Cassandra server are located in different regions, resulting in different timestamps. 2. Among Cassandra servers (or replicas) is also possible to have different timestamps, so above scenario can occur. Even we use NTP to synchronize the clock, the scenario can happen at least the different in milliseconds. That means that the timestamps for writes are derived either from a single Cassandra server clock, or a single app server clock. These clocks can flow backwards. What do you think? Ibrahim On Mon, Sep 7, 2015 at 1:26 AM, Jeff Jirsa <jeff.ji...@crowdstrike.com> wrote: > > Yes, your scenario can occur, and will occur if your clocks are not sync’d. > > Either you sync your clocks to appropriate tolerances, or you don’t write > without checking the existing value (with LWT). There is no other > resolution in cassandra – there are no vector clocks to allow you to manage > the conflict on your own at this point. > > > From: ibrahim El-sanosi > Reply-To: "user@cassandra.apache.org" > Date: Sunday, September 6, 2015 at 11:28 AM > > To: "user@cassandra.apache.org" > Subject: Re: Is Cassandra really Strong consistency? > > > > > > Yes, LWT is another case and different compared to what my scenario is > about. I am not talking about LWT and CAS, it is true that LWT uses logical > clock by utilising Paxos. But my scenario is talking about using timestamp > and Last-Write-Wins. > > > > > > If anyone can read the above scenario and confirm whether this can occur > or not, if it is possible, how current Cassandra can solve it? > > > > Regards, > > > > Ibrahim > > > > > > If anyone can read the above scenario and confirm whether this can occur > or not, if it is possible, how current Cassandra can solve it? > > > > Regards, > > > > Ibrahim > > On Sun, Sep 6, 2015 at 5:57 PM, Jeff Jirsa <jeff.ji...@crowdstrike.com> > wrote: > >> In the cases where NTP and client timestamps with microsecond resolution >> is insufficient, LWT “IF EXISTS, IF NOT EXISTS” is generally used. >> >> >> From: ibrahim El-sanosi >> Reply-To: "user@cassandra.apache.org" >> Date: Sunday, September 6, 2015 at 7:40 AM >> To: "user@cassandra.apache.org" >> >> Subject: Re: Is Cassandra really Strong consistency? >> >> >> >> I have done some research about “timestamps could jump back and forth >> arbitrarily if you talk to different nodes”. >> >> To summarise, it is possible in Cassandra for following scenario can >> happen in sequence: >> >> >> >> 1. Process A writes w1 with timestamp t=2 >> 2. Process B reads w1 >> 3. Process B writes w2 with timestamp t=1 >> 4. Process B reads w1, but expected w2 >> >> If the system clock goes backwards for any reason, Cassandra’s session >> consistency guarantees no longer hold, even consistency level is write/read >> CL = QOURUM or write CL = ALL and read CL =one. >> >> >> >> Moreover, even we use NTP, the problem above can occur. That means that >> the timestamps for writes are derived either from a single Cassandra server >> clock, or a single app server clock. These clocks can flow backwards, for a >> number of “reasons”: >> >> - *Hardware wonkiness can push clocks days or centuries into the >> future or past.* >> - *Virtualization can wreak havoc on kernel timekeeping.* >> - *Misconfigured nodes may not have NTP enabled, or may not be able >> to reach upstream sources.* >> - *Upstream NTP servers can lie.* >> - *When the problem is identified and fixed, NTP corrects large time >> differentials by jumping the clock discontinously to the correct time.* >> - *Even when perfectly synchronized, POSIX time itself is not >> monotonic*. >> >> >> >> If you want to read more this link can give you a lot hints. >> >> >> >> Regards, >> >> >> >> Ibrahim >> >> On Sun, Sep 6, 2015 at 2:01 PM, Edouard COLE <edouard.c...@rgsystem.com> >> wrote: >> >>> @ibrahim: When saying "clocks should be synchronized", it includes >>> Cassandra nodes AND clients >>> >>> NTP is the way to go >>> >>> Le 6 sept. 2015 à 14:56, Laing, Michael <michael.la...@nytimes.com> a >>> écrit : >>> >>> https://en.wikipedia.org/wiki/Network_Time_Protocol >>> >>> On Sun, Sep 6, 2015 at 8:23 AM, ibrahim El-sanosi < >>> ibrahimsaba...@gmail.com> wrote: >>> >>>> Assume the Cassandra cluster is located in somewhere in US. Clients >>>> that connect from different part of the world will have different timestamp >>>> (if we rely on client timestamp to store write) or If a coordinator is >>>> responsible for generating timestamp during the write, it also may have >>>> different time among replicas, resulting in write conflict can occur and >>>> impossible to resolve. >>>> >>>> >>>> >>>> When you are saying “Clocks should be synchronized”, does Cassandra >>>> synchronize the clock if so how can you refer me to any related article? >>>> >>>> >>>> >>>> Regards, >>>> >>>> >>>> >>>> Ibrahim >>>> >>>> On Sun, Sep 6, 2015 at 1:23 PM, Daniel Schulz < >>>> danielschulz2...@hotmail.com> wrote: >>>> >>>>> Cassandra is not changing clock settings; it does use it to omit >>>>> TTL'ed rows in compaction phases. So make sure your nodes agree on the >>>>> very >>>>> same time using e.g. NTP. It is very crucial for data integrity on most >>>>> distributed systems. >>>>> >>>>> ------------------------------ >>>>> Date: Sun, 6 Sep 2015 13:10:14 +0100 >>>>> Subject: Re: Is Cassandra really Strong consistency? >>>>> From: ibrahimsaba...@gmail.com >>>>> To: user@cassandra.apache.org >>>>> >>>>> >>>>> Do you mean Cassandra does synchronize the clock across all the >>>>> cluster, if yes how it does so, or could you refer me to any related >>>>> article? >>>>> >>>>> Thank you >>>>> >>>>> >>>>> Ibrahim >>>>> >>>>> On Sun, Sep 6, 2015 at 1:00 PM, Laing, Michael < >>>>> michael.la...@nytimes.com> wrote: >>>>> >>>>> I think I saw this before. >>>>> >>>>> Clocks must be synchronized. >>>>> >>>>> On Sun, Sep 6, 2015 at 7:28 AM, ibrahim El-sanosi < >>>>> ibrahimsaba...@gmail.com> wrote: >>>>> >>>>> Hi folks, >>>>> >>>>> Assume we have 4-nodes cluster N1, N2, N3, and N4 and replication >>>>> factor is 3. When write CL =ALL and read CL=ONE: >>>>> >>>>> Client c1 sends W1 = [k1,V1] to N1 (a coordinator). A coordinator >>>>> (N1) generates timestamp Mon 05-09-2015 11:30:40,200 (according to its >>>>> local clock) and assigned it to W1 and sends the W1 to N2, N3, and N4. >>>>> After few seconds, Client c2 sends W2 = [K1, V2] to N4 (a coordinator). A >>>>> coordinator (N4) generates timestamp Mon 05-09-2015 11:30:38,200 >>>>> (according >>>>> to its local clock, but assume here N4 clock a bit behind, nearly 2 >>>>> seconds) and assigned it to W2 and sends the W2 to N2, N3, and N4 >>>>> (itself). >>>>> >>>>> As we have write CL =ALL and read CL = ONE. Now, Client c2 wants to >>>>> read K1, connects to a coordinator N1, a coordinator sends read K1 to N2, >>>>> picking latest timestamp which is [K1, V1]:Mon 05-09-2015 11:30:40,200. >>>>> >>>>> So in this scenario, the latest data that wrote to the replicas is >>>>> [K1, V2] which should be the correct one, but it reads [K1,V1] because of >>>>> divert clock. >>>>> >>>>> Can such scenario occur? >>>>> >>>>> Thank you >>>>> >>>>> >>>>> >>>>> >>>> >>> >> >
