Re: Column value not getting updated
Interesting that you are finding excessive drift from public time servers. I only once saw that problem with AWS' time servers. To be conservative I sometimes recommend that clients spool up their own time server, but realize IT will also drift if the public time servers do! Somewhat different if in your own DC, but same time server drift issues. Google has resorted to putting tier one time server(s) (cesium clock or whatever) in every data center due to the public drift issues. Does anyone know if AWS' time server is now stratum 1 backed? However, it is better to have two (at least) in AWS, make sure their private IP's are not in the same 24 CIDR subnet! Of course this can get troublesome if load sharing between e.g. AWS East and West. *...* *“Life should not be a journey to the grave with the intention of arriving safely in apretty and well preserved body, but rather to skid in broadside in a cloud of smoke,thoroughly used up, totally worn out, and loudly proclaiming “Wow! What a Ride!” - Hunter ThompsonDaemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* On Tue, Mar 31, 2015 at 10:49 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: Thanks Mark. A great post indeed and saved me a lot of trouble. - Saurabh From: Mark Greene green...@gmail.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Tuesday, March 31, 2015 at 10:15 PM To: user@cassandra.apache.org user@cassandra.apache.org Subject: Re: Column value not getting updated Hey Saurabh, We're actually preparing for this ourselves and spinning up our own NTP server pool. The public NTP pools have a lot of drift and should not be relied upon for cluster technology that is sensitive to time skew like C*. The folks at Logentries did a great write up about this which we used as a guide. - https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-1-the-problem/ - https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-2-solutions/ -Mark On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: That’s what I found out that the clocks were not in sync. But I have setup NTP on all 3 nodes and would expect the clocks to be in sync. From: Nate McCall n...@thelastpickle.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Tuesday, March 31, 2015 at 2:50 PM To: Cassandra Users user@cassandra.apache.org Subject: Re: Column value not getting updated You would see that if the servers' clocks were out of sync. Make sure the time on the servers is in sync or set the client timestamps explicitly. On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. - I am using QueryBuilder API and not CQL directly. - I am using the consistency level of QUORUM for everything – insert, update and read. - Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Column value not getting updated
Hey Saurabh, We're actually preparing for this ourselves and spinning up our own NTP server pool. The public NTP pools have a lot of drift and should not be relied upon for cluster technology that is sensitive to time skew like C*. The folks at Logentries did a great write up about this which we used as a guide. - https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-1-the-problem/ - https://blog.logentries.com/2014/03/synchronizing-clocks-in-a-cassandra-cluster-pt-2-solutions/ -Mark On Tue, Mar 31, 2015 at 5:59 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: That’s what I found out that the clocks were not in sync. But I have setup NTP on all 3 nodes and would expect the clocks to be in sync. From: Nate McCall n...@thelastpickle.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Tuesday, March 31, 2015 at 2:50 PM To: Cassandra Users user@cassandra.apache.org Subject: Re: Column value not getting updated You would see that if the servers' clocks were out of sync. Make sure the time on the servers is in sync or set the client timestamps explicitly. On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. - I am using QueryBuilder API and not CQL directly. - I am using the consistency level of QUORUM for everything – insert, update and read. - Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Column value not getting updated
You would see that if the servers' clocks were out of sync. Make sure the time on the servers is in sync or set the client timestamps explicitly. On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. - I am using QueryBuilder API and not CQL directly. - I am using the consistency level of QUORUM for everything – insert, update and read. - Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Column value not getting updated
It's not enough to set up ntp, you're going to need to force the time to sync. ntp is only meant to correct for drift. You can either use ntpdate or I think there's a flag for ntpd (that I can't remember and am in a rush out the door) that you can use to force it to adjust to the correct time. On Tue, Mar 31, 2015 at 3:02 PM Saurabh Sethi saurabh_se...@symantec.com wrote: That’s what I found out that the clocks were not in sync. But I have setup NTP on all 3 nodes and would expect the clocks to be in sync. From: Nate McCall n...@thelastpickle.com Reply-To: user@cassandra.apache.org user@cassandra.apache.org Date: Tuesday, March 31, 2015 at 2:50 PM To: Cassandra Users user@cassandra.apache.org Subject: Re: Column value not getting updated You would see that if the servers' clocks were out of sync. Make sure the time on the servers is in sync or set the client timestamps explicitly. On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi saurabh_se...@symantec.com wrote: I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. - I am using QueryBuilder API and not CQL directly. - I am using the consistency level of QUORUM for everything – insert, update and read. - Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Re: Column value not getting updated
That’s what I found out that the clocks were not in sync. But I have setup NTP on all 3 nodes and would expect the clocks to be in sync. From: Nate McCall n...@thelastpickle.commailto:n...@thelastpickle.com Reply-To: user@cassandra.apache.orgmailto:user@cassandra.apache.org user@cassandra.apache.orgmailto:user@cassandra.apache.org Date: Tuesday, March 31, 2015 at 2:50 PM To: Cassandra Users user@cassandra.apache.orgmailto:user@cassandra.apache.org Subject: Re: Column value not getting updated You would see that if the servers' clocks were out of sync. Make sure the time on the servers is in sync or set the client timestamps explicitly. On Tue, Mar 31, 2015 at 3:23 PM, Saurabh Sethi saurabh_se...@symantec.commailto:saurabh_se...@symantec.com wrote: I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. * I am using QueryBuilder API and not CQL directly. * I am using the consistency level of QUORUM for everything – insert, update and read. * Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh -- - Nate McCall Austin, TX @zznate Co-Founder Sr. Technical Consultant Apache Cassandra Consulting http://www.thelastpickle.com
Column value not getting updated
I have written a unit test that creates a column family, inserts a row in that column family and then updates the value of one of the columns. After updating, unit test immediately tries to read the updated value for that column, but Cassandra returns the old value. * I am using QueryBuilder API and not CQL directly. * I am using the consistency level of QUORUM for everything – insert, update and read. * Cassandra is running as a 3 node cluster with replication factor of 3. Anyone has any idea what is going on here? Thanks, Saurabh
