> time (UTC) 0 1 2 3 4 5 6 > 7 8 9 10 11 12 13 > Good value 88 44 26 35 26 86 187 251 > 455 389 473 367 453 373 > C* counter 149 82 45 68 38 146 329 414 > 746 566 473 377 453 373 > I finished my Cassandra 1.1.6 upgrades at 9:30 UTC.
This looks like the counters were more out of sync before the upgrade than after? Do you know if your client is retrying counter operations ? (I saw some dropped messages in the S1 log). S1 shows a lot of Commit Log replay going on. Reading your timeline below this sounds like the auto restart catching you out. Cheers ----------------- Aaron Morton Freelance Cassandra Developer New Zealand @aaronmorton http://www.thelastpickle.com On 16/11/2012, at 10:22 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > Here is an example of the increase for some counter (counting events per hour) > > time (UTC) 0 1 2 3 4 5 6 > 7 8 9 10 11 12 13 > Good value 88 44 26 35 26 86 187 251 > 455 389 473 367 453 373 > C* counter 149 82 45 68 38 146 329 414 > 746 566 473 377 453 373 > > I finished my Cassandra 1.1.6 upgrades at 9:30 UTC. > > I found wrong values since the day before at 20:00 UTC (counters from hours > before are good) > > Here are the logs from the output: > Server 1: http://pastebin.com/WyCm6Ef5 (This one is from the same server as > the first bash history on my first mail) > Server 2: http://pastebin.com/gBe2KL2b (This one is from the same server as > the second bash history on my first mail) > > Alain > > 2012/11/15 aaron morton <aa...@thelastpickle.com> > Can you provide an example of the increase ? > > Can you provide the log from startup ? > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Developer > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 16/11/2012, at 3:21 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > >> We had an issue with counters over-counting even using the nodetool drain >> command before upgrading... >> >> Here is my bash history >> >> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >> 70 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak >> 71 sudo apt-get install cassandra >> 72 nodetool disablethrift >> 73 nodetool drain >> 74 service cassandra stop >> 75 cat /etc/cassandra/cassandra-env.sh >> /etc/cassandra/cassandra-env.sh.bak >> 76 vim /etc/cassandra/cassandra-env.sh >> 77 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >> 78 vim /etc/cassandra/cassandra.yaml >> 79 service cassandra start >> >> So I think I followed these steps >> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps >> >> I merged my conf files with an external tool so consider I merged my conf >> files on steps 76 and 78. >> >> I saw that the "sudo apt-get install cassandra" stop the server and restart >> it automatically. So it updated without draining and restart before I had >> the time to reconfigure the conf files. Is this "normal" ? Is there a way to >> avoid it ? >> >> So for the second node I decided to try to stop C*before the upgrade. >> >> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >> 126 cp /etc/cassandra/cassandra-env.sh /etc/cassandra/cassandra-env.sh.bak >> 127 nodetool disablegossip >> 128 nodetool disablethrift >> 129 nodetool drain >> 130 service cassandra stop >> 131 sudo apt-get install cassandra >> >> //131 : This restarted cassandra >> >> 132 nodetool disablethrift >> 133 nodetool disablegossip >> 134 nodetool drain >> 135 service cassandra stop >> 136 cat /etc/cassandra/cassandra-env.sh >> /etc/cassandra/cassandra-env.sh.bak >> 137 cim /etc/cassandra/cassandra-env.sh >> 138 vim /etc/cassandra/cassandra-env.sh >> 139 cat /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >> 140 vim /etc/cassandra/cassandra.yaml >> 141 service cassandra start >> >> After both of these updates I saw my current counters increase without any >> reason. >> >> Did I do anything wrong ? >> >> Alain >> > >
