Alain, My understanding is that drain ensures that all memtables are flushed, so that there is no data in the commitlog that is isn't in an sstable. A marker is saved that indicates the commit logs should not be replayed. Commitlogs are only removed from disk periodically (after commitlog_total_space_in_mb is exceeded?).
With 1.1.5/6, all nanotime commitlogs are replayed on startup regardless of whether they've been flushed. So in our case manually removing all the commitlogs after a drain was the only way to prevent their replay. Mike On Tue, Nov 20, 2012 at 5:19 AM, Alain RODRIGUEZ <arodr...@gmail.com> wrote: > @Mike > > I am glad to see I am not the only one with this issue (even if I am sorry > it happened to you of course.). > > Isn't drain supposed to clear the commit logs ? Did removing them worked > properly ? > > I his warning to C* users, Jonathan Ellis told that a drain would avoid > this issue, It seems like it doesn't. > > @Rob > > You understood precisely the 2 issues I met during the upgrade. I am sad > to see none of them is yet resolved and probably wont. > > > 2012/11/20 Mike Heffner <m...@librato.com> > >> Alain, >> >> We performed a 1.1.3 -> 1.1.6 upgrade and found that all the logs >> replayed regardless of the drain. After noticing this on the first node, we >> did the following: >> >> * nodetool flush >> * nodetool drain >> * service cassandra stop >> * mv /path/to/logs/*.log /backup/ >> * apt-get install cassandra >> <restarts automatically> >> >> I also agree that starting C* after an upgrade/install seems quite broken >> if it was already stopped before the install. However annoying, I have >> found this to be the default for most Ubuntu daemon packages. >> >> Mike >> >> >> On Thu, Nov 15, 2012 at 9:21 AM, Alain RODRIGUEZ <arodr...@gmail.com>wrote: >> >>> We had an issue with counters over-counting even using the nodetool >>> drain command before upgrading... >>> >>> Here is my bash history >>> >>> 69 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >>> 70 cp /etc/cassandra/cassandra-env.sh >>> /etc/cassandra/cassandra-env.sh.bak >>> 71 sudo apt-get install cassandra >>> 72 nodetool disablethrift >>> 73 nodetool drain >>> 74 service cassandra stop >>> 75 cat /etc/cassandra/cassandra-env.sh >>> /etc/cassandra/cassandra-env.sh.bak >>> 76 vim /etc/cassandra/cassandra-env.sh >>> 77 cat /etc/cassandra/cassandra.yaml >>> /etc/cassandra/cassandra.yaml.bak >>> 78 vim /etc/cassandra/cassandra.yaml >>> 79 service cassandra start >>> >>> So I think I followed these steps >>> http://www.datastax.com/docs/1.1/install/upgrading#upgrade-steps >>> >>> I merged my conf files with an external tool so consider I merged my >>> conf files on steps 76 and 78. >>> >>> I saw that the "sudo apt-get install cassandra" stop the server and >>> restart it automatically. So it updated without draining and restart before >>> I had the time to reconfigure the conf files. Is this "normal" ? Is there a >>> way to avoid it ? >>> >>> So for the second node I decided to try to stop C*before the upgrade. >>> >>> 125 cp /etc/cassandra/cassandra.yaml /etc/cassandra/cassandra.yaml.bak >>> 126 cp /etc/cassandra/cassandra-env.sh >>> /etc/cassandra/cassandra-env.sh.bak >>> 127 nodetool disablegossip >>> 128 nodetool disablethrift >>> 129 nodetool drain >>> 130 service cassandra stop >>> 131 sudo apt-get install cassandra >>> >>> //131 : This restarted cassandra >>> >>> 132 nodetool disablethrift >>> 133 nodetool disablegossip >>> 134 nodetool drain >>> 135 service cassandra stop >>> 136 cat /etc/cassandra/cassandra-env.sh >>> /etc/cassandra/cassandra-env.sh.bak >>> 137 cim /etc/cassandra/cassandra-env.sh >>> 138 vim /etc/cassandra/cassandra-env.sh >>> 139 cat /etc/cassandra/cassandra.yaml >>> /etc/cassandra/cassandra.yaml.bak >>> 140 vim /etc/cassandra/cassandra.yaml >>> 141 service cassandra start >>> >>> After both of these updates I saw my current counters increase without >>> any reason. >>> >>> Did I do anything wrong ? >>> >>> Alain >>> >>> >> >> >> -- >> >> Mike Heffner <m...@librato.com> >> Librato, Inc. >> >> >> > -- Mike Heffner <m...@librato.com> Librato, Inc.
