On Fri, Nov 21, 2014 at 3:11 AM, André Cruz <andre.c...@co.sapo.pt> wrote:
> Can it be that they were all in the middle of a compaction (Leveled > compaction) and the new sstables were written but the old ones were not > deleted? Will Cassandra blindly pick up old and new sstables when it > restarts? > Yes. https://issues.apache.org/jira/browse/CASSANDRA-6756 Also note linked tickets there like : https://issues.apache.org/jira/browse/CASSANDRA-6503 6503 is fix version 1.2.14 ... > 1- What is the correct sequence of commands to bring down a node safely? I > know that “drain" was used here, because it is in the log. I’ve read > somewhere that drain should not be used and “disablethrift”, > “disablegossip”, “flush” and then waiting a while is the correct way. > "drain" is the canonical answer. But "drain" has historically not always worked. In general when it hasn't worked, it has flushed properly but not marked clean properly. https://issues.apache.org/jira/browse/CASSANDRA-4446 https://issues.apache.org/jira/browse/CASSANDRA-5911 5911 is "too late for 1.2" and will not be merged there. > 2- Why won’t repair propagate this column value to the other nodes? > Repairs have run everyday and the value is still missing on the other nodes. > No idea. Are you sure it's not expired via TTL or masked in some other way? When you ask that node for it at CL.ONE, do you get this value? > 3- If I have these rogue sstables loaded this seems like a time bomb. Down > the line I will again delete columns that will reappear after some time. Is > there a way I can find these sstables that should not be there? I thought > the timestamp of the file would help but this zombie column is present on > one of the latest sstables. > Unfortunately, not really. You'd need the patch from CASSANDRA-6568 and you'd need to continuously have been capturing your live SStable set to determine when https://issues.apache.org/jira/browse/CASSANDRA-6568 The experience you're having is, AFAICT, irrecoverably fatal to consistency. This is why, at the summit this year, when Apple announced it had encountered and fixed like 5 bugs of this type, I summarized their talk in chats as "the talk where Apple showed that, despite what you've heard about quorum reads and writes, Cassandra has never stored data consistently except by fortunate accident." =Rob http://twitter.com/rcolidba
