[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13057298#comment-13057298 ] Jim Ancona commented on CASSANDRA-2773: --- We have deployed and tested 0.7.6 plus this patch to the affected cluster. The cluster restarted successfully and the tests that caused the original failure ran successfully. In addition, functional tests of our applications show no regressions. I also reviewed the Cassandra system logs after the testing and saw no errors or obvious problems. Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log, v1-0001-allow-deleting-a-rowand-updating-indexed-columns-init-.txt, v1-0002-CASSANDRA-2773-Add-unit-tests-to-verfy-fix-cherry-pick.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055611#comment-13055611 ] Sylvain Lebresne commented on CASSANDRA-2773: - Actually, there isn't really much risk for data loss given that as Jonathan said, if you hit that, it's fairly easy to go back to 0.7.5, fix client code and upgrade again. Granted this is not user friendly and not something you should expect from a minor upgrade, but let at least set the record straight on the data loss part. That being said, I don't think the patch on this ticket could screw up indexes more that we use to prior to 0.7.6, so maybe we can commit to 0.7.7 on that ground. I'd still suggest fixing client code in the meantime. Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055615#comment-13055615 ] Jonathan Ellis commented on CASSANDRA-2773: --- bq. I don't think the patch on this ticket could screw up indexes more that we use to prior to 0.7.6 That's a valid way to frame the issue. I'm good to commit for 0.7.7 if Jim can test the patch first, since he's the only one we've heard of hitting this in 0.7.x. (Specifically, we want to make sure that if we query WHERE foo = X we don't get results back where foo is something other than X. Ideally you'd start with an empty database, or at least drop + recreate indexes first to make sure the results aren't contaminated w/ corrupt entries from pre-0.7.6.) Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055721#comment-13055721 ] Jim Ancona commented on CASSANDRA-2773: --- I applied the 0.8 patch and added a couple of tests to ColumnFamilyStoreTest. The tests trigger the UnsupportedOperationException in 0.7.6 and return the correct values with the patch applied. Would you like me to test the same scenario against an actual patched server? Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log, v1-0001-allow-deleting-a-rowand-updating-indexed-columns-init-.txt, v1-0002-CASSANDRA-2773-Add-unit-tests-to-verfy-fix-cherry-pick.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13055771#comment-13055771 ] Jonathan Ellis commented on CASSANDRA-2773: --- Thanks for the test case, Jim. If by the same scenario, you mean the workload that left your commitlog throwing exceptions, then yes please. Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log, v1-0001-allow-deleting-a-rowand-updating-indexed-columns-init-.txt, v1-0002-CASSANDRA-2773-Add-unit-tests-to-verfy-fix-cherry-pick.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054582#comment-13054582 ] Jonathan Ellis commented on CASSANDRA-2773: --- I explained on-list that I'm not comfortable committing this to 0.7. The critical thing we need to deliver in 0.7 at this stage is stability. The rarity of the problem combined with the complexity of the code involved leads me to conclude that it's better to live with a bug we know how to avoid, than risk introducing new ones. Remember, the risk of new bugs affects _everyone_, while the fix can only benefit those who were generating the unusual mutation pattern here. It's our responsibility to take a balanced view. For the rare people who do find themselves affected here, your options include - stay on 0.7.6 - drain the commitlog with an earlier version before upgrading, then re-upgrade to 0.7.6 after fixing your code to not generate problematic mutations - run an unofficial build with this patch included - upgrade to 0.8.2 when released Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13054630#comment-13054630 ] Jim Ancona commented on CASSANDRA-2773: --- I disagree. This is a bug which allows a client to make a cluster unresponsive by performing a seemingly innocuous series of operations. If that happens, the cluster is un-restartable without loss of data. I wouldn't call a release where this can occur stable. So if the goal for 0.7 is stability... WRT fixing your code to not generate problematic mutations, this may be difficult to do. I have so far identified code that does deletes followed by updates in the same mutation, but I haven't yet found any updates followed by deletes. Are we sure that only the update-followed-by-delete scenario is problematic? In any case, even after reviewing all our code for the relevant scenarios, I would not feel comfortable deploying an 0.7 release with this vulnerability to production. The risk of a catastrophic failure is too great. Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Critical Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt, cassandra.log I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053941#comment-13053941 ] Jonathan Ellis commented on CASSANDRA-2773: --- Committed, with comment. (I wish CF objects could be immutable AND efficient...) Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Minor Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13053989#comment-13053989 ] Hudson commented on CASSANDRA-2773: --- Integrated in Cassandra-0.8 #187 (See [https://builds.apache.org/job/Cassandra-0.8/187/]) allow deleting a rowand updating indexed columns init in the same mutation patch by jbellis; reviewed by slebresne for CASSANDRA-2773 jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1138959 Files : * /cassandra/branches/cassandra-0.8/CHANGES.txt * /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/Table.java Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Minor Fix For: 0.8.2 Attachments: 2773-v2.txt, 2773.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2773) Index manager cannot support deleting and inserting into a row in the same mutation
[ https://issues.apache.org/jira/browse/CASSANDRA-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13051871#comment-13051871 ] Sylvain Lebresne commented on CASSANDRA-2773: - Hum, we cannot remove the column from cf in ignoreObsoleteMutations() because cf is the original column family from the row mutation and that's racy with commit log write (à la CASSANDRA-2604). We should clone the column family, but maybe it's simpler to add validation logic after all ? In any case, it could be worth it adding some comment in Table.apply() or Table.ignoreObsoleteMutations(). Index manager cannot support deleting and inserting into a row in the same mutation - Key: CASSANDRA-2773 URL: https://issues.apache.org/jira/browse/CASSANDRA-2773 Project: Cassandra Issue Type: Bug Affects Versions: 0.7.0 Reporter: Boris Yen Assignee: Jonathan Ellis Priority: Minor Fix For: 0.8.2 Attachments: 2773.txt I use hector 0.8.0-1 and cassandra 0.8. 1. create mutator by using hector api, 2. Insert a few columns into the mutator for key key1, cf standard. 3. add a deletion to the mutator to delete the record of key1, cf standard. 4. repeat 2 and 3 5. execute the mutator. the result: the connection seems to be held by the sever forever, it never returns. when I tried to restart the cassandra I saw unsupportedexception : Index manager cannot support deleting and inserting into a row in the same mutation. and the cassandra is dead forever, unless I delete the commitlog. I would expect to get an exception when I execute the mutator, not after I restart the cassandra. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
