Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
Hello, After a 'DROP TABLE' command that returns errors={}, last_host=127.0.0.1 (like most DROP commands do) from CQLSH with C* 2.1.0-rc2, I stopped C*. And I can not start one node. It says : ERROR 09:18:34 Exception encountered during startup java.lang.NullPointerException: null at org.apache.cassandra.db.Directories.init(Directories.java:191) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:553) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:455) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] java.lang.NullPointerException at org.apache.cassandra.db.Directories.init(Directories.java:191) at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:553) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:455) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544) Exception encountered during startup: null I do not now if it can help. Fabrice LARCHER 2014-07-18 7:23 GMT+02:00 Fabrice Larcher fabrice.larc...@level5.fr: Hello, I still experience a similar issue after a 'DROP KEYSPACE' command with C* 2.1-rc3. Connection to the node may fail after a 'DROP'. But I did not see this issue with 2.1-rc1 (- it seems like to be a regression brought with 2.1-rc2). Fabrice LARCHER 2014-07-17 9:19 GMT+02:00 Benedict Elliott Smith belliottsm...@datastax.com: Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the problem persists? You may see the last exception, but it shouldn't affect the stability of the cluster. If either of the other exceptions persist, please file a ticket. On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs ty...@datastax.com wrote: This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959, but that was fixed for 2.1.0-rc1. Is there any chance you can put together a script to reproduce the issue? On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan pavel.ko...@cortica.com wrote: It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
It would be nice if you can try with 2.1.0-rc5 (there has been quite a bit of bug fixes since rc2). If you can still reproduce that NPE there, please do open a jira ticket with the reproduction steps. On Thu, Aug 7, 2014 at 11:29 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, After a 'DROP TABLE' command that returns errors={}, last_host=127.0.0.1 (like most DROP commands do) from CQLSH with C* 2.1.0-rc2, I stopped C*. And I can not start one node. It says : ERROR 09:18:34 Exception encountered during startup java.lang.NullPointerException: null at org.apache.cassandra.db.Directories.init(Directories.java:191) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:553) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:455) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] java.lang.NullPointerException at org.apache.cassandra.db.Directories.init(Directories.java:191) at org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:553) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:455) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:544) Exception encountered during startup: null I do not now if it can help. Fabrice LARCHER 2014-07-18 7:23 GMT+02:00 Fabrice Larcher fabrice.larc...@level5.fr: Hello, I still experience a similar issue after a 'DROP KEYSPACE' command with C* 2.1-rc3. Connection to the node may fail after a 'DROP'. But I did not see this issue with 2.1-rc1 (- it seems like to be a regression brought with 2.1-rc2). Fabrice LARCHER 2014-07-17 9:19 GMT+02:00 Benedict Elliott Smith belliottsm...@datastax.com: Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the problem persists? You may see the last exception, but it shouldn't affect the stability of the cluster. If either of the other exceptions persist, please file a ticket. On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs ty...@datastax.com wrote: This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959, but that was fixed for 2.1.0-rc1. Is there any chance you can put together a script to reproduce the issue? On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan pavel.ko...@cortica.com wrote: It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2]
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the problem persists? You may see the last exception, but it shouldn't affect the stability of the cluster. If either of the other exceptions persist, please file a ticket. On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs ty...@datastax.com wrote: This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959, but that was fixed for 2.1.0-rc1. Is there any chance you can put together a script to reproduce the issue? On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan pavel.ko...@cortica.com wrote: It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55] Then, I can not connect to the Cluster anymore from my app (Java Driver 2.1-SNAPSHOT) and got in application logs : com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) (...) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I can still connect through CQLSH but if I run (again) a DROP KEYSPACE command from CQLSH, I get the following error : errors={}, last_host=127.0.0.1 Now, on a 2 nodes cluster I also have a similar issue but the error's stacktrace is different
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
Hello, I still experience a similar issue after a 'DROP KEYSPACE' command with C* 2.1-rc3. Connection to the node may fail after a 'DROP'. But I did not see this issue with 2.1-rc1 (- it seems like to be a regression brought with 2.1-rc2). Fabrice LARCHER 2014-07-17 9:19 GMT+02:00 Benedict Elliott Smith belliottsm...@datastax.com : Also https://issues.apache.org/jira/browse/CASSANDRA-7437 and https://issues.apache.org/jira/browse/CASSANDRA-7465 for rc3, although the CounterCacheKey assertion looks like an independent (though comparatively benign) bug I will file a ticket for. Can you try this against rc3 to see if the problem persists? You may see the last exception, but it shouldn't affect the stability of the cluster. If either of the other exceptions persist, please file a ticket. On Thu, Jul 17, 2014 at 1:41 AM, Tyler Hobbs ty...@datastax.com wrote: This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959, but that was fixed for 2.1.0-rc1. Is there any chance you can put together a script to reproduce the issue? On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan pavel.ko...@cortica.com wrote: It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55] Then, I can not connect to the Cluster anymore from my app (Java Driver 2.1-SNAPSHOT) and got in application logs : com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) (...) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
This looks like https://issues.apache.org/jira/browse/CASSANDRA-6959, but that was fixed for 2.1.0-rc1. Is there any chance you can put together a script to reproduce the issue? On Thu, Jul 10, 2014 at 8:51 AM, Pavel Kogan pavel.ko...@cortica.com wrote: It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55] Then, I can not connect to the Cluster anymore from my app (Java Driver 2.1-SNAPSHOT) and got in application logs : com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) (...) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I can still connect through CQLSH but if I run (again) a DROP KEYSPACE command from CQLSH, I get the following error : errors={}, last_host=127.0.0.1 Now, on a 2 nodes cluster I also have a similar issue but the error's stacktrace is different : From application logs : 17971 [Cassandra Java Driver worker-2] WARN com.datastax.driver.core.Cluster - No schema agreement from live replicas after 1 ms. The schema may not be up to date on some nodes. From system.log : INFO [SharedPool-Worker-2] 2014-07-10 09:04:53,434 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MigrationStage:1] 2014-07-10 09:04:56,553 CommitLogSegmentManager.java:304 - Failed waiting for a forced recycle of in-use commit log segments java.lang.AssertionError: null
C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55] Then, I can not connect to the Cluster anymore from my app (Java Driver 2.1-SNAPSHOT) and got in application logs : com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) (...) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I can still connect through CQLSH but if I run (again) a DROP KEYSPACE command from CQLSH, I get the following error : errors={}, last_host=127.0.0.1 Now, on a 2 nodes cluster I also have a similar issue but the error's stacktrace is different : From application logs : 17971 [Cassandra Java Driver worker-2] WARN com.datastax.driver.core.Cluster - No schema agreement from live replicas after 1 ms. The schema may not be up to date on some nodes. From system.log : INFO [SharedPool-Worker-2] 2014-07-10 09:04:53,434 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MigrationStage:1] 2014-07-10 09:04:56,553 CommitLogSegmentManager.java:304 - Failed waiting for a forced recycle of in-use commit log segments java.lang.AssertionError: null at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecycleAll(CommitLogSegmentManager.java:299) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:160) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.DefsTables.dropColumnFamily(DefsTables.java:516) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.DefsTables.mergeColumnFamilies(DefsTables.java:300) [apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at
Re: C* 2.1-rc2 gets unstable after a 'DROP KEYSPACE' command ?
It seems that memtable tries to flush itself to SSTable of not existing keyspace. I don't know why it is happens, but probably running nodetool flush before drop should prevent this issue. Pavel On Thu, Jul 10, 2014 at 4:09 AM, Fabrice Larcher fabrice.larc...@level5.fr wrote: Hello, I am using the 'development' version 2.1-rc2. With one node (=localhost), I get timeouts trying to connect to C* after running a 'DROP KEYSPACE' command. I have following error messages in system.log : INFO [SharedPool-Worker-3] 2014-07-09 16:29:36,578 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MemtableFlushWriter:6] 2014-07-09 16:29:37,178 CassandraDaemon.java:166 - Exception in thread Thread[MemtableFlushWriter:6,5,main] java.lang.RuntimeException: Last written key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) = current key DecoratedKey(91e7f660-076f-11e4-a36d-28d2444c0b1b, 52446dde90244ca49789b41671e4ca7c) writing into ./../data/data/test_main/user-911d5360076f11e4812d3d4ba97474ac/test_main-user.user_account-tmp-ka-1-Data.db at org.apache.cassandra.io.sstable.SSTableWriter.beforeAppend(SSTableWriter.java:172) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:215) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:351) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_55] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_55] at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_55] Then, I can not connect to the Cluster anymore from my app (Java Driver 2.1-SNAPSHOT) and got in application logs : com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:65) at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:258) at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:174) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52) at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36) (...) Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /127.0.0.1:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout during read)) at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:103) at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:175) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I can still connect through CQLSH but if I run (again) a DROP KEYSPACE command from CQLSH, I get the following error : errors={}, last_host=127.0.0.1 Now, on a 2 nodes cluster I also have a similar issue but the error's stacktrace is different : From application logs : 17971 [Cassandra Java Driver worker-2] WARN com.datastax.driver.core.Cluster - No schema agreement from live replicas after 1 ms. The schema may not be up to date on some nodes. From system.log : INFO [SharedPool-Worker-2] 2014-07-10 09:04:53,434 MigrationManager.java:319 - Drop Keyspace 'test_main' (...) ERROR [MigrationStage:1] 2014-07-10 09:04:56,553 CommitLogSegmentManager.java:304 - Failed waiting for a forced recycle of in-use commit log segments java.lang.AssertionError: null at org.apache.cassandra.db.commitlog.CommitLogSegmentManager.forceRecycleAll(CommitLogSegmentManager.java:299) ~[apache-cassandra-2.1.0-rc2.jar:2.1.0-rc2] at org.apache.cassandra.db.commitlog.CommitLog.forceRecycleAllSegments(CommitLog.java:160)