[jira] [Commented] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099920#comment-14099920 ] Leonid Shalupov commented on CASSANDRA-7754: Another one: 01:04:42.171 [MemtableFlushWriter:4] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:4,5,main] org.apache.cassandra.io.FSWriteError: java.io.FileNotFoundException: /xxx/data/ideace/ideaindex-3457657025a211e4af4261d16c1df9f6/ideace-ideaindex-tmp-ka-35-CompressionInfo.db (No such file or directory) at org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:264) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.FileUtils.closeQuietly(FileUtils.java:222) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:347) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:331) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:375) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65] Caused by: java.io.FileNotFoundException: /xxx/data/ideace/ideaindex-3457657025a211e4af4261d16c1df9f6/ideace-ideaindex-tmp-ka-35-CompressionInfo.db (No such file or directory) at java.io.FileOutputStream.open(Native Method) ~[na:1.7.0_65] at java.io.FileOutputStream.init(FileOutputStream.java:221) ~[na:1.7.0_65] at java.io.FileOutputStream.init(FileOutputStream.java:110) ~[na:1.7.0_65] at org.apache.cassandra.io.compress.CompressionMetadata$Writer.close(CompressionMetadata.java:351) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.compress.CompressedSequentialWriter.close(CompressedSequentialWriter.java:260) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] ... 12 common frames omitted 01:04:42.479 [MemtableFlushWriter:4] ERROR o.a.cassandra.service.StorageService - Stopping gossiper 01:04:44.480 [MemtableFlushWriter:4] ERROR o.a.cassandra.service.StorageService - Stopping RPC server 01:04:44.481 [MemtableFlushWriter:4] ERROR o.a.cassandra.service.StorageService - Stopping native transport FileNotFoundException in MemtableFlushWriter Key: CASSANDRA-7754 URL: https://issues.apache.org/jira/browse/CASSANDRA-7754 Project: Cassandra Issue Type: Bug Environment: Linux, OpenJDK 1.7 Reporter: Leonid Shalupov Exception in cassandra logs, after upgrade to 2.1: [MemtableFlushWriter:91] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:91,5,main] java.lang.RuntimeException: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:75) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:550) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:134) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:383) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:330) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
[jira] [Updated] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonid Shalupov updated CASSANDRA-7754: --- Priority: Critical (was: Major) FileNotFoundException in MemtableFlushWriter Key: CASSANDRA-7754 URL: https://issues.apache.org/jira/browse/CASSANDRA-7754 Project: Cassandra Issue Type: Bug Environment: Linux, OpenJDK 1.7 Reporter: Leonid Shalupov Priority: Critical Exception in cassandra logs, after upgrade to 2.1: [MemtableFlushWriter:91] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:91,5,main] java.lang.RuntimeException: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:75) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:550) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:134) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:383) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:330) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65] Caused by: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_65] at java.io.RandomAccessFile.init(RandomAccessFile.java:241) ~[na:1.7.0_65] at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:71) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] ... 14 common frames omitted -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[
https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099923#comment-14099923
]
Leonid Shalupov commented on CASSANDRA-7754:
Found on WARN level, seems to be related:
01:04:44.399 [SharedPool-Worker-3] DEBUG o.a.c.service.FileCacheService -
Evicting cold readers for
/xxx/data/ideace/content-38cb3a5025a211e4af4261d16c1df9f6/ideace-content-ka-16-Data.db
01:04:44.426 [SharedPool-Worker-3] WARN
o.a.c.c.AbstractTracingAwareExecutorService - Uncaught exception on thread
Thread[SharedPool-Worker-3,10,main]: {}
java.lang.RuntimeException: java.lang.RuntimeException:
java.io.FileNotFoundException:
/xxx/data/ideace/content-38cb3a5025a211e4af4261d16c1df9f6/ideace-content-ka-16-Data.db
(No such file or directory)
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2068)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_65]
at
org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:163)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:103)
[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException:
/xxx/data/ideace/content-38cb3a5025a211e4af4261d16c1df9f6/ideace-content-ka-16-Data.db
(No such file or directory)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createReader(CompressedPoolingSegmentedFile.java:59)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:40)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1689)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:93)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.readIndexedColumns(SSTableNamesIterator.java:209)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:142)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.columniterator.SSTableNamesIterator.init(SSTableNamesIterator.java:59)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:89)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.CollationController.collectTimeOrderedData(CollationController.java:125)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:59)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1873)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1681)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:345)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:55)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1393)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2065)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
... 4 common frames omitted
Caused by: java.io.FileNotFoundException:
/xxx/data/ideace/content-38cb3a5025a211e4af4261d16c1df9f6/ideace-content-ka-16-Data.db
(No such file or directory)
at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_65]
at java.io.RandomAccessFile.init(RandomAccessFile.java:241)
~[na:1.7.0_65]
at
org.apache.cassandra.io.util.RandomAccessReader.init(RandomAccessReader.java:58)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.init(CompressedRandomAccessReader.java:76)
~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5]
at
[jira] [Resolved] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benedict resolved CASSANDRA-7754. - Resolution: Not a Problem [~shalupov] the first exception you posted is occurring during creation of the initial file for writing, the last exception you posted is not related to the other two, and the middle exception appears to be thrown during abort of a write due to some other error which then finds the data it had been writing is now missing, so I suggest you most likely have some problems with your file system. I would check your ACLs are all in order, and look for background cleanup / archive processes. I currently doubt there is a problem with C* from the information you've posted, especially as this code is exercised regularly and we haven't seen any issues elsewhere, but if after further investigation you continue to be convinced there is a bug, please reopen the ticket with some more information and reproduction steps so we can try to replicate it ourselves. FileNotFoundException in MemtableFlushWriter Key: CASSANDRA-7754 URL: https://issues.apache.org/jira/browse/CASSANDRA-7754 Project: Cassandra Issue Type: Bug Environment: Linux, OpenJDK 1.7 Reporter: Leonid Shalupov Priority: Critical Exception in cassandra logs, after upgrade to 2.1: [MemtableFlushWriter:91] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:91,5,main] java.lang.RuntimeException: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:75) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:550) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:134) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:383) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:330) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65] Caused by: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_65] at java.io.RandomAccessFile.init(RandomAccessFile.java:241) ~[na:1.7.0_65] at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:71) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] ... 14 common frames omitted -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099932#comment-14099932 ] Marcus Eriksson commented on CASSANDRA-7754: You could try not running OpenJDK as well FileNotFoundException in MemtableFlushWriter Key: CASSANDRA-7754 URL: https://issues.apache.org/jira/browse/CASSANDRA-7754 Project: Cassandra Issue Type: Bug Environment: Linux, OpenJDK 1.7 Reporter: Leonid Shalupov Priority: Critical Exception in cassandra logs, after upgrade to 2.1: [MemtableFlushWriter:91] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:91,5,main] java.lang.RuntimeException: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:75) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:550) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:134) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:383) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:330) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65] Caused by: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_65] at java.io.RandomAccessFile.init(RandomAccessFile.java:241) ~[na:1.7.0_65] at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:71) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] ... 14 common frames omitted -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (CASSANDRA-7786) Cassandra is shutting down out of no apparent reason
Or Sher created CASSANDRA-7786: -- Summary: Cassandra is shutting down out of no apparent reason Key: CASSANDRA-7786 URL: https://issues.apache.org/jira/browse/CASSANDRA-7786 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.9 Reporter: Or Sher We've recently start facing an issue when one of the C* node in our dev and CI cluster (Thanks god didn't happen yet in Prod) is shutting down from time to time without any exceptions or errors. There is usually something like that in the logs: INFO [MemoryMeter:1] 2014-08-15 01:32:43,266 Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.597030881851438 (just-counted was 14.596825396825396). calculation took 2ms for 84 cells INFO [StorageServiceShutdownHook] 2014-08-15 01:40:58,954 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,007 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,011 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-15 01:41:01,011 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/192.168.27.241] 2014-08-15 01:41:01,012 MessagingService.java (line 923) MessagingService has terminated the accept() thread INFO [main] 2014-08-17 09:50:56,647 CassandraDaemon.java (line 135) Logging initialized You can see the last line in the log is usually written at least 5 minutes before the shutdown, sometimes 30 minutes before. I can't reproduce it as I have know idea why is that happening and how the attack this issue. I believe I'm not the only one suffering from this issue as there was a thread about such behavior in the user mail distribution. Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7786) Cassandra is shutting down out of no apparent reason
[ https://issues.apache.org/jira/browse/CASSANDRA-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099943#comment-14099943 ] Or Sher commented on CASSANDRA-7786: There is also nothing in the /var/log/messages indicating on a kernel oom. Cassandra is shutting down out of no apparent reason Key: CASSANDRA-7786 URL: https://issues.apache.org/jira/browse/CASSANDRA-7786 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.9 Reporter: Or Sher We've recently start facing an issue when one of the C* node in our dev and CI cluster (Thanks god didn't happen yet in Prod) is shutting down from time to time without any exceptions or errors. There is usually something like that in the logs: INFO [MemoryMeter:1] 2014-08-15 01:32:43,266 Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.597030881851438 (just-counted was 14.596825396825396). calculation took 2ms for 84 cells INFO [StorageServiceShutdownHook] 2014-08-15 01:40:58,954 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,007 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,011 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-15 01:41:01,011 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/192.168.27.241] 2014-08-15 01:41:01,012 MessagingService.java (line 923) MessagingService has terminated the accept() thread INFO [main] 2014-08-17 09:50:56,647 CassandraDaemon.java (line 135) Logging initialized You can see the last line in the log is usually written at least 5 minutes before the shutdown, sometimes 30 minutes before. I can't reproduce it as I have know idea why is that happening and how the attack this issue. I believe I'm not the only one suffering from this issue as there was a thread about such behavior in the user mail distribution. Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7754) FileNotFoundException in MemtableFlushWriter
[ https://issues.apache.org/jira/browse/CASSANDRA-7754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14099955#comment-14099955 ] Leonid Shalupov commented on CASSANDRA-7754: Thanks! I've found a race condition with cleanup of the data directory in my own code. FileNotFoundException in MemtableFlushWriter Key: CASSANDRA-7754 URL: https://issues.apache.org/jira/browse/CASSANDRA-7754 Project: Cassandra Issue Type: Bug Environment: Linux, OpenJDK 1.7 Reporter: Leonid Shalupov Priority: Critical Exception in cassandra logs, after upgrade to 2.1: [MemtableFlushWriter:91] ERROR o.a.c.service.CassandraDaemon - Exception in thread Thread[MemtableFlushWriter:91,5,main] java.lang.RuntimeException: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:75) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:104) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:99) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.init(SSTableWriter.java:550) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.sstable.SSTableWriter.init(SSTableWriter.java:134) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:383) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:330) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:314) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1054) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_65] at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_65] Caused by: java.io.FileNotFoundException: /xxx/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-tmp-ka-186-Index.db (No such file or directory) at java.io.RandomAccessFile.open(Native Method) ~[na:1.7.0_65] at java.io.RandomAccessFile.init(RandomAccessFile.java:241) ~[na:1.7.0_65] at org.apache.cassandra.io.util.SequentialWriter.init(SequentialWriter.java:71) ~[cassandra-all-2.1.0-rc5.jar:2.1.0-rc5] ... 14 common frames omitted -- This message was sent by Atlassian JIRA (v6.2#6252)
Git Push Summary
Repository: cassandra Updated Tags: refs/tags/2.1.0-rc6-tentative [deleted] 397c0b7c0
Git Push Summary
Repository: cassandra Updated Tags: refs/tags/2.1.0-rc6-tentative [created] d087317fd
git commit: (Hadoop) allow ACFRW to limit nodes to local DC
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.0 115bbe435 - b87741c07
(Hadoop) allow ACFRW to limit nodes to local DC
patch by Robbie Strickland; reviewed by Aleksey Yeschenko for
CASSANDRA-7252
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b87741c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b87741c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b87741c0
Branch: refs/heads/cassandra-2.0
Commit: b87741c077e74b2ae3fda3da2417dc1965c0c4ed
Parents: 115bbe4
Author: Robbie Strickland rostrickl...@gmail.com
Authored: Sun Aug 17 20:40:39 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:40:39 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 63 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 45 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 987c227..94169c1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
* (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
* Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/src/java/org/apache/cassandra/client/RingCache.java
--
diff --git a/src/java/org/apache/cassandra/client/RingCache.java
b/src/java/org/apache/cassandra/client/RingCache.java
index 3308471..cc9b1b2 100644
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@ -61,44 +61,47 @@ public class RingCache
public void refreshEndpointMap()
{
-try {
+try
+{
+Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
-Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+rangeMap = ArrayListMultimap.create();
-ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
-rangeMap = ArrayListMultimap.create();
-
-for (TokenRange range : ring)
+for (TokenRange range : ring)
+{
+Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+RangeToken r = new RangeToken(left, right, partitioner);
+for (String host : range.endpoints)
{
-Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
-Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
-RangeToken r = new RangeToken(left, right,
partitioner);
-for (String host : range.endpoints)
+try
+{
+rangeMap.put(r, InetAddress.getByName(host));
+}
+catch (UnknownHostException e)
{
-try
-{
-rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
-{
-throw new AssertionError(e); // host strings are
IPs
-}
+throw new AssertionError(e); // host strings are IPs
}
}
}
-catch (InvalidRequestException e)
-{
-throw new RuntimeException(e);
-}
-catch (IOException e)
-{
-throw new RuntimeException(e);
-}
-catch (TException e)
-{
-logger.debug(Error contacting seed list +
ConfigHelper.getOutputInitialAddress(conf) + + e.getMessage());
-}
}
+catch (InvalidRequestException e)
+{
+throw new RuntimeException(e);
+}
+
[1/2] git commit: (Hadoop) allow ACFRW to limit nodes to local DC
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1 ff9c63163 - 35999b3c8
(Hadoop) allow ACFRW to limit nodes to local DC
patch by Robbie Strickland; reviewed by Aleksey Yeschenko for
CASSANDRA-7252
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b87741c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b87741c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b87741c0
Branch: refs/heads/cassandra-2.1
Commit: b87741c077e74b2ae3fda3da2417dc1965c0c4ed
Parents: 115bbe4
Author: Robbie Strickland rostrickl...@gmail.com
Authored: Sun Aug 17 20:40:39 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:40:39 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 63 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 45 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 987c227..94169c1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
* (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
* Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/src/java/org/apache/cassandra/client/RingCache.java
--
diff --git a/src/java/org/apache/cassandra/client/RingCache.java
b/src/java/org/apache/cassandra/client/RingCache.java
index 3308471..cc9b1b2 100644
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@ -61,44 +61,47 @@ public class RingCache
public void refreshEndpointMap()
{
-try {
+try
+{
+Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
-Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+rangeMap = ArrayListMultimap.create();
-ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
-rangeMap = ArrayListMultimap.create();
-
-for (TokenRange range : ring)
+for (TokenRange range : ring)
+{
+Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+RangeToken r = new RangeToken(left, right, partitioner);
+for (String host : range.endpoints)
{
-Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
-Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
-RangeToken r = new RangeToken(left, right,
partitioner);
-for (String host : range.endpoints)
+try
+{
+rangeMap.put(r, InetAddress.getByName(host));
+}
+catch (UnknownHostException e)
{
-try
-{
-rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
-{
-throw new AssertionError(e); // host strings are
IPs
-}
+throw new AssertionError(e); // host strings are IPs
}
}
}
-catch (InvalidRequestException e)
-{
-throw new RuntimeException(e);
-}
-catch (IOException e)
-{
-throw new RuntimeException(e);
-}
-catch (TException e)
-{
-logger.debug(Error contacting seed list +
ConfigHelper.getOutputInitialAddress(conf) + + e.getMessage());
-}
}
+catch (InvalidRequestException e)
+{
+throw new RuntimeException(e);
+}
+
[2/2] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/client/RingCache.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35999b3c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35999b3c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35999b3c
Branch: refs/heads/cassandra-2.1
Commit: 35999b3c835f4eb36180595593d0def7c7dae084
Parents: ff9c631 b87741c
Author: Aleksey Yeschenko alek...@apache.org
Authored: Sun Aug 17 20:47:58 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:47:58 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 58 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 40 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/35999b3c/CHANGES.txt
--
diff --cc CHANGES.txt
index 113de3e,94169c1..b8132ba
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,7 +1,21 @@@
-2.0.10
+2.1.1
+ * Add max live/tombstoned cells to nodetool cfstats output (CASSANDRA-7731)
+ * Validate IPv6 wildcard addresses properly (CASSANDRA-7680)
+ * (cqlsh) Error when tracing query (CASSANDRA-7613)
+ * Avoid IOOBE when building SyntaxError message snippet (CASSANDRA-7569)
+ * SSTableExport uses correct validator to create string representation of
partition
+ keys (CASSANDRA-7498)
+ * Avoid NPEs when receiving type changes for an unknown keyspace
(CASSANDRA-7689)
+ * Add support for custom 2i validation (CASSANDRA-7575)
+ * Pig support for hadoop CqlInputFormat (CASSANDRA-6454)
+ * Add listen_interface and rpc_interface options (CASSANDRA-7417)
+ * Improve schema merge performance (CASSANDRA-7444)
+ * Adjust MT depth based on # of partition validating (CASSANDRA-5263)
+ * Optimise NativeCell comparisons (CASSANDRA-6755)
+ * Configurable client timeout for cqlsh (CASSANDRA-7516)
+ * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111)
+Merged from 2.0:
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
- * (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
- * Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
from system_traces (CASSANDRA-7641)
* (Hadoop) Add CqlOutputFormat (CASSANDRA-6927)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/35999b3c/src/java/org/apache/cassandra/client/RingCache.java
--
diff --cc src/java/org/apache/cassandra/client/RingCache.java
index 9b04518,cc9b1b2..c3dbda5
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@@ -61,44 -61,47 +61,42 @@@ public class RingCach
public void refreshEndpointMap()
{
- try {
+ try
+ {
+ Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
- Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+ String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+ rangeMap = ArrayListMultimap.create();
- ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
- rangeMap = ArrayListMultimap.create();
-
- for (TokenRange range : ring)
+ for (TokenRange range : ring)
+ {
+ Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+ Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+ RangeToken r = new RangeToken(left, right, partitioner);
+ for (String host : range.endpoints)
{
- Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
- Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
- RangeToken r = new RangeToken(left, right,
partitioner);
- for (String host : range.endpoints)
+ try
+ {
+ rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
++} catch
[2/2] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1.0
Merge branch 'cassandra-2.0' into cassandra-2.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cb772e54 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cb772e54 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cb772e54 Branch: refs/heads/cassandra-2.1.0 Commit: cb772e54461e79146e79d677b8721941a936f954 Parents: d087317 b87741c Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:49:00 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:49:00 2014 +0300 -- --
[1/2] git commit: (Hadoop) allow ACFRW to limit nodes to local DC
Repository: cassandra
Updated Branches:
refs/heads/cassandra-2.1.0 d087317fd - cb772e544
(Hadoop) allow ACFRW to limit nodes to local DC
patch by Robbie Strickland; reviewed by Aleksey Yeschenko for
CASSANDRA-7252
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b87741c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b87741c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b87741c0
Branch: refs/heads/cassandra-2.1.0
Commit: b87741c077e74b2ae3fda3da2417dc1965c0c4ed
Parents: 115bbe4
Author: Robbie Strickland rostrickl...@gmail.com
Authored: Sun Aug 17 20:40:39 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:40:39 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 63 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 45 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 987c227..94169c1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
* (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
* Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/src/java/org/apache/cassandra/client/RingCache.java
--
diff --git a/src/java/org/apache/cassandra/client/RingCache.java
b/src/java/org/apache/cassandra/client/RingCache.java
index 3308471..cc9b1b2 100644
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@ -61,44 +61,47 @@ public class RingCache
public void refreshEndpointMap()
{
-try {
+try
+{
+Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
-Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+rangeMap = ArrayListMultimap.create();
-ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
-rangeMap = ArrayListMultimap.create();
-
-for (TokenRange range : ring)
+for (TokenRange range : ring)
+{
+Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+RangeToken r = new RangeToken(left, right, partitioner);
+for (String host : range.endpoints)
{
-Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
-Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
-RangeToken r = new RangeToken(left, right,
partitioner);
-for (String host : range.endpoints)
+try
+{
+rangeMap.put(r, InetAddress.getByName(host));
+}
+catch (UnknownHostException e)
{
-try
-{
-rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
-{
-throw new AssertionError(e); // host strings are
IPs
-}
+throw new AssertionError(e); // host strings are IPs
}
}
}
-catch (InvalidRequestException e)
-{
-throw new RuntimeException(e);
-}
-catch (IOException e)
-{
-throw new RuntimeException(e);
-}
-catch (TException e)
-{
-logger.debug(Error contacting seed list +
ConfigHelper.getOutputInitialAddress(conf) + + e.getMessage());
-}
}
+catch (InvalidRequestException e)
+{
+throw new RuntimeException(e);
+}
+
[2/2] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7035cfcc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7035cfcc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7035cfcc Branch: refs/heads/cassandra-2.1 Commit: 7035cfccfcaf48ff64722942430ca8036e2e1e02 Parents: 35999b3 cb772e5 Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:49:58 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:49:58 2014 +0300 -- --
[1/2] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1.0
Repository: cassandra Updated Branches: refs/heads/cassandra-2.1 35999b3c8 - 7035cfccf Merge branch 'cassandra-2.0' into cassandra-2.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cb772e54 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cb772e54 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cb772e54 Branch: refs/heads/cassandra-2.1 Commit: cb772e54461e79146e79d677b8721941a936f954 Parents: d087317 b87741c Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:49:00 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:49:00 2014 +0300 -- --
[5/5] git commit: Merge branch 'cassandra-2.1' into trunk
Merge branch 'cassandra-2.1' into trunk Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/4e334ab8 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/4e334ab8 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/4e334ab8 Branch: refs/heads/trunk Commit: 4e334ab82858972b768920dc2f752ea1f283ec2c Parents: fe05727 7035cfc Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:50:42 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:50:42 2014 +0300 -- CHANGES.txt | 1 + .../org/apache/cassandra/client/RingCache.java | 58 ++-- .../apache/cassandra/hadoop/ConfigHelper.java | 11 3 files changed, 40 insertions(+), 30 deletions(-) -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e334ab8/CHANGES.txt -- http://git-wip-us.apache.org/repos/asf/cassandra/blob/4e334ab8/src/java/org/apache/cassandra/hadoop/ConfigHelper.java --
[3/5] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1.0
Merge branch 'cassandra-2.0' into cassandra-2.1.0 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/cb772e54 Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/cb772e54 Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/cb772e54 Branch: refs/heads/trunk Commit: cb772e54461e79146e79d677b8721941a936f954 Parents: d087317 b87741c Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:49:00 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:49:00 2014 +0300 -- --
[4/5] git commit: Merge branch 'cassandra-2.1.0' into cassandra-2.1
Merge branch 'cassandra-2.1.0' into cassandra-2.1 Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/7035cfcc Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/7035cfcc Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/7035cfcc Branch: refs/heads/trunk Commit: 7035cfccfcaf48ff64722942430ca8036e2e1e02 Parents: 35999b3 cb772e5 Author: Aleksey Yeschenko alek...@apache.org Authored: Sun Aug 17 20:49:58 2014 +0300 Committer: Aleksey Yeschenko alek...@apache.org Committed: Sun Aug 17 20:49:58 2014 +0300 -- --
[1/5] git commit: (Hadoop) allow ACFRW to limit nodes to local DC
Repository: cassandra
Updated Branches:
refs/heads/trunk fe0572778 - 4e334ab82
(Hadoop) allow ACFRW to limit nodes to local DC
patch by Robbie Strickland; reviewed by Aleksey Yeschenko for
CASSANDRA-7252
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/b87741c0
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/b87741c0
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/b87741c0
Branch: refs/heads/trunk
Commit: b87741c077e74b2ae3fda3da2417dc1965c0c4ed
Parents: 115bbe4
Author: Robbie Strickland rostrickl...@gmail.com
Authored: Sun Aug 17 20:40:39 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:40:39 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 63 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 45 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/CHANGES.txt
--
diff --git a/CHANGES.txt b/CHANGES.txt
index 987c227..94169c1 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,4 +1,5 @@
2.0.10
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
* (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
* Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
http://git-wip-us.apache.org/repos/asf/cassandra/blob/b87741c0/src/java/org/apache/cassandra/client/RingCache.java
--
diff --git a/src/java/org/apache/cassandra/client/RingCache.java
b/src/java/org/apache/cassandra/client/RingCache.java
index 3308471..cc9b1b2 100644
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@ -61,44 +61,47 @@ public class RingCache
public void refreshEndpointMap()
{
-try {
+try
+{
+Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
-Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+rangeMap = ArrayListMultimap.create();
-ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
-rangeMap = ArrayListMultimap.create();
-
-for (TokenRange range : ring)
+for (TokenRange range : ring)
+{
+Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+RangeToken r = new RangeToken(left, right, partitioner);
+for (String host : range.endpoints)
{
-Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
-Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
-RangeToken r = new RangeToken(left, right,
partitioner);
-for (String host : range.endpoints)
+try
+{
+rangeMap.put(r, InetAddress.getByName(host));
+}
+catch (UnknownHostException e)
{
-try
-{
-rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
-{
-throw new AssertionError(e); // host strings are
IPs
-}
+throw new AssertionError(e); // host strings are IPs
}
}
}
-catch (InvalidRequestException e)
-{
-throw new RuntimeException(e);
-}
-catch (IOException e)
-{
-throw new RuntimeException(e);
-}
-catch (TException e)
-{
-logger.debug(Error contacting seed list +
ConfigHelper.getOutputInitialAddress(conf) + + e.getMessage());
-}
}
+catch (InvalidRequestException e)
+{
+throw new RuntimeException(e);
+}
+catch
[2/5] git commit: Merge branch 'cassandra-2.0' into cassandra-2.1
Merge branch 'cassandra-2.0' into cassandra-2.1
Conflicts:
CHANGES.txt
src/java/org/apache/cassandra/client/RingCache.java
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/35999b3c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/35999b3c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/35999b3c
Branch: refs/heads/trunk
Commit: 35999b3c835f4eb36180595593d0def7c7dae084
Parents: ff9c631 b87741c
Author: Aleksey Yeschenko alek...@apache.org
Authored: Sun Aug 17 20:47:58 2014 +0300
Committer: Aleksey Yeschenko alek...@apache.org
Committed: Sun Aug 17 20:47:58 2014 +0300
--
CHANGES.txt | 1 +
.../org/apache/cassandra/client/RingCache.java | 58 ++--
.../apache/cassandra/hadoop/ConfigHelper.java | 11
3 files changed, 40 insertions(+), 30 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/35999b3c/CHANGES.txt
--
diff --cc CHANGES.txt
index 113de3e,94169c1..b8132ba
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@@ -1,20 -1,7 +1,21 @@@
-2.0.10
+2.1.1
+ * Add max live/tombstoned cells to nodetool cfstats output (CASSANDRA-7731)
+ * Validate IPv6 wildcard addresses properly (CASSANDRA-7680)
+ * (cqlsh) Error when tracing query (CASSANDRA-7613)
+ * Avoid IOOBE when building SyntaxError message snippet (CASSANDRA-7569)
+ * SSTableExport uses correct validator to create string representation of
partition
+ keys (CASSANDRA-7498)
+ * Avoid NPEs when receiving type changes for an unknown keyspace
(CASSANDRA-7689)
+ * Add support for custom 2i validation (CASSANDRA-7575)
+ * Pig support for hadoop CqlInputFormat (CASSANDRA-6454)
+ * Add listen_interface and rpc_interface options (CASSANDRA-7417)
+ * Improve schema merge performance (CASSANDRA-7444)
+ * Adjust MT depth based on # of partition validating (CASSANDRA-5263)
+ * Optimise NativeCell comparisons (CASSANDRA-6755)
+ * Configurable client timeout for cqlsh (CASSANDRA-7516)
+ * Include snippet of CQL query near syntax error in messages (CASSANDRA-7111)
+Merged from 2.0:
+ * (Hadoop) allow ACFRW to limit nodes to local DC (CASSANDRA-7252)
- * (cqlsh) Wait up to 10 sec for a tracing session (CASSANDRA-7222)
- * Fix NPE in FileCacheService.sizeInBytes (CASSANDRA-7756)
* (cqlsh) cqlsh should automatically disable tracing when selecting
from system_traces (CASSANDRA-7641)
* (Hadoop) Add CqlOutputFormat (CASSANDRA-6927)
http://git-wip-us.apache.org/repos/asf/cassandra/blob/35999b3c/src/java/org/apache/cassandra/client/RingCache.java
--
diff --cc src/java/org/apache/cassandra/client/RingCache.java
index 9b04518,cc9b1b2..c3dbda5
--- a/src/java/org/apache/cassandra/client/RingCache.java
+++ b/src/java/org/apache/cassandra/client/RingCache.java
@@@ -61,44 -61,47 +61,42 @@@ public class RingCach
public void refreshEndpointMap()
{
- try {
+ try
+ {
+ Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
- Cassandra.Client client =
ConfigHelper.getClientFromOutputAddressList(conf);
+ String keyspace = ConfigHelper.getOutputKeyspace(conf);
+ ListTokenRange ring = ConfigHelper.getOutputLocalDCOnly(conf)
+ ? client.describe_local_ring(keyspace)
+ : client.describe_ring(keyspace);
+ rangeMap = ArrayListMultimap.create();
- ListTokenRange ring =
client.describe_ring(ConfigHelper.getOutputKeyspace(conf));
- rangeMap = ArrayListMultimap.create();
-
- for (TokenRange range : ring)
+ for (TokenRange range : ring)
+ {
+ Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
+ Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
+ RangeToken r = new RangeToken(left, right, partitioner);
+ for (String host : range.endpoints)
{
- Token? left =
partitioner.getTokenFactory().fromString(range.start_token);
- Token? right =
partitioner.getTokenFactory().fromString(range.end_token);
- RangeToken r = new RangeToken(left, right,
partitioner);
- for (String host : range.endpoints)
+ try
+ {
+ rangeMap.put(r, InetAddress.getByName(host));
-}
-catch (UnknownHostException e)
++} catch (UnknownHostException e)
[jira] [Commented] (CASSANDRA-7252) RingCache cannot be configured to use local DC only
[ https://issues.apache.org/jira/browse/CASSANDRA-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100018#comment-14100018 ] Aleksey Yeschenko commented on CASSANDRA-7252: -- Committed the dc-local part as https://github.com/apache/cassandra/commit/b87741c077e74b2ae3fda3da2417dc1965c0c4ed, with minor formatting changes. The rest, however, should be split into separate tickets. RingCache cannot be configured to use local DC only --- Key: CASSANDRA-7252 URL: https://issues.apache.org/jira/browse/CASSANDRA-7252 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Robbie Strickland Assignee: Robbie Strickland Labels: patch Fix For: 2.0.10, 2.1.1 Attachments: cassandra-2.0.7-7252-2.txt, cassandra-2.0.7-7252.txt RingCache always calls describe_ring, returning the entire cluster. Considering it's used in the context of writing from Hadoop (which is typically in a multi-DC configuration), this is often not desirable behavior. In some cases there may be high-latency connections between the analytics DC and other DCs. I am attaching a patch that adds an optional config value to tell RingCache to use local nodes only, in which case it calls describe_local_ring instead. It also adds helpful failed host information to IOExceptions thrown in AbstractColumnFamilyOutputFormat.createAuthenticatedClient, CqlRecordWriter, and ColumnFamilyRecordWriter. This allows a user to more easily solve related connectivity issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100032#comment-14100032 ] Aleksey Yeschenko commented on CASSANDRA-7577: -- Won't do. Pretty sure you can configure libedit to enable recursive search, the proper way, instead of throwing a warning to install readline. We didn't add libedit support to cqlsh for nothing. cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7252) RingCache cannot be configured to use local DC only
[ https://issues.apache.org/jira/browse/CASSANDRA-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100040#comment-14100040 ] Robbie Strickland commented on CASSANDRA-7252: -- [~iamaleksey] Do you want me to create another ticket for the address option? RingCache cannot be configured to use local DC only --- Key: CASSANDRA-7252 URL: https://issues.apache.org/jira/browse/CASSANDRA-7252 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Robbie Strickland Assignee: Robbie Strickland Labels: patch Fix For: 2.0.10, 2.1.1 Attachments: cassandra-2.0.7-7252-2.txt, cassandra-2.0.7-7252.txt RingCache always calls describe_ring, returning the entire cluster. Considering it's used in the context of writing from Hadoop (which is typically in a multi-DC configuration), this is often not desirable behavior. In some cases there may be high-latency connections between the analytics DC and other DCs. I am attaching a patch that adds an optional config value to tell RingCache to use local nodes only, in which case it calls describe_local_ring instead. It also adds helpful failed host information to IOExceptions thrown in AbstractColumnFamilyOutputFormat.createAuthenticatedClient, CqlRecordWriter, and ColumnFamilyRecordWriter. This allows a user to more easily solve related connectivity issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7785) Add command to display the current logged-in user.
[
https://issues.apache.org/jira/browse/CASSANDRA-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100046#comment-14100046
]
Mikhail Stepura commented on CASSANDRA-7785:
[~aploetz] does it make sense to add an username into the cqlsh promt? (like
{{Aaron@cqlsh:stress}}
Add command to display the current logged-in user.
--
Key: CASSANDRA-7785
URL: https://issues.apache.org/jira/browse/CASSANDRA-7785
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Aaron Ploetz
Priority: Trivial
Labels: lhf
Fix For: 2.1.1
Attachments: 2.1.1-CASSANDRA-7785.txt
Currently, a user in cqlsh cannot see which user they are logged-in as. When
you have a cqlsh that has been open for a few hours, sometimes it is helpful
to be able to type a quick command and see your current user.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[
https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100068#comment-14100068
]
Robert Stupp commented on CASSANDRA-7577:
-
bq. configure libedit to enable recursive search
nope - neither {{bind ^R em-inc-search-prev}} in {{~/.editrc}} nor
{{readline.parse_and_bind(bind ^R em-inc-search-prev)}} in {{~/.pystartup}}
or in {{cqlsh}} work.
Recommendation on https://pypi.python.org/pypi/readline is to ??As the
alternatives to GNU readline do not have fully equivalent functionality, it is
useful to add proper readline support to these platforms.??
cqlsh: CTRL-R history search not working on OSX
---
Key: CASSANDRA-7577
URL: https://issues.apache.org/jira/browse/CASSANDRA-7577
Project: Cassandra
Issue Type: Bug
Environment: OSX - plain Terminal program
C* 2.0.x, 2.1, trunk
Reporter: Robert Stupp
Assignee: Robert Stupp
Priority: Minor
Fix For: 2.0.11, 2.1.1
Attachments: 7577-2.0.txt, 7577-2.1.txt
_recursive-history-search_ via ctrl-R does not work in cqlsh. The history
itself works via cursor up/down.
It works on Linux (and I guess on Windows with cygwin) but not on my Mac.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[
https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100068#comment-14100068
]
Robert Stupp edited comment on CASSANDRA-7577 at 8/17/14 7:12 PM:
--
bq. configure libedit to enable recursive search
nope - neither {{bind ^R em-inc-search-prev}} in {{\~/.editrc}} nor
{{readline.parse_and_bind(bind ^R em-inc-search-prev)}} in {{\~/.pystartup}}
or in {{cqlsh}} work.
Recommendation on https://pypi.python.org/pypi/readline is to ??As the
alternatives to GNU readline do not have fully equivalent functionality, it is
useful to add proper readline support to these platforms.??
was (Author: snazy):
bq. configure libedit to enable recursive search
nope - neither {{bind ^R em-inc-search-prev}} in {{~/.editrc}} nor
{{readline.parse_and_bind(bind ^R em-inc-search-prev)}} in {{~/.pystartup}}
or in {{cqlsh}} work.
Recommendation on https://pypi.python.org/pypi/readline is to ??As the
alternatives to GNU readline do not have fully equivalent functionality, it is
useful to add proper readline support to these platforms.??
cqlsh: CTRL-R history search not working on OSX
---
Key: CASSANDRA-7577
URL: https://issues.apache.org/jira/browse/CASSANDRA-7577
Project: Cassandra
Issue Type: Bug
Environment: OSX - plain Terminal program
C* 2.0.x, 2.1, trunk
Reporter: Robert Stupp
Assignee: Robert Stupp
Priority: Minor
Fix For: 2.0.11, 2.1.1
Attachments: 7577-2.0.txt, 7577-2.1.txt
_recursive-history-search_ via ctrl-R does not work in cqlsh. The history
itself works via cursor up/down.
It works on Linux (and I guess on Windows with cygwin) but not on my Mac.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (CASSANDRA-7743) Possible C* OOM issue during long running test
[
https://issues.apache.org/jira/browse/CASSANDRA-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100080#comment-14100080
]
Kishan Karunaratne commented on CASSANDRA-7743:
---
I'm running rc5 + the patch, and the issue still shows up.
I patched rc5 with the one file, and ran ant realclean jar to compile. I hope
this command didn't re-pull from git.
$ free -m
total used free sharedbuffers cached
Mem: 3702 2667 1035 0 1144
-/+ buffers/cache: 2520 1181
Swap:0 0 0
$ head -n 4 /proc/meminfo
MemTotal:3791292 kB
MemFree: 1060548 kB
Buffers:1280 kB
Cached: 148968 kB
Possible C* OOM issue during long running test
--
Key: CASSANDRA-7743
URL: https://issues.apache.org/jira/browse/CASSANDRA-7743
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Google Compute Engine, n1-standard-1
Reporter: Pierre Laporte
Assignee: Benedict
Fix For: 2.1 rc6
During a long running test, we ended up with a lot of
java.lang.OutOfMemoryError: Direct buffer memory errors on the Cassandra
instances.
Here is an example of stacktrace from system.log :
{code}
ERROR [SharedPool-Worker-1] 2014-08-11 11:09:34,610 ErrorMessage.java:218 -
Unexpected exception during request
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_25]
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:123)
~[na:1.7.0_25]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
~[na:1.7.0_25]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:251)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:112)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
{code}
The test consisted of a 3-nodes cluster of n1-standard-1 GCE instances (1
vCPU, 3.75 GB RAM) running cassandra-2.1.0-rc5, and a n1-standard-2 instance
running the test.
After ~2.5 days, several requests start to fail and we see the previous
stacktraces in the system.log file.
The output from linux ‘free’ and ‘meminfo’ suggest that there is still memory
available.
{code}
$ free -m
total used free sharedbuffers cached
Mem: 3702 3532169 0161854
-/+ buffers/cache: 2516 1185
Swap:0 0 0
$ head -n 4 /proc/meminfo
MemTotal:3791292 kB
MemFree: 173568 kB
Buffers: 165608 kB
Cached: 874752 kB
{code}
These errors do not affect all the queries we run. The cluster is still
responsive but is unable to display tracing information using cqlsh :
{code}
$ ./bin/nodetool --host 10.240.137.253
[jira] [Updated] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-7577: - Attachment: 7577-proper.txt It does work for me (had to brew uninstall python just to verify that it works). cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7252) RingCache cannot be configured to use local DC only
[ https://issues.apache.org/jira/browse/CASSANDRA-7252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100092#comment-14100092 ] Aleksey Yeschenko commented on CASSANDRA-7252: -- bq. Do you want me to create another ticket for the address option? If you feel like it and still need it - go for it. (This doesn't mean I'm for it - I haven't looked into the need for it, myself). RingCache cannot be configured to use local DC only --- Key: CASSANDRA-7252 URL: https://issues.apache.org/jira/browse/CASSANDRA-7252 Project: Cassandra Issue Type: Bug Components: Hadoop Reporter: Robbie Strickland Assignee: Robbie Strickland Labels: patch Fix For: 2.0.10, 2.1.1 Attachments: cassandra-2.0.7-7252-2.txt, cassandra-2.0.7-7252.txt RingCache always calls describe_ring, returning the entire cluster. Considering it's used in the context of writing from Hadoop (which is typically in a multi-DC configuration), this is often not desirable behavior. In some cases there may be high-latency connections between the analytics DC and other DCs. I am attaching a patch that adds an optional config value to tell RingCache to use local nodes only, in which case it calls describe_local_ring instead. It also adds helpful failed host information to IOExceptions thrown in AbstractColumnFamilyOutputFormat.createAuthenticatedClient, CqlRecordWriter, and ColumnFamilyRecordWriter. This allows a user to more easily solve related connectivity issues. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100113#comment-14100113 ] Robert Stupp commented on CASSANDRA-7577: - Sure you don't have a readline in /Library/Python/2.7/site-packages ? cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[
https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100113#comment-14100113
]
Robert Stupp edited comment on CASSANDRA-7577 at 8/17/14 9:10 PM:
--
Sure you don't have a readline in /Library/Python/2.7/site-packages ?
What does {{print readline.__doc__}} tell on your machine?
was (Author: snazy):
Sure you don't have a readline in /Library/Python/2.7/site-packages ?
cqlsh: CTRL-R history search not working on OSX
---
Key: CASSANDRA-7577
URL: https://issues.apache.org/jira/browse/CASSANDRA-7577
Project: Cassandra
Issue Type: Bug
Environment: OSX - plain Terminal program
C* 2.0.x, 2.1, trunk
Reporter: Robert Stupp
Assignee: Robert Stupp
Priority: Minor
Fix For: 2.0.11, 2.1.1
Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt
_recursive-history-search_ via ctrl-R does not work in cqlsh. The history
itself works via cursor up/down.
It works on Linux (and I guess on Windows with cygwin) but not on my Mac.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-7577: - Attachment: ctrl-r-libedit.png cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt, ctrl-r-libedit.png, ctrl-r-readline.png _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Yeschenko updated CASSANDRA-7577: - Attachment: ctrl-r-readline.png cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt, ctrl-r-libedit.png, ctrl-r-readline.png _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7577) cqlsh: CTRL-R history search not working on OSX
[ https://issues.apache.org/jira/browse/CASSANDRA-7577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100120#comment-14100120 ] Aleksey Yeschenko commented on CASSANDRA-7577: -- Yes, I'm sure. See the attached screenshots. Also CASSANDRA-3597. cqlsh: CTRL-R history search not working on OSX --- Key: CASSANDRA-7577 URL: https://issues.apache.org/jira/browse/CASSANDRA-7577 Project: Cassandra Issue Type: Bug Environment: OSX - plain Terminal program C* 2.0.x, 2.1, trunk Reporter: Robert Stupp Assignee: Robert Stupp Priority: Minor Fix For: 2.0.11, 2.1.1 Attachments: 7577-2.0.txt, 7577-2.1.txt, 7577-proper.txt, ctrl-r-libedit.png, ctrl-r-readline.png _recursive-history-search_ via ctrl-R does not work in cqlsh. The history itself works via cursor up/down. It works on Linux (and I guess on Windows with cygwin) but not on my Mac. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7785) Add command to display the current logged-in user.
[ https://issues.apache.org/jira/browse/CASSANDRA-7785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100177#comment-14100177 ] Aaron Ploetz commented on CASSANDRA-7785: - So, instead of adding a new subcommand, just show the user as a part of the prompt? That's a good idea. It would be consistent with how many Linux prompts display it. Long usernames could make that prompt take up more space. But if that's preferable to adding another subcommand, I can certainly do it and resubmit. Add command to display the current logged-in user. -- Key: CASSANDRA-7785 URL: https://issues.apache.org/jira/browse/CASSANDRA-7785 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Aaron Ploetz Priority: Trivial Labels: lhf Fix For: 2.1.1 Attachments: 2.1.1-CASSANDRA-7785.txt Currently, a user in cqlsh cannot see which user they are logged-in as. When you have a cqlsh that has been open for a few hours, sometimes it is helpful to be able to type a quick command and see your current user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7743) Possible C* OOM issue during long running test
[
https://issues.apache.org/jira/browse/CASSANDRA-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100197#comment-14100197
]
Benedict commented on CASSANDRA-7743:
-
Did you see the actual error, or have more info than meminfo? Because that is
not at all conclusive by itself.
Possible C* OOM issue during long running test
--
Key: CASSANDRA-7743
URL: https://issues.apache.org/jira/browse/CASSANDRA-7743
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Google Compute Engine, n1-standard-1
Reporter: Pierre Laporte
Assignee: Benedict
Fix For: 2.1 rc6
During a long running test, we ended up with a lot of
java.lang.OutOfMemoryError: Direct buffer memory errors on the Cassandra
instances.
Here is an example of stacktrace from system.log :
{code}
ERROR [SharedPool-Worker-1] 2014-08-11 11:09:34,610 ErrorMessage.java:218 -
Unexpected exception during request
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_25]
at java.nio.DirectByteBuffer.init(DirectByteBuffer.java:123)
~[na:1.7.0_25]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
~[na:1.7.0_25]
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:434)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:179)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:168)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.buffer.PoolArena.allocate(PoolArena.java:98)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:251)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:155)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:146)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:107)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:112)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:507)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:464)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:378)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:350)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
~[netty-all-4.0.20.Final.jar:4.0.20.Final]
at java.lang.Thread.run(Thread.java:724) ~[na:1.7.0_25]
{code}
The test consisted of a 3-nodes cluster of n1-standard-1 GCE instances (1
vCPU, 3.75 GB RAM) running cassandra-2.1.0-rc5, and a n1-standard-2 instance
running the test.
After ~2.5 days, several requests start to fail and we see the previous
stacktraces in the system.log file.
The output from linux ‘free’ and ‘meminfo’ suggest that there is still memory
available.
{code}
$ free -m
total used free sharedbuffers cached
Mem: 3702 3532169 0161854
-/+ buffers/cache: 2516 1185
Swap:0 0 0
$ head -n 4 /proc/meminfo
MemTotal:3791292 kB
MemFree: 173568 kB
Buffers: 165608 kB
Cached: 874752 kB
{code}
These errors do not affect all the queries we run. The cluster is still
responsive but is unable to display tracing information using cqlsh :
{code}
$ ./bin/nodetool --host 10.240.137.253 status duration_test
Datacenter: DC1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.240.98.27925.17 KB 256 100.0%
41314169-eff5-465f-85ea-d501fd8f9c5e RAC1
UN 10.240.137.253 1.1 MB 256 100.0%
c706f5f9-c5f3-4d5e-95e9-a8903823827e RAC1
UN 10.240.72.183 896.57 KB 256 100.0%
[jira] [Commented] (CASSANDRA-7220) Nodes hang with 100% CPU load
[ https://issues.apache.org/jira/browse/CASSANDRA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100232#comment-14100232 ] Robert Rudduck commented on CASSANDRA-7220: --- I am experiencing the same symptoms, but do not have any special cluster settings. 4 Nodes (VMs), nearly all writes with very few reads. Randomly after a day or two a node will appear offline in OpsCenter, but logging into the VM the cassandra process is running @ 100% cpu but is not responding. I can provide logs if needed. Thanks. - Robert Nodes hang with 100% CPU load - Key: CASSANDRA-7220 URL: https://issues.apache.org/jira/browse/CASSANDRA-7220 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.7 4 nodes cluster on 12 core machines Reporter: Robert Stupp Assignee: Ryan McGuire Attachments: c-12-read-100perc-cpu.zip I've ran a test that both reads and writes rows. After some time, all writes succeeded and all reads stopped. Two of the four nodes have 16 of 16 threads of the ReadStage thread pool running. The number of pending task continuouly grows on these nodes. I have attached outputs of the stack traces and some diagnostic output from nodetool tpstats nodetool status shows all nodes as UN. I had run that test previously without any issues in with the same configuration. Some specials from cassandra.yaml: - key_cache_size_in_mb: 1024 - row_cache_size_in_mb: 8192 The nodes running at 100% CPU are node2 and node3. node1node4 are fine. I'm not sure if it is reproducable - but it's definitly not a good behaviour. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7519) Further stress improvements to generate more realistic workloads
[ https://issues.apache.org/jira/browse/CASSANDRA-7519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100233#comment-14100233 ] T Jake Luciani commented on CASSANDRA-7519: --- I'm not very keen on the new labels you've chosen for the insert section of the yaml file, They should be more verbose. Batch size - number of unique partitions to update in a single operation This should mention partitions in it no? partitions_per_batch maybe? Batch count - number of batches we aim to split the update up into Does this mean the number of batches to split a operation of N partitions into? If so, then perhaps batch_split_count? I plan to run some test workloads to double check the logic, but first cut of the code looked good. I left a couple comments on the github branch Further stress improvements to generate more realistic workloads Key: CASSANDRA-7519 URL: https://issues.apache.org/jira/browse/CASSANDRA-7519 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Benedict Assignee: Benedict Priority: Minor Labels: tools Fix For: 2.1.1 We generally believe that the most common workload is for reads to exponentially prefer most recently written data. However as stress currently behaves we have two id generation modes: sequential and random (although random can be distributed). I propose introducing a new mode which is somewhat like sequential, except we essentially 'look back' from the current id by some amount defined by a distribution. I may possibly make the position only increment as it's first written to also, so that this mode can be run from a clean slate with a mixed workload. This should allow is to generate workloads that are more representative. At the same time, I will introduce a timestamp value generator for primary key columns that is strictly ascending, i.e. has some random component but is based off of the actual system time (or some shared monotonically increasing state) so that we can again generate a more realistic workload. This may be challenging to tie in with the new procedurally generated partitions, but I'm sure it can be done without too much difficulty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7519) Further stress improvements to generate more realistic workloads
[ https://issues.apache.org/jira/browse/CASSANDRA-7519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100237#comment-14100237 ] Benedict commented on CASSANDRA-7519: - bq. I plan to run some test workloads to double check the logic, but first cut of the code looked good. I left a couple comments on the github branch Thanks! bq. I'm not very keen on the new labels you've chosen for the insert section of the yaml file, They should be more verbose Nomenclature is always tricky, certainly not fixed on them. Although by making these more verbose we'll need to make the command line correspondingly more verbose to keep them in sync, which I'm not super keen on, but not too fussed about either. bq. partitions_per_batch maybe? perhaps partitions_per_operation? because per_batch implies we might change the number of partitions between batches, whereas we work with the same partitions for the duration of an 'operation' (the n= declared on command line)... bq. batch_split_count batches_per_operation? Further stress improvements to generate more realistic workloads Key: CASSANDRA-7519 URL: https://issues.apache.org/jira/browse/CASSANDRA-7519 Project: Cassandra Issue Type: Improvement Components: Tools Reporter: Benedict Assignee: Benedict Priority: Minor Labels: tools Fix For: 2.1.1 We generally believe that the most common workload is for reads to exponentially prefer most recently written data. However as stress currently behaves we have two id generation modes: sequential and random (although random can be distributed). I propose introducing a new mode which is somewhat like sequential, except we essentially 'look back' from the current id by some amount defined by a distribution. I may possibly make the position only increment as it's first written to also, so that this mode can be run from a clean slate with a mixed workload. This should allow is to generate workloads that are more representative. At the same time, I will introduce a timestamp value generator for primary key columns that is strictly ascending, i.e. has some random component but is based off of the actual system time (or some shared monotonically increasing state) so that we can again generate a more realistic workload. This may be challenging to tie in with the new procedurally generated partitions, but I'm sure it can be done without too much difficulty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7705) Safer Resource Management
[ https://issues.apache.org/jira/browse/CASSANDRA-7705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100260#comment-14100260 ] Benedict commented on CASSANDRA-7705: - Linked four related tickets Safer Resource Management - Key: CASSANDRA-7705 URL: https://issues.apache.org/jira/browse/CASSANDRA-7705 Project: Cassandra Issue Type: Improvement Components: Core Reporter: Benedict Fix For: 3.0 We've had a spate of bugs recently with bad reference counting. these can have potentially dire consequences, generally either randomly deleting data or giving us infinite loops. Since in 2.1 we only reference count resources that are relatively expensive and infrequently managed (or in places where this safety is probably not as necessary, e.g. SerializingCache), we could without any negative consequences (and only slight code complexity) introduce a safer resource management scheme for these more expensive/infrequent actions. Basically, I propose when we want to acquire a resource we allocate an object that manages the reference. This can only be released once; if it is released twice, we fail immediately at the second release, reporting where the bug is (rather than letting it continue fine until the next correct release corrupts the count). The reference counter remains the same, but we obtain guarantees that the reference count itself is never badly maintained, although code using it could mistakenly release its own handle early (typically this is only an issue when cleaning up after a failure, in which case under the new scheme this would be an innocuous error) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7220) Nodes hang with 100% CPU load
[ https://issues.apache.org/jira/browse/CASSANDRA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100261#comment-14100261 ] Benedict commented on CASSANDRA-7220: - Any other exceptions in the logs? Looks related to CASSANDRA-7262, CASSANDRA-7704, CASSANDRA-7705. It's likely this has been fixed in a newer release. Nodes hang with 100% CPU load - Key: CASSANDRA-7220 URL: https://issues.apache.org/jira/browse/CASSANDRA-7220 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.7 4 nodes cluster on 12 core machines Reporter: Robert Stupp Assignee: Ryan McGuire Attachments: c-12-read-100perc-cpu.zip I've ran a test that both reads and writes rows. After some time, all writes succeeded and all reads stopped. Two of the four nodes have 16 of 16 threads of the ReadStage thread pool running. The number of pending task continuouly grows on these nodes. I have attached outputs of the stack traces and some diagnostic output from nodetool tpstats nodetool status shows all nodes as UN. I had run that test previously without any issues in with the same configuration. Some specials from cassandra.yaml: - key_cache_size_in_mb: 1024 - row_cache_size_in_mb: 8192 The nodes running at 100% CPU are node2 and node3. node1node4 are fine. I'm not sure if it is reproducable - but it's definitly not a good behaviour. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (CASSANDRA-7220) Nodes hang with 100% CPU load
[ https://issues.apache.org/jira/browse/CASSANDRA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Rudduck updated CASSANDRA-7220: -- Attachment: system.log That is possible - we are running 2.0.8 at the moment IIRC. Attached is the system log of a node that just recently exhibited the same behavior, although it did not stay at 100% CPU for very long after it stopped responding. It is possible that it is not related to this bug directly although the general symptoms seem similar. If I need to open a new issue let me know. Nodes hang with 100% CPU load - Key: CASSANDRA-7220 URL: https://issues.apache.org/jira/browse/CASSANDRA-7220 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.7 4 nodes cluster on 12 core machines Reporter: Robert Stupp Assignee: Ryan McGuire Attachments: c-12-read-100perc-cpu.zip, system.log I've ran a test that both reads and writes rows. After some time, all writes succeeded and all reads stopped. Two of the four nodes have 16 of 16 threads of the ReadStage thread pool running. The number of pending task continuouly grows on these nodes. I have attached outputs of the stack traces and some diagnostic output from nodetool tpstats nodetool status shows all nodes as UN. I had run that test previously without any issues in with the same configuration. Some specials from cassandra.yaml: - key_cache_size_in_mb: 1024 - row_cache_size_in_mb: 8192 The nodes running at 100% CPU are node2 and node3. node1node4 are fine. I'm not sure if it is reproducable - but it's definitly not a good behaviour. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7220) Nodes hang with 100% CPU load
[ https://issues.apache.org/jira/browse/CASSANDRA-7220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100290#comment-14100290 ] Benedict commented on CASSANDRA-7220: - [~rarudduck] it looks like the issue that killed your server was OOM. I can't see a reason for this in the logs, so it's possible you simply need to increase your heap size, however upgrading may help as there are a LOT of exceptions related to CASSANDRA-7756 logged, and it's possible that's somehow causing a knock on effect of some kind. Nodes hang with 100% CPU load - Key: CASSANDRA-7220 URL: https://issues.apache.org/jira/browse/CASSANDRA-7220 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.7 4 nodes cluster on 12 core machines Reporter: Robert Stupp Assignee: Ryan McGuire Attachments: c-12-read-100perc-cpu.zip, system.log I've ran a test that both reads and writes rows. After some time, all writes succeeded and all reads stopped. Two of the four nodes have 16 of 16 threads of the ReadStage thread pool running. The number of pending task continuouly grows on these nodes. I have attached outputs of the stack traces and some diagnostic output from nodetool tpstats nodetool status shows all nodes as UN. I had run that test previously without any issues in with the same configuration. Some specials from cassandra.yaml: - key_cache_size_in_mb: 1024 - row_cache_size_in_mb: 8192 The nodes running at 100% CPU are node2 and node3. node1node4 are fine. I'm not sure if it is reproducable - but it's definitly not a good behaviour. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (CASSANDRA-7786) Cassandra is shutting down out of no apparent reason
[ https://issues.apache.org/jira/browse/CASSANDRA-7786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14100293#comment-14100293 ] Benedict commented on CASSANDRA-7786: - Are you sure you haven't somehow sent the message over JMX / nodeprobe somehow? Possibly a script accidentally has the command embedded? There doesn't seem to be any code path that could shutdown the server without first logging an exception. Cassandra is shutting down out of no apparent reason Key: CASSANDRA-7786 URL: https://issues.apache.org/jira/browse/CASSANDRA-7786 Project: Cassandra Issue Type: Bug Components: Core Environment: C* 2.0.9 Reporter: Or Sher We've recently start facing an issue when one of the C* node in our dev and CI cluster (Thanks god didn't happen yet in Prod) is shutting down from time to time without any exceptions or errors. There is usually something like that in the logs: INFO [MemoryMeter:1] 2014-08-15 01:32:43,266 Memtable.java (line 481) CFS(Keyspace='system', ColumnFamily='sstable_activity') liveRatio is 14.597030881851438 (just-counted was 14.596825396825396). calculation took 2ms for 84 cells INFO [StorageServiceShutdownHook] 2014-08-15 01:40:58,954 ThriftServer.java (line 141) Stop listening to thrift clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,007 Server.java (line 182) Stop listening for CQL clients INFO [StorageServiceShutdownHook] 2014-08-15 01:40:59,011 Gossiper.java (line 1279) Announcing shutdown INFO [StorageServiceShutdownHook] 2014-08-15 01:41:01,011 MessagingService.java (line 683) Waiting for messaging service to quiesce INFO [ACCEPT-/192.168.27.241] 2014-08-15 01:41:01,012 MessagingService.java (line 923) MessagingService has terminated the accept() thread INFO [main] 2014-08-17 09:50:56,647 CassandraDaemon.java (line 135) Logging initialized You can see the last line in the log is usually written at least 5 minutes before the shutdown, sometimes 30 minutes before. I can't reproduce it as I have know idea why is that happening and how the attack this issue. I believe I'm not the only one suffering from this issue as there was a thread about such behavior in the user mail distribution. Any thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
