[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13092394#comment-13092394 ] Wojciech Meler commented on CASSANDRA-2973: --- I've examined log files and found one broken sstable for which I have pre-cleanup and pre-scrub snapshots. It's too big for test case (8GB). I'll try to isolate the problem. fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler Assignee: Sylvain Lebresne after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13085698#comment-13085698 ] Wojciech Meler commented on CASSANDRA-2973: --- I'm back. As Michał said cluster history was: 0.7.0 - 0.7.2 - 0.7.3 - 0.7.4 - 0.8.0 - 0.8.1. It started with 6 nodes. After migrating to 0.8.0 cluster grew to 12, and after 0.8.1 to 18 nodes. It's hard to say which CF got read errors, but exceptions from scrub suggests that it was mta_logs which is plain CF. fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler Assignee: Sylvain Lebresne after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073478#comment-13073478 ] Sylvain Lebresne commented on CASSANDRA-2973: - Also, what is the story of that cluster. Is it a straight up 0.8 cluster, or was it upgraded from 0.7 ? fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler Assignee: Sylvain Lebresne after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073500#comment-13073500 ] Michał Kowalczuk commented on CASSANDRA-2973: - Hello, Wojciech is my colleague. This cluster was upgraded from 0.7. I'm not sure which CF was getting read errors, though. And as far as I know, Wojciech will be back on 15th August, not October. fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler Assignee: Sylvain Lebresne after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073145#comment-13073145 ] Wojciech Meler commented on CASSANDRA-2973: --- seen on 12 machines - on all that cleanup was run - it is hard to believe that hardware error caused that fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073165#comment-13073165 ] Jonathan Ellis commented on CASSANDRA-2973: --- agreed :) I don't suppose you have a pre-cleanup snapshot? fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073166#comment-13073166 ] Jonathan Ellis commented on CASSANDRA-2973: --- also: do you have the log from when scrub ran? fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073234#comment-13073234 ] Wojciech Meler commented on CASSANDRA-2973: --- BINGO :) I've run scrub in rolling way :) So I've done prescrub snapshots of 4 remaining nodes. All logs are collected. (Un)fortunately I'm going on vacation so I can give you all you need on 15'th October. Data set is quite big - about 140GB per node, so I'll need some instructions what to extract for you. Snapshot file list from 12th node: -rw-r--r-- 2 cassandra cassandra 49714745792 Jul 30 02:20 fdb-g-1715-Data.db -rw-r--r-- 2 cassandra cassandra 4815616 Jul 30 02:20 fdb-g-1715-Filter.db -rw-r--r-- 2 cassandra cassandra61556583 Jul 30 02:20 fdb-g-1715-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 02:20 fdb-g-1715-Statistics.db -rw-r--r-- 2 cassandra cassandra 3450580822 Jul 30 02:22 fdb-g-1717-Data.db -rw-r--r-- 2 cassandra cassandra 904096 Jul 30 02:22 fdb-g-1717-Filter.db -rw-r--r-- 2 cassandra cassandra18024560 Jul 30 02:22 fdb-g-1717-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 02:22 fdb-g-1717-Statistics.db -rw-r--r-- 2 cassandra cassandra 3213527243 Jul 30 02:25 fdb-g-1721-Data.db -rw-r--r-- 2 cassandra cassandra 904336 Jul 30 02:25 fdb-g-1721-Filter.db -rw-r--r-- 2 cassandra cassandra18026968 Jul 30 02:25 fdb-g-1721-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 02:25 fdb-g-1721-Statistics.db -rw-r--r-- 2 cassandra cassandra98385535 Jul 30 03:20 fdb-g-1723-Data.db -rw-r--r-- 2 cassandra cassandra 247456 Jul 30 03:20 fdb-g-1723-Filter.db -rw-r--r-- 2 cassandra cassandra 3130599 Jul 30 03:20 fdb-g-1723-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 03:20 fdb-g-1723-Statistics.db -rw-r--r-- 2 cassandra cassandra 106856768 Jul 30 09:41 fdb-g-1734-Data.db -rw-r--r-- 2 cassandra cassandra 195376 Jul 30 09:41 fdb-g-1734-Filter.db -rw-r--r-- 2 cassandra cassandra 2713645 Jul 30 09:41 fdb-g-1734-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 09:41 fdb-g-1734-Statistics.db -rw-r--r-- 2 cassandra cassandra 111863768 Jul 30 17:14 fdb-g-1745-Data.db -rw-r--r-- 2 cassandra cassandra 194416 Jul 30 17:14 fdb-g-1745-Filter.db -rw-r--r-- 2 cassandra cassandra 2761017 Jul 30 17:14 fdb-g-1745-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 17:14 fdb-g-1745-Statistics.db -rw-r--r-- 2 cassandra cassandra 845886467 Jul 30 20:16 fdb-g-1750-Data.db -rw-r--r-- 2 cassandra cassandra 902896 Jul 30 20:16 fdb-g-1750-Filter.db -rw-r--r-- 2 cassandra cassandra10419730 Jul 30 20:16 fdb-g-1750-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 20:16 fdb-g-1750-Statistics.db -rw-r--r-- 2 cassandra cassandra40226865 Jul 30 21:12 fdb-g-1751-Data.db -rw-r--r-- 2 cassandra cassandra 66608 Jul 30 21:12 fdb-g-1751-Filter.db -rw-r--r-- 2 cassandra cassandra 1319135 Jul 30 21:12 fdb-g-1751-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 21:12 fdb-g-1751-Statistics.db -rw-r--r-- 2 cassandra cassandra 10904996806 Jul 30 02:41 mailcache-g-3708-Data.db -rw-r--r-- 2 cassandra cassandra 254656 Jul 30 02:41 mailcache-g-3708-Filter.db -rw-r--r-- 2 cassandra cassandra 5073034 Jul 30 02:41 mailcache-g-3708-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 02:41 mailcache-g-3708-Statistics.db -rw-r--r-- 2 cassandra cassandra 18281871008 Jul 30 02:55 mailcache-g-3712-Data.db -rw-r--r-- 2 cassandra cassandra 968176 Jul 30 02:55 mailcache-g-3712-Filter.db -rw-r--r-- 2 cassandra cassandra11904976 Jul 30 02:55 mailcache-g-3712-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 02:55 mailcache-g-3712-Statistics.db -rw-r--r-- 2 cassandra cassandra35390850 Jul 30 04:41 mailcache-g-3717-Data.db -rw-r--r-- 2 cassandra cassandra 640 Jul 30 04:41 mailcache-g-3717-Filter.db -rw-r--r-- 2 cassandra cassandra 12533 Jul 30 04:41 mailcache-g-3717-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 04:41 mailcache-g-3717-Statistics.db -rw-r--r-- 2 cassandra cassandra 9067927959 Jul 30 11:02 mailcache-g-3727-Data.db -rw-r--r-- 2 cassandra cassandra 296416 Jul 30 11:02 mailcache-g-3727-Filter.db -rw-r--r-- 2 cassandra cassandra 3578551 Jul 30 11:02 mailcache-g-3727-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 11:02 mailcache-g-3727-Statistics.db -rw-r--r-- 2 cassandra cassandra67351540 Jul 30 16:08 mailcache-g-3738-Data.db -rw-r--r-- 2 cassandra cassandra1512 Jul 30 16:08 mailcache-g-3738-Filter.db -rw-r--r-- 2 cassandra cassandra 29793 Jul 30 16:08 mailcache-g-3738-Index.db -rw-r--r-- 2 cassandra cassandra4276 Jul 30 16:08 mailcache-g-3738-Statistics.db -rw-r--r-- 2 cassandra cassandra57132425 Jul 30 16:14 mailcache-g-3739-Data.db -rw-r--r-- 2 cassandra
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073235#comment-13073235 ] Wojciech Meler commented on CASSANDRA-2973: --- :((( just noticed that pre-scrub snapshots are taken by default I've done snapshot on Jul 24th, cleanup on Jun 29th so don't know if it will help... fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073237#comment-13073237 ] Wojciech Meler commented on CASSANDRA-2973: --- Scrub log from 9th node 2011-07-30 21:39:55:386 CEST INFO [CompactionExecutor:172][org.apache.cassandra.db.compaction.CompactionManager] Scrubbing SSTableReader(path='/var/lib/cassandra/data/mail/mta_logs-g-733-Data.db') 2011-07-30 21:39:55:469 CEST WARN [CompactionExecutor:172][org.apache.cassandra.db.compaction.CompactionManager] Non-fatal error reading row (stacktrace follows) java.io.IOError: java.io.IOException: Impossible row size 7933455022436196 at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:719) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:633) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65) at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:250) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: Impossible row size 7933455022436196 ... 9 more 2011-07-30 21:39:55:470 CEST INFO [CompactionExecutor:172][org.apache.cassandra.db.compaction.CompactionManager] Retrying from row index; data is -8 bytes starting at 547199 2011-07-30 21:39:55:470 CEST WARN [CompactionExecutor:172][org.apache.cassandra.db.compaction.CompactionManager] Retry failed too. Skipping to next row (retry's stacktrace follows) java.io.IOError: java.io.EOFException: bloom filter claims to be 909194543 bytes, longer than entire row size -8 at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:90) at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:748) at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:633) at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:65) at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:250) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.EOFException: bloom filter claims to be 909194543 bytes, longer than entire row size -8 at org.apache.cassandra.io.sstable.IndexHelper.defreezeBloomFilter(IndexHelper.java:111) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.init(SSTableIdentityIterator.java:119) ... 10 more 2011-07-30 21:39:55:482 CEST WARN [CompactionExecutor:172][org.apache.cassandra.db.compaction.CompactionManager] Non-fatal error reading row (stacktrace follows) # ls -l /var/lib/cassandra/data/mail/mta_logs-g-733* -rw-r--r-- 1 cassandra cassandra 0 Jul 30 21:42 /var/lib/cassandra/data/mail/mta_logs-g-733-Compacted -rw-r--r-- 3 cassandra cassandra 8398108921 Jul 29 21:37 /var/lib/cassandra/data/mail/mta_logs-g-733-Data.db -rw-r--r-- 3 cassandra cassandra2547136 Jul 29 21:37 /var/lib/cassandra/data/mail/mta_logs-g-733-Filter.db -rw-r--r-- 3 cassandra cassandra 50895929 Jul 29 21:37 /var/lib/cassandra/data/mail/mta_logs-g-733-Index.db -rw-r--r-- 3 cassandra cassandra 4276 Jul 29 21:37 /var/lib/cassandra/data/mail/mta_logs-g-733-Statistics.db # ls -l /var/lib/cassandra/data/mail/snapshots/1312053521562/mta_logs-g-733* -rw-r--r-- 3 cassandra cassandra 8398108921 Jul 29 21:37 /var/lib/cassandra/data/mail/snapshots/1312053521562/mta_logs-g-733-Data.db -rw-r--r-- 3 cassandra cassandra2547136 Jul 29 21:37 /var/lib/cassandra/data/mail/snapshots/1312053521562/mta_logs-g-733-Filter.db -rw-r--r-- 3 cassandra cassandra 50895929 Jul 29 21:37 /var/lib/cassandra/data/mail/snapshots/1312053521562/mta_logs-g-733-Index.db -rw-r--r-- 3 cassandra cassandra 4276 Jul 29 21:37 /var/lib/cassandra/data/mail/snapshots/1312053521562/mta_logs-g-733-Statistics.db # ls -ld /var/lib/cassandra/data/mail/snapshots/1312053521562/ drwxr-xr-x 2 cassandra cassandra 8192 Jul 30 21:18 /var/lib/cassandra/data/mail/snapshots/1312053521562/ Why files in snapshot
[jira] [Commented] (CASSANDRA-2973) fatal errrors after nodetool cleanup
[ https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073126#comment-13073126 ] Jonathan Ellis commented on CASSANDRA-2973: --- Is it reproducible? If not, it's quite possible that it's transient hardware-caused corruption. fatal errrors after nodetool cleanup Key: CASSANDRA-2973 URL: https://issues.apache.org/jira/browse/CASSANDRA-2973 Project: Cassandra Issue Type: Bug Affects Versions: 0.8.1 Reporter: Wojciech Meler after adding nodes to cluster running cleanup I get scaring exceptions in log: 2011-07-30 00:00:05:506 CEST ERROR [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal exception in thread Thread[ReadStage:2335,5,main] java.io.IOError: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:80) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91) at org.apache.cassandra.db.columniterator.SSTableSliceIterator.init(SSTableSliceIterator.java:67) at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66) at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80) at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189) at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146) at org.apache.cassandra.db.Table.getRow(Table.java:385) at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61) at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69) at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.io.IOException: mmap segment underflow; remaining is 4394 but 60165 requested at org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117) at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389) at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368) at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194) at org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83) at org.apache.cassandra.db.columniterator.IndexedSliceReader.init(IndexedSliceReader.java:73) ... 14 more exceptions disappeared after running scrub -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
