[
https://issues.apache.org/jira/browse/CASSANDRA-2973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073234#comment-13073234
]
Wojciech Meler commented on CASSANDRA-2973:
-------------------------------------------
BINGO :) I've run scrub in "rolling" way :)
So I've done prescrub snapshots of 4 remaining nodes.
All logs are collected. (Un)fortunately I'm going on vacation so I can give you
all you need on 15'th October.
Data set is quite big - about 140GB per node, so I'll need some instructions
what to extract for you.
Snapshot file list from 12th node:
-rw-r--r-- 2 cassandra cassandra 49714745792 Jul 30 02:20 fdb-g-1715-Data.db
-rw-r--r-- 2 cassandra cassandra 4815616 Jul 30 02:20 fdb-g-1715-Filter.db
-rw-r--r-- 2 cassandra cassandra 61556583 Jul 30 02:20 fdb-g-1715-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:20
fdb-g-1715-Statistics.db
-rw-r--r-- 2 cassandra cassandra 3450580822 Jul 30 02:22 fdb-g-1717-Data.db
-rw-r--r-- 2 cassandra cassandra 904096 Jul 30 02:22 fdb-g-1717-Filter.db
-rw-r--r-- 2 cassandra cassandra 18024560 Jul 30 02:22 fdb-g-1717-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:22
fdb-g-1717-Statistics.db
-rw-r--r-- 2 cassandra cassandra 3213527243 Jul 30 02:25 fdb-g-1721-Data.db
-rw-r--r-- 2 cassandra cassandra 904336 Jul 30 02:25 fdb-g-1721-Filter.db
-rw-r--r-- 2 cassandra cassandra 18026968 Jul 30 02:25 fdb-g-1721-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:25
fdb-g-1721-Statistics.db
-rw-r--r-- 2 cassandra cassandra 98385535 Jul 30 03:20 fdb-g-1723-Data.db
-rw-r--r-- 2 cassandra cassandra 247456 Jul 30 03:20 fdb-g-1723-Filter.db
-rw-r--r-- 2 cassandra cassandra 3130599 Jul 30 03:20 fdb-g-1723-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 03:20
fdb-g-1723-Statistics.db
-rw-r--r-- 2 cassandra cassandra 106856768 Jul 30 09:41 fdb-g-1734-Data.db
-rw-r--r-- 2 cassandra cassandra 195376 Jul 30 09:41 fdb-g-1734-Filter.db
-rw-r--r-- 2 cassandra cassandra 2713645 Jul 30 09:41 fdb-g-1734-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 09:41
fdb-g-1734-Statistics.db
-rw-r--r-- 2 cassandra cassandra 111863768 Jul 30 17:14 fdb-g-1745-Data.db
-rw-r--r-- 2 cassandra cassandra 194416 Jul 30 17:14 fdb-g-1745-Filter.db
-rw-r--r-- 2 cassandra cassandra 2761017 Jul 30 17:14 fdb-g-1745-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 17:14
fdb-g-1745-Statistics.db
-rw-r--r-- 2 cassandra cassandra 845886467 Jul 30 20:16 fdb-g-1750-Data.db
-rw-r--r-- 2 cassandra cassandra 902896 Jul 30 20:16 fdb-g-1750-Filter.db
-rw-r--r-- 2 cassandra cassandra 10419730 Jul 30 20:16 fdb-g-1750-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:16
fdb-g-1750-Statistics.db
-rw-r--r-- 2 cassandra cassandra 40226865 Jul 30 21:12 fdb-g-1751-Data.db
-rw-r--r-- 2 cassandra cassandra 66608 Jul 30 21:12 fdb-g-1751-Filter.db
-rw-r--r-- 2 cassandra cassandra 1319135 Jul 30 21:12 fdb-g-1751-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12
fdb-g-1751-Statistics.db
-rw-r--r-- 2 cassandra cassandra 10904996806 Jul 30 02:41
mailcache-g-3708-Data.db
-rw-r--r-- 2 cassandra cassandra 254656 Jul 30 02:41
mailcache-g-3708-Filter.db
-rw-r--r-- 2 cassandra cassandra 5073034 Jul 30 02:41
mailcache-g-3708-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:41
mailcache-g-3708-Statistics.db
-rw-r--r-- 2 cassandra cassandra 18281871008 Jul 30 02:55
mailcache-g-3712-Data.db
-rw-r--r-- 2 cassandra cassandra 968176 Jul 30 02:55
mailcache-g-3712-Filter.db
-rw-r--r-- 2 cassandra cassandra 11904976 Jul 30 02:55
mailcache-g-3712-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:55
mailcache-g-3712-Statistics.db
-rw-r--r-- 2 cassandra cassandra 35390850 Jul 30 04:41
mailcache-g-3717-Data.db
-rw-r--r-- 2 cassandra cassandra 640 Jul 30 04:41
mailcache-g-3717-Filter.db
-rw-r--r-- 2 cassandra cassandra 12533 Jul 30 04:41
mailcache-g-3717-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 04:41
mailcache-g-3717-Statistics.db
-rw-r--r-- 2 cassandra cassandra 9067927959 Jul 30 11:02
mailcache-g-3727-Data.db
-rw-r--r-- 2 cassandra cassandra 296416 Jul 30 11:02
mailcache-g-3727-Filter.db
-rw-r--r-- 2 cassandra cassandra 3578551 Jul 30 11:02
mailcache-g-3727-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 11:02
mailcache-g-3727-Statistics.db
-rw-r--r-- 2 cassandra cassandra 67351540 Jul 30 16:08
mailcache-g-3738-Data.db
-rw-r--r-- 2 cassandra cassandra 1512 Jul 30 16:08
mailcache-g-3738-Filter.db
-rw-r--r-- 2 cassandra cassandra 29793 Jul 30 16:08
mailcache-g-3738-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:08
mailcache-g-3738-Statistics.db
-rw-r--r-- 2 cassandra cassandra 57132425 Jul 30 16:14
mailcache-g-3739-Data.db
-rw-r--r-- 2 cassandra cassandra 1224 Jul 30 16:14
mailcache-g-3739-Filter.db
-rw-r--r-- 2 cassandra cassandra 23985 Jul 30 16:14
mailcache-g-3739-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:14
mailcache-g-3739-Statistics.db
-rw-r--r-- 2 cassandra cassandra 3832970739 Jul 30 20:17
mailcache-g-3748-Data.db
-rw-r--r-- 2 cassandra cassandra 107056 Jul 30 20:17
mailcache-g-3748-Filter.db
-rw-r--r-- 2 cassandra cassandra 1660163 Jul 30 20:17
mailcache-g-3748-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:17
mailcache-g-3748-Statistics.db
-rw-r--r-- 2 cassandra cassandra 314153777 Jul 30 20:49
mailcache-g-3749-Data.db
-rw-r--r-- 2 cassandra cassandra 6808 Jul 30 20:49
mailcache-g-3749-Filter.db
-rw-r--r-- 2 cassandra cassandra 135155 Jul 30 20:49
mailcache-g-3749-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:49
mailcache-g-3749-Statistics.db
-rw-r--r-- 2 cassandra cassandra 211709838 Jul 30 21:12
mailcache-g-3750-Data.db
-rw-r--r-- 2 cassandra cassandra 4664 Jul 30 21:12
mailcache-g-3750-Filter.db
-rw-r--r-- 2 cassandra cassandra 92163 Jul 30 21:12
mailcache-g-3750-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12
mailcache-g-3750-Statistics.db
-rw-r--r-- 2 cassandra cassandra 8404555561 Jul 30 01:34
mta_logs-g-1485-Data.db
-rw-r--r-- 2 cassandra cassandra 2551456 Jul 30 01:34
mta_logs-g-1485-Filter.db
-rw-r--r-- 2 cassandra cassandra 50978377 Jul 30 01:34
mta_logs-g-1485-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:34
mta_logs-g-1485-Statistics.db
-rw-r--r-- 2 cassandra cassandra 3759973372 Jul 30 01:37
mta_logs-g-1486-Data.db
-rw-r--r-- 2 cassandra cassandra 1611616 Jul 30 01:37
mta_logs-g-1486-Filter.db
-rw-r--r-- 2 cassandra cassandra 32166666 Jul 30 01:37
mta_logs-g-1486-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:37
mta_logs-g-1486-Statistics.db
-rw-r--r-- 2 cassandra cassandra 4179780390 Jul 30 01:41
mta_logs-g-1487-Data.db
-rw-r--r-- 2 cassandra cassandra 1507936 Jul 30 01:41
mta_logs-g-1487-Filter.db
-rw-r--r-- 2 cassandra cassandra 30052328 Jul 30 01:41
mta_logs-g-1487-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 01:41
mta_logs-g-1487-Statistics.db
-rw-r--r-- 2 cassandra cassandra 138623813 Jul 30 15:41
mta_logs-g-1507-Data.db
-rw-r--r-- 2 cassandra cassandra 429616 Jul 30 15:41
mta_logs-g-1507-Filter.db
-rw-r--r-- 2 cassandra cassandra 5115547 Jul 30 15:41
mta_logs-g-1507-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 15:41
mta_logs-g-1507-Statistics.db
-rw-r--r-- 2 cassandra cassandra 493377854 Jul 30 18:15
mta_logs-g-1513-Data.db
-rw-r--r-- 2 cassandra cassandra 1190896 Jul 30 18:15
mta_logs-g-1513-Filter.db
-rw-r--r-- 2 cassandra cassandra 13534807 Jul 30 18:15
mta_logs-g-1513-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 18:15
mta_logs-g-1513-Statistics.db
-rw-r--r-- 2 cassandra cassandra 36751337 Jul 30 19:14
mta_logs-g-1514-Data.db
-rw-r--r-- 2 cassandra cassandra 107152 Jul 30 19:14
mta_logs-g-1514-Filter.db
-rw-r--r-- 2 cassandra cassandra 2130178 Jul 30 19:14
mta_logs-g-1514-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 19:14
mta_logs-g-1514-Statistics.db
-rw-r--r-- 2 cassandra cassandra 32420585 Jul 30 20:14
mta_logs-g-1515-Data.db
-rw-r--r-- 2 cassandra cassandra 97784 Jul 30 20:14
mta_logs-g-1515-Filter.db
-rw-r--r-- 2 cassandra cassandra 1943186 Jul 30 20:14
mta_logs-g-1515-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:14
mta_logs-g-1515-Statistics.db
-rw-r--r-- 2 cassandra cassandra 30340652 Jul 30 21:12
mta_logs-g-1516-Data.db
-rw-r--r-- 2 cassandra cassandra 90888 Jul 30 21:12
mta_logs-g-1516-Filter.db
-rw-r--r-- 2 cassandra cassandra 1809686 Jul 30 21:12
mta_logs-g-1516-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12
mta_logs-g-1516-Statistics.db
-rw-r--r-- 2 cassandra cassandra 574037723 Jul 30 02:26
quota_counters-g-476-Data.db
-rw-r--r-- 2 cassandra cassandra 3778336 Jul 30 02:26
quota_counters-g-476-Filter.db
-rw-r--r-- 2 cassandra cassandra 74766796 Jul 30 02:26
quota_counters-g-476-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:26
quota_counters-g-476-Statistics.db
-rw-r--r-- 2 cassandra cassandra 66750073 Jul 30 16:14
quota_counters-g-500-Data.db
-rw-r--r-- 2 cassandra cassandra 382512 Jul 30 16:14
quota_counters-g-500-Filter.db
-rw-r--r-- 2 cassandra cassandra 7587562 Jul 30 16:14
quota_counters-g-500-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:14
quota_counters-g-500-Statistics.db
-rw-r--r-- 2 cassandra cassandra 65704662 Jul 30 16:15
quota_counters-g-503-Data.db
-rw-r--r-- 2 cassandra cassandra 409456 Jul 30 16:15
quota_counters-g-503-Filter.db
-rw-r--r-- 2 cassandra cassandra 7134471 Jul 30 16:15
quota_counters-g-503-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:15
quota_counters-g-503-Statistics.db
-rw-r--r-- 2 cassandra cassandra 496890534 Jul 30 16:20
quota_counters-g-505-Data.db
-rw-r--r-- 2 cassandra cassandra 7479616 Jul 30 16:20
quota_counters-g-505-Filter.db
-rw-r--r-- 2 cassandra cassandra 53756417 Jul 30 16:20
quota_counters-g-505-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 16:20
quota_counters-g-505-Statistics.db
-rw-r--r-- 2 cassandra cassandra 101189405 Jul 30 19:13
quota_counters-g-508-Data.db
-rw-r--r-- 2 cassandra cassandra 683296 Jul 30 19:13
quota_counters-g-508-Filter.db
-rw-r--r-- 2 cassandra cassandra 10281151 Jul 30 19:13
quota_counters-g-508-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 19:13
quota_counters-g-508-Statistics.db
-rw-r--r-- 2 cassandra cassandra 36645335 Jul 30 20:49
quota_counters-g-509-Data.db
-rw-r--r-- 2 cassandra cassandra 182984 Jul 30 20:49
quota_counters-g-509-Filter.db
-rw-r--r-- 2 cassandra cassandra 3633104 Jul 30 20:49
quota_counters-g-509-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 20:49
quota_counters-g-509-Statistics.db
-rw-r--r-- 2 cassandra cassandra 10071816 Jul 30 21:12
quota_counters-g-510-Data.db
-rw-r--r-- 2 cassandra cassandra 50536 Jul 30 21:12
quota_counters-g-510-Filter.db
-rw-r--r-- 2 cassandra cassandra 1004098 Jul 30 21:12
quota_counters-g-510-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 21:12
quota_counters-g-510-Statistics.db
-rw-r--r-- 2 cassandra cassandra 118 Jul 30 02:25 quota-g-6-Data.db
-rw-r--r-- 2 cassandra cassandra 496 Jul 30 02:25 quota-g-6-Filter.db
-rw-r--r-- 2 cassandra cassandra 33 Jul 30 02:25 quota-g-6-Index.db
-rw-r--r-- 2 cassandra cassandra 4276 Jul 30 02:25
quota-g-6-Statistics.db
error log sample from that node:
java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
10648 but 25701 requested
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: mmap segment underflow; remaining is 10648 but
25701 requested
at
org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
at
org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194)
at
org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73)
... 14 more
2011-07-30 21:24:48:340 CEST ERROR
[ReadStage:1993][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal
exception in thread Thread[ReadStage:1993,5,main]
java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
27299 but 28532 requested
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: mmap segment underflow; remaining is 27299 but
28532 requested
at
org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389)
at
org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
at
org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194)
at
org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73)
... 14 more
2011-07-30 21:26:43:823 CEST ERROR
[ReadStage:1972][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal
exception in thread Thread[ReadStage:1972,5,main]
java.lang.AssertionError
at
org.apache.cassandra.io.util.MappedFileDataInput.skipBytes(MappedFileDataInput.java:145)
at
org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:227)
at
org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:72)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2011-07-30 21:26:53:869 CEST ERROR
[ReadStage:1978][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal
exception in thread Thread[ReadStage:1978,5,main]
java.lang.AssertionError
at
org.apache.cassandra.io.util.MappedFileDataInput.skipBytes(MappedFileDataInput.java:145)
at
org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:227)
at
org.apache.cassandra.io.sstable.IndexHelper.skipBloomFilter(IndexHelper.java:50)
at
org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:72)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
at
org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
at
org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at
org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
at
org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
at
org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
at org.apache.cassandra.db.Table.getRow(Table.java:385)
at
org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
> fatal errrors after nodetool cleanup
> ------------------------------------
>
> Key: CASSANDRA-2973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2973
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.1
> Reporter: Wojciech Meler
>
> after adding nodes to cluster & running cleanup I get scaring exceptions in
> log:
> 2011-07-30 00:00:05:506 CEST ERROR
> [ReadStage:2335][org.apache.cassandra.service.AbstractCassandraDaemon] Fatal
> exception in thread Thread[ReadStage:2335,5,main]
> java.io.IOError: java.io.IOException: mmap segment underflow; remaining is
> 4394 but 60165 requested
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:80)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:91)
> at
> org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:67)
> at
> org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
> at
> org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:80)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1292)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1189)
> at
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1146)
> at org.apache.cassandra.db.Table.getRow(Table.java:385)
> at
> org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:61)
> at
> org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:69)
> at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.io.IOException: mmap segment underflow; remaining is 4394
> but 60165 requested
> at
> org.apache.cassandra.io.util.MappedFileDataInput.readBytes(MappedFileDataInput.java:117)
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:389)
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:368)
> at
> org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:194)
> at
> org.apache.cassandra.io.sstable.IndexHelper.deserializeIndex(IndexHelper.java:83)
> at
> org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:73)
> ... 14 more
> exceptions disappeared after running scrub
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
