[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14188434#comment-14188434 ] Jonathan Ellis commented on CASSANDRA-5256: --- That is a different error; please open a new ticket. Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Assignee: Jonathan Ellis Priority: Critical Labels: compaction Fix For: 1.2.2 Attachments: 5256-v2.txt, 5256-v4.txt, 5256-v5.txt, 5256.txt When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- I've invoked the `nodetool compact` three times; this occurred after each. The node has been up for a couple days accepting writes and has not been restarted. Here's the server's log since it was started a few days ago: https://gist.github.com/cscotta/4956472/raw/95e7cbc68de1aefaeca11812cbb98d5d46f534e8/cassandra.log Here's the
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187189#comment-14187189
]
Nikolai Grigoriev commented on CASSANDRA-5256:
--
I have just got this problem on multiple nodes. Cassandra 2.0.10 (DSE 4.5.2).
Should I reopen?
{code}
ERROR [CompactionExecutor:1196] 2014-10-28 17:14:50,124 CassandraDaemon.java
(line 199) Exception in thread Thread[CompactionExecutor:1196,1,main]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:259)
at org.apache.cassandra.io.util.Memory.getInt(Memory.java:211)
at
org.apache.cassandra.io.sstable.IndexSummary.getIndex(IndexSummary.java:79)
at
org.apache.cassandra.io.sstable.IndexSummary.getKey(IndexSummary.java:84)
at
org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:58)
at
org.apache.cassandra.io.sstable.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:692)
at
org.apache.cassandra.io.sstable.SSTableReader.estimatedKeysForRanges(SSTableReader.java:663)
at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:328)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:354)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:125)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:113)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:192)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
Memory was freed AssertionError During Major Compaction
-
Key: CASSANDRA-5256
URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0
Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
java version 1.6.0_30
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
Ubuntu 12.04.2 LTS
Reporter: C. Scott Andreas
Assignee: Jonathan Ellis
Priority: Critical
Labels: compaction
Fix For: 1.2.2
Attachments: 5256-v2.txt, 5256-v4.txt, 5256-v5.txt, 5256.txt
When initiating a major compaction with `./nodetool -h localhost compact`, an
AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
(line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
at
org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
at
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581291#comment-13581291 ] Jonathan Ellis commented on CASSANDRA-5256: --- I think there are two problems here: # I was wrong, we're not starting with non-compacting candidates (this is the case for STCS, but not LCS, which checks for compacting-ness later on). # The assert is broken in a concurrent compactions world; it's possible that the sstable we're checking for overlaps with, gets compacted away already by a different task before that assert fires. v5 edits findDroppableSSTable to check for compacting status, and removes the getOverlappingSSTables assert. Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Assignee: Jonathan Ellis Priority: Critical Labels: compaction Fix For: 1.2.2 Attachments: 5256.txt, 5256-v2.txt, 5256-v4.txt When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13581905#comment-13581905 ] Sylvain Lebresne commented on CASSANDRA-5256: - +1 on v5. Nits: the comment of CompactionTask.execute is now outdated about the unmarking part (I would also probably have move the unmarking part in AbsractCompactionTask rather than CompactionTask as that felt safer to me in principle, but that doesn't really matter). Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Assignee: Jonathan Ellis Priority: Critical Labels: compaction Fix For: 1.2.2 Attachments: 5256.txt, 5256-v2.txt, 5256-v4.txt, 5256-v5.txt When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- I've invoked the `nodetool compact` three times; this occurred after each. The node has been up
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13579534#comment-13579534 ] Yuki Morishita commented on CASSANDRA-5256: --- Overall, looks good and simplifies things. One problem I had was CompactionsTest was throwing AssertionError on testSingleSSTableCompactionWithLeveledCompaction, when trying to find overlapping sstables to determine droppable ones. Patch for that pushed to https://github.com/yukim/cassandra/commits/5256-3. Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Assignee: Jonathan Ellis Priority: Critical Labels: compaction Fix For: 1.2.2 Attachments: 5256.txt, 5256-v2.txt When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- I've invoked the `nodetool compact` three times; this occurred after each. The node has
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13579565#comment-13579565
]
Sylvain Lebresne commented on CASSANDRA-5256:
-
lgtm, +1.
Nits:
* we could also move the try \{ task.execute() \} finally \{
task.unmarkSSTables(); \} inside AbstractCompactionTask to 1) avoid repeating
it 3 times and 2) making sure we don't forget about it.
* maybe the ctor of AbstractCompactionTask could assert that the sstable it
gets are 'marked compacting' to avoid that kind of mistake in the future.
Memory was freed AssertionError During Major Compaction
-
Key: CASSANDRA-5256
URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0
Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
java version 1.6.0_30
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
Ubuntu 12.04.2 LTS
Reporter: C. Scott Andreas
Assignee: Jonathan Ellis
Priority: Critical
Labels: compaction
Fix For: 1.2.2
Attachments: 5256.txt, 5256-v2.txt
When initiating a major compaction with `./nodetool -h localhost compact`, an
AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
(line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
at
org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71)
at
org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
---
I've invoked the `nodetool compact`
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13579718#comment-13579718
]
Jonathan Ellis commented on CASSANDRA-5256:
---
bq. One problem I had was CompactionsTest was throwing AssertionError on
testSingleSSTableCompactionWithLeveledCompaction, when trying to find
overlapping sstables to determine droppable ones.
I'm worried this patch fixes a symptom instead of a deeper bug:
# We already start with {{SetSSTableReader candidates =
cfs.getUncompactingSSTables();}}; fixing a race by adding additional checks
seems like it will just make the race less likely rather than eliminate it
entirely
# I'm not sure why compacting status should affect getOverlappingSSTables
For the record, here's the stack trace in question:
{noformat}
[junit] Caused by: java.util.concurrent.ExecutionException:
java.lang.AssertionError
[junit] at
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
[junit] at java.util.concurrent.FutureTask.get(FutureTask.java:83)
[junit] at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:373)
[junit] Caused by: java.lang.AssertionError
[junit] at
org.apache.cassandra.db.ColumnFamilyStore.getOverlappingSSTables(ColumnFamilyStore.java:882)
[junit] at
org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:184)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:296)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getMaximalTask(LeveledCompactionStrategy.java:107)
[junit] at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:95)
[junit] at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:185)
{noformat}
Memory was freed AssertionError During Major Compaction
-
Key: CASSANDRA-5256
URL: https://issues.apache.org/jira/browse/CASSANDRA-5256
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.2.0
Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu
Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
java version 1.6.0_30
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
Ubuntu 12.04.2 LTS
Reporter: C. Scott Andreas
Assignee: Jonathan Ellis
Priority: Critical
Labels: compaction
Fix For: 1.2.2
Attachments: 5256.txt, 5256-v2.txt
When initiating a major compaction with `./nodetool -h localhost compact`, an
AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory:
ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java
(line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime]
java.lang.AssertionError: Memory was freed
at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146)
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116)
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176)
at
org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88)
at
org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327)
at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755)
at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792)
at
org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114)
at
org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101)
at
org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109)
at
org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578694#comment-13578694 ] Jonathan Ellis commented on CASSANDRA-5256: --- Is this LeveledCompactionStrategy or SizeTiered? Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Priority: Critical When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- I've invoked the `nodetool compact` three times; this occurred after each. The node has been up for a couple days accepting writes and has not been restarted. Here's the server's log since it was started a few days ago: https://gist.github.com/cscotta/4956472/raw/95e7cbc68de1aefaeca11812cbb98d5d46f534e8/cassandra.log Here's the code being used to issue writes to the datastore: https://gist.github.com/cscotta/20cbd36c2503c71d06e9 --- Configuration: One node, one keyspace, one column
[jira] [Commented] (CASSANDRA-5256) Memory was freed AssertionError During Major Compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-5256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13578723#comment-13578723 ] Brandon Williams commented on CASSANDRA-5256: - It's STS. Memory was freed AssertionError During Major Compaction - Key: CASSANDRA-5256 URL: https://issues.apache.org/jira/browse/CASSANDRA-5256 Project: Cassandra Issue Type: Bug Components: Core Affects Versions: 1.2.0 Environment: Linux ashbdrytest01p 3.2.0-37-generic #58-Ubuntu SMP Thu Jan 24 15:28:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux java version 1.6.0_30 Java(TM) SE Runtime Environment (build 1.6.0_30-b12) Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode) Ubuntu 12.04.2 LTS Reporter: C. Scott Andreas Priority: Critical When initiating a major compaction with `./nodetool -h localhost compact`, an AssertionError is thrown in the CompactionExecutor from o.a.c.io.util.Memory: ERROR [CompactionExecutor:41495] 2013-02-14 14:38:35,720 CassandraDaemon.java (line 133) Exception in thread Thread[CompactionExecutor:41495,1,RMI Runtime] java.lang.AssertionError: Memory was freed at org.apache.cassandra.io.util.Memory.checkPosition(Memory.java:146) at org.apache.cassandra.io.util.Memory.getLong(Memory.java:116) at org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:176) at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:88) at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:327) at java.io.RandomAccessFile.readInt(RandomAccessFile.java:755) at java.io.RandomAccessFile.readLong(RandomAccessFile.java:792) at org.apache.cassandra.utils.BytesReadTracker.readLong(BytesReadTracker.java:114) at org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:101) at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:92) at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149) at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:235) at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:109) at org.apache.cassandra.db.compaction.PrecompactedRow.init(PrecompactedRow.java:93) at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:162) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76) at org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57) at org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114) at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138) at org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:158) at org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at org.apache.cassandra.db.compaction.CompactionTask.execute(CompactionTask.java:71) at org.apache.cassandra.db.compaction.CompactionManager$6.runMayThrow(CompactionManager.java:342) at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) --- I've invoked the `nodetool compact` three times; this occurred after each. The node has been up for a couple days accepting writes and has not been restarted. Here's the server's log since it was started a few days ago: https://gist.github.com/cscotta/4956472/raw/95e7cbc68de1aefaeca11812cbb98d5d46f534e8/cassandra.log Here's the code being used to issue writes to the datastore: https://gist.github.com/cscotta/20cbd36c2503c71d06e9 --- Configuration: One node, one keyspace, one column family. ~60 writes/second of data
