[jira] [Commented] (CASSANDRA-8269) Large number of system hints other CF's cause heap to fill and run OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200315#comment-14200315
]
Aleksey Yeschenko commented on CASSANDRA-8269:
--
I think CASSANDRA-6988 (since 2.0.11) would make this a non-issue.
Large number of system hints other CF's cause heap to fill and run OOM
Key: CASSANDRA-8269
URL: https://issues.apache.org/jira/browse/CASSANDRA-8269
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: DSE 4.5.0 with Apache Cassandra 2.0.5
Reporter: Jose Martinez Poblete
Attachments: alln01-ats-cas2-java_1414110068_Leak_Suspects.zip,
system.log
A 3 node cluster with large amount of sstables for system.hints and other 3
user tables was coming down regularly with OOM on system log showing up the
following:
{noformat}
ERROR [OptionalTasks:1] 2014-10-23 18:51:29,052 CassandraDaemon.java (line
199) Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:187)
at
org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:122)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:229)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:203)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.io.sstable.SSTableScanner.hasNext(SSTableScanner.java:183)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
at
org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:74)
at
org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1586)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1643)
at
org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:513)
at
org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:91)
at
org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:173)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
A heapdump would show the following:
{noformat}
Class Name
| Shallow Heap | Retained Heap | Percentage
--
java.lang.Thread @ 0x67b292138 OptionalTasks:1 Thread
| 104 | 4,901,485,768 | 58.60%
|- org.apache.cassandra.utils.MergeIterator$ManyToOne @ 0x7b9dc4ad8
| 40 | 4,900,817,312 | 58.59%
| |- java.util.ArrayList @ 0x6f05f15f0
| 24 | 403,635,848 | 4.83%
| |- org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator @
0x7b5fe7078| 40 |29,669,312 | 0.35%
| | |- org.apache.cassandra.db.RowIndexEntry$IndexedEntry @ 0x7b7caaa28
| 32 |26,770,264 | 0.32%
| | |- org.apache.cassandra.db.RowIndexEntry$IndexedEntry @ 0x7b7f6e670
| 32 | 2,898,864 | 0.03%
| | | '- java.util.ArrayList @ 0x7b7caaae0
| 24 |
[jira] [Commented] (CASSANDRA-8269) Large number of system hints other CF's cause heap to fill and run OOM
[
https://issues.apache.org/jira/browse/CASSANDRA-8269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14200323#comment-14200323
]
Michael Shuler commented on CASSANDRA-8269:
---
I believe CASSANDRA-6998 is the correct ticket number from the first comment.
Large number of system hints other CF's cause heap to fill and run OOM
Key: CASSANDRA-8269
URL: https://issues.apache.org/jira/browse/CASSANDRA-8269
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: DSE 4.5.0 with Apache Cassandra 2.0.5
Reporter: Jose Martinez Poblete
Attachments: alln01-ats-cas2-java_1414110068_Leak_Suspects.zip,
system.log
A 3 node cluster with large amount of sstables for system.hints and other 3
user tables was coming down regularly with OOM on system log showing up the
following:
{noformat}
ERROR [OptionalTasks:1] 2014-10-23 18:51:29,052 CassandraDaemon.java (line
199) Exception in thread Thread[OptionalTasks:1,5,main]
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cassandra.io.sstable.IndexHelper$IndexInfo.deserialize(IndexHelper.java:187)
at
org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:122)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:229)
at
org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator.computeNext(SSTableScanner.java:203)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.io.sstable.SSTableScanner.hasNext(SSTableScanner.java:183)
at
org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:144)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.init(MergeIterator.java:87)
at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:46)
at
org.apache.cassandra.db.RowIteratorFactory.getIterator(RowIteratorFactory.java:74)
at
org.apache.cassandra.db.ColumnFamilyStore.getSequentialIterator(ColumnFamilyStore.java:1586)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1709)
at
org.apache.cassandra.db.ColumnFamilyStore.getRangeSlice(ColumnFamilyStore.java:1643)
at
org.apache.cassandra.db.HintedHandOffManager.scheduleAllDeliveries(HintedHandOffManager.java:513)
at
org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:91)
at
org.apache.cassandra.db.HintedHandOffManager$1.run(HintedHandOffManager.java:173)
at
org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{noformat}
A heapdump would show the following:
{noformat}
Class Name
| Shallow Heap | Retained Heap | Percentage
--
java.lang.Thread @ 0x67b292138 OptionalTasks:1 Thread
| 104 | 4,901,485,768 | 58.60%
|- org.apache.cassandra.utils.MergeIterator$ManyToOne @ 0x7b9dc4ad8
| 40 | 4,900,817,312 | 58.59%
| |- java.util.ArrayList @ 0x6f05f15f0
| 24 | 403,635,848 | 4.83%
| |- org.apache.cassandra.io.sstable.SSTableScanner$KeyScanningIterator @
0x7b5fe7078| 40 |29,669,312 | 0.35%
| | |- org.apache.cassandra.db.RowIndexEntry$IndexedEntry @ 0x7b7caaa28
| 32 |26,770,264 | 0.32%
| | |- org.apache.cassandra.db.RowIndexEntry$IndexedEntry @ 0x7b7f6e670
| 32 | 2,898,864 | 0.03%
| | | '- java.util.ArrayList @ 0x7b7caaae0
| 24 |
