[
https://issues.apache.org/jira/browse/CASSANDRA-5716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Ellis updated CASSANDRA-5716:
--------------------------------------
Affects Version/s: (was: 1.2.6)
2.0 beta 1
Fix Version/s: (was: 1.2.6)
2.0
Issue Type: Bug (was: Improvement)
> Remark on cassandra-5273 : Hanging system after OutOfMemory. Server cannot
> die due to uncaughtException handling
> ----------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5716
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5716
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 2.0 beta 1
> Environment: linux
> Reporter: Ignace Desimpel
> Priority: Minor
> Fix For: 2.0
>
>
> Possible incorrect handling of an OOM as a result of modifications made for
> issue cassandra-5273.
> I could reproduce the OOM, with the patch of Cassandra-5273 applied.
> The good news is that, at least in my case, it works fine : the system did
> die !
>
> However, due to multiple uncaughtException handling, multiple threads are
> calling the exitThread.start() routine, causing an IllegalStateException.
> There are some other exceptions also, but that seems logical. Also, after
> calling the start() function, the thread(s) will continue to run, and that
> could not be wanted.
>
> Below I pasted the stack trace.
> Just for your information, after all this works, and I could restart the
> Cassandra server and redo the OOM
>
> 2013-06-27 16:28:15.384 Unable to reduce heap usage since there are no dirty
> column families
> java.lang.OutOfMemoryError: Java heap space
> Dumping heap to java_pid4808.hprof ...
> Heap dump file created [278960302 bytes in 2.659 secs]
> 2013-06-27 16:28:42.655 Exception in thread Thread[qtp1564441079-31,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread Thread[qtp1564441079-36,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread Thread[qtp1564441079-30,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread Thread[GossipTasks:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread
> Thread[PERIODIC-COMMIT-LOG-SYNCER,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655 Exception in thread
> Thread[metrics-meter-tick-thread-2,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.656 Exception in thread
> Thread[EXPIRING-MAP-REAPER:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at
> java.util.concurrent.ConcurrentHashMap$EntrySet.iterator(ConcurrentHashMap.java:1202)
> ~[na:1.6.0_29]
> at org.apache.cassandra.utils.ExpiringMap$1.run(ExpiringMap.java:88)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
> ~[thrift/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [na:1.6.0_29]
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> [na:1.6.0_29]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_29]
> 2013-06-27 16:28:42.656 Exception in thread Thread[ScheduledTasks:1,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at com.yammer.metrics.stats.Snapshot.<init>(Snapshot.java:30)
> ~[metrics-core-2.0.3.jar:na]
> at
> com.yammer.metrics.stats.ExponentiallyDecayingSample.getSnapshot(ExponentiallyDecayingSample.java:107)
> ~[metrics-core-2.0.3.jar:na]
> at
> org.apache.cassandra.locator.DynamicEndpointSnitch.updateScores(DynamicEndpointSnitch.java:237)
> ~[thrift/:na]
> at
> org.apache.cassandra.locator.DynamicEndpointSnitch.access$0(DynamicEndpointSnitch.java:217)
> ~[thrift/:na]
> at
> org.apache.cassandra.locator.DynamicEndpointSnitch$1.run(DynamicEndpointSnitch.java:71)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:75)
> ~[thrift/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> [na:1.6.0_29]
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> [na:1.6.0_29]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> [na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_29]
> 2013-06-27 16:28:42.656 Exception in thread Thread[Thrift:3,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at java.util.BitSet.initWords(BitSet.java:144) ~[na:1.6.0_29]
> at java.util.BitSet.<init>(BitSet.java:139) ~[na:1.6.0_29]
> at org.apache.cassandra.thrift.Column.<init>(Column.java:135)
> ~[thrift/:na]
> at org.apache.cassandra.thrift.Column.<init>(Column.java:158)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.CassandraServer.thriftifyColumns(CassandraServer.java:184)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.CassandraServer.thriftifyColumnFamily(CassandraServer.java:285)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.CassandraServer.thriftifyKeySlices(CassandraServer.java:1071)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.CassandraServer.get_range_slices(CassandraServer.java:960)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:3454)
> ~[thrift/:na]
> at
> org.apache.cassandra.thrift.Cassandra$Processor$get_range_slices.getResult(Cassandra.java:1)
> ~[thrift/:na]
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
> ~[libthrift-0.7.0.jar:0.7.0]
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
> ~[libthrift-0.7.0.jar:0.7.0]
> at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:199)
> ~[thrift/:na]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ~[na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) ~[na:1.6.0_29]
> 2013-06-27 16:28:42.655 Exception in thread
> Thread[COMMIT-LOG-ALLOCATOR,5,main]
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.655
> java.lang.OutOfMemoryError: Java heap space
> 2013-06-27 16:28:42.656 Failed deleting temp components for
> ..\cassandra\dbdatafile\system\schema_keyspaces\system-schema_keyspaces-tmp-ic-16
> org.apache.cassandra.io.FSWriteError: java.io.IOException: Failed to delete
> C:\develop\configs\AnnotationServer7\bin\..\cassandra\dbdatafile\system\schema_keyspaces\system-schema_keyspaces-tmp-ic-16-Statistics.db
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:112)
> ~[thrift/:na]
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:103)
> ~[thrift/:na]
> at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:146)
> ~[thrift/:na]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:309)
> ~[thrift/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:509)
> [thrift/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:448)
> [thrift/:na]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> [thrift/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> [thrift/:na]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_29]
> Caused by: java.io.IOException: Failed to delete
> C:\develop\configs\AnnotationServer7\bin\..\cassandra\dbdatafile\system\schema_keyspaces\system-schema_keyspaces-tmp-ic-16-Statistics.db
> ... 11 common frames omitted
> 2013-06-27 16:28:42.656 Exception in thread Thread[FlushWriter:2,5,main]
> org.apache.cassandra.io.FSWriteError: java.io.IOException: Failed to delete
> C:\develop\configs\AnnotationServer7\bin\..\cassandra\dbdatafile\system\schema_keyspaces\system-schema_keyspaces-tmp-ic-16-Statistics.db
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:112)
> ~[thrift/:na]
> at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:103)
> ~[thrift/:na]
> at org.apache.cassandra.io.sstable.SSTable.delete(SSTable.java:146)
> ~[thrift/:na]
> at
> org.apache.cassandra.io.sstable.SSTableWriter.abort(SSTableWriter.java:309)
> ~[thrift/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.writeSortedContents(Memtable.java:509)
> ~[thrift/:na]
> at
> org.apache.cassandra.db.Memtable$FlushRunnable.runWith(Memtable.java:448)
> ~[thrift/:na]
> at
> org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
> ~[thrift/:na]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[thrift/:na]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ~[na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) ~[na:1.6.0_29]
> Caused by: java.io.IOException: Failed to delete
> C:\develop\configs\AnnotationServer7\bin\..\cassandra\dbdatafile\system\schema_keyspaces\system-schema_keyspaces-tmp-ic-16-Statistics.db
> ... 11 common frames omitted
> 2013-06-27 16:28:42.656 Stopping gossiper
> 2013-06-27 16:28:42.656 Stopping gossip by operator request
> 2013-06-27 16:28:42.656 Announcing shutdown
> 2013-06-27 16:28:44.684 Stopping RPC server
> 2013-06-27 16:28:44.684 Stop listening to thrift clients
> 2013-06-27 16:29:06.743 Exception in thread Thread[OptionalTasks:1,5,main]
> java.lang.IllegalThreadStateException: null
> at java.lang.Thread.start(Thread.java:638) ~[na:1.6.0_29]
> at
> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:189)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:213)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.logExceptionsAfterExecute(DebuggableThreadPoolExecutor.java:196)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor.afterExecute(DebuggableScheduledThreadPoolExecutor.java:46)
> ~[thrift/:na]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> ~[na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) ~[na:1.6.0_29]
> 2013-06-27 16:29:06.743 Exception in thread Thread[ScheduledTasks:1,5,main]
> java.lang.IllegalThreadStateException: null
> at java.lang.Thread.start(Thread.java:638) [na:1.6.0_29]
> at
> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:189)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:213)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
> ~[thrift/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> ~[na:1.6.0_29]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_29]
> 2013-06-27 16:29:06.743 Exception in thread
> Thread[EXPIRING-MAP-REAPER:1,5,main]
> java.lang.IllegalThreadStateException: null
> at java.lang.Thread.start(Thread.java:638) [na:1.6.0_29]
> at
> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:189)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:213)
> ~[thrift/:na]
> at
> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79)
> ~[thrift/:na]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
> ~[na:1.6.0_29]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
> ~[na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> [na:1.6.0_29]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> [na:1.6.0_29]
> at java.lang.Thread.run(Thread.java:662) [na:1.6.0_29]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
