Hello, We are running ONAP through OOM in a K8s cluster and recently we have been getting this error;
INFO 13:50:48 Initializing aaigraph.graphindex_lock_ INFO 13:50:48 Opening /opt/cassandra/data/data/aaigraph/graphindex_lock_-0af6b620867f11e99a58ab2e80056111/aaigraph-graphindex_lock_-ka-5 (102 bytes) java.lang.OutOfMemoryError: Java heap space Dumping heap to java_pid1.hprof ... Unable to create java_pid1.hprof: Permission denied ERROR 13:51:06 Exception in thread Thread[SSTableBatchOpen:1,5,main] java.lang.OutOfMemoryError: Java heap space at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:76) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:69) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:47) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile.createPooledReader(CompressedPoolingSegmentedFile.java:95) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:62) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:1902) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(SimpleSliceReader.java:57) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:245) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:268) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2034) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1877) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:353) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:347) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:370) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:353) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:68) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.cql3.QueryProcessor.executeInternal(QueryProcessor.java:317) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.db.SystemKeyspace.getSSTableReadMeter(SystemKeyspace.java:972) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader$GlobalTidy.ensureReadMeter(SSTableReader.java:2388) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier.setup(SSTableReader.java:2204) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader.setup(SSTableReader.java:2145) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:491) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:384) ~[apache-cassandra-2.1.22.jar:2.1.22] at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:531) ~[apache-cassandra-2.1.22.jar:2.1.22] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_292] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_292] java.lang.OutOfMemoryError: Java heap space Which is preventing access to the AAI data. This came out of nowhere after K8s decided to restart the pods. Any ideas how to fix this? WIll increasing the heap size through kubecttl edit to the stateful set delete the data? Thanks -=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#23294): https://lists.onap.org/g/onap-discuss/message/23294 Mute This Topic: https://lists.onap.org/mt/83278742/21656 Group Owner: [email protected] Unsubscribe: https://lists.onap.org/g/onap-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
