More heap-space. See the documentation for memory tuning: memory-tuning <https://ignite.apache.org/docs/latest/perf-and-troubleshooting/memory-tuning#generic-gc-settings>
You might also want to check that your query is iterating over the records in your cache rather than trying to read the whole thing into memory. Regards, Stephen > On 10 Mar 2021, at 17:44, Josh Katz <[email protected]> > wrote: > > Please see the log, it seems to be OutOfMemory from JVM on Windows Server > 2016. > How can we address this issue? > > ^-- System thread pool [active=0, idle=7, qSize=0] > [08:35:20,163][INFO][grid-timeout-worker-#22%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][IgniteKernal%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56] > FreeList [name=default##FreeList, buckets=256, dataPages=2657, reusePages=0] > [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] > <NonEquitiesYoY> Failed to run query [qry=GridCacheQueryInfo [loc=false, > trans=null, rdc=null, qry=GridCacheQueryAdapter [type=SCAN, clsName=null, > clause=null, filter=null, limit=0, transform=null, part=null, incMeta=false, > pageSize=1024, timeout=0, incBackups=false, forceLocal=false, dedup=false, > prj=null, keepBinary=true, subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, > taskHash=0, mvccSnapshot=null, dataPageScanEnabled=null], locFut=null, > sndId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, reqId=9, incMeta=false, > all=false], node=10276c3c-a550-490e-bc48-edf9596dd1dd] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314) > [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheDistributedQueryManager] > <NonEquitiesYoY> Failed to run query. > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314) > [08:35:21,897][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][GridCacheIoManager] > Failed processing message [senderId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, > msg=GridCacheQueryRequest [id=9, cacheName=NonEquitiesYoY, type=SCAN, > fields=false, clause=null, limit=0, clsName=null, keyValFilter=null, > rdc=null, trans=null, pageSize=1024, incBackups=false, cancel=false, > incMeta=false, all=false, keepBinary=true, > subjId=7d026e80-e6c9-4ac0-8c93-3cb1f56b0054, taskHash=0, part=-1, > topVer=AffinityTopologyVersion [topVer=163, minorTopVer=0], > mvccSnapshot=null, flags=0, super=GridCacheIdMessage [cacheId=-1867634189, > super=GridCacheMessage [msgId=10, depInfo=null, > lastAffChangedTopVer=AffinityTopologyVersion [topVer=158, minorTopVer=1], > err=null, skipPrepare=false]]]] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314) > [08:35:22,163][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] > Critical system error detected. Will be handled accordingly to configured > handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.OutOfMemoryError: Java heap space]] > java.lang.OutOfMemoryError: Java heap space > at > org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) > at > org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl.toCacheObject(CacheObjectBinaryProcessorImpl.java:1344) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:625) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:445) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncomplete(CacheDataRowAdapter.java:356) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.doInitFromLink(CacheDataRowAdapter.java:261) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:160) > at > org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:131) > at > org.apache.ignite.internal.processors.cache.tree.DataRow.<init>(DataRow.java:55) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:130) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:419) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ForwardCursor.fillFromBuffer0(BPlusTree.java:5787) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5553) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5479) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findLowerUnbounded(BPlusTree.java:1048) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1098) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:164) > at > org.apache.ignite.internal.processors.cache.tree.CacheDataTree.find(CacheDataTree.java:63) > at > org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.find(BPlusTree.java:1080) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2929) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2899) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2893) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.cursor(IgniteCacheOffheapManagerImpl.java:2881) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$3.onHasNext(IgniteCacheOffheapManagerImpl.java:942) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.advance(GridCacheQueryManager.java:3062) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager$ScanQueryIterator.onHasNext(GridCacheQueryManager.java:3028) > at > org.apache.ignite.internal.util.GridCloseableIteratorAdapter.hasNextX(GridCloseableIteratorAdapter.java:53) > at > org.apache.ignite.internal.util.lang.GridIteratorAdapter.hasNext(GridIteratorAdapter.java:45) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.runQuery(GridCacheQueryManager.java:1314) > [08:35:22,179][WARNING][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][CacheDiagnosticManager] > Page locks dump: > > Thread=[name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=66], state=TIMED_WAITING > Locked pages = [] > Locked pages log: > name=exchange-worker-#48%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=7400], state=RUNNABLE > Locked pages = [] > Locked pages log: > name=query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=11], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-0-#1%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=12], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-1-#2%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=13], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-2-#3%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=14], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-3-#4%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=15], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-4-#5%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=16], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-5-#6%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=17], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-6-#7%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > Thread=[name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%, > id=18], state=WAITING > Locked pages = [] > Locked pages log: > name=sys-stripe-7-#8%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56% > time=(1615394122179, 2021-03-10 08:35:22.179) > > > > [08:35:22,179][SEVERE][query-#7335%ignite-instance-9329856b-236a-4482-a29e-d9e7ace51c56%][] > JVM will be halted immediately due to the failure: > [failureCtx=FailureContext [type=CRITICAL_ERROR, > err=java.lang.OutOfMemoryError: Java heap space]] > > > > From: Stephen Darlington <[email protected] > <mailto:[email protected]>> > Sent: Wednesday, March 10, 2021 9:35 AM > To: [email protected] <mailto:[email protected]> > Subject: Re: Cluster keeps crashing with SEVERE message > > This is an EXTERNAL EMAIL. Stop and think before clicking a link or opening > attachments. > > We’d need to see more of the stack trace and see your configuration before > this could be diagnosed. > > It looks like a node abruptly left the cluster. If you’re losing data it > seems like you’ve not configured any backups. > > > On 10 Mar 2021, at 16:53, Josh Katz <[email protected] > <mailto:[email protected]>> wrote: > > After we load 35 million records from the database into the cluster. When we > query it using a scan query we get the following SEVERE error, > After that the node are restarting and the caches all show 0 keys and we have > to restart the initial load. > Is there a way to salvage the data so we don’t have to reload every time? For > example, if we add persistence at this stage the initial load will be slower > correct? > How can we avoid seeing this error? “[SEVERE] Failed to process selector key” > > <image001.png> > > Thanks, > Josh > > > Please follow the hyperlink to important disclosures. > https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html > <https://www.dodgeandcox.com/disclosures/email_disclosure_funds.html>
