We are running Ignite in LOCAL mode currently and have persistence enabled. For
this specific test, the off heap memory was set to as low as possible (~10 MB)
to test on-disk performance. We have multiple caches sharing the same
DataRegion. For the test, we basically write synchronously to cache A, write
asynchronously to cache B (using putAll), then read synchronously from cache A
(using getAll). When we get to doing this for about 50K key-value pairs, we get
an OOM exception only if the async write to cache B is enabled. We also don't
get the exception if we raise the off-heap memory allocated. The error is as
following:
Dec 03, 2019 11:44:04 AM java.util.logging.LogManager$RootLogger log
SEVERE: Critical system error detected. Will be handled accordingly to
configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0,
super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED,
SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext
[type=CRITICAL_ERROR, err=class o.a.i.i.mem.IgniteOutOfMemoryException: Failed
to find a page for eviction [segmentCapacity=317, loaded=125, maxDirtyPages=94,
dirtyPages=125, cpPages=0, pinnedInSegment=0, failedToPrepare=126]
Out of memory in data region [name=default, initSize=10.5 MiB, maxSize=10.5
MiB, persistenceEnabled=true] Try the following:
^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
^-- Enable eviction or expiration policies]]
class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Failed to find
a page for eviction [segmentCapacity=317, loaded=125, maxDirtyPages=94,
dirtyPages=125, cpPages=0, pinnedInSegment=0, failedToPrepare=126]
Out of memory in data region [name=default, initSize=10.5 MiB, maxSize=10.5
MiB, persistenceEnabled=true] Try the following:
^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
^-- Enable eviction or expiration policies
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$Segment.tryToFindSequentially(PageMemoryImpl.java:2427)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$Segment.removePageForReplacement(PageMemoryImpl.java:2321)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl$Segment.access$900(PageMemoryImpl.java:1930)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:744)
at
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:683)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.acquirePage(DataStructure.java:144)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.compareKeys(CacheDataTree.java:213)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.compare(CacheDataTree.java:164)
at
org.apache.ignite.internal.processors.cache.tree.CacheDataTree.compare(CacheDataTree.java:49)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:5105)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:5025)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1300(BPlusTree.java:90)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:291)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600)
at
org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
at
org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1243)
at
org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1227)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.find(IgniteCacheOffheapManagerImpl.java:2754)
at
org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.find(GridCacheOffheapManager.java:1953)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.read(IgniteCacheOffheapManagerImpl.java:670)
at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.getAllInternal(GridLocalAtomicCache.java:405)
at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache.access$000(GridLocalAtomicCache.java:86)
at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache$2.call(GridLocalAtomicCache.java:351)
at
org.apache.ignite.internal.processors.cache.local.atomic.GridLocalAtomicCache$2.call(GridLocalAtomicCache.java:349)
at
org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6817)
at
org.apache.ignite.internal.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
So a couple of questions regarding this:
-Any idea what could cause this? My understanding was that this OOM exception
should only happen if persistence is enabled. I am guessing that this has to do
with sharing the DataRegion across separate caches.
-What is the best-practice for DataRegion/cache allocation? One DataRegion per
cache? Is it always a bad idea to share a DataRegion across multiple caches?
