[
https://issues.apache.org/jira/browse/IGNITE-8917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16718685#comment-16718685
]
Semen Boikov commented on IGNITE-8917:
--------------------------------------
For non-atomic caches or when entry is updated on atomic cache backups memory
is evicted right before single entry is updated, and it works more or less
reliable (see usages IgniteCacheDatabaseSharedManager.ensureFreeSpace). But now
putAll operation for atomic cache on primary node calls
IgniteCacheDatabaseSharedManager.ensureFreeSpace not before each entry update,
but only once for putAll. It is implemented this way since putAll first locks
all entries and then updates each entry, and it is not safe to call
'ensureFreeSpace' for each entry while entries locks are held (it can cause
deadlock).
I tried to implement some workaround to predict how much space is needed for
putAll and evict this amount of pages before putAll, but it does not work
reliable. First I don't see how to predict how many pages will be needed to
update BPlustTree and free lists, also before entry is locked we don't know is
it will be update or insert + for invoke operation result is now known in
advance.
I believe if thread-per-partition is implemented (IGNITE-4682) then it won't be
necessary to lock all entries before update, and ensureFreeSpace can be called
right before each entry update.
> IgniteOutOfMemory when performing putAll with configured pageEvictionMode
> -------------------------------------------------------------------------
>
> Key: IGNITE-8917
> URL: https://issues.apache.org/jira/browse/IGNITE-8917
> Project: Ignite
> Issue Type: Bug
> Components: cache
> Affects Versions: 2.5
> Reporter: Denis Mekhanikov
> Assignee: Semen Boikov
> Priority: Major
> Attachments: IgnitePdsPageEvictionBatchTest.java
>
>
> When {{pageEvictionMode}} is configured and {{IgniteCache#putAll(...)}}
> operations are performed, then page eviction may not be applied properly. It
> results in IgniteOutOfMemory being thrown.
> PFA test, reproducing the problem.
> Exception:
> {noformat}
> Critical system error detected. Will be handled accordingly to configured
> handler [hnd=class o.a.i.failure.NoOpFailureHandler,
> failureCtx=FailureContext [type=CRITICAL_ERROR, err=class
> o.a.i.i.mem.IgniteOutOfMemoryException: Out of memory in data region
> [name=evicting-data-region, initSize=256.0 MiB, maxSize=256.0 MiB,
> persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies]]
> class org.apache.ignite.internal.mem.IgniteOutOfMemoryException: Out of
> memory in data region [name=evicting-data-region, initSize=256.0 MiB,
> maxSize=256.0 MiB, persistenceEnabled=false] Try the following:
> ^-- Increase maximum off-heap memory size (DataRegionConfiguration.maxSize)
> ^-- Enable Ignite persistence (DataRegionConfiguration.persistenceEnabled)
> ^-- Enable eviction or expiration policies
> at
> org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:305)
> at
> org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.allocateDataPage(AbstractFreeList.java:464)
> at
> org.apache.ignite.internal.processors.cache.persistence.freelist.AbstractFreeList.insertDataRow(AbstractFreeList.java:491)
> at
> org.apache.ignite.internal.processors.cache.persistence.RowStore.addRow(RowStore.java:97)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:1325)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.update(GridCacheMapEntry.java:4955)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4750)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry$AtomicCacheUpdateClosure.call(GridCacheMapEntry.java:4464)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:3083)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2977)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1726)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invokeDown(BPlusTree.java:1703)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1610)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1272)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:370)
> at
> org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:1755)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAtomicCache.java:2420)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.update(GridDhtAtomicCache.java:1882)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1735)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:299)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:812)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:664)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:248)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAll0(GridDhtAtomicCache.java:1068)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.putAll0(GridDhtAtomicCache.java:648)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2811)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1147)
> at
> org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:836)
> at
> org.apache.ignite.internal.processors.cache.persistence.db.IgnitePdsPageEvictionBatchTest.testPageEvictionPutAll(IgnitePdsPageEvictionBatchTest.java:57)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2086)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:140)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2001)
> at java.lang.Thread.run(Thread.java:748){noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
