Hi Andrey,
Thanks for your replay. It still confused me for:
1) for storm worker process, If it is because of OOM and crashed. it should dump the heap. for I set -XX:+HeapDumpOnOutOfMemoryError
but it didn't. For storm worker, it behaves like a normal fatal error which make storm worker restart.
2) It did make ignite server heap dump. for I analyzed the dump, it is ignite server's. (seeing from PID and java command etc.)
anyone can explain? Thanks.
Thanks
Shawn
On 3/31/2018 01:36，Andrey Mashenkov<[email protected]> wrote：
Hi Shawn,1. Ignite use off heap to store cache entries. Client store no cache data.
Cache in LOCAL mode can be used on client side, and it should use offheap of course.All data client retreive from server will be in offheap2. It is not IgniteOutOfMemory error, but JVM OOM.So, try to investigate if there is a memory leak in your code.On Fri, Mar 30, 2018 at 6:36 AM, shawn.du <[email protected]> wrote:Hi,My Ignite client heap OOM yesterday. This is the first time we encounter this issue.My ignite client colocates within Storm worker process. this issue cause storm worker restart.I have several questions about it: our ignite version is 2.3.01) if ignite in client mode, it use offheap? how to set the max onheap/offheap memory to use.2) our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker to dump the heap.but we get a ignite server's heap dump. ignite server didn't die. The ignite server's heap dump is very small. only have 200M.which process is OOM? worker or ignite server?This is logs: Thanks in advance.Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node. at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.UpdateErrors. addFailedKeys(UpdateErrors. java:124) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic. GridNearAtomicUpdateResponse. addFailedKeys( GridNearAtomicUpdateResponse. java:342) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache. updateAllAsyncInternal0( GridDhtAtomicCache.java:1784) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache. updateAllAsyncInternal( GridDhtAtomicCache.java:1627) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache. processNearAtomicUpdateRequest (GridDhtAtomicCache.java:3054) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache. access$400(GridDhtAtomicCache. java:129) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache$ 5.apply(GridDhtAtomicCache. java:265) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.distributed. dht.atomic.GridDhtAtomicCache$ 5.apply(GridDhtAtomicCache. java:260) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager. processMessage( GridCacheIoManager.java:1060) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager.onMessage0( GridCacheIoManager.java:579) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager. handleMessage( GridCacheIoManager.java:378) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager. handleMessage( GridCacheIoManager.java:304) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager.access$100( GridCacheIoManager.java:99) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache. GridCacheIoManager$1. onMessage(GridCacheIoManager. java:293) ~[stormjar.jar:?] at org.apache.ignite.internal.managers.communication. GridIoManager.invokeListener( GridIoManager.java:1555) ~[stormjar.jar:?] at org.apache.ignite.internal.managers.communication. GridIoManager. processRegularMessage0( GridIoManager.java:1183) ~[stormjar.jar:?] at org.apache.ignite.internal.managers.communication. GridIoManager.access$4200( GridIoManager.java:126) ~[stormjar.jar:?] at org.apache.ignite.internal.managers.communication. GridIoManager$9.run( GridIoManager.java:1090) ~[stormjar.jar:?] at org.apache.ignite.internal.util.StripedExecutor$Stripe. run(StripedExecutor.java:505) ~[stormjar.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151] Suppressed: java.lang.OutOfMemoryError: Java heap spaceat org.apache.ignite.internal.processors.cache. IncompleteCacheObject.<init>( IncompleteCacheObject.java:44) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cacheobject. IgniteCacheObjectProcessorImpl .toCacheObject( IgniteCacheObjectProcessorImpl .java:191) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. CacheDataRowAdapter. readIncompleteValue( CacheDataRowAdapter.java:404) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. CacheDataRowAdapter. readFragment( CacheDataRowAdapter.java:248) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. CacheDataRowAdapter. initFromLink( CacheDataRowAdapter.java:174) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. CacheDataRowAdapter. initFromLink( CacheDataRowAdapter.java:102) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2RowFactory.getRow( H2RowFactory.java:62) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. io.H2ExtrasInnerIO. getLookupRow(H2ExtrasInnerIO. java:114) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. io.H2ExtrasInnerIO. getLookupRow(H2ExtrasInnerIO. java:36) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2Tree.getRow(H2Tree.java:123) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2Tree.getRow(H2Tree.java:40) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.getRow( BPlusTree.java:4372) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2Tree.compare(H2Tree.java: 200) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2Tree.compare(H2Tree.java:40) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.compare( BPlusTree.java:4359) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree. findInsertionPoint(BPlusTree. java:4279) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.access$1500( BPlusTree.java:81) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree$Search.run0( BPlusTree.java:261) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree$GetPageHandler. run(BPlusTree.java:4697) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree$GetPageHandler. run(BPlusTree.java:4682) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.util.PageHandler. readPage(PageHandler.java:158) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. DataStructure.read( DataStructure.java:319) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.removeDown( BPlusTree.java:1823) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.removeDown( BPlusTree.java:1842) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.doRemove( BPlusTree.java:1752) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.persistence. tree.BPlusTree.remove( BPlusTree.java:1578) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.database. H2TreeIndex.remove( H2TreeIndex.java:216) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.opt. GridH2Table.doUpdate( GridH2Table.java:504) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2.opt. GridH2Table.update( GridH2Table.java:423) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query.h2. IgniteH2Indexing.remove( IgniteH2Indexing.java:580) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.query. GridQueryProcessor.remove( GridQueryProcessor.java:2334) ~[stormjar.jar:?] at org.apache.ignite.internal.processors.cache.query. GridCacheQueryManager.remove( GridCacheQueryManager.java: 461) ThanksShawn
--Best regards,
Andrey V. Mashenkov
