Hi,
My Ignite client heap OOM yesterday. This is the first time we encounter this issue.
My ignite client colocates within Storm worker process. this issue cause storm worker restart.
I have several questions about it: our ignite version is 2.3.0
1) if ignite in client mode, it use offheap? how to set the max onheap/offheap memory to use.
2) our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker to dump the heap.
but we get a ignite server's heap dump. ignite server didn't die. The ignite server's heap dump is very small. only have 200M.
which process is OOM? worker or ignite server?
This is logs: Thanks in advance.
Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[stormjar.jar:?]
at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[stormjar.jar:?]
at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[stormjar.jar:?]
at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[stormjar.jar:?]
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[stormjar.jar:?]
at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[stormjar.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Suppressed: java.lang.OutOfMemoryError: Java heap space
at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.<init>(IncompleteCacheObject.java:44) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toCacheObject(IgniteCacheObjectProcessorImpl.java:191) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:404) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:248) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:114) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:36) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:123) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:40) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4372) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:200) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:40) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4359) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:81) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:261) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:319) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1823) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.removeDown(BPlusTree.java:1842) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doRemove(BPlusTree.java:1752) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.remove(BPlusTree.java:1578) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.remove(H2TreeIndex.java:216) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:504) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:423) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:580) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2334) ~[stormjar.jar:?]
at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:461)
Thanks
Shawn
