Re: Ignite Client Heap out of Memory issue

[Andrey Mashenkov]

Hi Shawn,

The OOM error occurs on remote server node, there were not sufficient
memory to process request,
but other threads were not affected by this.
Looks like, Ignite was able to recover from the error as it was suppressed
and  reply to client has been sended.


On Mon, Apr 2, 2018 at 8:22 AM, shawn.du  wrote:

> Hi Andrey,
>
> Thanks for your replay. It still confused me for:
> 1) for storm worker process, If it is  because of OOM and crashed. it
> should dump the heap. for I set  -XX:+HeapDumpOnOutOfMemoryError
> but it didn't.  For storm worker, it behaves like a normal fatal error
> which make storm worker restart.
> 2) It did make ignite server heap dump. for I analyzed the dump, it is
> ignite server's. (seeing from PID and java command etc.)
>
> anyone can explain? Thanks.
>
> Thanks
> Shawn
>
> On 3/31/2018 01:36，Andrey Mashenkov
>  wrote：
>
> Hi Shawn,
>
> 1. Ignite use off heap to store cache entries. Client store no cache data.
> Cache in LOCAL mode can be used on client side, and it should use offheap
> of course.
>
> All data client retreive from server will be in offheap
>
> 2. It is not IgniteOutOfMemory error, but JVM OOM.
> So, try to investigate if there is a memory leak in your code.
>
> On Fri, Mar 30, 2018 at 6:36 AM, shawn.du  wrote:
>
>> Hi,
>>
>> My Ignite client heap OOM yesterday.  This is the first time we encounter
>> this issue.
>>
>> My ignite client colocates within Storm worker process. this issue cause
>> storm worker restart.
>> I have several questions about it: our ignite version is 2.3.0
>> 1) if ignite in client mode, it use offheap? how to set the max
>> onheap/offheap memory to use.
>> 2) our storm worker have 8G memory, ignite client print OOM, it doesn't
>> trigger storm worker to dump the heap.
>> but we get a ignite server's heap dump.  ignite server didn't die.
>> The ignite server's heap dump is very small. only have 200M.
>> which process is OOM? worker or ignite server?
>>
>> This is logs:  Thanks in advance.
>>
>>   Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update
>> keys on primary node.
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridNearAtomicUpdateRes
>> ponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.upda
>> teAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.upda
>> teAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.proc
>> essNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache.acce
>> ss$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.
>> apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.
>> apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.pro
>> cessors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.man
>> agers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.man
>> agers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
>> ~[stormjar.jar:?]
>> at org.apache.ignite.internal.man
>> 

Re: Ignite Client Heap out of Memory issue

[shawn.du]

Hi Andrey,Thanks for your replay. It still confused me for:1) for storm worker process, If it is  because of OOM and crashed. it should dump the heap. for I set  -XX:+HeapDumpOnOutOfMemoryError    but it didn't.  For storm worker, it behaves like a normal fatal error which make storm worker restart.
2) It did make ignite server heap dump. for I analyzed the dump, it is ignite server's. (seeing from PID and java command etc.) anyone can explain? Thanks.






ThanksShawn





On 3/31/2018 01:36，Andrey Mashenkov wrote： 


Hi Shawn,1. Ignite use off heap to store cache entries. Client store no cache data. Cache in LOCAL mode can be used on client side, and it should use offheap of course.All data client retreive from server will be in offheap2. It is not IgniteOutOfMemory error, but JVM OOM. So, try to investigate if there is a memory leak in your code.On Fri, Mar 30, 2018 at 6:36 AM, shawn.du  wrote:







Hi,My Ignite client heap OOM yesterday.  This is the first time we encounter this issue.My ignite client colocates within Storm worker process. this issue cause storm worker restart.I have several questions about it: our ignite version is 2.3.01) if ignite in client mode, it use offheap? how to set the max onheap/offheap memory to use.2) our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker to dump the heap.    but we get a ignite server's heap dump.  ignite server didn't die.  The ignite server's heap dump is very small. only have 200M.    which process is OOM? worker or ignite server?This is logs:  Thanks in advance.  Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[stormjar.jar:?]                at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[stormjar.jar:?]                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]                Suppressed: java.lang.OutOfMemoryError: Java heap space                        at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) ~[stormjar.jar:?]                        at 

Re: Ignite Client Heap out of Memory issue

[Andrey Mashenkov]

Hi Shawn,

1. Ignite use off heap to store cache entries. Client store no cache data.
Cache in LOCAL mode can be used on client side, and it should use offheap
of course.

All data client retreive from server will be in offheap

2. It is not IgniteOutOfMemory error, but JVM OOM.
So, try to investigate if there is a memory leak in your code.

On Fri, Mar 30, 2018 at 6:36 AM, shawn.du  wrote:

> Hi,
>
> My Ignite client heap OOM yesterday.  This is the first time we encounter
> this issue.
>
> My ignite client colocates within Storm worker process. this issue cause
> storm worker restart.
> I have several questions about it: our ignite version is 2.3.0
> 1) if ignite in client mode, it use offheap? how to set the max
> onheap/offheap memory to use.
> 2) our storm worker have 8G memory, ignite client print OOM, it doesn't
> trigger storm worker to dump the heap.
> but we get a ignite server's heap dump.  ignite server didn't die.
> The ignite server's heap dump is very small. only have 200M.
> which process is OOM? worker or ignite server?
>
> This is logs:  Thanks in advance.
>
>   Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update
> keys on primary node.
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.UpdateErrors.
> addFailedKeys(UpdateErrors.java:124) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.
> addFailedKeys(GridNearAtomicUpdateResponse.java:342) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> updateAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache.
> access$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.distributed.dht.atomic.GridDhtAtomicCache$
> 5.apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager.access$100(GridCacheIoManager.java:99)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.processors.cache.
> GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.invokeListener(GridIoManager.java:1555) ~[stormjar.jar:?]
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.processRegularMessage0(GridIoManager.java:1183)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.managers.communication.
> GridIoManager.access$4200(GridIoManager.java:126) ~[stormjar.jar:?]
> at org.apache.ignite.internal.managers.communication.
> GridIoManager$9.run(GridIoManager.java:1090) ~[stormjar.jar:?]
> at 
> org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
> ~[stormjar.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
> Suppressed: java.lang.OutOfMemoryError: Java heap space
> at org.apache.ignite.internal.processors.cache.
> IncompleteCacheObject.(IncompleteCacheObject.java:44)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cacheobject.IgniteCacheObjectProcessorImpl.toCacheObject(
> IgniteCacheObjectProcessorImpl.java:191) ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:404)
> ~[stormjar.jar:?]
> at org.apache.ignite.internal.
> 

Ignite Client Heap out of Memory issue

[shawn.du]

Hi,My Ignite client heap OOM yesterday.  This is the first time we encounter this issue.My ignite client colocates within Storm worker process. this issue cause storm worker restart.I have several questions about it: our ignite version is 2.3.01) if ignite in client mode, it use offheap? how to set the max onheap/offheap memory to use.2) our storm worker have 8G memory, ignite client print OOM, it doesn't trigger storm worker to dump the heap.    but we get a ignite server's heap dump.  ignite server didn't die.  The ignite server's heap dump is very small. only have 200M.    which process is OOM? worker or ignite server?This is logs:  Thanks in advance.  Suppressed: org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node.                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.UpdateErrors.addFailedKeys(UpdateErrors.java:124) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFailedKeys(GridNearAtomicUpdateResponse.java:342) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1784) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1627) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3054) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:129) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:265) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:260) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99) ~[stormjar.jar:?]                at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) ~[stormjar.jar:?]                at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090) ~[stormjar.jar:?]                at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505) ~[stormjar.jar:?]                at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]                Suppressed: java.lang.OutOfMemoryError: Java heap space                        at org.apache.ignite.internal.processors.cache.IncompleteCacheObject.(IncompleteCacheObject.java:44) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.cacheobject.IgniteCacheObjectProcessorImpl.toCacheObject(IgniteCacheObjectProcessorImpl.java:191) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readIncompleteValue(CacheDataRowAdapter.java:404) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.readFragment(CacheDataRowAdapter.java:248) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:174) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[stormjar.jar:?]                        at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:62) ~[stormjar.jar:?]                        at