Hi Alex,
Here is the full stack trace
[INFO][tcp-disco-sock-reader-#130][TcpDiscoverySpi] Finished serving remote
node connection
[INFO][tcp-disco-sock-reader-#653][TcpDiscoverySpi] Started serving remote
node connection
[SEVERE][tcp-disco-sock-reader-#130][TcpDiscoverySpi] Runtime error caught
during grid runnable execution: Socket reader [id=313,
name=tcp-disco-sock-reader-#130,
nodeId=35a7ca47-3245-4f9f-8114-9b65c6d5e9bf]
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at
java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3450)
at
java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3358)
at
java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3170)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1850)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1527)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
at
org.apache.ignite.internal.util.IgniteUtils.readMap(IgniteUtils.java:5146)
at
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:617)
at
java.io.ObjectInputStream.readExternalData(ObjectInputStream.java:2063)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2012)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1536)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2232)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2156)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2014)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1536)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:423)
at java.util.ArrayList.readObject(ArrayList.java:791)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1058)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2123)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2014)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1536)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2232)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2156)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2014)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1536)
[INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming
connection
[INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming
connection
[INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new
thread for connection
[INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery accepted incoming
connection
[INFO][tcp-disco-sock-reader-#654][TcpDiscoverySpi] Started serving remote
node connection
[INFO][tcp-disco-srvr-#3][TcpDiscoverySpi] TCP discovery spawning a new
thread for connection
Thanks,
Amol
On Sat, Aug 4, 2018 at 3:28 AM, Alex Plehanov <[email protected]>
wrote:
> Offheap and heap memory regions are used for different purposes and can't
> replace each other. You can't get rid of OOME in heap by increasing offheap
> memory.
> Can you provide full exception stack trace?
>
> 2018-08-03 20:55 GMT+03:00 Amol Zambare <[email protected]>:
>
>> Thanks Alex and Denis
>>
>> We have configured off heap memory to 100GB and we have 10 nodes ignite
>> cluster. However when we are running spark job we see following error in
>> the ignite logs. When we run the spark job heap utilization on most of the
>> ignite nodes is increasing significantly though we are using off heap
>> storage. We have set JVM heap size on each ignite node to 50GB. Please
>> suggest.
>>
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>> at java.util.Arrays.copyOf(Arrays.java:3332)
>> at java.lang.AbstractStringBuilde
>> r.ensureCapacityInternal(AbstractStringBuilder.java:124)
>>
>>
>> On Fri, Aug 3, 2018 at 4:16 AM, Alex Plehanov <[email protected]>
>> wrote:
>>
>>> "Non-heap memory ..." metrics in visor have nothing to do with offheap
>>> memory allocated for data regions. "Non-heap memory" returned by visor
>>> it's JVM managed memory regions other then heap used for internal JVM
>>> purposes (JIT compiler, etc., see [1]). Memory allocated in offheap by
>>> Ignite for data regions (via "unsafe") not included into this metrics. Some
>>> data region related metrics in visor were implemented in Ignite 2.4.
>>>
>>> [1] https://docs.oracle.com/javase/8/docs/api/java/lang/manageme
>>> nt/MemoryMXBean.html
>>>
>>
>>
>
