Ignite in client Mode: outofMemory error with TCP discovery

[mahesh76private]

Hi, 

Whenever, a client loses connection with ignite cluster, it throws out an
OutOfMemory error. 
Can you please see if you can explain this behavior?

The ignite client had 1G memory set on XMS and XMX... and a very few clients
active. 
Now it is quite possible that ignite clients lose connection with ignite 
cluster frequently.

Any insights into the issue will be deeply appreciated.




15-05-2019 15:30:35.024 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 5063 milliseconds.
15-05-2019 15:30:38.958 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3664 milliseconds.
15-05-2019 15:30:42.783 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3666 milliseconds.
15-05-2019 15:30:46.522 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3630 milliseconds.
15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
org.apache.ignite.internal.diagnostic.warning - Failed to wait for partition
map exchange [topVer=AffinityTopologyVersion [topVer=751, minorTopVer=1],
node=604f46ce-3693-404c-b180-29b2b6dcf8ed]. Consider changing
TransactionConfiguration.txTimeoutOnPartitionMapSynchronization to non
default value to avoid this message. Dumping pending objects that might be
the cause: 
15-05-2019 15:30:50.163 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3591 milliseconds.
15-05-2019 15:30:50.163 [exchange-worker-#38] WARN 
org.apache.ignite.internal.diagnostic.warning - Ready affinity version:
AffinityTopologyVersion [topVer=751, minorTopVer=0]
15-05-2019 15:30:57.519 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 3648 milliseconds.
15-05-2019 15:31:04.762 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 7242 milliseconds.
15-05-2019 15:31:31.284 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 22505 milliseconds.
15-05-2019 15:31:38.924 [jvm-pause-detector-worker] WARN
 org.apache.ignite.internal.IgniteKernal.warning - Possible too long
JVM pause: 11482 milliseconds.
15-05-2019 15:31:47.483 [grid-timeout-worker-#23] INFO
 org.apache.ignite.internal.IgniteKernal.info - 
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
    ^-- Node [id=604f46ce, uptime=05:40:44.720]
    ^-- H/N/C [hosts=7, nodes=7, CPUs=56]
    ^-- CPU [cur=100%, avg=0.63%, GC=30.2%]
    ^-- PageMemory [pages=0]
    ^-- Heap [used=2025MB, free=1.08%, comm=2048MB]
    ^-- Off-heap [used=0MB, free=-1%, comm=0MB]
    ^-- Outbound messages queue [size=0]
    ^-- Public thread pool [active=0, idle=0, qSize=0]
    ^-- System thread pool [active=1, idle=0, qSize=0]
15-05-2019 15:31:54.864 [tcp-client-disco-sock-writer-#2]
ERROR
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error - Failed to send
message: null
java.io.IOException: Failed to get acknowledge for message:
TcpDiscoveryClientMetricsUpdateMessage [super=TcpDiscoveryAbstractMessage
[sndNodeId=null, id=1fe38eaba61-604f46ce-3693-404c-b180-29b2b6dcf8ed,
verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null,
isClient=true]]
        at
org.apache.ignite.spi.discovery.tcp.ClientImpl$SocketWriter.body(ClientImpl.java:1398)
        at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "TcpDiscoverySpi.timer"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Catalina-utility-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "AsyncFileHandlerWriter-1259475182"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "jvm-pause-detector-worker"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-2-#26"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ajp-nio-8009-ClientPoller-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-1-#30"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "exchange-worker-#38"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "Catalina-utility-2"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "tcp-client-disco-sock-reader-#3"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-0-#29"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "session-timeout-worker-#34"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "nio-acceptor-client-listener-#33"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "nio-acceptor-tcp-comm-#28"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-timeout-worker-#23"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "http-nio-8080-ClientPoller-0"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-3-#27"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-3-#32"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-client-listener-2-#31"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-1-#25"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "grid-nio-worker-tcp-comm-0-#24"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "tcp-comm-worker-#1"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "sys-#261"
Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "ignite-update-noti



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/