[
https://issues.apache.org/jira/browse/IGNITE-16136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Albrecht updated IGNITE-16136:
------------------------------------
Description:
We are experiencing thread pool starvations and after some time out of memory
exceptions in some of our ignite client nodes while the server node seems to be
running without any problems. It seems like all sys threads are stuck when
calling MarshallerContextImpl.getClassName. Which in turn leads to a growing
worker queue.
First warnings regarding the thread pool starvation:
{code:java}
10.12.21 11:22:34.603 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:27:34.654 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:32:34.713 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:37:34.764 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:42:34.796 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:47:34.839 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
{code}
Out of memory error leading to a crash of the application:
{code}
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "https-openssl-nio-16443-ClientPoller"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "ajp-nio-16009-ClientPoller"
11-Dec-2021 03:07:24.446 SEVERE [Catalina-utility-1]
org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async
timeouts
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
{code}
The queue full of messages:
!image-2021-12-15-21-17-47-652.png!
It seems like all sys threads are stuck while waiting at:
{code}
sys-#170
at jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
at java.util.concurrent.locks.LockSupport.park()V (LockSupport.java:323)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(Z)Ljava/lang/Object;
(GridFutureAdapter.java:178)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get()Ljava/lang/Object;
(GridFutureAdapter.java:141)
at
org.apache.ignite.internal.MarshallerContextImpl.getClassName(BI)Ljava/lang/String;
(MarshallerContextImpl.java:379)
at
org.apache.ignite.internal.MarshallerContextImpl.getClass(ILjava/lang/ClassLoader;)Ljava/lang/Class;
(MarshallerContextImpl.java:344)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(Ljava/util/concurrent/ConcurrentMap;ILjava/lang/ClassLoader;Lorg/apache/ignite/marshaller/MarshallerContext;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedMarshallerIdMapper;)Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor;
(OptimizedMarshallerUtils.java:264)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:341)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(Ljava/lang/Object;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V
(OptimizedObjectInputStream.java:519)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:611)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:954)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.GridEventConsumeHandler$EventWrapper.readExternal(Ljava/io/ObjectInput;)V
(GridEventConsumeHandler.java:558)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(Ljava/lang/reflect/Constructor;Ljava/lang/reflect/Method;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:555)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:949)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
java.util.concurrent.ConcurrentLinkedDeque.readObject(Ljava/io/ObjectInputStream;)V
(ConcurrentLinkedDeque.java:1588)
at
jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Method.java:566)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:604)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:954)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(OptimizedMarshaller.java:228)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(AbstractNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(Lorg/apache/ignite/internal/binary/streams/BinaryInputStream;Lorg/apache/ignite/internal/binary/BinaryContext;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(BinaryUtils.java:1762)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
(BinaryReaderExImpl.java:1965)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
(BinaryReaderExImpl.java:1717)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(I)Ljava/lang/Object;
(BinaryReaderExImpl.java:1985)
at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
(BinaryFieldAccessor.java:703)
at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
(BinaryFieldAccessor.java:188)
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)Ljava/lang/Object;
(BinaryClassDescriptor.java:875)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
(BinaryReaderExImpl.java:1765)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
(BinaryReaderExImpl.java:1717)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(GridBinaryMarshaller.java:313)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(BinaryMarshaller.java:102)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(AbstractNodeNameAwareMarshaller.java:82)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(Lorg/apache/ignite/marshaller/Marshaller;[BLjava/lang/ClassLoader;)Ljava/lang/Object;
(IgniteUtils.java:10168)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$7.onMessage(Ljava/util/UUID;Ljava/lang/Object;B)V
(GridContinuousProcessor.java:269)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(Ljava/lang/Byte;Lorg/apache/ignite/internal/managers/communication/GridMessageListener;Ljava/util/UUID;Ljava/lang/Object;)V
(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(Lorg/apache/ignite/internal/managers/communication/GridIoManager;Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
(GridIoManager.java:127)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run()V
(GridIoManager.java:1093)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run()V
(ThreadPoolExecutor.java:628)
at java.lang.Thread.run()V (Thread.java:829)
{code}
Screenshot of sys threads stacktrace:
!image-2021-12-15-21-13-43-775.png!
was:
We are experiencing thread pool starvations and after some time out of memory
exceptions in some of our ignite client nodes while the server node seems to be
running without any problems. It seems like all sys threads are stuck when
calling MarshallerContextImpl.getClassName. Which in turn leads to a growing
worker queue.
{code:java}
10.12.21 11:22:34.603 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:27:34.654 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:32:34.713 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:37:34.764 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:42:34.796 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
10.12.21 11:47:34.839 [WARN ]
IgniteKernal.warning(127): Possible thread pool starvation detected (no task
completed in last 30000ms, is system thread pool size large enough?)
{code}
{code}
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "https-openssl-nio-16443-ClientPoller"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler
in thread "ajp-nio-16009-ClientPoller"
11-Dec-2021 03:07:24.446 SEVERE [Catalina-utility-1]
org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async
timeouts
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
{code}
!image-2021-12-15-21-17-47-652.png!
It seems like all sys threads are stuck while waiting at:
{code}
sys-#170
at jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
at java.util.concurrent.locks.LockSupport.park()V (LockSupport.java:323)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(Z)Ljava/lang/Object;
(GridFutureAdapter.java:178)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get()Ljava/lang/Object;
(GridFutureAdapter.java:141)
at
org.apache.ignite.internal.MarshallerContextImpl.getClassName(BI)Ljava/lang/String;
(MarshallerContextImpl.java:379)
at
org.apache.ignite.internal.MarshallerContextImpl.getClass(ILjava/lang/ClassLoader;)Ljava/lang/Class;
(MarshallerContextImpl.java:344)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(Ljava/util/concurrent/ConcurrentMap;ILjava/lang/ClassLoader;Lorg/apache/ignite/marshaller/MarshallerContext;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedMarshallerIdMapper;)Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor;
(OptimizedMarshallerUtils.java:264)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:341)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(Ljava/lang/Object;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V
(OptimizedObjectInputStream.java:519)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:611)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:954)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.GridEventConsumeHandler$EventWrapper.readExternal(Ljava/io/ObjectInput;)V
(GridEventConsumeHandler.java:558)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(Ljava/lang/reflect/Constructor;Ljava/lang/reflect/Method;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:555)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:949)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
java.util.concurrent.ConcurrentLinkedDeque.readObject(Ljava/io/ObjectInputStream;)V
(ConcurrentLinkedDeque.java:1588)
at
jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Unknown Source)
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(DelegatingMethodAccessorImpl.java:43)
at
java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
(Method.java:566)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
(OptimizedObjectInputStream.java:604)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
(OptimizedClassDescriptor.java:954)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
(OptimizedObjectInputStream.java:346)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
(OptimizedObjectInputStream.java:198)
at java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
(ObjectInputStream.java:484)
at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
(ObjectInputStream.java:451)
at
org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(OptimizedMarshaller.java:228)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(AbstractNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(Lorg/apache/ignite/internal/binary/streams/BinaryInputStream;Lorg/apache/ignite/internal/binary/BinaryContext;Ljava/lang/ClassLoader;)Ljava/lang/Object;
(BinaryUtils.java:1762)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
(BinaryReaderExImpl.java:1965)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
(BinaryReaderExImpl.java:1717)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(I)Ljava/lang/Object;
(BinaryReaderExImpl.java:1985)
at
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
(BinaryFieldAccessor.java:703)
at
org.apache.ignite.internal.binary.BinaryFieldAccessor.read(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
(BinaryFieldAccessor.java:188)
at
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)Ljava/lang/Object;
(BinaryClassDescriptor.java:875)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
(BinaryReaderExImpl.java:1765)
at
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
(BinaryReaderExImpl.java:1717)
at
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(GridBinaryMarshaller.java:313)
at
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(BinaryMarshaller.java:102)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal([BLjava/lang/ClassLoader;)Ljava/lang/Object;
(AbstractNodeNameAwareMarshaller.java:82)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(Lorg/apache/ignite/marshaller/Marshaller;[BLjava/lang/ClassLoader;)Ljava/lang/Object;
(IgniteUtils.java:10168)
at
org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$7.onMessage(Ljava/util/UUID;Ljava/lang/Object;B)V
(GridContinuousProcessor.java:269)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(Ljava/lang/Byte;Lorg/apache/ignite/internal/managers/communication/GridMessageListener;Ljava/util/UUID;Ljava/lang/Object;)V
(GridIoManager.java:1569)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
(GridIoManager.java:1197)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(Lorg/apache/ignite/internal/managers/communication/GridIoManager;Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
(GridIoManager.java:127)
at org.apache.ignite.internal.managers.communication.GridIoManager$9.run()V
(GridIoManager.java:1093)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
(ThreadPoolExecutor.java:1128)
at java.util.concurrent.ThreadPoolExecutor$Worker.run()V
(ThreadPoolExecutor.java:628)
at java.lang.Thread.run()V (Thread.java:829)
{code}
!image-2021-12-15-21-13-43-775.png!
> System Thread pool starvation and out of memory
> -----------------------------------------------
>
> Key: IGNITE-16136
> URL: https://issues.apache.org/jira/browse/IGNITE-16136
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 2.7.6
> Reporter: David Albrecht
> Priority: Critical
> Attachments: image-2021-12-15-21-13-43-775.png,
> image-2021-12-15-21-17-47-652.png
>
>
> We are experiencing thread pool starvations and after some time out of memory
> exceptions in some of our ignite client nodes while the server node seems to
> be running without any problems. It seems like all sys threads are stuck when
> calling MarshallerContextImpl.getClassName. Which in turn leads to a growing
> worker queue.
>
> First warnings regarding the thread pool starvation:
> {code:java}
> 10.12.21 11:22:34.603 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:27:34.654 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:32:34.713 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:37:34.764 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:42:34.796 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> 10.12.21 11:47:34.839 [WARN ]
> IgniteKernal.warning(127): Possible thread pool starvation detected (no task
> completed in last 30000ms, is system thread pool size large enough?)
> {code}
> Out of memory error leading to a crash of the application:
> {code}
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "https-openssl-nio-16443-ClientPoller"
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "ajp-nio-16009-ClientPoller"
> 11-Dec-2021 03:07:24.446 SEVERE [Catalina-utility-1]
> org.apache.coyote.AbstractProtocol.startAsyncTimeout Error processing async
> timeouts
> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java heap space
> {code}
> The queue full of messages:
> !image-2021-12-15-21-17-47-652.png!
> It seems like all sys threads are stuck while waiting at:
> {code}
> sys-#170
> at jdk.internal.misc.Unsafe.park(ZJ)V (Native Method)
> at java.util.concurrent.locks.LockSupport.park()V (LockSupport.java:323)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(Z)Ljava/lang/Object;
> (GridFutureAdapter.java:178)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get()Ljava/lang/Object;
> (GridFutureAdapter.java:141)
> at
> org.apache.ignite.internal.MarshallerContextImpl.getClassName(BI)Ljava/lang/String;
> (MarshallerContextImpl.java:379)
> at
> org.apache.ignite.internal.MarshallerContextImpl.getClass(ILjava/lang/ClassLoader;)Ljava/lang/Class;
> (MarshallerContextImpl.java:344)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshallerUtils.classDescriptor(Ljava/util/concurrent/ConcurrentMap;ILjava/lang/ClassLoader;Lorg/apache/ignite/marshaller/MarshallerContext;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedMarshallerIdMapper;)Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor;
> (OptimizedMarshallerUtils.java:264)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:341)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:198)
> at
> java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
> (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
> (ObjectInputStream.java:451)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readFields(Ljava/lang/Object;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$ClassFields;)V
> (OptimizedObjectInputStream.java:519)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
> (OptimizedObjectInputStream.java:611)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
> (OptimizedClassDescriptor.java:954)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:346)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:198)
> at
> java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
> (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
> (ObjectInputStream.java:451)
> at
> org.apache.ignite.internal.GridEventConsumeHandler$EventWrapper.readExternal(Ljava/io/ObjectInput;)V
> (GridEventConsumeHandler.java:558)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readExternalizable(Ljava/lang/reflect/Constructor;Ljava/lang/reflect/Method;)Ljava/lang/Object;
> (OptimizedObjectInputStream.java:555)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
> (OptimizedClassDescriptor.java:949)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:346)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:198)
> at
> java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
> (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
> (ObjectInputStream.java:451)
> at
> java.util.concurrent.ConcurrentLinkedDeque.readObject(Ljava/io/ObjectInputStream;)V
> (ConcurrentLinkedDeque.java:1588)
> at
> jdk.internal.reflect.GeneratedMethodAccessor268.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Unknown Source)
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (DelegatingMethodAccessorImpl.java:43)
> at
> java.lang.reflect.Method.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;
> (Method.java:566)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readSerializable(Ljava/lang/Class;Ljava/util/List;Ljava/lang/reflect/Method;Lorg/apache/ignite/internal/marshaller/optimized/OptimizedClassDescriptor$Fields;)Ljava/lang/Object;
> (OptimizedObjectInputStream.java:604)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedClassDescriptor.read(Lorg/apache/ignite/internal/marshaller/optimized/OptimizedObjectInputStream;)Ljava/lang/Object;
> (OptimizedClassDescriptor.java:954)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObject0()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:346)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedObjectInputStream.readObjectOverride()Ljava/lang/Object;
> (OptimizedObjectInputStream.java:198)
> at
> java.io.ObjectInputStream.readObject(Ljava/lang/Class;)Ljava/lang/Object;
> (ObjectInputStream.java:484)
> at java.io.ObjectInputStream.readObject()Ljava/lang/Object;
> (ObjectInputStream.java:451)
> at
> org.apache.ignite.internal.marshaller.optimized.OptimizedMarshaller.unmarshal0(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
> (OptimizedMarshaller.java:228)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Ljava/io/InputStream;Ljava/lang/ClassLoader;)Ljava/lang/Object;
> (AbstractNodeNameAwareMarshaller.java:94)
> at
> org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(Lorg/apache/ignite/internal/binary/streams/BinaryInputStream;Lorg/apache/ignite/internal/binary/BinaryContext;Ljava/lang/ClassLoader;)Ljava/lang/Object;
> (BinaryUtils.java:1762)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
> (BinaryReaderExImpl.java:1965)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
> (BinaryReaderExImpl.java:1717)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.readField(I)Ljava/lang/Object;
> (BinaryReaderExImpl.java:1985)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.read0(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
> (BinaryFieldAccessor.java:703)
> at
> org.apache.ignite.internal.binary.BinaryFieldAccessor.read(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)V
> (BinaryFieldAccessor.java:188)
> at
> org.apache.ignite.internal.binary.BinaryClassDescriptor.read(Lorg/apache/ignite/internal/binary/BinaryReaderExImpl;)Ljava/lang/Object;
> (BinaryClassDescriptor.java:875)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0()Ljava/lang/Object;
> (BinaryReaderExImpl.java:1765)
> at
> org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize()Ljava/lang/Object;
> (BinaryReaderExImpl.java:1717)
> at
> org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize([BLjava/lang/ClassLoader;)Ljava/lang/Object;
> (GridBinaryMarshaller.java:313)
> at
> org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0([BLjava/lang/ClassLoader;)Ljava/lang/Object;
> (BinaryMarshaller.java:102)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal([BLjava/lang/ClassLoader;)Ljava/lang/Object;
> (AbstractNodeNameAwareMarshaller.java:82)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(Lorg/apache/ignite/marshaller/Marshaller;[BLjava/lang/ClassLoader;)Ljava/lang/Object;
> (IgniteUtils.java:10168)
> at
> org.apache.ignite.internal.processors.continuous.GridContinuousProcessor$7.onMessage(Ljava/util/UUID;Ljava/lang/Object;B)V
> (GridContinuousProcessor.java:269)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(Ljava/lang/Byte;Lorg/apache/ignite/internal/managers/communication/GridMessageListener;Ljava/util/UUID;Ljava/lang/Object;)V
> (GridIoManager.java:1569)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
> (GridIoManager.java:1197)
> at
> org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(Lorg/apache/ignite/internal/managers/communication/GridIoManager;Lorg/apache/ignite/internal/managers/communication/GridIoMessage;Ljava/util/UUID;)V
> (GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run()V
> (GridIoManager.java:1093)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
> (ThreadPoolExecutor.java:1128)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run()V
> (ThreadPoolExecutor.java:628)
> at java.lang.Thread.run()V (Thread.java:829)
> {code}
> Screenshot of sys threads stacktrace:
> !image-2021-12-15-21-13-43-775.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
