[
https://issues.apache.org/jira/browse/ARTEMIS-2623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307042#comment-17307042
]
Francesco Nigro commented on ARTEMIS-2623:
------------------------------------------
[~mahendra s] if you would like the broker to handle OOM you should set the JVM
to do it ie by using {{-XX:+ExitOnOutOfMemoryError}} or
{{-XX:+CrashOnOutOfMemoryError}}: the issue raised by [~kwall] is related to
OOM caused by native Thread creation that are not handled correctly by the
mentioned JVM parameters.
> Artemis continues to run in broken state after OOME (thread)
> ------------------------------------------------------------
>
> Key: ARTEMIS-2623
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2623
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.11.0
> Environment: openjdk 11.0.3 2019-04-16 LTS
> OpenJDK Runtime Environment 18.9 (build 11.0.3+7-LTS)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.3+7-LTS, mixed mode, sharing)
> Reporter: Keith Wall
> Priority: Major
>
> If Artemis runs out of threads with the JVM signalling this with an OOME
> error, the process continues in a broken state logging exceptions. It would
> be better if the process exited so the running environment (Kubernetes, job
> control etc) can restart it.
> Note that stack trace shows a {{ClassCastException}} where an attempt to made
> to assign the OOME (Error) to an Exception.
> I read from [JDK-8155004|https://bugs.openjdk.java.net/browse/JDK-8155004]
> that the JVM flag {{XxxxOnOutOfMemoryError}} is not applied to this sort of
> OOME.
> {noformat}
> [104223.528s][warning][os,thread] Failed to start thread - pthread_create
> failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> 2020-02-19T14:41:04.437Z WARN [ProtonServerSenderContext]
> java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast
> to class java.lang.Exception (java.lang.OutOfMemoryError and
> java.lang.Exception are in module java.base of loader 'bootstrap')
> java.lang.ClassCastException: class java.lang.OutOfMemoryError cannot be cast
> to class java.lang.Exception (java.lang.OutOfMemoryError and
> java.lang.Exception are in module java.base of loader 'bootstrap')
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.individualAcknowledge(ServerConsumerImpl.java:998)
> at
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPSessionCallback.ack(AMQPSessionCallback.java:399)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:621)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at
> org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at
> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> 2020-02-19T14:41:04.437Z WARN [ProtonHandler] AMQ119007: error acknowledging
> message
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class
> java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception
> (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base
> of loader 'bootstrap')
> org.apache.activemq.artemis.protocol.amqp.exceptions.ActiveMQAMQPIllegalStateException:
> AMQ119007: error acknowledging message
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPMessage@51dd2f4c, class
> java.lang.OutOfMemoryError cannot be cast to class java.lang.Exception
> (java.lang.OutOfMemoryError and java.lang.Exception are in module java.base
> of loader 'bootstrap')
> at
> org.apache.activemq.artemis.protocol.amqp.proton.ProtonServerSenderContext.onMessage(ProtonServerSenderContext.java:624)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.onDelivery(AMQPConnectionContext.java:583)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.Events.dispatch(Events.java:92)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.dispatch(ProtonHandler.java:485)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.flush(ProtonHandler.java:285)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.inputBuffer(ProtonHandler.java:242)
> at
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.inputBuffer(AMQPConnectionContext.java:180)
> at
> org.apache.activemq.artemis.protocol.amqp.broker.ActiveMQProtonRemotingConnection.bufferReceived(ActiveMQProtonRemotingConnection.java:145)
> at
> org.apache.activemq.artemis.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:654)
> at
> org.apache.activemq.artemis.core.remoting.impl.netty.ActiveMQChannelHandler.channelRead(ActiveMQChannelHandler.java:73)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1470)
> at
> io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1219)
> at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1266)
> at
> io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
> at
> io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
> at
> io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:796)
> at
> io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:427)
> at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:328)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> at
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> [104223.571s][warning][os,thread] Failed to start thread - pthread_create
> failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
