[
https://issues.apache.org/jira/browse/ARTEMIS-3117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17284009#comment-17284009
]
Sebastian T commented on ARTEMIS-3117:
--------------------------------------
{code:yaml}
- threadName: Thread-16 (activemq-netty-threads)
threadId: 115
threadState: RUNNABLE
blockedTime: 1485418
blockedCount: 64007
waitedTime: 77377
waitedCount: 507941
lockName: null
lockOwnerId: -1
lockOwnerName: null
daemon: true
inNative: false
suspended: false
priority: 5
stackTrace: - java.util.Arrays.fill:3494
- sun.security.provider.DigestBase.engineReset:182
- sun.security.provider.DigestBase.engineUpdate:112
- java.security.MessageDigest$Delegate.engineUpdate:623
- java.security.MessageDigest.update:355
- java.security.MessageDigest.digest:430
- com.sun.crypto.provider.PKCS12PBECipherCore.derive:119
- com.sun.crypto.provider.PKCS12PBECipherCore.derive:69
- com.sun.crypto.provider.HmacPKCS12PBESHA1.engineInit:134
- javax.crypto.Mac.chooseProvider:366
- javax.crypto.Mac.init:465
- sun.security.pkcs12.PKCS12KeyStore.lambda$engineLoad$2:2151
-
sun.security.pkcs12.PKCS12KeyStore$$Lambda$617/0x00007f82725bc1b0.tryOnce:-1
- sun.security.pkcs12.PKCS12KeyStore$RetryWithZero.run:295
- sun.security.pkcs12.PKCS12KeyStore.engineLoad:2149
- sun.security.util.KeyStoreDelegator.engineLoad:243
- java.security.KeyStore.load:1479
-
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeystore:265
-
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadTrustManagerFactory:213
-
org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createNettyContext:171
-
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.loadOpenSslEngine:654
-
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.getSslHandler:529
-
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor$4.initChannel:415
- io.netty.channel.ChannelInitializer.initChannel:129
- io.netty.channel.ChannelInitializer.handlerAdded:112
- io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded:938
- io.netty.channel.DefaultChannelPipeline.callHandlerAdded0:609
- io.netty.channel.DefaultChannelPipeline.access$100:46
-
io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute:1463
- io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers:1115
- io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded:650
- io.netty.channel.AbstractChannel$AbstractUnsafe.register0:502
- io.netty.channel.AbstractChannel$AbstractUnsafe.access$200:417
- io.netty.channel.AbstractChannel$AbstractUnsafe$1.run:474
- io.netty.util.concurrent.AbstractEventExecutor.safeExecute:164
- io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks:472
- io.netty.channel.epoll.EpollEventLoop.run:384
- io.netty.util.concurrent.SingleThreadEventExecutor$4.run:989
- io.netty.util.internal.ThreadExecutorMap$2.run:74
- org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run:118
lockedMonitors: - className: java.lang.Object
identityHashCode: 2036147318
lockedStackDepth: 9
lockedStackFrame: javax.crypto.Mac.chooseProvider:366
- className: sun.security.pkcs12.PKCS12KeyStore
identityHashCode: 1078305195
lockedStackDepth: 14
lockedStackFrame: sun.security.pkcs12.PKCS12KeyStore.engineLoad:2149
- className:
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor
identityHashCode: 1316121135
lockedStackDepth: 21
lockedStackFrame:
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.getSslHandler:529
lockedSynchronizers: []
lockInfo: null
- threadName: Thread-17 (activemq-netty-threads)
threadId: 116
threadState: BLOCKED
blockedTime: 1501712
blockedCount: 87086
waitedTime: 17546
waitedCount: 106476
lockName:
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor@4e726a2f
lockOwnerId: 115
lockOwnerName: Thread-16 (activemq-netty-threads)
daemon: true
inNative: false
suspended: false
priority: 5
stackTrace:
-
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor:getSslHandler:528
-
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor$4:initChannel:415
- io.netty.channel.ChannelInitializer:initChannel:129
- io.netty.channel.ChannelInitializer:handlerAdded:112
- io.netty.channel.AbstractChannelHandlerContext:callHandlerAdded:938
- io.netty.ch
io.netty.channel.DefaultChannelPipeline:methodName:access$100:46
-
io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask:execute:1463
- io.netty.channel.DefaultChannelPipeline:callHandlerAddedForAllHandlers:1115
- io.netty.channel.DefaultChannelPipeline:invokeHandlerAddedIfNeeded:650
- io.netty.channel.AbstractChannel$AbstractUnsafe:register0:502
- io.netty.channel.AbstractChannel$AbstractUnsafe:access$200:417
- io.netty.channel.AbstractChannel$AbstractUnsafe$1:run:474
- io.netty.util.concurrent.AbstractEventExecutor:safeExecute:164
- io.netty.util.concurrent.SingleThreadEventExecutor:runAllTasks:472
- io.netty.channel.epoll.EpollEventLoop:run:384
- io.netty.util.concurrent.SingleThreadEventExecutor$4:run:989
- io.netty.util.internal.ThreadExecutorMap$2:run:74
- org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1:run:118
lockedMonitors: []
lockedSynchronizers: []
lockInfo: className:
org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor
identityHashCode: 1316121135
{code}
> Performance degradation when switching from JDK8 to JDK11
> ---------------------------------------------------------
>
> Key: ARTEMIS-3117
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3117
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Broker
> Affects Versions: 2.16.0
> Environment: Amazon Linux 2, Amazon Corretto (OpenJDK 11), AMQP over
> TLS via BoringSSL
> Reporter: Sebastian T
> Priority: Major
> Attachments: broker.xml, image-2021-02-12-21-39-32-185.png,
> image-2021-02-12-21-40-21-125.png, image-2021-02-12-21-44-26-271.png,
> image-2021-02-12-21-46-52-006.png, image-2021-02-12-21-47-02-387.png,
> image-2021-02-12-21-47-57-301.png, image-2021-02-12-22-01-07-044.png
>
>
> Since it was announced that probably Artemis 2.18.0 will require Java 11 we
> upgraded the JVM of one of our broker clusters from OpenJDK 8 to OpenJDK 11
> and are seeing a noticable performance degradation which results in higher
> CPU usage and higher latency.
> We are monitoring request/reply round trip duration with a custom distributed
> qpid-jms based healthcheck applications. Here is a graphic that shows the
> effect when we switched the JDK:
> !image-2021-02-12-21-39-32-185.png!
> CPU Usage of the broker process:
> !image-2021-02-12-22-01-07-044.png|width=874,height=262!
>
> The broker itself is also monitored via Dynatrace, there I can see that after
> upgrading to JDK 11 the broker process spend 21% of CPU time locking while in
> JDK it only spent 3.2%.
> *JDK 8:*
> !image-2021-02-12-21-40-21-125.png|width=1247,height=438!
>
> *JDK 11:*
> *!image-2021-02-12-21-44-26-271.png|width=1197,height=420!*
>
> *A method hotspot breakdown reveals this:*
> !image-2021-02-12-21-47-02-387.png|width=1271,height=605!
> !image-2021-02-12-21-47-57-301.png|width=1059,height=627!
> Maybe I am misinterpreting the charts but the root cause seems to be
> somewhere in {{org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1}}
> and/or in
> {{org.apache.activemq.artemis.core.remoting.impl.netty.NettyAcceptor.getSslHandler}}
> I currently cannot pinpoint the exact line number.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
