[
https://issues.apache.org/jira/browse/ARTEMIS-3271?focusedWorklogId=597221&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-597221
]
ASF GitHub Bot logged work on ARTEMIS-3271:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 15/May/21 14:10
Start Date: 15/May/21 14:10
Worklog Time Spent: 10m
Work Description: franz1981 commented on pull request #3558:
URL: https://github.com/apache/activemq-artemis/pull/3558#issuecomment-841664877
@clebertsuconic
I'm getting some weird issue while under heavy load...
```
2021-05-15 10:00:49,256 WARN
[org.apache.activemq.artemis.utils.critical.CriticalMeasure] Component
org.apache.activemq.artemis.core.server.impl.QueueImpl is expired on path 0
2021-05-15 10:00:49,258 INFO [org.apache.activemq.artemis.core.server]
AMQ224107: The Critical Analyzer detected slow paths on the broker. It is
recommended that you enable trace logs on
org.apache.activemq.artemis.utils.critical while you troubleshoot this issue.
You should disable the trace logs when you have finished troubleshooting.
2021-05-15 10:00:49,258 ERROR [org.apache.activemq.artemis.core.server]
AMQ224079: The process for the virtual machine will be killed, as component
QueueImpl[name=q_7, postOffice=PostOfficeImpl
[server=ActiveMQServerImpl::serverUUID=64e3f254-b585-11eb-8f02-000f532921b8],
temp=false]@9a33f70 is not responsive
2021-05-15 10:00:50,429 WARN [org.apache.activemq.artemis.core.server]
AMQ222199: Thread dump:
*******************************************************************************
Complete Thread dump
```
```
"Thread-17
(ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@15bbf42f)"
Id=142 RUNNABLE
at io.netty.channel.epoll.Native.eventFdWrite(Native Method)
at io.netty.channel.epoll.EpollEventLoop.wakeup(EpollEventLoop.java:186)
at
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:849)
at
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
at
io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
at
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
at
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
at
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
at
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:311)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:364)
at
org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:288)
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:418)
- locked java.lang.Object@7755ba9d
at
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:329)
at
org.apache.activemq.artemis.core.protocol.core.impl.CoreSessionCallback.sendMessage(CoreSessionCallback.java:128)
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1170)
at
org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:514)
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:3747)
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.deliver(QueueImpl.java:3689)
- locked org.apache.activemq.artemis.core.server.impl.QueueImpl@9a33f70
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.deliverDirect(QueueImpl.java:3643)
at
org.apache.activemq.artemis.core.server.impl.QueueImpl.addTail(QueueImpl.java:1218)
at
org.apache.activemq.artemis.core.server.impl.RoutingContextImpl.internalprocessReferences(RoutingContextImpl.java:182)
at
org.apache.activemq.artemis.core.server.impl.RoutingContextImpl.processReferences(RoutingContextImpl.java:177)
at
org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl$2.done(PostOfficeImpl.java:1568)
at
org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl$1.run(OperationContextImpl.java:279)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
at
org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
at
org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$37/599984672.run(Unknown
Source)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Number of locked synchronizers = 2
- java.util.concurrent.locks.ReentrantLock$NonfairSync@2e770b23
- java.util.concurrent.ThreadPoolExecutor$Worker@6cf16158
```
The stack trace show just a single thread in `RUNNABLE` state in that same
path...so probably there is some error related how we collect the metric...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 597221)
Time Spent: 2.5h (was: 2h 20m)
> Avoid Multi Thread measurement on Critical Analyzer
> ---------------------------------------------------
>
> Key: ARTEMIS-3271
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3271
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.17.0
> Reporter: Clebert Suconic
> Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The Critical Analyzer could be called from multiple Threads.
>
> As a result, I am afraid there could be a situation that left time was
> updated at a wrong pace.. leaving to inconsistencies.
>
>
> I'm improving it to only consider a single thread from all the calls. Any
> calls from multi-threading should be ignored... All we need is to capture one
> thread hanging and detect a dead lock.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
