[
https://issues.apache.org/jira/browse/ARTEMIS-4522?focusedWorklogId=893425&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-893425
]
ASF GitHub Bot logged work on ARTEMIS-4522:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 01/Dec/23 13:56
Start Date: 01/Dec/23 13:56
Worklog Time Spent: 10m
Work Description: MrEasy commented on PR #4699:
URL:
https://github.com/apache/activemq-artemis/pull/4699#issuecomment-1836160916
Hi,
> When you hit another feature not working because the thread pool is
saturated, are you going to add another pool to that feature also?
If you reach the configured limit, I think everyone is clear that you will
get delays until resources are freed. Our goal here was to address the issue
that was more unexpected to us, which lead to the big 10 seconds delays based
on the flow-control explicitly.
You could argue, that if going into slow-consumer-handling is not aiming for
highest throughput anyhow, but having consumer-window-size>0 this has another
disadvantage for us (see lengthy explanatin by Sascha in slack).
So goal for us here was to stick to the slow-consumer-handling, but avoiding
the big 10 seconds influence of the full pool on it.
> Having a warn that the pool is saturated and suggestion to increase would
probably be a better feature IMO.
I think that would also be an excellent improvement, since now it is not
easily visible when the client thread-pool reaches its limit.
Issue Time Tracking
-------------------
Worklog Id: (was: 893425)
Time Spent: 40m (was: 0.5h)
> Slow-Consumer-Handling may get delayed significantly on occupied pool
> ---------------------------------------------------------------------
>
> Key: ARTEMIS-4522
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4522
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Reporter: Rico Neubauer
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Slack discussion with [~jbertram] see here:
> [https://the-asf.slack.com/archives/CFL910J30/p1699520764243169]
> h2. Initial Description
> consumers-window-size set to 0 and there some queues, that are heavily used
> and have consumers attached, that allow for a concurrency of 500 messages
> (larger than thread-pool-client-size).
> After some time - unclear what triggers it - the behavior becomes bad in the
> way that from those queues only 1-5 messages are fetched and immediately
> processed, but then there is a pause of 10 seconds before the next 1-5
> messages are fetched. This continues.
> In the thread-dumps, several like the following can be seen:
> {noformat}
> "Thread-0 (ActiveMQ-client-global-threads)" #640 daemon prio=5 os_prio=0
> cpu=899719.36ms elapsed=354425.44s tid=0x00007f904c02e850 nid=0x150d29
> runnable [0x00007f8fdbae5000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
> - parking to wait for <0x00000003f6c9d1d0> (a
> org.apache.activemq.artemis.utils.AbstractLatch$CountSync)
> at
> java.util.concurrent.locks.LockSupport.parkNanos([email protected]/LockSupport.java:252)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:717)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos([email protected]/AbstractQueuedSynchronizer.java:1074)
> at
> org.apache.activemq.artemis.utils.AbstractLatch.await(AbstractLatch.java:115)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.startSlowConsumer(ClientConsumerImpl.java:869)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl.callOnMessage(ClientConsumerImpl.java:1025)
> at
> org.apache.activemq.artemis.core.client.impl.ClientConsumerImpl$Runner.run(ClientConsumerImpl.java:1154)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:42)
> at
> org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:31)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:65)
> at
> org.apache.activemq.artemis.utils.actors.ProcessorBase$$Lambda$2008/0x00000007ce20c000.run(Unknown
> Source)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
> at
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
> {noformat}
> which corresponds to code location with a 10 seconds sleep here:
> [https://github.com/apache/activemq-artemis/blob/main/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/impl/ClientConsumerImpl.java#L854|https://github.com/seeburger-ag/activemq-artemis/blob/main/artemis-core-client/src/main/java/org/apache/activemq/artemis/core/client/impl/ClientConsumerImpl.java#L854]
> It seems it always needs to wait until the 10 seconds are reached, which is
> unexpected, since the latch should count-down before, after having sent the
> credit to the server. We could assume some bug, that maybe increments the
> latch, but misses to decrement, so it is not able to reach 0 ever again, but
> that's speculative.
> h2. Evaluation
> See slack for details, it boils down to the thread-pool in use being out of
> free threads, so the slow-consumer-handling is not able to run in good time.
> Affected versions: all since at least 2.22.0 up to current - probably since
> "ever", involved code derives from HornetQ.
> h2. Possible Solutions
> We experimented a bit and found a solution for us - or actually 2 variants.
> * No. 1 just uses an own single-threaded executor for flow-control, whose
> only job is to guarantee the credit was sent before continuing with other
> operations when starting/resetting slow-consumers.
> This looks simple and works perfectly in the test-case as well as in the real
> product in stress-tests
> [https://github.com/seeburger-ag/activemq-artemis/commit/5b94560fc55fabcc15f138fbec6b2f855508cc7e]
> * No. 2 has a similar idea, but goes the full way of adding an own
> thread-pool for flow-control and adding it like the others.
> This is more fitting into the style of the existing thread-pools and allows
> its configuration.
> [https://github.com/seeburger-ag/activemq-artemis/commit/b4d558c5fc90f3a442391ebcf4976e8d6ec7adb0]
> and [~jbertram] said:
> {quote}I think either one would be fine. Create a Jira and send a PR.
> {quote}
> So here is the Jira and I will push No. 2 as a pull-request:
> https://github.com/apache/activemq-artemis/pull/4699
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
