[
https://issues.apache.org/jira/browse/ARTEMIS-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Gustafson updated ARTEMIS-4453:
------------------------------------
Description:
I found what I believe to be a bug related to flow control with symmetric
clusters. I wrote [this
example|https://github.com/acgustafson/artemis-flowcontrol-issue] to reproduce
the behavior I am seeing. The test program does the following:
* Create 2 clustered embedded brokers
* Create 3000 topics with 1 producer per topic sending 1 message every 30
seconds with 1500 producers on each broker.
* Create 12 consumers on a wildcard that matches all 3000 topics with 6
consumers on each broker
* Create 1 additional "statistics" wildcard consumer to log how many messages
it has consumed over the previous 30 seconds
Shortly after running this program the "statistics consumer" shows it's only
consuming 1500 messages per 30 seconds. The broker's logs show the
$artemis.internal.sf queues are being blocked on flow control. From what I can
tell the broker never seems to recover from this, even after killing all of the
producers/consumers and restarting with a much lower number of
producers/consumers.
This behaviour started appearing for me in 2.22.0 after
https://issues.apache.org/jira/browse/ARTEMIS-3805 changed the default
producer-window-size to 1MB from -1.
I can reproduce the issue using the latest snapshot. If you set the
producer-window-size back to -1 the issue no longer appears.
Program to reproduce: [https://github.com/acgustafson/artemis-flowcontrol-issue]
Discussion of this issue in slack at
[https://the-asf.slack.com/archives/CFL910J30/p1696356372797149?thread_ts=1694705189.957139&cid=CFL910J30]
was:
I found what I believe to be a bug related to flow control with symmetric
clusters. I wrote [this
example|https://github.com/acgustafson/artemis-flowcontrol-issue] to reproduce
the behavior I am seeing. The test program does the following:
* Create 2 clustered embedded brokers
* Create 3000 topics with 1 producer per topic sending 1 message every 30
seconds
* Create 12 consumers on a wildcard that matches all 3000 topics.
* Create 1 additional "statistics" wildcard consumer to log how many messages
it has consumed over the previous 30 seconds
Shortly after running this program the "statistics consumer" shows it's only
consuming 1500 messages per 30 seconds. The broker's logs show the
$artemis.internal.sf queues are being blocked on flow control. From what I can
tell the broker never seems to recover from this, even after killing all of the
producers/consumers and restarting with a much lower number of
producers/consumers.
This behaviour started appearing for me in 2.22.0 after
https://issues.apache.org/jira/browse/ARTEMIS-3805 changed the default
producer-window-size to 1MB from -1.
I can reproduce the issue using the latest snapshot. If you set the
producer-window-size back to -1 the issue no longer appears.
Program to reproduce: [https://github.com/acgustafson/artemis-flowcontrol-issue]
Discussion of this issue in slack at
https://the-asf.slack.com/archives/CFL910J30/p1696356372797149?thread_ts=1694705189.957139&cid=CFL910J30
> Bridge blocked by flow control, seemingly forever
> -------------------------------------------------
>
> Key: ARTEMIS-4453
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4453
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, Clustering
> Reporter: Andy Gustafson
> Priority: Major
>
> I found what I believe to be a bug related to flow control with symmetric
> clusters. I wrote [this
> example|https://github.com/acgustafson/artemis-flowcontrol-issue] to
> reproduce the behavior I am seeing. The test program does the following:
> * Create 2 clustered embedded brokers
> * Create 3000 topics with 1 producer per topic sending 1 message every 30
> seconds with 1500 producers on each broker.
> * Create 12 consumers on a wildcard that matches all 3000 topics with 6
> consumers on each broker
> * Create 1 additional "statistics" wildcard consumer to log how many
> messages it has consumed over the previous 30 seconds
> Shortly after running this program the "statistics consumer" shows it's only
> consuming 1500 messages per 30 seconds. The broker's logs show the
> $artemis.internal.sf queues are being blocked on flow control. From what I
> can tell the broker never seems to recover from this, even after killing all
> of the producers/consumers and restarting with a much lower number of
> producers/consumers.
> This behaviour started appearing for me in 2.22.0 after
> https://issues.apache.org/jira/browse/ARTEMIS-3805 changed the default
> producer-window-size to 1MB from -1.
>
> I can reproduce the issue using the latest snapshot. If you set the
> producer-window-size back to -1 the issue no longer appears.
>
> Program to reproduce:
> [https://github.com/acgustafson/artemis-flowcontrol-issue]
>
> Discussion of this issue in slack at
> [https://the-asf.slack.com/archives/CFL910J30/p1696356372797149?thread_ts=1694705189.957139&cid=CFL910J30]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
