[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=866120=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-866120 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 17/Jun/23 00:20 Start Date: 17/Jun/23 00:20 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4511: URL: https://github.com/apache/activemq-artemis/pull/4511#issuecomment-1595508793 tests are good.. I'm merging this Issue Time Tracking --- Worklog Id: (was: 866120) Time Spent: 2.5h (was: 2h 20m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=866121=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-866121 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 17/Jun/23 00:20 Start Date: 17/Jun/23 00:20 Worklog Time Spent: 10m Work Description: clebertsuconic merged PR #4511: URL: https://github.com/apache/activemq-artemis/pull/4511 Issue Time Tracking --- Worklog Id: (was: 866121) Time Spent: 2h 40m (was: 2.5h) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=866103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-866103 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 20:14 Start Date: 16/Jun/23 20:14 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4511: URL: https://github.com/apache/activemq-artemis/pull/4511#issuecomment-1595259485 @gtully either merge it or close if you prefer not to do this please? Thanks Issue Time Tracking --- Worklog Id: (was: 866103) Time Spent: 2h 20m (was: 2h 10m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=866102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-866102 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 20:14 Start Date: 16/Jun/23 20:14 Worklog Time Spent: 10m Work Description: clebertsuconic opened a new pull request, #4511: URL: https://github.com/apache/activemq-artemis/pull/4511 (no comment) Issue Time Tracking --- Worklog Id: (was: 866102) Time Spent: 2h 10m (was: 2h) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=866056=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-866056 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 14:44 Start Date: 16/Jun/23 14:44 Worklog Time Spent: 10m Work Description: gtully merged PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509 Issue Time Tracking --- Worklog Id: (was: 866056) Time Spent: 2h (was: 1h 50m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865979=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865979 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 10:11 Start Date: 16/Jun/23 10:11 Worklog Time Spent: 10m Work Description: gtully commented on code in PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#discussion_r1232058024 ## artemis-server/src/main/java/org/apache/activemq/artemis/core/server/Queue.java: ## @@ -389,6 +389,8 @@ default int retryMessages(Filter filter, Integer expectedHits) throws Exception boolean hasMatchingConsumer(Message message); Review Comment: agree, I was leaving it unchanged but calling it what it is PendingMessageCount is best, sorted. thanks. Issue Time Tracking --- Worklog Id: (was: 865979) Time Spent: 1h 50m (was: 1h 40m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[
https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865975=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865975
]
ASF GitHub Bot logged work on ARTEMIS-4314:
---
Author: ASF GitHub Bot
Created on: 16/Jun/23 10:08
Start Date: 16/Jun/23 10:08
Worklog Time Spent: 10m
Work Description: gtully commented on code in PR #4509:
URL: https://github.com/apache/activemq-artemis/pull/4509#discussion_r1232055299
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/server/federation/FederatedQueueConsumerImpl.java:
##
@@ -155,6 +167,67 @@ private synchronized void connect() throws Exception {
}
}
+ interface QueueHandle {
+ long getMessageCount();
+ int getCreditWindow();
+ }
+
+ private QueueHandle createQueueHandle(ActiveMQServer server,
ClientSession.QueueQuery queryResult) {
+ final Queue queue = server.locateQueue(queryResult.getName());
+ int creditWindow = DEFAULT_CONSUMER_WINDOW_SIZE;
+
+ final Integer defaultConsumerWindowSize =
queryResult.getDefaultConsumerWindowSize();
+ if (defaultConsumerWindowSize != null) {
+ creditWindow = defaultConsumerWindowSize.intValue();
+ if (creditWindow <= 0) {
+creditWindow = DEFAULT_CONSUMER_WINDOW_SIZE;
+logger.trace("{} override non positive queue consumerWindowSize
with {}.", this, creditWindow);
+ }
+ }
+
+ final int finalCreditWindow = creditWindow;
+ return new QueueHandle() {
+ @Override
+ public long getMessageCount() {
+return queue.getMessageCountForRing();
+ }
+
+ @Override
+ public int getCreditWindow() {
+return finalCreditWindow;
+ }
+ };
+ }
+
+ private void scheduleCreditOnEmpty(final int delay, final QueueHandle
handle) {
+ if (handle != null) {
Review Comment:
agree, that check was just defensive but unnecessary, the consumer can go
stale with a pending check but that is already covered and any failure will
result in the session/consumer getting recreated, so all that in necessary is
that we don't reschedule in that case.
thanks for the feedback.
Issue Time Tracking
---
Worklog Id: (was: 865975)
Time Spent: 1h 40m (was: 1.5h)
> Federation, support consumerWindowSize zero and federate in batches only when
> the local queue is has excess capacity
>
>
> Key: ARTEMIS-4314
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4314
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Federation
>Affects Versions: 2.28.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>Priority: Major
> Fix For: 2.29.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Dual queue federation, where clusters federate in both direction can suffer
> from message flip flopping once the priority adjustment kicks in.
> If there is a large backlog, the lower priority federation consumer is in
> play once all of the local consumer credit is exhausted and the backlog can
> drain to the other cluster.
> If demand is low there, the process can repeat. limiting the rate of the
> federation consumer can help but it is not ideal b/c when there is no local
> demand, we want to have a high rate of migration.
>
> A possible solution is to have the federation consumer manage its own credit
> and only flow messages when the local queue has capacity. Then flow a batch
> of messages, and await again that the local queue has capacity. In this way,
> there is no thundering herd effect, but there is also fast migration of
> messages once there is demand.
> the consumerWindowSize=0 is already in play for consumer.receive calls and
> there is already a defaultConsumerWindowSize for an address. These can be
> combined to realise batchFederationOnCapacity semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865973=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865973 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 10:05 Start Date: 16/Jun/23 10:05 Worklog Time Spent: 10m Work Description: gtully commented on PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#issuecomment-1594440095 thanks @clebertsuconic I have used the queue executor to sync on the metrics, great input thanks. That will make it more responsive, missing a trigger would incur an unnecessary delay. Issue Time Tracking --- Worklog Id: (was: 865973) Time Spent: 1.5h (was: 1h 20m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865903=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865903 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 16/Jun/23 01:14 Start Date: 16/Jun/23 01:14 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#issuecomment-1593914208 disclaimer.. I didn't test my changes.. those are just suggestions. Issue Time Tracking --- Worklog Id: (was: 865903) Time Spent: 1h 20m (was: 1h 10m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865884=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865884 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 15/Jun/23 21:04 Start Date: 15/Jun/23 21:04 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#issuecomment-1593721724 and BTW I wasn't able to use the ActiveMQSCheduledComponent as I suggested because of some of the state needed in the calls. Issue Time Tracking --- Worklog Id: (was: 865884) Time Spent: 1h 10m (was: 1h) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865881=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865881 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 15/Jun/23 21:03 Start Date: 15/Jun/23 21:03 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#issuecomment-1593720505 @gtully for some reason I am not being able to make a PR to your repo. Github is not recognizing your user on the filters and giving me a hard time. So, I created a branch you can pull the commit: https://github.com/clebertsuconic/activemq-artemis/tree/_4314 Basically you will endup in this calling flowControl method. which needs to be protected and using the same thread used to deliver. it's not just that state.. any call to flowControl has to be done through the same thread used to deliver... otherwise you will have races on the number of credits. Issue Time Tracking --- Worklog Id: (was: 865881) Time Spent: 50m (was: 40m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865882=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865882 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 15/Jun/23 21:03 Start Date: 15/Jun/23 21:03 Worklog Time Spent: 10m Work Description: clebertsuconic commented on PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#issuecomment-1593721108 another option would be to use a sycnhronize block everywhere to make sure the flow control is correct.. so it's better to just use the same executor guaranteeing the proper threading I think. Issue Time Tracking --- Worklog Id: (was: 865882) Time Spent: 1h (was: 50m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[
https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865879=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865879
]
ASF GitHub Bot logged work on ARTEMIS-4314:
---
Author: ASF GitHub Bot
Created on: 15/Jun/23 20:51
Start Date: 15/Jun/23 20:51
Worklog Time Spent: 10m
Work Description: tabish121 commented on code in PR #4509:
URL: https://github.com/apache/activemq-artemis/pull/4509#discussion_r1231516909
##
artemis-server/src/main/java/org/apache/activemq/artemis/core/server/federation/FederatedQueueConsumerImpl.java:
##
@@ -155,6 +167,67 @@ private synchronized void connect() throws Exception {
}
}
+ interface QueueHandle {
+ long getMessageCount();
+ int getCreditWindow();
+ }
+
+ private QueueHandle createQueueHandle(ActiveMQServer server,
ClientSession.QueueQuery queryResult) {
+ final Queue queue = server.locateQueue(queryResult.getName());
+ int creditWindow = DEFAULT_CONSUMER_WINDOW_SIZE;
+
+ final Integer defaultConsumerWindowSize =
queryResult.getDefaultConsumerWindowSize();
+ if (defaultConsumerWindowSize != null) {
+ creditWindow = defaultConsumerWindowSize.intValue();
+ if (creditWindow <= 0) {
+creditWindow = DEFAULT_CONSUMER_WINDOW_SIZE;
+logger.trace("{} override non positive queue consumerWindowSize
with {}.", this, creditWindow);
+ }
+ }
+
+ final int finalCreditWindow = creditWindow;
+ return new QueueHandle() {
+ @Override
+ public long getMessageCount() {
+return queue.getMessageCountForRing();
+ }
+
+ @Override
+ public int getCreditWindow() {
+return finalCreditWindow;
+ }
+ };
+ }
+
+ private void scheduleCreditOnEmpty(final int delay, final QueueHandle
handle) {
+ if (handle != null) {
Review Comment:
Personally I'd treat a null handle being passed here as a terminal event and
throw since it's probably gonna break something if you try and schedule for
credit and don't actually schedule anything to replenish credit.
Issue Time Tracking
---
Worklog Id: (was: 865879)
Time Spent: 40m (was: 0.5h)
> Federation, support consumerWindowSize zero and federate in batches only when
> the local queue is has excess capacity
>
>
> Key: ARTEMIS-4314
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4314
> Project: ActiveMQ Artemis
> Issue Type: Improvement
> Components: Federation
>Affects Versions: 2.28.0
>Reporter: Gary Tully
>Assignee: Gary Tully
>Priority: Major
> Fix For: 2.29.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Dual queue federation, where clusters federate in both direction can suffer
> from message flip flopping once the priority adjustment kicks in.
> If there is a large backlog, the lower priority federation consumer is in
> play once all of the local consumer credit is exhausted and the backlog can
> drain to the other cluster.
> If demand is low there, the process can repeat. limiting the rate of the
> federation consumer can help but it is not ideal b/c when there is no local
> demand, we want to have a high rate of migration.
>
> A possible solution is to have the federation consumer manage its own credit
> and only flow messages when the local queue has capacity. Then flow a batch
> of messages, and await again that the local queue has capacity. In this way,
> there is no thundering herd effect, but there is also fast migration of
> messages once there is demand.
> the consumerWindowSize=0 is already in play for consumer.receive calls and
> there is already a defaultConsumerWindowSize for an address. These can be
> combined to realise batchFederationOnCapacity semantics.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4314) Federation, support consumerWindowSize zero and federate in batches only when the local queue is has excess capacity
[ https://issues.apache.org/jira/browse/ARTEMIS-4314?focusedWorklogId=865870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-865870 ] ASF GitHub Bot logged work on ARTEMIS-4314: --- Author: ASF GitHub Bot Created on: 15/Jun/23 20:07 Start Date: 15/Jun/23 20:07 Worklog Time Spent: 10m Work Description: clebertsuconic commented on code in PR #4509: URL: https://github.com/apache/activemq-artemis/pull/4509#discussion_r1231485362 ## artemis-server/src/main/java/org/apache/activemq/artemis/core/server/Queue.java: ## @@ -389,6 +389,8 @@ default int retryMessages(Filter filter, Integer expectedHits) throws Exception boolean hasMatchingConsumer(Message message); Review Comment: If you're using this outside of ring, I would rename this to something else... getRawMessageCount (terrible name I know.. but anything removing the "ring" term) Issue Time Tracking --- Worklog Id: (was: 865870) Time Spent: 0.5h (was: 20m) > Federation, support consumerWindowSize zero and federate in batches only when > the local queue is has excess capacity > > > Key: ARTEMIS-4314 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4314 > Project: ActiveMQ Artemis > Issue Type: Improvement > Components: Federation >Affects Versions: 2.28.0 >Reporter: Gary Tully >Assignee: Gary Tully >Priority: Major > Fix For: 2.29.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Dual queue federation, where clusters federate in both direction can suffer > from message flip flopping once the priority adjustment kicks in. > If there is a large backlog, the lower priority federation consumer is in > play once all of the local consumer credit is exhausted and the backlog can > drain to the other cluster. > If demand is low there, the process can repeat. limiting the rate of the > federation consumer can help but it is not ideal b/c when there is no local > demand, we want to have a high rate of migration. > > A possible solution is to have the federation consumer manage its own credit > and only flow messages when the local queue has capacity. Then flow a batch > of messages, and await again that the local queue has capacity. In this way, > there is no thundering herd effect, but there is also fast migration of > messages once there is demand. > the consumerWindowSize=0 is already in play for consumer.receive calls and > there is already a defaultConsumerWindowSize for an address. These can be > combined to realise batchFederationOnCapacity semantics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
