Justin,

Thanks again for your attention to this.  I have verified many times now that 
message groups work in our test harness --
When I enable message groups the test succeeds, and when I disable message 
groups the test fails.

Your concern about message groups was: “Even in the case of message grouping, 
the messages are not directed to a specific consumer as much as they are just 
directed at _just one consumer (i.e. any consumer) at a time_ since the goal is 
just for the messages to be processed in order.”

In our case, all three types of RPC request messages for a given task 
(startLongThing, isLongThingFinished, and getLongThingResult) get the same 
message group ID.  Logically the RPC message stream would look like:

REQUEST: {"request": "start_long_thing", "message_group":"123456"}
RESPONSE: {"task_id":"1"}
REQUEST: {"request": "is_long_thing_finished", "task_id":"1", 
"message_group":"123456"}
RESPONSE: {"finished":"false"}
REQUEST: {"request": "is_long_thing_finished", "task_id":"1", 
"message_group":"123456"}
RESPONSE: {"finished":"false"}
...
REQUEST: {"request": "is_long_thing_finished", "task_id":"1", 
"message_group":"123456"}
RESPONSE: {"finished":"true"}
REQUEST: {"request": "get_long_thing_result", "task_id":"1", 
"message_group":"123456"}
RESPONSE: {"result":{...}}

All of the "is_long_thing_finished" requests are polling at a rate of, let’s 
say, once per second.

As far as our application is concerned, the important thing is that all of 
these messages with the same message group Id are deliver to the same consumer. 
 It doesn’t matter which one.  I believe that your description of how message 
groups work does indeed match our needs.

However, there are some things that concern me about the behavior of message 
groups which are not documented, and I’m hoping you or someone on the group can 
fill in the blanks for me.

My fundamental question is: Do message group IDs expire?
Every task that we create gets a unique message group ID.  Over the course of 
time, the cumulative set of unique message group IDs is unbounded.

This leads to the first question – if the broker tracks message group IDs 
“forever”, will this cause a memory leak in the broker as the tracking of these 
IDs accumulates?

The flip side of that question: if the broker expires message group IDs and 
eventually forgets about them, how much time is a message group ID available 
before it expires?  Does the tracking of the message group continue to get 
renewed so long as it is used periodically?

Thanks
John





[rg] <https://www.redpointglobal.com/>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

34 Washington Street, Suite 205 Wellesley Hills, MA 02481

M: +1 7209385761<tel:+1%207209385761> | 
john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>
From: Justin Bertram <jbert...@apache.org>
Sent: Saturday, December 9, 2023 10:49 PM
To: users@activemq.apache.org
Subject: Re: Is it possible to route message to a specific consumer?

*** [Caution] This email is from an external source. Please use caution 
responding, opening attachments or clicking embedded links. ***

I wouldn't expect message grouping to help given the information about your 
use-case that you've provided so far. I outlined the reasons for this in my 
previous email. However, if you say that it's working for you then there's 
almost certainly something about your use-case that I don't understand.

Typically in request/response use-cases the requester specifies a "correlation 
ID" to allow it to keep track of the exchange of messages. Also, in cases with 
multiple brokers it's typical to cluster those brokers together so that 
messages can be shared transparently across the different brokers or perhaps to 
use connection routers [1] to "shard" applications to particular brokers or 
sets of brokers so that the applications can get reconnected back to the proper 
broker(s) in case they get disconnected assuming that the brokers contain 
application-related state information. I assume you're using something like 
this already, but if not it may be worth looking into.


Justin

[1] 
https://activemq.apache.org/components/artemis/documentation/latest/connection-routers.html#connection-routers


Justin

On Fri, Dec 8, 2023 at 5:26 PM John Lilley 
<john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>>
 wrote:
Hi Justin,

Thanks for the response!

Our use case is something like this.  Suppose we have a nice RPC client wrapped 
around message queues and their services.  Some methods take a long time, and 
we want to report status along the way, so we’ve arranged for such methods to 
return a “task ID”:

String taskID = client.startLongThing();
while (!client.isLongThingFinished(taskID)) { wait, or print progress… }
client.getLongThingResult(taskID);

The problem we hit is that the long-running tasks are not stateless.  They 
start, execute and finish within the same server process.  So the calls to 
startLongThing(), isLongThingFinished(), and getLongThingResult() must all be 
serviced by the same process.  If we have multiple HA instances of a service 
running in a Kubernetes cluster, the isLongThingFinished() requests will be 
handle by a random server that may not know about the task.

Do you think that the “message group” attribute is a reasonable approach?  In 
JMS:
message.setStringProperty("JMSXGroupID", "grouplabel");

I’ve implemented and tested this and it does seem to work.  Do you know of any 
caveats or cautions?

Thanks
John




[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,MiJvWl9nWaaSsiJBFyc7CGAQMRDxVHUdDSfPqakKAOm_SablTwk4IQKKszbYlDIh5j4aSqYxuECDLsDRtkQVkoSrqFbw7DnV0Oi9x8uMfXhoDTfu1yRjKFO_DpM,&typo=1>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

34 Washington Street, Suite 205 Wellesley Hills, MA 02481

M: +1 7209385761<tel:+1%207209385761> | 
john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>
From: Justin Bertram <jbert...@apache.org<mailto:jbert...@apache.org>>
Sent: Thursday, December 7, 2023 10:56 PM
To: users@activemq.apache.org<mailto:users@activemq.apache.org>
Subject: Re: Is it possible to route message to a specific consumer?

*** [Caution] This email is from an external source. Please use caution 
responding, opening attachments or clicking embedded links. ***

> Is there any kind of “consumer affinity” that can be attached to a message, 
> which directs the broker to select a specific consumer?

I may be forgetting something, but I believe the answer to this question is no. 
The broker is broadly designed to support decoupled, asynchronous messaging 
where consumers and producers know essentially nothing about each other and the 
broker just receives messages and makes them available for consumption. Message 
consumption is fundamentally consumer-driven. The broker itself doesn't choose 
where the messages go except in some limited sense in particular use-cases. 
Even in the case of message grouping, the messages are not directed to a 
specific consumer as much as they are just directed at _just one consumer (i.e. 
any consumer) at a time_ since the goal is just for the messages to be 
processed in order.

You can add specific properties to specific messages and then JMS consumers can 
use selectors to either ignore or choose those messages for consumption and in 
this way you can ensure that certain consumers get certain messages, but both 
the producer and consumer need to know about the properties. Furthermore, if 
all the applications use a single queue then they can start interfering with 
each other. For example, if one application fails to consume its messages then 
messages will begin accumulating and may prevent other applications from 
getting their messages since queues are by nature first-in-first-out data 
structures.

I'd need to know more about the specifics of your use-case before I could make 
further recommendations.


Justin

On Thu, Dec 7, 2023 at 1:26 PM John Lilley 
<john.lil...@redpointglobal.com.invalid<mailto:john.lil...@redpointglobal.com.invalid>>
 wrote:
Oooh, maybe I answered my own question:
https://activemq.apache.org/components/artemis/documentation/1.0.0/message-grouping.html
This looks like what I’m after.  Any advice around its use?

John




[rg]<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,RLCz7uLdUVWU-4PdO0mzCFR75gph7wgFAJpvnoG5yg55RbBDZxZfAUVZs-ERB9VGZHVUZK6w6h0r3FI5bjkHA50Th_z9pS5huRB5AZIUhxN5nTIQlTbVa2sLLA,,&typo=1>

John Lilley

Data Management Chief Architect, Redpoint Global Inc.

34 Washington Street, Suite 205 Wellesley Hills, MA 02481

M: +1 7209385761<tel:+1%207209385761> | 
john.lil...@redpointglobal.com<mailto:john.lil...@redpointglobal.com>
From: John Lilley 
<john.lil...@redpointglobal.com.INVALID<mailto:john.lil...@redpointglobal.com.INVALID>>
Sent: Thursday, December 7, 2023 11:58 AM
To: users@activemq.apache.org<mailto:users@activemq.apache.org>
Subject: Is it possible to route message to a specific consumer?

*** [Caution] This email is from an external source. Please use caution 
responding, opening attachments or clicking embedded links. ***

Greetings,

We are in the process of transforming our Java-based services into 
Kubernetes-based microservice swarms with HA topology.  So every service will 
have multiple instances.  The good news is, AMQ supports this splendidly, 
because the message broker is very good about load-balancing between consumers 
and handling failed consumers.

However, we have a few cases of “long running tasks” that a single service 
instance is responsible for.  In those cases, we need to be able to route a 
request to a specific consumer instance.  We could use specially-named queues 
for each consumer instance, but this seems awkward and hopefully not necessary.

Is there any kind of “consumer affinity” that can be attached to a message, 
which directs the broker to select a specific consumer?
If not, other ideas are certainly welcome.

We are using OpenJDK 17

This is our maven dependency
<dependency>
    <groupId>org.apache.activemq</groupId>
    <artifactId>artemis-jms-client-all</artifactId>
    <version>2.30.0</version>
</dependency>


Thanks
John



John Lilley 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

Data Management Chief Architect, Redpoint Global Inc. 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

34 Washington Street, Suite 205 Wellesley Hills, MA 02481 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

M: +1 7209385761 | john.lil...@redpointglobal.com 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential 
and is intended solely for the use of the individual(s) to whom it is 
addressed. If you believe you received this e-mail in error, please notify the 
sender immediately, delete the e-mail from your computer and do not copy, print 
or disclose it to anyone else. If you properly received this e-mail as a 
customer, partner or vendor of Redpoint, you should maintain its contents in 
confidence subject to the terms and conditions of your agreement(s) with 
Redpoint. 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential 
and is intended solely for the use of the individual(s) to whom it is 
addressed. If you believe you received this e-mail in error, please notify the 
sender immediately, delete the e-mail from your computer and do not copy, print 
or disclose it to anyone else. If you properly received this e-mail as a 
customer, partner or vendor of Redpoint, you should maintain its contents in 
confidence subject to the terms and conditions of your agreement(s) with 
Redpoint. 
<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fwww.redpointglobal.com%2f&c=E,1,PCllWqn1FN806xiChO-uNijPVawTv9Y3krF28r_sBUJsVHuM4SlcKCNVMTp_Qq94AnDC39XAs-pcWHhVnAPvZZRJbL8kXpIdP0TQ85wOYAroxX0px-0h_pKn&typo=1>

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential 
and is intended solely for the use of the individual(s) to whom it is 
addressed. If you believe you received this e-mail in error, please notify the 
sender immediately, delete the e-mail from your computer and do not copy, print 
or disclose it to anyone else. If you properly received this e-mail as a 
customer, partner or vendor of Redpoint, you should maintain its contents in 
confidence subject to the terms and conditions of your agreement(s) with 
Redpoint.

PLEASE NOTE: This e-mail from Redpoint Global Inc. (“Redpoint”) is confidential 
and is intended solely for the use of the individual(s) to whom it is 
addressed. If you believe you received this e-mail in error, please notify the 
sender immediately, delete the e-mail from your computer and do not copy, print 
or disclose it to anyone else. If you properly received this e-mail as a 
customer, partner or vendor of Redpoint, you should maintain its contents in 
confidence subject to the terms and conditions of your agreement(s) with 
Redpoint.

Reply via email to