Thanks all, it helps I saw this https://issues.apache.org/jira/browse/QPID-5165 while running the "test" but did not get the whole picture.
Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the mapping group-id / consumers looks tricky (at least to me)...especially if you consider things like elasticity, network failures. no? On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <[email protected]> wrote: > Thanks! > > ________________________________ > From: Ken Giusti <[email protected]> > Sent: Friday, September 29, 2017 3:54:20 PM > To: users; [email protected] > Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 > > At the moment the qpid dispatch router ignores all group related > message properties when computing the route for a destination. > > I've opened a corresponding feature request against the router to > track support for message groups: > > https://issues.apache.org/jira/browse/DISPATCH-843 > > On Fri, Sep 29, 2017 at 9:23 AM, Keith W <[email protected]> wrote: > > A JIRA has been raised for this problem: > > > > https://issues.apache.org/jira/browse/QPID-7937 > > > > On 29 September 2017 at 11:47, Rob Godfrey <[email protected]> > wrote: > >> On 29 September 2017 at 10:35, Adel Boutros <[email protected]> > wrote: > >> > >>> Hello Rob, > >>> > >>> > >>> I would like to give my opinion on this. > >>> > >>> > >>> In our current use cases, we are configuring the brokers dynamically > using > >>> the REST API. We would like to have this possibility in the use case of > >>> Olivier. > >> > >> > >>> Also, having to restart the virtual host is damaging the High > Availability > >>> of the messaging system. This is because while the Virtual Host is > >>> restarting, no queues are available and the messages are inaccessible. > So I > >>> was wondering if restarting the virtual is a must or it could be fixed > in a > >>> Jira story? > >>> > >>> > >> > >> Sorry - my wording wasn't very clear. If you set this when you first > >> create the queue then you don't need to restart the vhost... however if > you > >> have an existing queue that you want to change, then the effect won't be > >> seen until the vhost is restarted (basically the queue sets itself up > to be > >> "group aware" or "not group aware" when it starts up, it can't change > while > >> it is running). I don't think this is really an issue for you in that > when > >> you create your queues with the REST API you just need to set this > >> attribute. > >> > >> > >>> > >>> Regarding the 2nd point of multiple brokers and dispatch router, if all > >>> brokers have the same queues configured with the appropriate grouping > >>> config, would that really break the feature? > >>> > >>> In our current use cases, all components have the same configuration > all > >>> the time (All brokers have same queues and same for the dispatch > router). > >>> > >>> So I would imagine if a consumer wants to consume a message with the > >>> correct group, the dispatch router will propagate this header to any > >>> available broker and will only get the corresponding messages. > >>> > >>> > >>> > >> The way grouping works is that the first time a message with a > particular > >> group id comes along, the broker will assign that group to a particular > >> consumer. Each broker will do this independently. So if you have two > >> brokers B1 and B2; and two consumers C1 and C2 and message M of group A > >> arrives on B1 whereas message N of group A arrives on B2; then B1 may > >> decide to associate group A with C1 whereas broker B2 decides to > associate > >> group A with consumer C2. So message groups with multiple brokers will > not > >> work behind a router *unless* the router is enhanced so that it is > aware of > >> message grouping (so all messages of the same group are sent to the same > >> broker) or the brokers somehow share information about the group -> > >> consumer assignment. > >> > >> -- Rob > >> > >> What do you think? > >>> > >>> > >>> Regards, > >>> > >>> Adel > >>> > >>> ________________________________ > >>> From: Rob Godfrey <[email protected]> > >>> Sent: Friday, September 29, 2017 9:14:52 AM > >>> To: [email protected] > >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 > >>> > >>> On 29 September 2017 at 01:09, Olivier Mallassi < > >>> [email protected]> > >>> wrote: > >>> > >>> > Gentleman, > >>> > > >>> > I will need your help. > >>> > > >>> > I have a use case where I would like to guarantee "consumer > affinity", > >>> > which is usually implemented using the JMSXGroupID (actually, I am > sure > >>> it > >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0) > >>> > > >>> > My test does the "classical" case: > >>> > for (i ... i < 100.) > >>> > Message textMsg = new TextM.... > >>> > textMsg.setStringProperty("JMSXGroupID", "groupA") > >>> > MessageProducer.send(textMsg); > >>> > > >>> > and I start two consumers (assuming only one would work). > >>> > > >>> > What I observe is that both consumers are receiving messages, acting > as > >>> > competing consumers. I also tried added the qpid.group_header_key > >>> property > >>> > to the queue but it does not change anything. > >>> > > >>> > > >>> > My setup is > >>> > - java broker 6.0.4 & 6.1.4 > >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0) > >>> > > >>> > Is this the expected behaviour? Any ideas? > >>> > > >>> > > >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message > >>> Grouping functionality was originally built around the AMQP 0-x > protocols > >>> and is looking for a message group in the application headers of the > >>> message. In AMQP 1.0 there is a dedicated property for this and the > JMS > >>> client is (correctly from an AMQP 1.0 point of view) placing the group > id > >>> there. > >>> > >>> As a work around instead of using JMSXGroupID you could use any other > (non > >>> JMSX prefixed) header, e.g.: > >>> > >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA") > >>> > >>> Note that on the broker you need to configure the queue so it knows > which > >>> header to use; this is stored in the messageGroupKey property of the > queue > >>> (to "qpidBugWorkaround" in this case) . Note that after setting this > >>> property you will need to restart the virtual host before the change > takes > >>> effect. > >>> > >>> > >>> > > >>> > Complementary question: let's assume now, that there is the DR > between > >>> the > >>> > broker & the consumers. Does the DR "propagate" or support "grouping > >>> > messages"? (I assume it is depending if you route the link or > messages) > >>> > > >>> > >>> With a single broker and link routing, yes grouping messages should > >>> continue to work. > >>> > >>> If there are multiple brokers sharding the queue then it would not > work - > >>> to support this the router would need to be changed to recognize > grouping > >>> and send all messages with the same group id to the same broker > waypoint > >>> (also, in this case, you would need to use message routing for inbound > >>> messages and link routing for consuming messages). We have talked > about > >>> support for behaviour like this before, but there is nothing > implemented > >>> yet as far as I know. > >>> > >>> -- Rob > >>> > >>> > >>> > > >>> > > >>> > thanks for your help > >>> > > >>> > Regards. > >>> > > >>> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > -- > -K > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
