On 30 September 2017 at 09:23, Olivier Mallassi <[email protected]> wrote: > Thanks all, it helps > I saw this https://issues.apache.org/jira/browse/QPID-5165 while running > the "test" but did not get the whole picture. >
The interesting bits here are all with the servers as already discusse, but just to be clear...the above JIRA isn't related to the current qpid-jms-client 0.25.0 release in any way, but rather a change for an older prototype client that was included in the Qpid 0.26 release some years ago. The current AMQP 1.0 Qpid JMS client JIRAs are at https://issues.apache.org/jira/browse/QPIDJMS > Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the > mapping group-id / consumers looks tricky (at least to me)...especially if > you consider things like elasticity, network failures. no? > > On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <[email protected]> wrote: > >> Thanks! >> >> ________________________________ >> From: Ken Giusti <[email protected]> >> Sent: Friday, September 29, 2017 3:54:20 PM >> To: users; [email protected] >> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 >> >> At the moment the qpid dispatch router ignores all group related >> message properties when computing the route for a destination. >> >> I've opened a corresponding feature request against the router to >> track support for message groups: >> >> https://issues.apache.org/jira/browse/DISPATCH-843 >> >> On Fri, Sep 29, 2017 at 9:23 AM, Keith W <[email protected]> wrote: >> > A JIRA has been raised for this problem: >> > >> > https://issues.apache.org/jira/browse/QPID-7937 >> > >> > On 29 September 2017 at 11:47, Rob Godfrey <[email protected]> >> wrote: >> >> On 29 September 2017 at 10:35, Adel Boutros <[email protected]> >> wrote: >> >> >> >>> Hello Rob, >> >>> >> >>> >> >>> I would like to give my opinion on this. >> >>> >> >>> >> >>> In our current use cases, we are configuring the brokers dynamically >> using >> >>> the REST API. We would like to have this possibility in the use case of >> >>> Olivier. >> >> >> >> >> >>> Also, having to restart the virtual host is damaging the High >> Availability >> >>> of the messaging system. This is because while the Virtual Host is >> >>> restarting, no queues are available and the messages are inaccessible. >> So I >> >>> was wondering if restarting the virtual is a must or it could be fixed >> in a >> >>> Jira story? >> >>> >> >>> >> >> >> >> Sorry - my wording wasn't very clear. If you set this when you first >> >> create the queue then you don't need to restart the vhost... however if >> you >> >> have an existing queue that you want to change, then the effect won't be >> >> seen until the vhost is restarted (basically the queue sets itself up >> to be >> >> "group aware" or "not group aware" when it starts up, it can't change >> while >> >> it is running). I don't think this is really an issue for you in that >> when >> >> you create your queues with the REST API you just need to set this >> >> attribute. >> >> >> >> >> >>> >> >>> Regarding the 2nd point of multiple brokers and dispatch router, if all >> >>> brokers have the same queues configured with the appropriate grouping >> >>> config, would that really break the feature? >> >>> >> >>> In our current use cases, all components have the same configuration >> all >> >>> the time (All brokers have same queues and same for the dispatch >> router). >> >>> >> >>> So I would imagine if a consumer wants to consume a message with the >> >>> correct group, the dispatch router will propagate this header to any >> >>> available broker and will only get the corresponding messages. >> >>> >> >>> >> >>> >> >> The way grouping works is that the first time a message with a >> particular >> >> group id comes along, the broker will assign that group to a particular >> >> consumer. Each broker will do this independently. So if you have two >> >> brokers B1 and B2; and two consumers C1 and C2 and message M of group A >> >> arrives on B1 whereas message N of group A arrives on B2; then B1 may >> >> decide to associate group A with C1 whereas broker B2 decides to >> associate >> >> group A with consumer C2. So message groups with multiple brokers will >> not >> >> work behind a router *unless* the router is enhanced so that it is >> aware of >> >> message grouping (so all messages of the same group are sent to the same >> >> broker) or the brokers somehow share information about the group -> >> >> consumer assignment. >> >> >> >> -- Rob >> >> >> >> What do you think? >> >>> >> >>> >> >>> Regards, >> >>> >> >>> Adel >> >>> >> >>> ________________________________ >> >>> From: Rob Godfrey <[email protected]> >> >>> Sent: Friday, September 29, 2017 9:14:52 AM >> >>> To: [email protected] >> >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 >> >>> >> >>> On 29 September 2017 at 01:09, Olivier Mallassi < >> >>> [email protected]> >> >>> wrote: >> >>> >> >>> > Gentleman, >> >>> > >> >>> > I will need your help. >> >>> > >> >>> > I have a use case where I would like to guarantee "consumer >> affinity", >> >>> > which is usually implemented using the JMSXGroupID (actually, I am >> sure >> >>> it >> >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0) >> >>> > >> >>> > My test does the "classical" case: >> >>> > for (i ... i < 100.) >> >>> > Message textMsg = new TextM.... >> >>> > textMsg.setStringProperty("JMSXGroupID", "groupA") >> >>> > MessageProducer.send(textMsg); >> >>> > >> >>> > and I start two consumers (assuming only one would work). >> >>> > >> >>> > What I observe is that both consumers are receiving messages, acting >> as >> >>> > competing consumers. I also tried added the qpid.group_header_key >> >>> property >> >>> > to the queue but it does not change anything. >> >>> > >> >>> > >> >>> > My setup is >> >>> > - java broker 6.0.4 & 6.1.4 >> >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0) >> >>> > >> >>> > Is this the expected behaviour? Any ideas? >> >>> > >> >>> > >> >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message >> >>> Grouping functionality was originally built around the AMQP 0-x >> protocols >> >>> and is looking for a message group in the application headers of the >> >>> message. In AMQP 1.0 there is a dedicated property for this and the >> JMS >> >>> client is (correctly from an AMQP 1.0 point of view) placing the group >> id >> >>> there. >> >>> >> >>> As a work around instead of using JMSXGroupID you could use any other >> (non >> >>> JMSX prefixed) header, e.g.: >> >>> >> >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA") >> >>> >> >>> Note that on the broker you need to configure the queue so it knows >> which >> >>> header to use; this is stored in the messageGroupKey property of the >> queue >> >>> (to "qpidBugWorkaround" in this case) . Note that after setting this >> >>> property you will need to restart the virtual host before the change >> takes >> >>> effect. >> >>> >> >>> >> >>> > >> >>> > Complementary question: let's assume now, that there is the DR >> between >> >>> the >> >>> > broker & the consumers. Does the DR "propagate" or support "grouping >> >>> > messages"? (I assume it is depending if you route the link or >> messages) >> >>> > >> >>> >> >>> With a single broker and link routing, yes grouping messages should >> >>> continue to work. >> >>> >> >>> If there are multiple brokers sharding the queue then it would not >> work - >> >>> to support this the router would need to be changed to recognize >> grouping >> >>> and send all messages with the same group id to the same broker >> waypoint >> >>> (also, in this case, you would need to use message routing for inbound >> >>> messages and link routing for consuming messages). We have talked >> about >> >>> support for behaviour like this before, but there is nothing >> implemented >> >>> yet as far as I know. >> >>> >> >>> -- Rob >> >>> >> >>> >> >>> > >> >>> > >> >>> > thanks for your help >> >>> > >> >>> > Regards. >> >>> > >> >>> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> > >> >> >> >> -- >> -K >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
