On 30 September 2017 at 10:23, Olivier Mallassi <[email protected]> wrote:
> Thanks all, it helps > I saw this https://issues.apache.org/jira/browse/QPID-5165 while running > the "test" but did not get the whole picture. > > Indeed, w/ multiple brokers + multiple DR, ensuring the consistency of the > mapping group-id / consumers looks tricky (at least to me)...especially if > you consider things like elasticity, network failures. no? > Yes - if you have multiple DRs then there would need to be some sort of communication between them to share a common view of which groups go to which brokers and this communication would need to survive failures / split brain scenarios / etc. Another potential avenue would be to somehow have the brokers allocate groups to consumers in a deterministic manner; but this would only work if the number (and identity) of the consumers was fixed and known beforehand. -- Rob > > On Fri, Sep 29, 2017 at 4:10 PM, Adel Boutros <[email protected]> > wrote: > > > Thanks! > > > > ________________________________ > > From: Ken Giusti <[email protected]> > > Sent: Friday, September 29, 2017 3:54:20 PM > > To: users; [email protected] > > Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 > > > > At the moment the qpid dispatch router ignores all group related > > message properties when computing the route for a destination. > > > > I've opened a corresponding feature request against the router to > > track support for message groups: > > > > https://issues.apache.org/jira/browse/DISPATCH-843 > > > > On Fri, Sep 29, 2017 at 9:23 AM, Keith W <[email protected]> wrote: > > > A JIRA has been raised for this problem: > > > > > > https://issues.apache.org/jira/browse/QPID-7937 > > > > > > On 29 September 2017 at 11:47, Rob Godfrey <[email protected]> > > wrote: > > >> On 29 September 2017 at 10:35, Adel Boutros <[email protected]> > > wrote: > > >> > > >>> Hello Rob, > > >>> > > >>> > > >>> I would like to give my opinion on this. > > >>> > > >>> > > >>> In our current use cases, we are configuring the brokers dynamically > > using > > >>> the REST API. We would like to have this possibility in the use case > of > > >>> Olivier. > > >> > > >> > > >>> Also, having to restart the virtual host is damaging the High > > Availability > > >>> of the messaging system. This is because while the Virtual Host is > > >>> restarting, no queues are available and the messages are > inaccessible. > > So I > > >>> was wondering if restarting the virtual is a must or it could be > fixed > > in a > > >>> Jira story? > > >>> > > >>> > > >> > > >> Sorry - my wording wasn't very clear. If you set this when you first > > >> create the queue then you don't need to restart the vhost... however > if > > you > > >> have an existing queue that you want to change, then the effect won't > be > > >> seen until the vhost is restarted (basically the queue sets itself up > > to be > > >> "group aware" or "not group aware" when it starts up, it can't change > > while > > >> it is running). I don't think this is really an issue for you in that > > when > > >> you create your queues with the REST API you just need to set this > > >> attribute. > > >> > > >> > > >>> > > >>> Regarding the 2nd point of multiple brokers and dispatch router, if > all > > >>> brokers have the same queues configured with the appropriate grouping > > >>> config, would that really break the feature? > > >>> > > >>> In our current use cases, all components have the same configuration > > all > > >>> the time (All brokers have same queues and same for the dispatch > > router). > > >>> > > >>> So I would imagine if a consumer wants to consume a message with the > > >>> correct group, the dispatch router will propagate this header to any > > >>> available broker and will only get the corresponding messages. > > >>> > > >>> > > >>> > > >> The way grouping works is that the first time a message with a > > particular > > >> group id comes along, the broker will assign that group to a > particular > > >> consumer. Each broker will do this independently. So if you have two > > >> brokers B1 and B2; and two consumers C1 and C2 and message M of > group A > > >> arrives on B1 whereas message N of group A arrives on B2; then B1 may > > >> decide to associate group A with C1 whereas broker B2 decides to > > associate > > >> group A with consumer C2. So message groups with multiple brokers > will > > not > > >> work behind a router *unless* the router is enhanced so that it is > > aware of > > >> message grouping (so all messages of the same group are sent to the > same > > >> broker) or the brokers somehow share information about the group -> > > >> consumer assignment. > > >> > > >> -- Rob > > >> > > >> What do you think? > > >>> > > >>> > > >>> Regards, > > >>> > > >>> Adel > > >>> > > >>> ________________________________ > > >>> From: Rob Godfrey <[email protected]> > > >>> Sent: Friday, September 29, 2017 9:14:52 AM > > >>> To: [email protected] > > >>> Subject: Re: Consumer affinity (JMSXGroupID), AMQP 1.0 > > >>> > > >>> On 29 September 2017 at 01:09, Olivier Mallassi < > > >>> [email protected]> > > >>> wrote: > > >>> > > >>> > Gentleman, > > >>> > > > >>> > I will need your help. > > >>> > > > >>> > I have a use case where I would like to guarantee "consumer > > affinity", > > >>> > which is usually implemented using the JMSXGroupID (actually, I am > > sure > > >>> it > > >>> > was working w/ AMQP 0.10 clients but here I am using AMQP 1.0) > > >>> > > > >>> > My test does the "classical" case: > > >>> > for (i ... i < 100.) > > >>> > Message textMsg = new TextM.... > > >>> > textMsg.setStringProperty("JMSXGroupID", "groupA") > > >>> > MessageProducer.send(textMsg); > > >>> > > > >>> > and I start two consumers (assuming only one would work). > > >>> > > > >>> > What I observe is that both consumers are receiving messages, > acting > > as > > >>> > competing consumers. I also tried added the qpid.group_header_key > > >>> property > > >>> > to the queue but it does not change anything. > > >>> > > > >>> > > > >>> > My setup is > > >>> > - java broker 6.0.4 & 6.1.4 > > >>> > - java clients using qpid-jms 0.25.0 (AMQP 1.0) > > >>> > > > >>> > Is this the expected behaviour? Any ideas? > > >>> > > > >>> > > > >>> This is a bug in the AMQP 1.0 behaviour of the broker. The Message > > >>> Grouping functionality was originally built around the AMQP 0-x > > protocols > > >>> and is looking for a message group in the application headers of the > > >>> message. In AMQP 1.0 there is a dedicated property for this and the > > JMS > > >>> client is (correctly from an AMQP 1.0 point of view) placing the > group > > id > > >>> there. > > >>> > > >>> As a work around instead of using JMSXGroupID you could use any other > > (non > > >>> JMSX prefixed) header, e.g.: > > >>> > > >>> textMsg.setStringProperty("qpidBugWorkaround", "groupA") > > >>> > > >>> Note that on the broker you need to configure the queue so it knows > > which > > >>> header to use; this is stored in the messageGroupKey property of the > > queue > > >>> (to "qpidBugWorkaround" in this case) . Note that after setting this > > >>> property you will need to restart the virtual host before the change > > takes > > >>> effect. > > >>> > > >>> > > >>> > > > >>> > Complementary question: let's assume now, that there is the DR > > between > > >>> the > > >>> > broker & the consumers. Does the DR "propagate" or support > "grouping > > >>> > messages"? (I assume it is depending if you route the link or > > messages) > > >>> > > > >>> > > >>> With a single broker and link routing, yes grouping messages should > > >>> continue to work. > > >>> > > >>> If there are multiple brokers sharding the queue then it would not > > work - > > >>> to support this the router would need to be changed to recognize > > grouping > > >>> and send all messages with the same group id to the same broker > > waypoint > > >>> (also, in this case, you would need to use message routing for > inbound > > >>> messages and link routing for consuming messages). We have talked > > about > > >>> support for behaviour like this before, but there is nothing > > implemented > > >>> yet as far as I know. > > >>> > > >>> -- Rob > > >>> > > >>> > > >>> > > > >>> > > > >>> > thanks for your help > > >>> > > > >>> > Regards. > > >>> > > > >>> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > -- > > -K > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > >
