RE: [Dispatch Router] multicast distribution
Yes we already thought of that as a fallback plan. I guess we will go for this approach. Thanks Gordon for your time. Olivier -Original Message- From: Gordon Sim Sent: mercredi 14 novembre 2018 13:15 To: users@qpid.apache.org Subject: Re: [Dispatch Router] multicast distribution On 14/11/18 11:59, VERMEULEN Olivier wrote: > No if the subscriber is still down the message should also be resent. > So you're right it's not simply the active subscribers, it would be all the > subscribers that were connected to the dispatch-router at some point and that > were not explicitly closed. > Actually it's not even limited to the dispatch-router that received the > message, we should also consider all the dispatch-routers inter-connected > with it. I think explicit management of the subscriptions might be a better way to approach the problem. For each subscriber, set up a unique subscription queue on each broker of relevance, and have a link from the broker sending messages out through an anycast address on the router network that matches the subscription identifier. Then the subscribe can simply attach a receiver on that identifier to the router network and that will trigger the flow of messages from all queues which can then be acknowledged end-to-end. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Dispatch Router] multicast distribution
On 14/11/18 11:59, VERMEULEN Olivier wrote: No if the subscriber is still down the message should also be resent. So you're right it's not simply the active subscribers, it would be all the subscribers that were connected to the dispatch-router at some point and that were not explicitly closed. Actually it's not even limited to the dispatch-router that received the message, we should also consider all the dispatch-routers inter-connected with it. I think explicit management of the subscriptions might be a better way to approach the problem. For each subscriber, set up a unique subscription queue on each broker of relevance, and have a link from the broker sending messages out through an anycast address on the router network that matches the subscription identifier. Then the subscribe can simply attach a receiver on that identifier to the router network and that will trigger the flow of messages from all queues which can then be acknowledged end-to-end. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
No if the subscriber is still down the message should also be resent. So you're right it's not simply the active subscribers, it would be all the subscribers that were connected to the dispatch-router at some point and that were not explicitly closed. Actually it's not even limited to the dispatch-router that received the message, we should also consider all the dispatch-routers inter-connected with it. Olivier -Original Message- From: Gordon Sim Sent: mardi 13 novembre 2018 15:33 To: users@qpid.apache.org Subject: Re: [Dispatch Router] multicast distribution On 13/11/18 13:58, VERMEULEN Olivier wrote: > I guess yes, all the active subscribers when the router receives the message. > If the message is not successfully acked by all of them, then yes the message > is requeued and sent again to all active subscribers. > If one of the subscribers disconnects in the middle of the process I would > say we are back to the previous case and the message is resent to all active > subscribers. What if another message is sent while the subscriber above remains disconnected? Does it just miss getting the message? If so, why is it so important that it didn't miss the first one? If not, that means that the set of subscribers to deliver to is not simply those that are active. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Dispatch Router] multicast distribution
On 13/11/18 13:58, VERMEULEN Olivier wrote: I guess yes, all the active subscribers when the router receives the message. If the message is not successfully acked by all of them, then yes the message is requeued and sent again to all active subscribers. If one of the subscribers disconnects in the middle of the process I would say we are back to the previous case and the message is resent to all active subscribers. What if another message is sent while the subscriber above remains disconnected? Does it just miss getting the message? If so, why is it so important that it didn't miss the first one? If not, that means that the set of subscribers to deliver to is not simply those that are active. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [Dispatch Router] multicast distribution
This sounds like an architecturally wrong fit for a dispatch router and like a better fit for a message broker. The router has no place to store messages and it is not an endpoint capable of taking ownership of messages. See docs/books/user-guide/theory_of_operation.adoc, section Message Routing/ Routing Patterns. Acknowledged deliveries are performed only with Anycast distribution and Multicast deliveries are unreliable. - Original Message - > From: "VERMEULEN Olivier" > To: users@qpid.apache.org > Sent: Tuesday, November 13, 2018 8:58:55 AM > Subject: RE: [Dispatch Router] multicast distribution > > I guess yes, all the active subscribers when the router receives the message. > If the message is not successfully acked by all of them, then yes the message > is requeued and sent again to all active subscribers. > If one of the subscribers disconnects in the middle of the process I would > say we are back to the previous case and the message is resent to all active > subscribers. > We don't really mind receiving the message twice in some corner cases. > > Olivier > > -Original Message- > From: Gordon Sim > Sent: lundi 12 novembre 2018 21:46 > To: users@qpid.apache.org > Subject: Re: [Dispatch Router] multicast distribution > > On 12/11/18 19:32, VERMEULEN Olivier wrote: > > when all receivers acknowledge it > > How is the set of 'all' receivers defined? All those active at the time the > message is received by the first router? (What if one of them had been > disconnected just before that message?) > > If a message is not successfully acked, would it be requeued and replayed to > all subscribers? > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional > commands, e-mail: users-h...@qpid.apache.org > > *** This e-mail contains information for the > intended recipient only. It may contain proprietary material or confidential > information. If you are not the intended recipient you are not authorized to > distribute, copy or use this e-mail or any attachment to it. Murex cannot > guarantee that it is virus free and accepts no responsibility for any loss > or damage arising from its use. If you have received this e-mail in error > please notify immediately the sender and delete the original email received, > any attachments and all copies from your system. > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
I guess yes, all the active subscribers when the router receives the message. If the message is not successfully acked by all of them, then yes the message is requeued and sent again to all active subscribers. If one of the subscribers disconnects in the middle of the process I would say we are back to the previous case and the message is resent to all active subscribers. We don't really mind receiving the message twice in some corner cases. Olivier -Original Message- From: Gordon Sim Sent: lundi 12 novembre 2018 21:46 To: users@qpid.apache.org Subject: Re: [Dispatch Router] multicast distribution On 12/11/18 19:32, VERMEULEN Olivier wrote: > when all receivers acknowledge it How is the set of 'all' receivers defined? All those active at the time the message is received by the first router? (What if one of them had been disconnected just before that message?) If a message is not successfully acked, would it be requeued and replayed to all subscribers? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Dispatch Router] multicast distribution
On 12/11/18 19:32, VERMEULEN Olivier wrote: when all receivers acknowledge it How is the set of 'all' receivers defined? All those active at the time the message is received by the first router? (What if one of them had been disconnected just before that message?) If a message is not successfully acked, would it be requeued and replayed to all subscribers? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
when all receivers acknowledge it -Original Message- From: Gordon Sim Sent: lundi 12 novembre 2018 17:38 To: users@qpid.apache.org Subject: Re: [Dispatch Router] multicast distribution On 12/11/18 16:35, VERMEULEN Olivier wrote: > So this use case (client acknowledged multicast with message routing) will > never be supported ? What is you intention? That the message be dequeued when any receiver acknowledges it? Or only when all receivers acknowledge it? Or that each receiver independently acknowledges, and the message will be resent only those receivers that did not acknowledge? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system.
Re: [Dispatch Router] multicast distribution
On 12/11/18 16:35, VERMEULEN Olivier wrote: So this use case (client acknowledged multicast with message routing) will never be supported ? What is you intention? That the message be dequeued when any receiver acknowledges it? Or only when all receivers acknowledge it? Or that each receiver independently acknowledges, and the message will be resent only those receivers that did not acknowledge? - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
So this use case (client acknowledged multicast with message routing) will never be supported ? -Original Message- From: Gordon Sim Sent: lundi 12 novembre 2018 17:20 To: users@qpid.apache.org Subject: Re: [Dispatch Router] multicast distribution On 12/11/18 15:50, VERMEULEN Olivier wrote: > Would it also work if the queue is sharded across multiple brokers? > Will my consumers receive the messages stored in all the brokers for a given > queue? No, each link would be routed to one broker. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org *** This e-mail contains information for the intended recipient only. It may contain proprietary material or confidential information. If you are not the intended recipient you are not authorized to distribute, copy or use this e-mail or any attachment to it. Murex cannot guarantee that it is virus free and accepts no responsibility for any loss or damage arising from its use. If you have received this e-mail in error please notify immediately the sender and delete the original email received, any attachments and all copies from your system. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [Dispatch Router] multicast distribution
On 12/11/18 15:50, VERMEULEN Olivier wrote: Would it also work if the queue is sharded across multiple brokers? Will my consumers receive the messages stored in all the brokers for a given queue? No, each link would be routed to one broker. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
Would it also work if the queue is sharded across multiple brokers?
Will my consumers receive the messages stored in all the brokers for a given
queue?
-Original Message-
From: Gordon Sim
Sent: lundi 12 novembre 2018 16:09
To: users@qpid.apache.org
Subject: Re: [Dispatch Router] multicast distribution
On 12/11/18 14:58, VERMEULEN Olivier wrote:
> address {
> prefix: queue_1
> waypoint: yes
> distribution: multicast
> }
This will not give you end to end acknowledgement with consuming clients. The
router will acknowledge each message received by the queue on receipt, causing
the message to be dequeued.
One option is to use link routing for topics. That establishes a distinct
association between the broker and the consumer, which is needed for
acknowledged transfer.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional
commands, e-mail: users-h...@qpid.apache.org
*** This e-mail contains information for the
intended recipient only. It may contain proprietary material or confidential
information. If you are not the intended recipient you are not authorized to
distribute, copy or use this e-mail or any attachment to it. Murex cannot
guarantee that it is virus free and accepts no responsibility for any loss or
damage arising from its use. If you have received this e-mail in error please
notify immediately the sender and delete the original email received, any
attachments and all copies from your system.
Re: [Dispatch Router] multicast distribution
On 12/11/18 14:58, VERMEULEN Olivier wrote:
address {
prefix: queue_1
waypoint: yes
distribution: multicast
}
This will not give you end to end acknowledgement with consuming
clients. The router will acknowledge each message received by the queue
on receipt, causing the message to be dequeued.
One option is to use link routing for topics. That establishes a
distinct association between the broker and the consumer, which is
needed for acknowledged transfer.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
RE: [Dispatch Router] multicast distribution
I tried to reduce the test case as much as possible.
I'm sending a message with the following code:
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);
Destination destination = session.createTopic("topic_1");
MessageProducer producer = session.createProducer(destination);
TextMessage message = session.createTextMessage("Hello");
producer.send(message);
session.close();
And I'm receiving it with:
Session session = connection.createSession(false, Session.CLIENT_ACKNOWLEDGE);
Queue subscription = session.createQueue("queue_1");
MessageConsumer consumer = session.createConsumer(subscription);
TextMessage message = (TextMessage) consumer.receiveNoWait();
System.out.println("Message received: " + message.getText());
//message.acknowledge();
session.close();
Note that I commented out the acknowledgement.
So I was expecting the same result if I launch this code twice.
But actually the second time I'm receiving NULL.
Here is the dispatch-router config:
router {
id: router.dell440srv.10104
mode: interior
workerThreads: 4
defaultDistribution: unavailable
}
listener {
host : 0.0.0.0
port: 10104
role: normal
saslMechanisms: ANONYMOUS
requireSsl: no
authenticatePeer: no
}
log {
module: DEFAULT
enable: info+
includeSource: false
outputFile: log/qpid.log
}
connector {
name: broker.dell440srv.5673
host: dell440srv
port: 5673
role: route-container
}
address {
prefix: topic_1
waypoint: yes
distribution: balanced
}
address {
prefix: queue_1
waypoint: yes
distribution: multicast
}
autoLink {
name: broker.dell440srv.5673.topic_1
addr: topic_1
connection: broker.dell440srv.5673
direction: out
}
autoLink {
name: broker.dell440srv.5673.queue_1
addr: queue_1
connection: broker.dell440srv.5673
direction: in
}
-Original Message-
From: Gordon Sim
Sent: lundi 12 novembre 2018 14:16
To: users@qpid.apache.org
Subject: Re: [Dispatch Router] multicast distribution
On 12/11/18 13:07, VERMEULEN Olivier wrote:
> Hello,
>
> I'm using 1 dispatch-router (1.3.0), 2 brokers (7.0.3) and a simple
> topic/queue configuration.
> I was testing the multicast distribution on the dispatch-router with 1
> producer on the topic and 2 listeners on the queue.
> In AUTO_ACK everything seems to work fine but when I switch to CLIENT_ACK I
> have a strange behavior.
> If I don't acknowledge the message in my listeners I would expect the message
> to go back to the broker (which is what I observed with the balanced
> distribution) but instead the message is just gone...
> Do you know if there is a bug with the multicast distribution? Or did I
> misconfigure something?
I'm not clear on what your configuration is exactly, but multicast distribution
in the router is not end-to-end acknowledged (it can't be, as there may be
multiple receivers). The router that first handles the messages will
acknowledge it to the producer.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional
commands, e-mail: users-h...@qpid.apache.org
*** This e-mail contains information for the
intended recipient only. It may contain proprietary material or confidential
information. If you are not the intended recipient you are not authorized to
distribute, copy or use this e-mail or any attachment to it. Murex cannot
guarantee that it is virus free and accepts no responsibility for any loss or
damage arising from its use. If you have received this e-mail in error please
notify immediately the sender and delete the original email received, any
attachments and all copies from your system.
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
Re: [Dispatch Router] multicast distribution
On 12/11/18 13:07, VERMEULEN Olivier wrote: Hello, I'm using 1 dispatch-router (1.3.0), 2 brokers (7.0.3) and a simple topic/queue configuration. I was testing the multicast distribution on the dispatch-router with 1 producer on the topic and 2 listeners on the queue. In AUTO_ACK everything seems to work fine but when I switch to CLIENT_ACK I have a strange behavior. If I don't acknowledge the message in my listeners I would expect the message to go back to the broker (which is what I observed with the balanced distribution) but instead the message is just gone... Do you know if there is a bug with the multicast distribution? Or did I misconfigure something? I'm not clear on what your configuration is exactly, but multicast distribution in the router is not end-to-end acknowledged (it can't be, as there may be multiple receivers). The router that first handles the messages will acknowledge it to the producer. - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
