Hello Ted,
thanks for your reply. My description was not precise enough, so I'll
have another try:
1. We have one server with a broker. The server has a reachable IP
address from all clients.
2. We have X number of clients that come and go. However, any client
must register himself to the system using https and the server owner
will approve the registration before the client is allowed to use the
AMQP messaging infrastructure. Until system administrator says that this
client is not valid any longer, the client is consider a legal
addressing entity in our system.
3. Clients live on un-stable networks, so they come and go - change IP
address "randomly" and will usually be NAT'ed
4. Therefore, the initial connection to the server is always done by the
client
5. On the client, there is always a broker, whos purpose is to act as a
service for message delivery from devices on the LAN and apps on the
client itself. Additionally, it provides queues for messages to every
registered "app" on the client LAN, and an outgoing queue for messages
from these apps/devices to the server. (There are reasons for why the
apps do not use the server broker directly - but I cannot go into details.)
6. The server will always know the unique identity of every single
client, and can address messages to each and every client (planning to
use a header exchange for this on the server)
7. On the server, we plan to add a queue for every single client, since
the messaging pattern might be quite different for every single one of them.
8. In our head, up till here, this looks good and "simple". We have done
some prototyping, and it seems to work as expected. But then we get into
group messages, which we probably still do not have enough understanding
of. However, this was the case and idea:
9. From time to time, we need to send large updates to a "random" set of
clients. The server system will know their ID, and they need to receive
the message at one point, no matter if they are on-line or not when the
message is delivered to the server's broker.
10. One solution could be to "fan out" the messages to every client's
queue in the group on the server. However, the message needs to be
persistent and durable. It is then our understanding that the queued
messages would be stored on disk - as one copy for each client not
online at the moment of queue delivery. Since there could be a lot of
huge messages that might have a long ttl, we risk using too much storage
space in the long run. Our hope/plan was to create a queue for that
particular group of clients on the fly, which - we think - would make
sure that there will always be only one memory and disk copy of the
message until it is delivered to all clients.
11. However, to achieve that, we would need to tell every client's
exchange in that group that it would need to bind to this group queue
also "now". And since we are sure that all the clients in the group will
most likely not be online at the time of message delivery to the
server's broker, we need to find a way to tell the clients' brokers to
bind to it as soon as the client actually gets online. That should
probably be doable using QMF.
12. We also have an idea that the queue could know how many clients it
should deliver its messages to in some way, and how many it has
delivered (its messages) to, and that the queue can be deleted
(automatically or semi-automatically) when all deliveries to all clients
have taken place. That might not be the case, though.
I do not know it this made it clearer. I see that a drawing of the
system probably would have helped a lot on the understanding... :)
You suggestion below to "browse-only" and "ring policy" for it is
something we need to look further into. You say that "messages are only
removed from the queue when they expire TTL or when the queue reaches it
limits." However, we would also need to delete the message from the
queue (and the queue itself when emptied) when all its intended clients
have ack'ed that they have received the messages in that group's queue.
I was hoping that the queue or message itself would know that in some
way, but I presume we need to delete it manually from the server's group
queue in some when that happens? In any case, in the long run, that
should prevent the queue from reaching its limit - which we must try to
avoid.
Thanks for your time and support.
BR,
Tor Rune Skoglund, [email protected]
Den 23. mai 2014 21:12, skrev Ted Ross:
I'm not sure exactly what you mean by "exchanges" on clients, but I'll
throw something out anyway.
I'm assuming that you've got one broker (you call 'server A') in a
central location/public IP. You have a queue on this broker that holds
messages pertaining to a topic and you want those messages delivered to
each of a number of clients that may come and go from time to time.
In this case, I would recommend using a single queue, probably with a
ring policy to ensure it doesn't get too big. The queue should be set
to "browse-only" so no single consumer removes any messages. Messages
are only removed from the queue when they expire TTL or when the queue
reaches its size limit.
Now, when a new consumer subscribes to the queue, all the messages will
be delivered immediately and any new messages will be delivered as they
arrive. This will be true independently for each subscriber to the queue.
Now, if by 'exchanges' you mean that the 'clients' are actually brokers
in a federation relationship, this will work as well. In this case, the
queue subscribers aren't end clients but an exchange on the downstream
broker that can route the messages any way they see fit. This will
require queue-routes in the federation.
I hope this helps,
-Ted
On 05/23/2014 09:18 AM, Tor Rune Skoglund wrote:
Hello List,
we want to bind an outgoing queue on server A to a number of exchanges
on a number of clients devices. It is not guaranteed that every single
client is online when the binding happens. In that case, we want those
clients that actually are online to receive the messages in that queue
asap, and the rest of clients get their exchanges to bind automatically
when they get online (and then start receiving the messages in that queue).
Is this controllable from the server side, e.g. by using QMF to
configure the remove exchanges when the client device actually gets
online? Any pointers to how to do this?
The queue should auto-delete when _all_ those client devices have
received the messages (or when the message times out due to ttl).
The client devices might live "everywhere" on the net, might change IP
addresses from connection till next connection, and could be on LANs
(i.e. NAT'ed), so it is always the client that initiates the TCP
connection to the server.
Appreciate any input on how to solve this! :)
- Tor Rune Skoglund, [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
