Hello,
Recently the topic of gigantic collectives came up and more or less as
usual its a painful experience.
The canonical example is:
/ --> MQ Cluster -> n x mco nodes
/
client -> MQ Cluster --> MQ Cluster -> n x mco nodes
\
\ --> MQ Cluster -> n x mco nodes
People built these with ActiveMQ/RabbitMQ and it does solve some of the
problems but not all, since the way subscriptions and duplications work
this becomes more and more expensive as it grows. More chatter between
MQ nodes and inevitably the config to create broadcast zones becomes
just crazy. Particularly the pressure on the persistant data stores in
ActiveMQ is a huge problem.
After some chats online here and offline I decided to finally do
something we've kind of wanted for a long time and that is to create a
federation of collectives.
Below the proposal that I have working and am busy implementing, I'd
love to hear what people with big or distant networks think about this.
I think it will greatly reduce the pain of using middleware in those
cases. Though I am only targeting Choria with this work.
You would now be able to build many completely standalone networks, your
london, tokyo, us-east, etc collectives would be standalone and they
would run federation brokers to enable client comms. You would only need
to worry about scaling middleware to their size and not the entire
combined size.
client ->
MQ Cluster ->
Federation Broker Cluster ->
MQ Cluster ->
Regional MCollective
A client can be configured to speak to many Federation Brokers and a
Federation Broker is a daemon that is entirely stateless and you can run
as many as you like, they scale up automatically and share the message
work load. You don't need anything like etcd or consul, there's no
shared state and it stores no state on disk or memory. Ideal for HA but
for cases where many people use MCollective concurrently.
So a deployment might be:
Federation Network:
Clients connect to a 3 x NATS cluster
London Network:
5 x NATS cluster
5 x Federation Brokers (one per NATS box for simplicity, could be 2
per etc)
Tokyo Network:
5 x NATS cluster
5 x Federation Brokers (one per NATS box for simplicity, could be 2
per etc)
etc
Every Federation broker - 10 in this case, connects to the Federation
Network. On the Federation network is only 10 subscriptions against n
queues where n is the number of networks - 2 here.
You configure:
choria.federation.networks = london, tokyo
and 'mco ping' sends 2 messages, 1 to each of the 'london' and 'tokyo'
network. Any 1 of the 5 x Federation Brokers pick up the message and
republish it.
You can do `CHORIA_NETWORK=china mco ping` and that client automatically
becomes federation aware if it was not configured so or it switches to
only use the china network etc.
Replies will be a stream back load shared across the 5 x Federation
Brokers in each region and these will republish the message into the
Federation Network back to the client.
Your networks are standalone and can continue to talk to normal
mcollective clients run in those localities, they continue to be fully
operational traditional collectives. You'd need to share CA infra
though for the moment, I can avoid this requirement eventually.
In the case of direct addressing to 10k nodes where the client would
previously have to send 10k messages the client would now send 50 *
networks messages and the clusters of federation brokers will do the
hard work - load balanced across them all - to produce the 10000
messages.
In my testing this actually speeds things up a fair bit since a lot of
the hard work of making the requests are now load shared across groups
of workers, its transparent and supports all mcollective features.
--
R.I.Pienaar / www.devco.net / @ripienaar
--
---
You received this message because you are subscribed to the Google Groups
"mcollective-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to mcollective-users+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
