I wrote up some early end user docs about how this might function, I've not yet written a released Federation Broker but the docs might paint a better picture:
http://choria.io/docs/configuration/federations/ On Sun, Mar 12, 2017, at 10:20, R.I.Pienaar wrote: > Hello, > > Recently the topic of gigantic collectives came up and more or less as > usual its a painful experience. > > The canonical example is: > > > / --> MQ Cluster -> n x mco nodes > / > client -> MQ Cluster --> MQ Cluster -> n x mco nodes > \ > \ --> MQ Cluster -> n x mco nodes > > People built these with ActiveMQ/RabbitMQ and it does solve some of the > problems but not all, since the way subscriptions and duplications work > this becomes more and more expensive as it grows. More chatter between > MQ nodes and inevitably the config to create broadcast zones becomes > just crazy. Particularly the pressure on the persistant data stores in > ActiveMQ is a huge problem. > > After some chats online here and offline I decided to finally do > something we've kind of wanted for a long time and that is to create a > federation of collectives. > > Below the proposal that I have working and am busy implementing, I'd > love to hear what people with big or distant networks think about this. > I think it will greatly reduce the pain of using middleware in those > cases. Though I am only targeting Choria with this work. > > You would now be able to build many completely standalone networks, your > london, tokyo, us-east, etc collectives would be standalone and they > would run federation brokers to enable client comms. You would only need > to worry about scaling middleware to their size and not the entire > combined size. > > client -> > MQ Cluster -> > Federation Broker Cluster -> > MQ Cluster -> > Regional MCollective > > > A client can be configured to speak to many Federation Brokers and a > Federation Broker is a daemon that is entirely stateless and you can run > as many as you like, they scale up automatically and share the message > work load. You don't need anything like etcd or consul, there's no > shared state and it stores no state on disk or memory. Ideal for HA but > for cases where many people use MCollective concurrently. > > So a deployment might be: > > Federation Network: > > Clients connect to a 3 x NATS cluster > > London Network: > > 5 x NATS cluster > 5 x Federation Brokers (one per NATS box for simplicity, could be 2 > per etc) > > Tokyo Network: > > 5 x NATS cluster > 5 x Federation Brokers (one per NATS box for simplicity, could be 2 > per etc) > > etc > > Every Federation broker - 10 in this case, connects to the Federation > Network. On the Federation network is only 10 subscriptions against n > queues where n is the number of networks - 2 here. > > You configure: > > choria.federation.networks = london, tokyo > > and 'mco ping' sends 2 messages, 1 to each of the 'london' and 'tokyo' > network. Any 1 of the 5 x Federation Brokers pick up the message and > republish it. > > You can do `CHORIA_NETWORK=china mco ping` and that client automatically > becomes federation aware if it was not configured so or it switches to > only use the china network etc. > > Replies will be a stream back load shared across the 5 x Federation > Brokers in each region and these will republish the message into the > Federation Network back to the client. > > Your networks are standalone and can continue to talk to normal > mcollective clients run in those localities, they continue to be fully > operational traditional collectives. You'd need to share CA infra > though for the moment, I can avoid this requirement eventually. > > In the case of direct addressing to 10k nodes where the client would > previously have to send 10k messages the client would now send 50 * > networks messages and the clusters of federation brokers will do the > hard work - load balanced across them all - to produce the 10000 > messages. > > In my testing this actually speeds things up a fair bit since a lot of > the hard work of making the requests are now load shared across groups > of workers, its transparent and supports all mcollective features. > > -- > R.I.Pienaar / www.devco.net / @ripienaar > > -- > > --- > You received this message because you are subscribed to the Google Groups > "mcollective-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- R.I.Pienaar / www.devco.net / @ripienaar -- --- You received this message because you are subscribed to the Google Groups "mcollective-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
