Hey Manohar, Sounds like this could be simplified by using Kafka as an event bus. Kafka is horizontally scalable and the Sarama golang library is excellent.
Regards, Ryan On Wed, Apr 26, 2017 at 11:33 AM, Manohar Kumar <forwarding...@gmail.com> wrote: > Hello, > > I am working on building a distributed system where a bunch of workers > (can scale to few thousands) are connected to a manager. Workers send > certain local events to the manager and the manager then should broadcast > that event to all other workers. Workers shouldn't miss any event (for > reasons like network issue, or they are rebooting etc.). For now the > master high availability is not a serious requirement. But would prefer a > design that doesn't over complicate things when master HA has to be added. > master and slaves are fully go code and unlikely to change. > > options I am considering are.. > > 1. workers connect to the master over TCP. Master writes the event to a > consistent store like etcd or consul. All the workers would setup a watch > on the store and go read it themselves. I don't know how well the watch > approach will scale as the number of workers increase. > > 2. workers connect to the master over TCP. Master uses a gossip mechanism > to send the event (for ex: using serf). This has the drawback that some > clients can receive the event much later others. But will scale better than > approach 1. This approach needs some form of reconciliation when a worker > reboots because it could have missed some events when it was down or > rebooting. > > 3. workers connect to the master over TCP. Master on getting an event from > worker 1 sends it over the TCP to all other workers. This needs lot of > 'reinventing the wheel' to guarantee that all workers get the event. > > Looking for suggestions for other approaches (message broker ? pubsub ?) > using open source go packages for this requirement. Thanks. > > Manohar. > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
