Hi Karan, Only one master can be elected leader in the current architecture. It's unlikely we're at a point where we need to balance work across masters to push scalability further. That comes with a lot of complexity, and we still have a lot of room for performance improvements on a single leader architecture.
There are successful clusters sized beyond 35k machines as well, so performance generally depend on the characteristics of the workloads. I believe folks running clusters this large generally use a high core count server (e.g. 24 cores), as to whether this is necessary is not clear to me and again depends on the workloads, but certainly the master will continue to improve its ability to leverage more cores for better performance. In terms of the performance issues you've experienced, if you could provide some data (e.g. flamegraphs of a backlogged master) that would help us start to get a better sense of what's happening. Ben On Thu, Mar 22, 2018 at 2:26 PM, Karan Pradhan <[email protected]> wrote: > Hi All, > > I had the following questions: > 1. > I was wondering if it is possible to have multiple Mesos masters as > elected masters in a Mesos cluster so that the load can be balanced amongst > the masters. Is there a way to achieve this? > In general, can there be a load balancer for the Mesos masters? > > 2. > I have seen spikes in the Mesos event queues while running spark SQL > workloads with multiple stages. So I was wondering what is a better way to > handle these scalability issues. I noticed that compute intensive machines > were able to deal with those workloads better. Is there a particular > hardware requirement or requirement for the number of masters for scaling a > Mesos cluster horizontally? After reading success stories which mention > that Mesos is deployed for ~10K machines, I was curious about the hardware > used and the number of masters in this case. > > It would be awesome if I could get some insight into these questions. > > Thanks, > Karan > >
