2015-06-17 17:33 GMT+08:00 Brian Candler <[email protected]>: > I have some questions about resource accounting in Mesos. > > In the [mesos architecture]( > http://mesos.apache.org/documentation/latest/mesos-architecture/) and the > [2010 white paper](https://www.cs.berkeley.edu/~alig/papers/mesos.pdf), > the following simple example is given: > > * A node declares it has 4 CPUs and 4GB RAM > * A resource offer is made to a framework > * The framework scheduler accepts (2 CPU, 1GB) for one task and (1 CPU, > 2GB) for another tasks > * Mesos then is able to offer the remaining 1 CPU and 1GB of RAM to > another framework. > > My question is, is this the total resource accounting which Mesos does, or > is it able to take account of the *measured* resource utilisation on a > particular node at the current point in time? > > Here are some examples of what I'm thinking of. > > (1) A node has 12 CPUs. If tasks each declare a usage of 1 CPU, will Mesos > only run 12 of them? > > Let's say some of those tasks only use 50% of a CPU on average (because > part of the time they are waiting on network or disk I/O); some tasks use > 0% CPU most of the time (because they are interactive and are waiting for > incoming queries) > > Clearly, if the remaining tasks are multi-threaded they will be able to > take up the spare CPU capacity. But if those 12 tasks were all > single-threaded processes, it seems to me that the CPU would be > underutilised. > > A partial solution might be if each task could request a fraction of a CPU. > But CPU fraction is Time-Division. I don't think it's possible to request a fraction.
> > (2) A task requests 2GB of RAM, but when running it actually uses only 1GB > of RAM. This means there is spare RAM, at least at the current point in > time, bearing in mind that the task could take up its allocation whenever > it choses. > > Conversely: a task requests 2GB of RAM usage, but actually uses 3GB > (assuming there is no hard cgroups enforcement to prevent it doing so). > > Third case: a sysadmin manually runs some process on the node which uses > some RAM, but the process was not started via Mesos. > > Would any of these scenarios result in the Mesos allocator making larger > or smaller resource offers, or would it always calculate RAM available = > (initial system RAM) - (total of accepted resource offers) ? > > (3) Can the executor set execution priorities for different tasks, so that > if the CPU is overcommitted, "real time" tasks can take precedence over > "batch" tasks? > > Where I'm coming from is trying to arrange for a single cluster which runs > some real time jobs, and all the remaining available resources are soaked > up running background batch jobs. > > The real-time jobs may be bursty interactive workloads - when not being > queried they are idle and the resources, but when the real-time jobs are > being queried they should not be slowed down significantly by the batch > jobs. The background jobs are mixed CPU and I/O, so to achieve full CPU > utilisation you need to run more of them than the number of CPU cores you > have. > > At the moment I'm using HTCondor but this only really supports batch > workloads where each task has a defined lifetime; I can't run the > long-running, real-time query tasks under the same framework. To get full > CPU utilisation I have to get Condor to lie about the number of CPU cores > are on each node, pretending there are more than there really are. > > Therefore I'm wondering whether Mesos would suit this better, maybe using > one framework for the persistent real-time workflows and one for the batch > workloads, but I really want to understand better how the resources would > be divided up between the frameworks and how to achieve both real-time > operation and good utilisation of remaining resources. > > Also, with HTCondor I have to take a lot of care to calculate (guess) > up-front the amount of RAM each task will take, and that is difficult, as > it depends on the algorithms used and the size of the incoming data sets. > Set the estimate too low, and processes can be killed by the OOM killer due > to RAM exhaustion. Set the estimate too high and the machine is > underutilised. It's made more complicated by the fact that the jobs use > mmap() on large shared databases, so running multiple instances of the same > task doesn't use N times as much memory as one task. > > I'm guessing that Mesos isn't going to free me from having to > pre-calculate resource requirements for tasks, but if it were able to > respond to *actual* RAM usage on the system, allow for some overcommitment, > and terminate lower-priority tasks if RAM is exhausted, that would be > extremely helpful. > > Thanks, > > Brian Candler. > >

