I have some questions about resource accounting in Mesos.
In the [mesos
architecture](http://mesos.apache.org/documentation/latest/mesos-architecture/)
and the [2010 white
paper](https://www.cs.berkeley.edu/~alig/papers/mesos.pdf), the
following simple example is given:
* A node declares it has 4 CPUs and 4GB RAM
* A resource offer is made to a framework
* The framework scheduler accepts (2 CPU, 1GB) for one task and (1 CPU,
2GB) for another tasks
* Mesos then is able to offer the remaining 1 CPU and 1GB of RAM to
another framework.
My question is, is this the total resource accounting which Mesos does,
or is it able to take account of the *measured* resource utilisation on
a particular node at the current point in time?
Here are some examples of what I'm thinking of.
(1) A node has 12 CPUs. If tasks each declare a usage of 1 CPU, will
Mesos only run 12 of them?
Let's say some of those tasks only use 50% of a CPU on average (because
part of the time they are waiting on network or disk I/O); some tasks
use 0% CPU most of the time (because they are interactive and are
waiting for incoming queries)
Clearly, if the remaining tasks are multi-threaded they will be able to
take up the spare CPU capacity. But if those 12 tasks were all
single-threaded processes, it seems to me that the CPU would be
underutilised.
A partial solution might be if each task could request a fraction of a CPU.
(2) A task requests 2GB of RAM, but when running it actually uses only
1GB of RAM. This means there is spare RAM, at least at the current point
in time, bearing in mind that the task could take up its allocation
whenever it choses.
Conversely: a task requests 2GB of RAM usage, but actually uses 3GB
(assuming there is no hard cgroups enforcement to prevent it doing so).
Third case: a sysadmin manually runs some process on the node which uses
some RAM, but the process was not started via Mesos.
Would any of these scenarios result in the Mesos allocator making larger
or smaller resource offers, or would it always calculate RAM available =
(initial system RAM) - (total of accepted resource offers) ?
(3) Can the executor set execution priorities for different tasks, so
that if the CPU is overcommitted, "real time" tasks can take precedence
over "batch" tasks?
Where I'm coming from is trying to arrange for a single cluster which
runs some real time jobs, and all the remaining available resources are
soaked up running background batch jobs.
The real-time jobs may be bursty interactive workloads - when not being
queried they are idle and the resources, but when the real-time jobs are
being queried they should not be slowed down significantly by the batch
jobs. The background jobs are mixed CPU and I/O, so to achieve full CPU
utilisation you need to run more of them than the number of CPU cores
you have.
At the moment I'm using HTCondor but this only really supports batch
workloads where each task has a defined lifetime; I can't run the
long-running, real-time query tasks under the same framework. To get
full CPU utilisation I have to get Condor to lie about the number of CPU
cores are on each node, pretending there are more than there really are.
Therefore I'm wondering whether Mesos would suit this better, maybe
using one framework for the persistent real-time workflows and one for
the batch workloads, but I really want to understand better how the
resources would be divided up between the frameworks and how to achieve
both real-time operation and good utilisation of remaining resources.
Also, with HTCondor I have to take a lot of care to calculate (guess)
up-front the amount of RAM each task will take, and that is difficult,
as it depends on the algorithms used and the size of the incoming data
sets. Set the estimate too low, and processes can be killed by the OOM
killer due to RAM exhaustion. Set the estimate too high and the machine
is underutilised. It's made more complicated by the fact that the jobs
use mmap() on large shared databases, so running multiple instances of
the same task doesn't use N times as much memory as one task.
I'm guessing that Mesos isn't going to free me from having to
pre-calculate resource requirements for tasks, but if it were able to
respond to *actual* RAM usage on the system, allow for some
overcommitment, and terminate lower-priority tasks if RAM is exhausted,
that would be extremely helpful.
Thanks,
Brian Candler.