2015-06-17 17:33 GMT+08:00 Brian Candler <[email protected]>:

> I have some questions about resource accounting in Mesos.
>
> In the [mesos architecture](
> http://mesos.apache.org/documentation/latest/mesos-architecture/) and the
> [2010 white paper](https://www.cs.berkeley.edu/~alig/papers/mesos.pdf),
> the following simple example is given:
>
> * A node declares it has 4 CPUs and 4GB RAM
> * A resource offer is made to a framework
> * The framework scheduler accepts (2 CPU, 1GB) for one task and (1 CPU,
> 2GB) for another tasks
> * Mesos then is able to offer the remaining 1 CPU and 1GB of RAM to
> another framework.
>
> My question is, is this the total resource accounting which Mesos does, or
> is it able to take account of the *measured* resource utilisation on a
> particular node at the current point in time?
>
> Here are some examples of what I'm thinking of.
>
> (1) A node has 12 CPUs. If tasks each declare a usage of 1 CPU, will Mesos
> only run 12 of them?
>
> Let's say some of those tasks only use 50% of a CPU on average (because
> part of the time they are waiting on network or disk I/O); some tasks use
> 0% CPU most of the time (because they are interactive and are waiting for
> incoming queries)
>
> Clearly, if the remaining tasks are multi-threaded they will be able to
> take up the spare CPU capacity. But if those 12 tasks were all
> single-threaded processes, it seems to me that the CPU would be
> underutilised.
>
> A partial solution might be if each task could request a fraction of a CPU.
>
But CPU fraction is Time-Division. I don't think it's possible to request a
fraction.


>
> (2) A task requests 2GB of RAM, but when running it actually uses only 1GB
> of RAM. This means there is spare RAM, at least at the current point in
> time, bearing in mind that the task could take up its allocation whenever
> it choses.
>
> Conversely: a task requests 2GB of RAM usage, but actually uses 3GB
> (assuming there is no hard cgroups enforcement to prevent it doing so).
>
> Third case: a sysadmin manually runs some process on the node which uses
> some RAM, but the process was not started via Mesos.
>
> Would any of these scenarios result in the Mesos allocator making larger
> or smaller resource offers, or would it always calculate RAM available =
> (initial system RAM) - (total of accepted resource offers) ?
>
> (3) Can the executor set execution priorities for different tasks, so that
> if the CPU is overcommitted, "real time" tasks can take precedence over
> "batch" tasks?
>
> Where I'm coming from is trying to arrange for a single cluster which runs
> some real time jobs, and all the remaining available resources are soaked
> up running background batch jobs.
>
> The real-time jobs may be bursty interactive workloads - when not being
> queried they are idle and the resources, but when the real-time jobs are
> being queried they should not be slowed down significantly by the batch
> jobs. The background jobs are mixed CPU and I/O, so to achieve full CPU
> utilisation you need to run more of them than the number of CPU cores you
> have.
>
> At the moment I'm using HTCondor but this only really supports batch
> workloads where each task has a defined lifetime; I can't run the
> long-running, real-time query tasks under the same framework. To get full
> CPU utilisation I have to get Condor to lie about the number of CPU cores
> are on each node, pretending there are more than there really are.
>
> Therefore I'm wondering whether Mesos would suit this better, maybe using
> one framework for the persistent real-time workflows and one for the batch
> workloads, but I really want to understand better how the resources would
> be divided up between the frameworks and how to achieve both real-time
> operation and good utilisation of remaining resources.
>
> Also, with HTCondor I have to take a lot of care to calculate (guess)
> up-front the amount of RAM each task will take, and that is difficult, as
> it depends on the algorithms used and the size of the incoming data sets.
> Set the estimate too low, and processes can be killed by the OOM killer due
> to RAM exhaustion. Set the estimate too high and the machine is
> underutilised. It's made more complicated by the fact that the jobs use
> mmap() on large shared databases, so running multiple instances of the same
> task doesn't use N times as much memory as one task.
>
> I'm guessing that Mesos isn't going to free me from having to
> pre-calculate resource requirements for tasks, but if it were able to
> respond to *actual* RAM usage on the system, allow for some overcommitment,
> and terminate lower-priority tasks if RAM is exhausted, that would be
> extremely helpful.
>
> Thanks,
>
> Brian Candler.
>
>

Reply via email to