On 06/17/2015 07:01 AM, Alex Rukletsov wrote:
Brian,
these are very good and nicely written questions, let me try to ask them.
Mesos built-in allocator does the bookkeeping based on the declared
resource consumption and not actual (you call it "measured"). This means
the example in Apache docs and the white paper is correct.
However, there is an ongoing work for adding oversubscription to Mesos.
I'm not an expert in this area, maybe Niklas Nielsen chimes in and
corrects me later, but IMO it is planned to measure the actual resource
consumption on each Mesos agent node and notify Mesos master (including
allocator) about extra free but revokable resources. Here
<https://docs.google.com/document/d/1pUnElxHy1uWfHY_FOvvRC73QaOGgdXE0OXN-gbxdXA0/edit#heading=h.yvd9qbi4swb4>
is the design doc for this feature, some code has been already landed in
Mesos master branch, check, for example,
include/mesos/slave/resource_estimator.hpp.
Ah, some excellent reading.
As far as I know we do not have execution priorities for tasks. We plan
to takle this problem from a different direction: introduce quota (i.e.
cluster-wide resource reservations) for production frameworks, which
guarantees a certain amount of resources can be used by the framework at
any time, together with introducing oversubscription for quota
resources, that are currently unused by the framework. Another effort
that aims to increase cluster utilization is optimistic offers, which
means offering same resources to multiple frameworks at the same time.
Please be advised, that both quota and optimistic offers are in the
early design phase right now and will definitely not land in Mesos 0.23.
And yes, to increase CPU utilization, you may also lie to Mesos master
about how many CPUs your agent nodes have : ).
Wow, I'm not sure how to 'chime in' on this. So please excuse my
injection to this thread, if anyone finds it 'off_beat'. I guess I'm
going to have to sign up to the dev list for mesos.
As many of you may recall, during the master-slave debate and a few
postings since, I have basically proposed the 'autonomy_function'. With
the idea of basically having (2) categories of lowest level nodes (1):
slaves that are strictly governed from above by the master and workers
that have some degree of autonomy. One of the purposes for this is so
the lowest level (worker) nodes can be able to obtain additional
resources from the pool, dynamically and release them back to the pool,
dynamically; all based on code within the worker nodes. Slave-nodes
would not have this ability. A further extention is that the worker
nodes via the concept of 'autonomy' be allowed to declare when they are
anticipating being ready for more (additional/new) work and not the
master they serve. Much Faster as the decisions are being made at
a lower (faster) level. Sure this involves the worker-nodes being aware
of their own resource utilization, in-situ. I think in the future, as
mesos matures, the master nodes will be involved in an ever expanding
role of oversight for security, bandwidwith, and planning/supervision of
the larger aspects of running a large, dynamic computation environment
that will continue to attract newer and more complex frameworks.
The "BUSS" where the CPU cycles, distributed ram and other low_level
resources exist, are better off (faster and more accurate) if we manage
them, at the "BUSS" level. All the way up the stack and back down again
for memory issues, is just not going to work. Particularly on big
science problems that are memory bounded. Read upon RDMA, its here now
and it's a game changer.
This 'top down' approach to memory and other lowlevel (hardware)
resource allocation ain't gonna work in the land of robust RDMA. Sure
it's only a gut feeling, but I'm working on this opportunity to organize
a proposal for consideration. This is but one example of what can be
achieved with the 'Autonomy_Function'; i.e. robust, dynamic resource
request/allocation, from the bottom up (worker-nodes).
You guys do know that RDMA is now in Ethernet, right? Infiniband
and other such interconnects will soon not have any advantage over
10+ gig E. It's a fundamental (quantum?) shift going on here with RDMA,
imo (nothing humble about it).
Hope this sheds some light on the topic.
I have much to read about what is planned so any additional resource
identifications are most welcome.
hth,
James
On Wed, Jun 17, 2015 at 11:47 AM, Brian Candler <[email protected]
<mailto:[email protected]>> wrote:
On 17/06/2015 10:33, Brian Candler wrote:
It's made more complicated by the fact that the jobs use mmap()
on large shared databases, so running multiple instances of the
same task doesn't use N times as much memory as one task.
Aside: combined with cgroups this gets hairy.
As I understand it, mmap() memory is charged to the first process
which touches it, and not to subsequent users of the same page. When
a process terminates, the charge gets passed to the parent cgroup.
Also: 3.2-vintage kernels have issues: even if only using cgroups
for accounting (no hard limits), it seems the OOM killer kicks in if
there are too many dirty pages waiting to be written to disk.
https://www-auth.cs.wisc.edu/lists/htcondor-users/2015-February/msg00087.shtml
https://www-auth.cs.wisc.edu/lists/htcondor-users/2015-February/msg00135.shtml
Just thought that might be of interest.