Hi, hopefully this helps to answers some of your resource questions: In general you can configure arbitrary resources when starting a slave see http://mesos.apache.org/documentation/latest/configuration/ and http://mesosphere.com/docs/reference/mesos-slave/ for details. So can you override the default, one other usecase could also be that you tie your resources to specific roles (i.e. two cpus are reserved for HDFS framework, two for framework x and so on) in the following format: --resources="cpus(hdfs):2; cpus(stage):2 mem(*):15360; disk(*):710534; ports(*):[31000-32000]"
There are certain resources (cpus,mem, disk, port) which are autodiscovered by mesos. These automatically detected values are basically what you saw with your resources. (see http://mesos.apache.org/documentation/attributes-resources/) Basically, when you want to use less than the detected amounts you should explicitly assign less resources to the default role cpus(*):2. Hope this helps, Joerg On Fri, Mar 27, 2015 at 5:34 PM, John Omernik <[email protected]> wrote: > I have a question about how Mesos manages resources (and then some > follow-up questions as well). I will use the power of story time to > explain my question. > > Say we have a Mesos node. The physical node is running Ubuntu, and > from a hardware perspective, it's a E3-1240V2 Intel Processor with 4 > physical cores, 32 GB of ram and total disk space of 64 GB. This disk > is split into two partitions of 14 GB (3.1 GB Free) and 40 GB (12 GB > Free) > > > When I see this node registered as a mesos slave, here is what is reported: > > CPUs: 8 > Memory: 30.4GB > Disk 8.6 GB > > So this is where I would like to further my understanding of Mesos. > How does it see and manage resources? > > 1. Obviously the Ubuntu OS takes resources. How does that factor into > Mesos' available resource calculations? Where does these calculations > even come from (I can't see how 8.6 GB is my available disk on a > node). It obviously shows hyperthreaded cores, not physical cores, and > doesn't reserve any cores for OS use, but based on the memory total, > is it reserving memory or is that a GB math discrepancy? > > 2. Does Mesos continually update available resources to take into > account resources that are not managed by Mesos? IE. If the node > registered with 30.4 GB available memory, and then an hour later some > process not managed by mesos take another GB of memory, is Mesos aware > of that? Or will Mesos try to give resources away that it may not > control and may not exist? What happens if that occurs? > > 3. Can you tell Mesos Slaves, when they start, to only take X > resources? I.e. let's say you were running HDFS or MapRFS outside of > Mesos on every node, you wanted to allocate 10% of memory always to > that, can you tell Mesos to not give away resources that are > statically assigned for something else? Obviously the idea here is to > allow Mesos to manage it all, but somethings like MapRFS really need > direct access to disks etc. Perhaps it's already doing that to take > into account the OS on the box? > > 4. Has anyone thought of coming up with a smallish linux distro > designed to minimize non Mesos resource usage (OS usage). Basically > the minimum to allow Mesos and Docker to run on a node, with the idea > that all else should be in Mesos. (Not taking into account my > HDFS/MapRFS question). > > Lots weird questions here, I understand, please feel free to point me > in the right direction on docs for how to understand some of this, > that said, if you know specifics, please share! > > John >
