> Am 27.06.2019 um 13:46 schrieb Dan Whitehouse <d.whiteho...@qmul.ac.uk>:
> First off, I am running UGE as opposed to SGE.
> We've got a couple of systems, one running 8.5.4 and the other 8.6.5.
> Users request memory resources in their job scripts by passing:
> "-l h_vmem=1G" (for example).
> We make use of a JSV and when this is set, what actually gets passed to 
> the scheduler is:
> "h_vmem=1G,m_mem_free=1G"
> We set "h_vmem_limit=true" in cgroups_params so it is enforced by cgroups.
> The thing that I am not entirely sure about is what we are actually 
> limiting here!
> If I write a program to malloc memory in a loop, then cgroups kills it 
> when it has allocated over 400G of ram (on a machine with about 24G).
> Looking at the output of qacct, it has used ~1G to do so. So my 
> assumption here is that cgroups is killing on memory used as opposed to 
> virtual memory allocated. Which of the two settings (h_vmem / 
> m_mem_free) is responsible for this, and what is the other one for?
> I'm sure this isn't the first time this has been asked, and for that I 
> apologise but I can't seem to find a clear explanation of this.

We don't use cgroups for now. But the allocation of memory is often delayed 
until you are really access the allocated space (even without cgroups in 
place). You could fill the allocated area with data, and test what happens then.

h_vmem will be enforced by the kernel, even without cgroups. In case you make 
h_vmem consumable and attach a sensible value to each exechost, SGE will also 
keep track of this and disallow further submissions.

With h_vmem used at jobsubmission, either the kernel or SGE will then notice 
that your job and the sum of all of its processes passed this limit and kill 
the job. SGE can do this, by using the additional group ID which is attached to 
all processes by a particular job, while the kernel might only watch a certain 

Using now an assigned cgroup, even the kernel can keep track of the overall 
memory consumption of sevreal processes in this assigned cgroup.

-- Reuti

> Thanks!
> -- 
> Dan Whitehouse
> Research Systems Administrator, IT Services
> Queen Mary University of London
> Mile End
> E1 4NS
> _______________________________________________
> users mailing list
> users@gridengine.org
> https://gridengine.org/mailman/listinfo/users

users mailing list

Reply via email to