On Thu, Aug 9, 2012 at 3:24 PM, Rhys Ulerich <[email protected]> wrote:
>> Would it make sense to have a setting in GNU >> Parallel that automatically run 'ulimit ' with the relevant amount of >> memory, so if you ask for X jobs to be run on a given server, then >> each job is only allowed 1/X'th of the memory on the machine. > > I like it. Maybe '--whatever 100%' is up to the limit of the machine > and '--whatever 100M' allows setting a specific size to be divided > amongst the jobs. Interesting idea. > I wondered if one could set a memory limit on a group so that the on > average the processes could use no more than 1/Xth of the memory but > any individual process might employ more than 1/X. Reading the ulimit > description in the bash(1) man page, I noticed this ulimit option: > > -v The maximum amount of virtual memory available to the shell > and, on some systems, to its children > > It seems the goal you have is to enforce that none of the processes > swap. Do you think a 'ulimit -v'-like behavior could be collectively > attached to the group or processes that GNU parallel spawns? If so, > could the virtual limit be set to be only the physical memory? Doing it per process would simply require a 'ulimit -v/-m' to be run as part of the command. This will work on remote machines, too. Doing it for the group of jobs will not work remotely, as each remote jobs is started through its own ssh. To run locally it requires changing the ulimit from inside perl. BSD::Resource has setrlimit which seems to do what we want. But I really do not want to require people to install yet another Perl module - especially not to get this fairly limited functionality. So I think the 'ulimit -v/-m' run as a wrapper of the command is currently the only realistic. - o - Should it take an argument? I tend to say no. The normal situation will be that you want a safety net that will save you if one of the jobs takes more the a 1/X'th of the RAM, where X is the number of jobs running on that machine; thus it should be computed on the machine running the jobs. And in the odd case that you do not want that, you can run 'ulimit' manually. What would be a reasonable name? --memlimit? --ramlimit? --maxram? --swap-safety-net? >> For most situations it will not make a difference, so I am interested >> in whether you will anticipate more surprise by having your big job >> killed or the happyness by not swapping the machine to death? > > The principle of least surprise says running too many memory hungry > processes should swap the machine into the ground, IMHO. After giving it some thoughts I tend to agree: The safety net should not be default. /Ole
