Hi Robert, Normally set the job environment's limit in the queue config, however the file descriptor limit is not part of the queue limit. I believe 4096 comes from the execd's environment, and it gets inherited by the shepherd, and then the job.
You have 2 options: 1) Change the environment of the execd - ie. when execd starts, make sure there's enough descriptor limit in the shell (or the init environment). 2) You can set S_DESCRIPTORS, H_DESCRIPTORS with the "execd_params" option in sge_conf: http://gridscheduler.sourceforge.net/htmlman/htmlman5/sge_conf.html Example: H_DESCRIPTORS=5000 Rayson On Tue, May 29, 2012 at 9:34 AM, Robert Hutton <[email protected]> wrote: > Hi All, > > We have some jobs that we'd like to run that need to open about 5000 files > simultaneously. When they're run outside the > grid they run fine but when run with qsub they fail. I've run ulimit -H -n > inside and outside to see why this might be. > On our head node (foxtrot): > > $ cat test.sh > #!/bin/bash > ulimit -H -n > $ ./test.sh > 65536 > $ qsub -q longrun.q@foxtrot -cwd -S /bin/bash ./test.sh > Your job 215227 ("test.sh") has been submitted > $ cat test.sh.o215227 > 4096 > $ grep nofile /etc/security/limits.conf > # - nofile - max number of open files > * soft nofile 16384 > * hard nofile 65536 > > So the above shows that there is a limit of 4096 being set by GridEngine on > the max number of open files, but I haven't > been able to find where this is being set in order to change it. Can anyone > point me in the right direction? I'm > running Ubuntu 12.04 with 6.2u5-4 from the Ubuntu repos and the Open Grid > Scheduler hwloc drop-in upgrade[1]. > > Thanks, > > Rob > > [1] http://gridscheduler.sourceforge.net/projects/hwloc/GridEnginehwloc.html > > -- > Robert Hutton > Senior Systems and Database Administrator > Centre for Genomics and Global Health <http://cggh.org> > The Wellcome Trust Centre for Human Genetics > Roosevelt Drive > Oxford > OX3 7BN > United Kingdom > Tel: +44 (0)1865 287721 > _______________________________________________ > users mailing list > [email protected] > https://gridengine.org/mailman/listinfo/users _______________________________________________ users mailing list [email protected] https://gridengine.org/mailman/listinfo/users
