Well, my main concern was rather the temporary files. Right now I have to move my Workspace directory b/c the shared storage where it was runs out of memory. Transferring like 150000 smaller files is taking it’s time. And also deleting them all at once again will require some time (I twice canceled all jobs and the OpenMOLE console and all these temporary files are for the moment still present).
> On 12 May 2015, at 14:35, Jonathan Passerat-Palmbach > <[email protected]> wrote: > > The problem with this is it's very local to our configuration at Imperial. > OpenMOLE is not intended to cope with local configurations and assumes that > the environment it's been assigned will scale > if not then, we need to restrict that environment locally (create a queue > with less machines, or pass a requirement that restrains the number of nodes, > ...) > > On 12/05/15 14:28, Andreas Schuh wrote: >> The main reason being to avoid too many temporary files being created in the >> sharedDirectory. For each job, this requires around 100-1024 KB. If the >> input files were no links, these would add to this as well. Given hundreds >> of thousands of jobs, this easily adds up. Further, I heard rumours in our >> lab that the SLURM head node/server would crash if more than 10000 jobs are >> submitted at once… Jonathan might have a better idea what’s true about this, >> though. > > this is a purely arbitrary limit set by myself :) > we could increase it as Slurm can handle way more jobs but I wasn't sure the > server hosting it could scale up and didn't want to change that during the > deadlines period > but now we can play with it :) > >> My main concern are the number of temporary files that are created (also >> counting towards the quota of inodes…). > shouldn't be a problem if you're not keeping your .openmole in your /homes > i don't know of any user quota on the other shared spaces > > > Cheers > J. >> >>> On 12 May 2015, at 09:34, Romain Reuillon <[email protected]> wrote: >>> >>> Hi Andrea, >>> >>> no, but why would you do that ? >>> >>> Romain >>> >>> Le 12/05/2015 10:27, Andreas Schuh a écrit : >>>> Hi, >>>> >>>> is it possible to limit the number of jobs submitted to the >>>> SLURM/Condor/PBS queue at a time ? >>>> >>>> Andreas >>>> _______________________________________________ >>>> OpenMOLE-users mailing list >>>> [email protected] >>>> http://fedex.iscpif.fr/mailman/listinfo/openmole-users >>> >> >> _______________________________________________ >> OpenMOLE-users mailing list >> [email protected] >> http://fedex.iscpif.fr/mailman/listinfo/openmole-users > > -- > Jonathan Passerat-Palmbach, PhD > Research Associate > Department of Computing > Imperial College London > > South Kensington Campus > Huxley Building - room 344 > 180 Queen's Gate > London SW7 2AZ > > > _______________________________________________ > OpenMOLE-users mailing list > [email protected] > http://fedex.iscpif.fr/mailman/listinfo/openmole-users _______________________________________________ OpenMOLE-users mailing list [email protected] http://fedex.iscpif.fr/mailman/listinfo/openmole-users
