On 6 February 2015 at 20:42, Ole Tange <o...@tange.dk> wrote: >>> On Fri, 6 Feb 2015 23:20 xmoon 2000 <xmoon2...@googlemail.com> wrote: >>>> >>>> I need to run about 4,000 jobs that each take around 20 seconds to >>>> complete > : >>>> On my 32 core machine works OK, BUT there is a "lull" in processing >>>> every few seconds as new jobs are started, once the current crop have >>>> completed. I assume this is due to an overhead in starting jobs that >>>> is only noticable because my jobs are so short. >>>> >>>> Is there any way I could make this more efficient, so my cores are >>>> fully utilised and getting through the whole process is faster? > > On Fri, Feb 6, 2015 at 3:56 PM, xmoon 2000 <xmoon2...@googlemail.com> wrote: >> I am running this in cygwin. I call parallel from the standard >> window/shell supplied by cygwin. > > My experience with Cygwin is limited, but I seem to remember that > spawning is expensive (in the order of 0.1 second), which is one of > the reasons for recommending MSYS. So if your jobs all finish at the > same time, GNU Parallel will need 3.2 seconds to start 32 new jobs. > Maybe you will have a better experience if you delay starting jobs by > 0.1 seconds: > > cat /tmp/parList | parallel -j 28 --delay 0.1 --eta; > > What is your reason for not using all 32 cores? > > If it is not due to the slow spawning, it could simply be due to disk > IO when printing. The standard reply for that is simply increase the > number of parallel jobs until all CPUs are 100% loaded (or disk IO is > killing your performance): > > cat /tmp/parList | parallel -j 40 --delay 0.1 --eta; > > > /Ole
Ole, 1. Reason for not using 32 cores is then I can't do anything else. I need to do emails, check web, simple stuff whilst waiting for jobs to finish. 2. I will try your delay - see if that helps. 3. When I run my own bash script which does a very simple approach to spawning jobs, then I do start upto 50 jobs at a time. However, counting how many processes are active on my jobs on cygwin/windows is time-expensive and not very accurate! (Well, for me anyway because I am staring so many little jobs). Moon
