Seems the right way would be to account for the number of CPUs used by a process. But then you've starting writing PBS.
Wrap the mgiza command with flock? On 11/30/11 22:19, Barry Haddow wrote: > Hi Tamas > > The behaviour seems reasonable to me. > > The flag --parallel tells train-model.perl to run processes simultaneously if > possible, so it will run giza in both directions at the same time. > > The --mgiza-cpus is just for mgiza. Why not just specify 4 cpus for mgiza, if > you only have 8 in total? > >>>> --mgiza --mgiza-cpus 8 --parallel on an 8-core machine, I actually get >>>> 16 giza processes. > That's odd, I'd expect 2 giza processes, each with 8 threads. > > cheers - Barry > > > On Wednesday 30 Nov 2011 22:05:40 Kádár Tamás (KTamas) wrote: >> Er, no. As far as I can observe, --parallel parallelizes some tasks >> that are possible into 2 threads, one for e2f, one for f2e. For step >> 2, giza++, it starts two threads, one for e2f and one for f2e. Then >> _that_ is parallelized by mgiza into whatever number of processes I >> tell it to (the original idea for mgiza is one process per CPU/CPU >> core). I beleive the solution would be not to parallelize step 2 from >> train-model.perl but leave it to mgiza if we're training with that. >> (Until then the workaround is to halve the number of cpus in the mgiza >> parameters I guess.) >> >> I have yet to (and will soon) test the performance differences between >> 16 processes on a 8-core system versus 8 processes on a 8-core system. >> In theory guess the latter should fare somewhat better. >> >> Best regards, >> Tamas >> >> 2011/11/30 Kenneth Heafield <mo...@kheafield.com>: >>> Wouldn't it run 64 processes? >>> >>> On 11/30/11 21:31, Kádár Tamás (KTamas) wrote: >>>> Hi >>>> >>>> With the option --parallel and running mgiza with whatever number of >>>> cpus, the training script parallelizes mgiza too... so if I run >>>> --mgiza --mgiza-cpus 8 --parallel on an 8-core machine, I actually get >>>> 16 giza processes. Shouldn't it either a) warn you about this behavior >>>> or b) not parallelize mgiza? Makes no sense to me to parallelize >>>> something that is already parallelized :) >>>> >>>> Just my $.02. >>>> >>>> Best regards, >>>> Tamas >>>> _______________________________________________ >>>> Moses-support mailing list >>>> Moses-support@mit.edu >>>> http://mailman.mit.edu/mailman/listinfo/moses-support >> >> _______________________________________________ >> Moses-support mailing list >> Moses-support@mit.edu >> http://mailman.mit.edu/mailman/listinfo/moses-support >> > > _______________________________________________ > Moses-support mailing list > Moses-support@mit.edu > http://mailman.mit.edu/mailman/listinfo/moses-support _______________________________________________ Moses-support mailing list Moses-support@mit.edu http://mailman.mit.edu/mailman/listinfo/moses-support
