On Fri, Oct 10, 2008 at 7:26 AM, Sven Hartrumpf <[EMAIL PROTECTED]> wrote: > Hi all. > > If I use xargs -P for parallel excution of commands distributed > by ssh over several machines, the start-up by xargs is too > fast (yes, not too slow :-) ) so that the load balancer > (that selects the best node for the current ssh job) will not > catch up with the newly started processes. > Therefore the processes are badly distributed. > > A solution would be be an xargs option like > --delay-parallel-start=<SECS> > which waits <SECS> seconds before starting a new parallel job. > > How can this be achieved currently?
I have seen good solutions to this problem that use pull-based approaches; machines collect chunks of work from some shared storage, complete it, and then go back for more. The advantage of this approach is that it works well for situations where backends are unequal in performance, or some of the machines have an independent workload. There are downsides too though, which is that there is a certain amount of additional complexity and you need to guard against cases like having 0 remaining backends... James.