I rarely run more than 200 jobs in parallel. But I was curious how many jobs I could safely run. This starts 30000 sleeps:
seq 1000000000 | parallel -N1000 -j120 --pipe parallel -j0 sleep I have let those stay in the background for a day now, and the only issue I have had so far is that 'ps aux' takes > 1 sec to run because it has to list > 30000 processes. I also tried going higher, but GNU Parallel does not seem to act predictably when hitting the system hits the 32767 process limit. But that might also be caused by the rest of the system acting weird when hitting that limit. So using the fan out approach (parallel --pipe parallel) GNU Parallel scales well to 30000 processes without any changes to my GNU/Linux system. If you are running GNU Parallel on another platform (and you are not afraid to crash it), it would be interesting what the (safe) limit is on other systems. /Ole
