Have you tried reading the documentation? It tells you to set an id when running parallel from a script.
The purpose of id is to group your programs into groups. It might be better if sem just refused to run from a script without an id. On 24 Jan 2017 10:57 pm, "D. R. Evans" <[email protected]> wrote: > I am about to embark on a project in which the jobs will be much more > computationally intensive than the ones that precipitated this thread (see > below). I really need to understand why the "sem -- wait" does not do what > I > expected, otherwise I'm liable to waste an awful lot of time by playing > safe > and running the python jobs in the new project in series rather than > parallel. > > Can someone please point me to what the problem might be? > > (Maybe my problem is simply in failing to understand what "sem --wait" is > supposed to do, but in the past it's always seemed to do what my reading of > the documentation suggests: pause the script until all the jobs started > with > "sem" complete.) > > For completeness, here is the original post: > > D. R. Evans wrote on 01/22/2017 07:22 PM: > > Could someone please explain the following to me? I expect that I'm > missing > > something simple, but I don't understand what I am seeing. > > > > I have a bash script, the relevant part of which looks like this: > > > > ---- > > > > for (( yr=$MIN_YEAR; yr <= $MAX_YEAR; yr++)) > > do > > > > echo "creating maps for $yr" > > > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > sem -j-1 rbn-posters-map.py [parameters to python script] > > > > done > > > > sem --wait > > > > echo "After wait" > > ps auxw | grep rbn > > > > ---- > > > > My understanding, which is obviously incomplete, was that the "sem > --wait" > > command would cause the script to block until all the various > "rbn-posters.py" > > jobs that had been started with sem had completed. Therefore, I expected > that > > the "ps" command would find none of those jobs still to be running. > > > > However, what actually happens, consistently, is that 3 such jobs are > running > > when the "ps" command is executed (on this machine there are four cores). > > > > I need to have all the jobs completed at the point where the "ps" is > currently > > placed, so that the next part of the script will run correctly. But at > the > > moment, because not all the "rbn-posters.py" jobs have completed, the > next > > part fails. > > > > I welcome enlightenment as to why the "sem --wait" isn't doing what I > > expected. (And bonus points if someone can explain how to modify the sem > > commands in the script so as to be sure that they have all completed at > the > > point where the "ps" is currently located, preferably while still > allowing > > parallel execution of the "rbn-posters.py" jobs.) > > > > I thank you very much. > > Doc > > -- > Web: http://www.sff.net/people/N7DR > >
