Re: OT: launching jobs in a combined serial parallel way
On Wed, 24 Jun 2009 21:28:53 -0400 Johan Kullstam kullstj...@comcast.net wrote: Kamaraju S Kusumanchi raju.mailingli...@gmail.com writes: I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. Use make. I like to use make for all sorts of things, not just compiles. And, since you are already using the existance of a file to check for done, you are practically there. Make with the -j parameter will spawn parallel jobs for you. Write a makefile similar this: I think that you also need to create the files, something like proga.out: proga touch proga.out progb.out: progb touch progb.out probc.out: proga.out progb.out progc Then you can do $ nohup make -j 2 Maybe a redirect of the text output to a file. Now you can walk away. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
On Fri, 26 Jun 2009 18:52:47 +0300 Micha Feigin mi...@post.tau.ac.il wrote: On Wed, 24 Jun 2009 21:28:53 -0400 Johan Kullstam kullstj...@comcast.net wrote: Kamaraju S Kusumanchi raju.mailingli...@gmail.com writes: I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. Use make. I like to use make for all sorts of things, not just compiles. And, since you are already using the existance of a file to check for done, you are practically there. Make with the -j parameter will spawn parallel jobs for you. Write a makefile similar this: I think that you also need to create the files, something like proga.out: proga touch proga.out progb.out: progb touch progb.out This may work even better probc.out: proga.out progb.out progc rm -f proga.out rm -f progb.out Then you can do $ nohup make -j 2 Maybe a redirect of the text output to a file. Now you can walk away. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
On Wed, Jun 24, 2009 at 10:05:20PM -0400, Scott Gifford wrote: Douglas A. Tutty dtu...@vianet.ca writes: On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. Just to clarify, wait(2) and the shell wait builtin do not poll, they instruct the kernel to put the process to sleep until a child process finishes, then wake it up and return from the wait call. Because of that they are very efficient. Still, if the programs in question are doing a great deal of work, the extra work required by polling will not be very significant in comparison. Isn't that just passing the polling on to the kernel? At some level, some process has to see if a pid exists, if not, wait a period, then check again. Doug. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Douglas A. Tutty dtu...@vianet.ca writes: On Wed, Jun 24, 2009 at 10:05:20PM -0400, Scott Gifford wrote: Douglas A. Tutty dtu...@vianet.ca writes: On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. Just to clarify, wait(2) and the shell wait builtin do not poll, they instruct the kernel to put the process to sleep until a child process finishes, then wake it up and return from the wait call. Because of that they are very efficient. Still, if the programs in question are doing a great deal of work, the extra work required by polling will not be very significant in comparison. Isn't that just passing the polling on to the kernel? At some level, some process has to see if a pid exists, if not, wait a period, then check again. I don't believe so. I believe that the parent process is suspended completely, then when the child process exits, the kernel checks to see if the parent process is currently waiting, then wakes it up. So the wakeup is initiated when the child process exits, not checked periodically. Perhaps a kernel hacker can confirm this, but looking in $KERNEL_SRC/kernel/exit.c, sys_exit() calls do_exit() which calls exit_notify(); that calls do_notify_parent() from signal.c, which re-activates the parent process. It's much like a blocking read(2) or select(2) is put to sleep by the kernel, to be woken up again when the condition it's waiting for occurs. Scott. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
In 20090625143028.ga7...@blitz.hooton, Douglas A. Tutty wrote: On Wed, Jun 24, 2009 at 10:05:20PM -0400, Scott Gifford wrote: Douglas A. Tutty dtu...@vianet.ca writes: While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. Just to clarify, wait(2) and the shell wait builtin do not poll. Because of that they are very efficient. Still, if the programs in question are doing a great deal of work, the extra work required by polling will not be very significant in comparison. Isn't that just passing the polling on to the kernel? At some level, some process has to see if a pid exists, if not, wait a period, then check again. No, it really doesn't. At the kernel level, it is possible to add a function to a list of functions to be run when another process exits. The wait/waitpid calls without a timeout basically do that and then put the waiting process to sleep. Once the does process exits, the kernel runs through the list of registered functions and eventually calls one that wakes up the sleeping process. True asynchronous / event-based programming is ultimately more efficient that polling solutions not because the polling is done elsewhere, but because the polling is eliminated entirely. (It's actually rare to see much of an end-to-end improvement in throughput by moving the polling around.) -- Boyd Stephen Smith Jr. ,= ,-_-. =. b...@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/\_/ signature.asc Description: This is a digitally signed message part.
Re: OT: launching jobs in a combined serial parallel way
Douglas A. Tutty wrote: On Wed, Jun 24, 2009 at 10:05:20PM -0400, Scott Gifford wrote: Douglas A. Tutty dtu...@vianet.ca writes: On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. Just to clarify, wait(2) and the shell wait builtin do not poll, they instruct the kernel to put the process to sleep until a child process finishes, then wake it up and return from the wait call. Because of that they are very efficient. Still, if the programs in question are doing a great deal of work, the extra work required by polling will not be very significant in comparison. Isn't that just passing the polling on to the kernel? At some level, some process has to see if a pid exists, if not, wait a period, then check again. Doug. No, the OS doesn't poll, either. It keeps track of waits and what they are waiting for, and when that event completes the os performs the required action. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. You also have the ability to have proga and progb only touch the file if they complete successfully. If you merely wait until their ps disappears, you don't know if they crashed or properly completed. Doug. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Doug writes: While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. You also have the ability to have proga and progb only touch the file if they complete successfully. Have each of them check for a completion file when they complete successfully. If it exists remove it and start progc. If not create a completion file and exit (there is a possible race condition here). Both should be started from the same script, of course, which should handle the completion file, errors and cleanup. No need to modify the programs. -- John Hasler -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Hello, On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. Douglas A. Tutty dtu...@vianet.ca: ] While you may think its terribly inefficient, it isn't really. A fancy ] wait function is just polling anyway, you're just making it overt. ] You also have the ability to have proga and progb only touch the file ] if they complete successfully. If you merely wait until their ps dis- ] appears, you don't know if they crashed or properly completed. Agreed. The original solution is, in my view, perfectly clean and fine. It's simple, and simple is good. Another problem with a ps(1)-monitoring solution, besides the one Douglas points out above, is that it doesn't scale; a script that uses ps(1) can't be run concurrently in multiple directories, should your requirements eventually expand to include that in future. Your solution, Kamaraju, can. Nota bene: Just as in recursion one tests for the stopping condition first, at the top of the function, I presume you're remembering to test for the existence of these semaphore files before launching proga and progb, and doing the right thing (however you define that, e.g. not proceeding at all, or alternatively just removing them and proceeding) if they're already there. Cheers, Eric -- The great French Marshal Lyautey once asked his gardener to plant a tree. The gardener objected that the tree was slow-growing and would not reach maturity for 100 years. The marshal replied: ''In that case, there is no time to lose, plant it this afternoon.'' --John F. Kennedy Eric De Mund | Y!: ead0002 e...@ixian.com | ICQ: 811788 -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
On Mon, Jun 22, 2009 at 5:17 PM, Kamaraju S Kusumanchiraju.mailingli...@gmail.com wrote: proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. I've taken to using flock for such things if I'm launching them from other scripts. I forget which package and I can't look right now (my machine died this morning). I think it's like this: flock /path/to/lockfile programtorun --options --to --programtorun I would do something like: flock /path/to/proga /path/to/proga --options_for_a flock /path/to/progb /path/to/progb --options_for_b flock /path/to/proga flock /path/to/progb /path/to/progc --options_for_c That is, use an flock on the program itself as the serialization point. Well, this works when the file I'm locking is a shell script in my ~/bin/ directory. Not so sure if it's a /usr/bin/ type of file without write access. In that case you could always create files in /tmp/ And there may be better ways to lock on multiple files than chaining flock like I did above. Read the man page. mrc -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
On Wed, Jun 24, 2009 at 4:46 PM, Mike Castledalgoda+deb...@gmail.com wrote: I've taken to using flock for such things if I'm launching them from other scripts. I forget which package and I can't look right now (my machine died this morning). To clarify, I meant to say: I've taken to using flock for such things if I'm launching them from different windows. mrc -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Kamaraju S Kusumanchi raju.mailingli...@gmail.com writes: I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. Use make. I like to use make for all sorts of things, not just compiles. And, since you are already using the existance of a file to check for done, you are practically there. Make with the -j parameter will spawn parallel jobs for you. Write a makefile similar this: proga.out: proga progb.out: progb probc.out: proga.out progb.out progc Then you can do $ nohup make -j 2 Maybe a redirect of the text output to a file. Now you can walk away. -- Johan KULLSTAM -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Douglas A. Tutty dtu...@vianet.ca writes: On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. Just to clarify, wait(2) and the shell wait builtin do not poll, they instruct the kernel to put the process to sleep until a child process finishes, then wake it up and return from the wait call. Because of that they are very efficient. Still, if the programs in question are doing a great deal of work, the extra work required by polling will not be very significant in comparison. -Scott. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Douglas A. Tutty wrote: On Mon, Jun 22, 2009 at 08:17:44PM -0400, Kamaraju S Kusumanchi wrote: Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. While you may think its terribly inefficient, it isn't really. A fancy wait function is just polling anyway, you're just making it overt. You also have the ability to have proga and progb only touch the file if they complete successfully. If you merely wait until their ps disappears, you don't know if they crashed or properly completed. Doug. Good point. I'll have to think about corner cases like these furthermore. The scenario you portray does sometimes occur. The suggestions I got in this thread are very helpful. Thanks guys. raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
OT: launching jobs in a combined serial parallel way
I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. Using Debian Lenny (Stable). thanks raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Hi, I didn't test that script, just writing it from memory, but maybe that gives you some ideas: #!/bin/bash prog1 prog2 while [ $(ps -A | grep -E 'prog1|prog2') == 0 ] do sleep 5 done prog3 more or less like that greetings, vitaminx 2009/6/23 Kamaraju S Kusumanchi raju.mailingli...@gmail.com I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. Using Debian Lenny (Stable). thanks raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org -- www ... http://www.callistix.net/ mail ... vitam...@callistix.net irc ... #chezpaeule @ euirc mud ... vitaminx @ aardmud
Re: OT: launching jobs in a combined serial parallel way
On 6/22/09 6:32 PM, me wrote: 2009/6/23 Kamaraju S Kusumanchiraju.mailingli...@gmail.com I have three programs - say proga, progb, progc. proga, progb are completely independent. They take couple of hours to finish. The time to complete proga, progb are not same. progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? I prefer to use nohup since sometimes I have to log out of the machine before the whole process finishes. Currently I have a shell script that works as below. 1) launch proga, progb in the background using nohup. 2) Ask proga, progb to write a file when they finish. 3) Every five minutes check if these files are present. If they are present, launch progc. This gets me going for now. But it looks terribly inefficient. I would appreciate if someone can provide a better solution. Using Debian Lenny (Stable). thanks raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org #!/bin/bash prog1 prog2 while [ $(ps -A | grep -E 'prog1|prog2') == 0 ] do sleep 5 done prog3 Are progs ab yours? If so, how about having each one check ps for the other one as part of it's exit. If the other one isn't running, start progc before exit. Or if stuck with a script, start ab; sleep an hour and a half; and then go into the 5 minute loop on ps | grep... -- Glenn English g...@slsware.com -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Kamaraju S Kusumanchi raju.mailingli...@gmail.com writes: [...] progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? In a shell script, run proga and progb in the background, then wait for them both with the wait builtin. Something like this: #!/bin/sh -e # -e flag will abort on any error proga # Start proga in background progb # Start progb in background wait # Wait for whichever finishes first wait # Wait for the other one progc # Now run progc in the foreground Scott. -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Scott Gifford wrote: Kamaraju S Kusumanchi raju.mailingli...@gmail.com writes: [...] progc should to be launched only after both proga, progb are finished. progc takes another couple of hours to finish. What is good way to automate this problem (that is no manual interaction)? In a shell script, run proga and progb in the background, then wait for them both with the wait builtin. Something like this: #!/bin/sh -e # -e flag will abort on any error proga # Start proga in background progb # Start progb in background wait # Wait for whichever finishes first wait # Wait for the other one progc # Now run progc in the foreground Scott. This is great. A little perusal of 'man wait' gave me a better idea. nohup proga pa=$! nohup progb pb=$! wait $pa $pb nohup progc Thanks a lot. Just what I am looking for. raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
Are progs ab yours? Yes. They are mine. If so, how about having each one check ps for the other one as part of it's exit. If the other one isn't running, start progc before exit. That is a nice idea. I have not thought about it. For now, I will go with Gifford's tip of using the shell's wait function. regards raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
Re: OT: launching jobs in a combined serial parallel way
me wrote: #!/bin/bash prog1 prog2 while [ $(ps -A | grep -E 'prog1|prog2') == 0 ] do sleep 5 done prog3 more or less like that You know, I actually started out with something similar. But it has limitations. Let's say you want to launch two sets of (proga, progb, progc). Then the ps + grep logic will grep for proga, progb from both the sets. That is why I shifted to writing a file at the end. In any case, thanks for the reply. raju -- Kamaraju S Kusumanchi http://malayamaarutham.blogspot.com/ -- To UNSUBSCRIBE, email to debian-user-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org
