On Mon, 2007-02-12 at 12:29 -0800, Michael Brailsford wrote: > If I had to synchronize multiple scripts, I think I would stop thinking about > shell scripting. The noclobber thing is bash specific, and most unix > distributions you see in the real world will not have bash, but instead they > all have korn shell. Korn shell is considered the de facto standard. > Unless, bash gets installed manually, which is generally frowned upon in > places where standard builds and standard configurations are more important > than the user's preference. I would suggest using something like perl. > Every unix machine I have ever seen in any setting in my experience has perl. >
Well my scripts are bash scripts, not korn (sh). And for my particular job, short of writing a client server application, the file locking method works very well. My scripts all have #!/bin/bash at the top, so I'm not worried about anything in the real world or other such things. And given my problem, I don't see any benefit of perl over bash. > > Of course, if you are stuck supporting an existing system and you cannot > change it, then good luck. This is a recently created system and, yes, synchronizing asynchronous bash scripts is one of the better solutions for the job at hand. In fact, the problem of synchronization is the same whether I'm doing it in perl or bash. To humor you, though, I will state the problem. Server A sshes to Server B and runs a script. For security reasons server A uses a ssh key to a non-privileged account on Server B, where is uses sudo to run a script that is limited in what it does (limit the root scope). The script's job is to tar up a particular folder and then delete the folder. For certain reasons, the script then uses the atd to schedule itself to run in 5 minutes. When the script finally runs, it checks to see if the folder really should be archived (in case the admin changed his mind during the last 5 minutes), and then proceeds the archive the folder and then delete the folder. The problem is that this archiving can take a long time. Further, server A can ssh in at any time and ask for another folder to be archived. Worst case server A sends up to 1500 folder archive requests over a period of 20 minutes. This is a rare event. Nonetheless, after the 5 minute sleep, it's possible to have hundreds of scripts trying to run tar and gzip simultaneously on folders ranging anywhere from 1 to 500 MB. Without creating something complicated, the solution is to get the scripts to synchronize themselves so that only one tar (current solution) is running at a time. I forcibly serialize them to one at a time only because it's the simplest and the need to serialize is rare anyway. So the locking solution works very very well. Completion time is of no consequence. The problem for me was that the main script was already coded up and functioning before the concurrency problem was thought about. And it's not a true concurrency problem either; it's just a matter of keeping the server load down. If I was doing it again, maybe setting up some kind of job queue and then polling the queue every minute or two with cron may have worked. But this file locking turned out to be a very simple and effective solution, given that concurrency would only be an issue once a year or so. > > -Michael > > ----- Original Message ---- > From: Bart Whiteley <[EMAIL PROTECTED]> > To: Provo Linux Users Group Mailing List <[email protected]> > Sent: Monday, February 12, 2007 2:20:03 PM > Subject: Re: synchronizing concurrent shell scripts > > On 2/12/07, Michael L Torrie <[EMAIL PROTECTED]> wrote: > > > > > > > > However, touch works well. Something like this has always worked for > > me. > > > > > > while [ -d /var/somelockfile ]; do sleep 1; done > > > touch /var/somelockfile > > > # do something > > > rm /var/somelockfile > > > > Sadly this will not work, at least if you goal is to synchronize > > multiple scripts running asynchronously. It would have worked in my > > case, but it's not a semaphore. Who wins when several processes make it > > past the while loop at about the same time? > > > > > > Right. This is why you should use the redirect '>' with noclobber instead. > With noclobber, the redirect works like an atomic test-and-set. > > /* > PLUG: http://plug.org, #utah on irc.freenode.net > Unsubscribe: http://plug.org/mailman/options/plug > Don't fear the penguin. > */ > > > > > /* > PLUG: http://plug.org, #utah on irc.freenode.net > Unsubscribe: http://plug.org/mailman/options/plug > Don't fear the penguin. > */ > /* PLUG: http://plug.org, #utah on irc.freenode.net Unsubscribe: http://plug.org/mailman/options/plug Don't fear the penguin. */
