On Wed, Sep 20, 2000 at 08:53:53PM -0600, Eric Whiting wrote:
> Thanks for the info.
>
> I have used the shell redirection stuff to capture stdout/stderr from
> programs. But this time I'm trying to do a parallel rsync from a single
> server to multiple clients all at the same time. The server has 2G RAM
> and can handle the concurrent rsyncs.
>
> My problem is probably a perl programming problem and not a rsync one,
> but the 'log' feature in rsync would help me a lot. The following
> skeleton of perl shows what I'm trying to do. The perl 'open' command is
> really a popen system call that starts a child process to handle the
> rsync.
(Pedantry: popen is a libc function, not a system call.)
> The reason I wanted to use the rsync logfile feature is it seems the
> children and parent all share the same stdout and the individual '>>'s
> don't catch the right output. If I start a parallel rsync to 8 hosts,
> all the stdout gets mangled.
>
> Any other ideas?
> eric
>
> FROM THE PERLIPC MAN PAGE
>
> Filehandles
>
> Both the main process and any child processes it forks
> share the same STDIN, STDOUT, and STDERR filehandles. If
> both processes try to access them at once, strange things
> can happen.
For detailed help you should go to comp.lang.perl or some similar
forum. But briefly:
Yes, the child and parent share the same initial filehandles.
However, the child first starts a shell, which handles >> and
replaces stdout with a filehandle newly opened appending to the
logfile.
I think your script has the problem that in the second phase you're
reading all of the output from the first host before starting to read
from the second. This means that all the later ones will block
waiting for output (after the first 4k) until the first command is
complete. The programs will not run in parallel at all.
Here's a simple shell script that really does do parallel uploads,
logs each to a different file, and monitors the results.
----
#! /bin/sh
# $sites and $upload_files are set by the caller, which in this
# case is a Makefile
pids=
for i in $sites
do
logfile=`echo $i |sed -e 's/:.*//'`
rsync -l -v -e ssh $upload_files $i >/tmp/$logfile &
echo "started $! in background"
pids="$pids $!"
done
failed=0
for i in $pids
do
if wait $i
then
echo $i completed OK
else
echo $i failed!
failed=1
fi
done
exit $failed
----
>
>
>
> FROM MY ATTEMPT AT A PARALLEL RSYNC IN PERL
>
> foreach $dir (@dirs) {
> # inner loop hosts -- do these all in parallel (at the same time)
> foreach $host (@hosts) {
>
> $cmd="/usr/local/bin/rsync -a --delete -e '/usr/local/bin/ssh '
> 'apps/.snapshot/*' /home/apps/${dir} ${host}:/home/apps\n";
>
> print "Working on $host $dir\n";
> open("$host","$cmd >>$log_dir/$host.log|"); # fork/sh/cmd magic is
> in the '|'
> }
>
> # wait for processes to finish -- blocks on the slow host -- zombies
> the others for a while
> foreach $host (@hosts) {
> while(<$host>) {
> $data{$host}.=$_; # try to save the data -- sure wish I could log
> to a fixed file instead.
> }
> close($host);
> }
> }
--
Martin Pool, Linuxcare, Inc.
+61 2 6262 8990
[EMAIL PROTECTED], http://www.linuxcare.com/
Linuxcare. Support for the revolution.
PGP signature
