Re: How to avoid copying empty directories?
I agree with this - a perl script to generate a list of includes from a list of
directories
is not hard to write (below in long and readable rather than compact perly
form), and what about --python and --awk:
#cat frag.pl
#!/usr/bin/perl -w
# frag.pl - perl fragment for generating include list for rsync
my $includes = ;
my %already = ();
my @pathlist = ;
foreach my $path (@pathlist) {
chop $path;
my @subdirs = split(/\//,$path);
my $dirs;
foreach my $dir (@subdirs) {
$dirs .= $dir./;
unless(defined($already{$dirs}) $already{$dirs} eq included) {
$includes .= --include .$dirs;
$already{$dirs} = included;
}
}
$includes .= --include $path/**;
}
$includes .= --exclude \*\ ;
print $includes,\n;
#cat test
dir1/local/src/dir1
dir2/local/bin
dir2/local/src
#cat test | ./frag.pl
--include dir1/ --include dir1/local/ --include dir1/local/src/ --include
dir1/local/src/dir1/ --include dir1/local/src/dir1/** --include dir2/ --include
dir2/local/ --include dir2/local/bin/ --include dir2/local/bin/** --include
dir2/local/src/ --include dir2/local/src/** --exclude *
Dave Dykstra wrote:
On Thu, Nov 29, 2001 at 09:23:18AM +1100, Martin Pool wrote:
On 28 Nov 2001, Dave Dykstra [EMAIL PROTECTED] wrote:
On Tue, Nov 27, 2001 at 04:03:58PM -0800, [EMAIL PROTECTED] wrote:
rsync -avu --include 'tmp1/*/*.c' --include */ --exclude * tmp1 tmp2
The above command copies all the empty directories under tmp1/ . Is there
any way to avoid it?
Currently the only way is to explicitly include the parent directories
you want instead of --include */.
A quick fix is to post-process the directory with
find tmp2 -type d -empty -depth -exec rmdir {} \;
I think the best fix is to add scripting support:
rsync -avu --perl 'filter() { ( -d -s ) || /\.c$/; }' here friendly:/tmp
Anybody else like this idea?
Nah, the best fix in this case is --files-from. I'm not too keen on the
scripting idea. Seems like more bloat to me. With --files-from people
can use whatever script they like to generate the list of files before
sending it to rsync.
- Dave
Not all files synched - hard link problems???
I am sorry if this has been covered before: I have done a couple of futile searches in the bug reporting system and is there any way to search the archive? I am having a strange symptom: I am synching directories (that have very long file names, by the time the full path is specified) and a lot of hard links. It seems that the directory is being copied piecemeal - that is, if I run rsync enough times, the entire contents ultimately get copied. It seems like I am running into some hard limit in the size of the filelist or something. I am running 2.4.6 on linux - the source directory is remote mounted on solaris, the destination is linux. For instance, I have a directory that is in the tree that contains 173 files at the source - most of which are hard links - here is the effect of an ls | wc on the destination after five succesive identical runs of rsync on the source (this is in a subdirectory of one of the directories in the command below). The directory did not exist before running the sync. [root@ks-s0-107-1- SC]# ls | wc 49 491286 [root@ks-s0-107-1- SC]# ls | wc 85 852234 [root@ks-s0-107-1- SC]# ls | wc 120 1203243 [root@ks-s0-107-1- SC]# ls | wc 152 1524112 [root@ks-s0-107-1- SC]# ls | wc 173 1734739 So that it seems to have synched 49, then 36, then 35, then 32, the finally the last 21 files in the directory. (The increment seems to vary if I try it again, i.e. from 49 to 90). I get no error mesages (that I can see). In fact, the first time I run the program, it seems to notice all the files and produces 163 messages that file blah-de-blah is a hard link, but then doesn't seem to make the link for file blah-de-blah - this behavior remains constant with each succesive run. Here is the rsync command (generated by a perl script and broken into little pieces by my mailer) rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats --progress -tpgoCHr -v -v --include install/ --include install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ --include install/redhat-7.1/zantaz_rpm_upgrade/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ --include install/kickstart/zantaz_upgrade/20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** --exclude * /net/swdev/staging/* [EMAIL PROTECTED]:/ Thanks, Dave
More - cause - not all files synched - program hangs in select
The bug has something to do with verbosity - it works fine without verbosity on. The irony, of course, is that one turns verbosity on to fix these things (I have applied the hang patch, BTW). The last thing I see is the match_report message, then the program hangs in select. Here are the symptoms: I am syncing a number of parallel directories that contain large numbers hard links. At the source: dir a: f1 f2 f3 f4 f5 dir b: f1 f2 f3 f4 f5 dir c: f1 f2 f3 f4 f5 f1 in each directory is a hard link to every other f1, etc. (note, there are actually about 10 directories of 100 files in each directory) After my first run of rsync, I get something like this. dir a: f1 (link count 3) f2 (3) f3 (1) f4 (1) f5 (1) dir b: f1 (3) f2 (3) dir c: f1 (3) f2 (3) After my second run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (1) f5 (1) dir b: f1 (3) f2 (3) f3 (3) dir c: f1 (3) f2 (3) f3 (3) After my third run of rsync, I get something like this. dir a: f1 (3) f2 (3) f3 (3) f4 (3) f5 (1) dir b: f1 (3) f2 (3) f3 (3) f4 (3) dir c: f1 (3) f2 (3) f3 (3) f4 (3) etc. So that dir a: is completely copied the first time, but only the beginnings of dir b and c. With each successive run, dirs b c ( d e f) fill up, until it really is done. Again, this doesn't happen if I turn verbosity entirely off. Perhaps somebody is closing some file descriptor - I noticed that the program was hanging on the remote box in select - and then somebody else (probably having to do with making links) is trying to write to it. Thus, the program hangs but the timing is not always the same and sometimes a few more or less links are made before it hangs. It seems that making the hard links is done in a second phase after the real file transfers are done, at which time something must be closed. I could look through the source code and fix it myself but maybe by the time I get in tomorrow somebody that knows it well will have done so. Dave Dave Madole wrote: I am sorry if this has been covered before: I have done a couple of futile searches in the bug reporting system and is there any way to search the archive? I am having a strange symptom: I am synching directories (that have very long file names, by the time the full path is specified) and a lot of hard links. It seems that the directory is being copied piecemeal - that is, if I run rsync enough times, the entire contents ultimately get copied. It seems like I am running into some hard limit in the size of the filelist or something. I am running 2.4.6 on linux - the source directory is remote mounted on solaris, the destination is linux. For instance, I have a directory that is in the tree that contains 173 files at the source - most of which are hard links - here is the effect of an ls | wc on the destination after five succesive identical runs of rsync on the source (this is in a subdirectory of one of the directories in the command below). The directory did not exist before running the sync. [root@ks-s0-107-1- SC]# ls | wc 49 491286 [root@ks-s0-107-1- SC]# ls | wc 85 852234 [root@ks-s0-107-1- SC]# ls | wc 120 1203243 [root@ks-s0-107-1- SC]# ls | wc 152 1524112 [root@ks-s0-107-1- SC]# ls | wc 173 1734739 So that it seems to have synched 49, then 36, then 35, then 32, the finally the last 21 files in the directory. (The increment seems to vary if I try it again, i.e. from 49 to 90). I get no error mesages (that I can see). In fact, the first time I run the program, it seems to notice all the files and produces 163 messages that file blah-de-blah is a hard link, but then doesn't seem to make the link for file blah-de-blah - this behavior remains constant with each succesive run. Here is the rsync command (generated by a perl script and broken into little pieces by my mailer) rsync -e 'ssh -l root -p 8989' --rsync-path /usr/bin/rsync --stats --progress -tpgoCHr -v -v --include install/ --include install/kickstart/ --include install/kickstart/zantaz_upgrade/ --include install/kickstart/zantaz_upgrade/20011121/ --include install/redhat-7.1/ --include install/redhat-7.1/zantaz_rpm_upgrade/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/ --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/ --include install/kickstart/zantaz_upgrade/20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121/** --include install/redhat-7.1/zantaz_rpm_upgrade/DS05_00_00-SAM-SUN-20011121-devel/** --exclude * /net/swdev/staging/* [EMAIL PROTECTED]:/ Thanks, Dave
include/exclude directory question
I want to send a subset of directories sepcified in include arguments to a client, but I am creating all the peer and parent directories as well, although they are empty - here is basically what I'm doing. assuming I have /staging/upgrade/dir1, /staging/upgrade/dir2 and /staging/upgrade/dir3 on the source tree. --include */ -include upgrade/dir1/** -include upgrade/dir3/** -exclude * /staging/* / what I get on the client is /ugrade/dir1/* /upgrade/dir2 (an empty directory) /upgrade/dir3/* How can I get rsync to filter out and stop creating the empty dir2 directory without specifucally excluding it? I've looked through about a years worth of past postings with likely looking subject lines but can't find the answer. Thanks, Dave
