include/exclude ?
All,
Could someone please help me resolve this:
[admx:test] $ ls
ERR01 ah01 ah02 an01 an02 mp01 mp02
[admx:test] $ ls {an,mp,ERR}*
ERR01 an01 an02 mp01 mp02
I want to rsync only the {an,mp,ERR}* files across using the following
command but do not see the expected results.
[admx:test] $ rsync -va --exclude=* --include={an,mp,ERR}* ./*
mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
I am running 'rsync version 2.3.1 protocol version 20' on a IBM
RS6000/F50 running AIX 4.3.1 on both sides.
Thanks
Nishi
Re: include/exclude ?
rsync doesn't do the odd wildcard syntax you show there... you'll have to
do --include=an* --include=mp* --include=ERR*
Tim Conway
[EMAIL PROTECTED]
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
.\n '
There are some who call me Tim?
Kapoor, Nishikant X [EMAIL PROTECTED]
Sent by: [EMAIL PROTECTED]
11/30/2001 09:09 AM
To: [EMAIL PROTECTED]
cc: (bcc: Tim Conway/LMT/SC/PHILIPS)
Subject:include/exclude ?
Classification:
All,
Could someone please help me resolve this:
[admx:test] $ ls
ERR01 ah01 ah02 an01 an02 mp01 mp02
[admx:test] $ ls {an,mp,ERR}*
ERR01 an01 an02 mp01 mp02
I want to rsync only the {an,mp,ERR}* files across using the following
command but do not see the expected results.
[admx:test] $ rsync -va --exclude=* --include={an,mp,ERR}* ./*
mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
I am running 'rsync version 2.3.1 protocol version 20' on a IBM
RS6000/F50 running AIX 4.3.1 on both sides.
Thanks
Nishi
Any chance of --no-detatch in the next release
This option is required for use on cygwin in server mode - so integrating it into the main code would be very helpful for us cygwinners. It is after all a rather small patch - doesn't clutter up the code that much :-) Max.
Re: open issue regarding local copies and -W
On 4 Sep 2001 21:24:33 +1000, Martin Poole [EMAIL PROTECTED] wrote: On 14 Aug 2001, Dave Dykstra [EMAIL PROTECTED] wrote: Martin, I want to remind you that there's still an open issue regarding copies when both source and destination are on the local machine (including over NFS): Thanks for the reminder. I hope the regression suite will help chase these things out and keep them out. I haven't had time yet to merge and test the nohang patch. I put a patch into the CVS to automatically imply the '-W' option in that case, and somebody said he was concerned about that because he found that with -W on local copies he had cases where rsync would lock up, and it wouldn't without -W (I presume because -W artificially slows it down). Should there be an option to turn -W back off, overriding that new default (or should the new default be taken back out)? There are a few other options that would be nice to be able to negate (--blocking-io comes to mind), maybe there should be a consistent way of specifying them. Should the bug that causes lockup with -W be investigated (or has it already been fixed with the Wayne Davison patches)? I suppose we should have the --no-* option syntax. That's pretty simple. This didn't make it into 2.5.0. Can you take care of it Martin? Maybe you thought I was going to do it. - Dave
Re: cannot create .hosts.b0WX1x : File exists
On Fri, Nov 30, 2001 at 12:08:19AM +0100, Rok Krulec wrote: Hello Dave, What version of sources is that which had mkstemp at line 121 of syscall.c? It's surprising that you could just replace one with the other, as mkstemp is supposed to open the file and mktemp is not supposed to. It sounds like you have some inconsistent version of the sources. rsync --version returns: rsync version 2.4.6 protocol version 24 Rsync 2.4.6 did not use mkstemp at all. A comment in receiver.c says mktemp is deliberately used here instead of mkstemp. and line 120 of syscall.c calls mktemp. I suspect you had a locally hacked version of rsync source in which someone changed the mktemp call to mkstemp, and you just changed it back. I don't have any 2.4 kernel systems with libc-2.2.1, but I tried a 2.4 kernel with a later libc and mktemp and mkstemp worked as expected. Ah, I bet I know what happened. I see that gcc warns when you use mktemp: the use of `mktemp' is dangerous, better use `mkstemp' so somebody person must have thought they could just replace one with the other without looking at the man page. - Dave Dykstra
Re: Why does one of there work and the other doesn't
Martin Pool wrote: Ian Kettleborough [EMAIL PROTECTED] wrote: 1. How much memory does each file to be copied need. Obvisiouly I have too many files. Hard to say exactly. On the order of a hundred bytes per file. I may have misunderstood the question, but maybe we should point out that, on the receiving end, each file needs at least an amount of *disk space* equal in size to the file as a new file is constructed before the old file is deleted. I am not sure which end the 100 bytes per file applies to, and I guess that is the RAM memory footprint?. Does rsync need 100 bytes for each file that might be transferred during a session (all files in the specified directory(ies)), or does it need only 100 bytes as it does one file at a time? Trying to learn, also, Randy Kramer
RE: include/exclude ?
It is finally working but I am not sure if I understand it right. It
seems to me that the order in which 'include' and 'exclude' exist in the
command line is making all the difference. Check out the following two
commands:
*** include is BEFORE exclude - Working fine ! ***
[admx:test] $ rsync -va --include=an* --exclude=* ./* mm2x::AppError
building file list ... done
an01
an02
wrote 178 bytes read 369 bytes 1094.00 bytes/sec
total size is 0 speedup is 0.00
*** include is AFTER exclude - Not Working ! ***
[admx:test] $ rsync -va --exclude=* --include=an* ./* mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
Unless I am missing something obvious, the order is all that is
different in two statements above.
Nishi
-Original Message-
From: [EMAIL PROTECTED] [SMTP:[EMAIL PROTECTED]]
rsync doesn't do the odd wildcard syntax you show there... you'll have
to
do --include=an* --include=mp* --include=ERR*
Tim Conway
[EMAIL PROTECTED]
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970)
,
.\n '
There are some who call me Tim?
Kapoor, Nishikant X [EMAIL PROTECTED]
All,
Could someone please help me resolve this:
[admx:test] $ ls
ERR01 ah01 ah02 an01 an02 mp01 mp02
[admx:test] $ ls {an,mp,ERR}*
ERR01 an01 an02 mp01 mp02
I want to rsync only the {an,mp,ERR}* files across using the
following
command but do not see the expected results.
[admx:test] $ rsync -va --exclude=* --include={an,mp,ERR}* ./*
mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
I am running 'rsync version 2.3.1 protocol version 20' on a IBM
RS6000/F50 running AIX 4.3.1 on both sides.
Thanks
Nishi
RE: include/exclude ?
You're right. the order dependency actually lets you create very complex
include/exclude rules. for each item, each --include and --exclude is
evaluated in commandline order, until the item has either passed all the
tests, and is thus included, or has been excluded, at which point
evaluation restarts with the next item. if you --exclude=* first, every
file gets excluded right away, and never gets checked to see if it matches
--include=an* .
Group: That's a correct synopsis of the behaviour, isn't it? If not,
don't let my error stand.
Tim Conway
[EMAIL PROTECTED]
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970),
.\n '
There are some who call me Tim?
Kapoor, Nishikant X [EMAIL PROTECTED]
11/30/2001 01:40 PM
To: Tim Conway/LMT/SC/PHILIPS@AMEC
Kapoor, Nishikant X [EMAIL PROTECTED]
cc: [EMAIL PROTECTED]
[EMAIL PROTECTED]
Subject:RE: include/exclude ?
Classification:
It is finally working but I am not sure if I understand it right. It
seems to me that the order in which 'include' and 'exclude' exist in the
command line is making all the difference. Check out the following two
commands:
*** include is BEFORE exclude - Working fine ! ***
[admx:test] $ rsync -va --include=an* --exclude=* ./* mm2x::AppError
building file list ... done
an01
an02
wrote 178 bytes read 369 bytes 1094.00 bytes/sec
total size is 0 speedup is 0.00
*** include is AFTER exclude - Not Working ! ***
[admx:test] $ rsync -va --exclude=* --include=an* ./* mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
Unless I am missing something obvious, the order is all that is
different in two statements above.
Nishi
-Original Message-
From: [EMAIL PROTECTED] [SMTP:[EMAIL PROTECTED]]
rsync doesn't do the odd wildcard syntax you show there... you'll have
to
do --include=an* --include=mp* --include=ERR*
Tim Conway
[EMAIL PROTECTED]
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(,
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970)
,
.\n '
There are some who call me Tim?
Kapoor, Nishikant X [EMAIL PROTECTED]
All,
Could someone please help me resolve this:
[admx:test] $ ls
ERR01 ah01 ah02 an01 an02 mp01 mp02
[admx:test] $ ls {an,mp,ERR}*
ERR01 an01 an02 mp01 mp02
I want to rsync only the {an,mp,ERR}* files across using the
following
command but do not see the expected results.
[admx:test] $ rsync -va --exclude=* --include={an,mp,ERR}* ./*
mm2x::AppError
building file list ... done
wrote 67 bytes read 337 bytes 269.33 bytes/sec
total size is 0 speedup is 0.00
I am running 'rsync version 2.3.1 protocol version 20' on a IBM
RS6000/F50 running AIX 4.3.1 on both sides.
Thanks
Nishi
Re: include/exclude ?
On Fri, Nov 30, 2001 at 03:18:16PM -0700, [EMAIL PROTECTED] wrote: You're right. the order dependency actually lets you create very complex include/exclude rules. for each item, each --include and --exclude is evaluated in commandline order, until the item has either passed all the tests, and is thus included, or has been excluded, at which point evaluation restarts with the next item. if you --exclude=* first, every file gets excluded right away, and never gets checked to see if it matches --include=an* . Group: That's a correct synopsis of the behaviour, isn't it? If not, don't let my error stand. Correct except that I would say that it has gone past all the tests rather than has passed. Saying has passed makes it sound like it matched all the patterns when actually you mean that it didn't match any of the patterns. - Dave
RE: Why does one of there work and the other doesn't
From: Randy Kramer [mailto:[EMAIL PROTECTED]] I am not sure which end the 100 bytes per file applies to, and I guess that is the RAM memory footprint?. Does rsync need 100 bytes for each file that might be transferred during a session (all files in the specified directory(ies)), or does it need only 100 bytes as it does one file at a time? Yes, the ~100 bytes is in RAM - I think a key point though is that the storage to hold the file list grows exponentially (doubling each time), so if you have a lot of files in the worst case you can use almost twice as much memory as needed. Here's an analysis I posted to the list a while back that I think is still probably valid for the current versions of rsync - a later followup noted that it didn't include an ~28 byte structure for each entry in the include/exclude list: - - - - - - - - - - - - - - - - - - - - - - - - - (a) How much memory, in bytes/file, does rsync allocate? This is only based on my informal code peeks in the past, so take it with a grain of salt - I don't know if anyone has done a more formal memory analysis. I believe that the major driving factors in memory usage that I can see is: 1. The per-file overhead in the filelist for each file in the system. The memory is kept for all files for the life of the rsync process. I believe this is 56 bytes per file (it's a file_list structure), but a critical point is that it is allocated initially for 1000 files, but then grows exponentially (doubling). So the space will grow as 1000, 2000, 4000, 8000 etc.. until it has enough room for the files necessary. This means you might, worst case, have just about twice as much memory as necessary, but it reduces the reallocation calls quite a bit. At ~56K per 1000 files, if you've got a file system with 1 files in it, you'll allocate room for 16000 and use up 896K. This growth pattern seems to occur on both sender and receiver of any given file list (e.g., I don't see a transfer of the total count over the wire used to optimize the allocation on the receiver). 2. The per-block overhead for the checksums for each file as it is processed. This memory exists only for the duration of one file. This is 32 bytes per file (a sum_buf) allocated as on memory chunk. This exists on the receiver as it is computed and transmitted, and on the sender as it receives it and uses it to match against the new file. 3. The match tables built to determine the delta between the original file and the new file. I haven't looked at closely at this section of code, but I believe we're basically talking about the hash table, which is going to be a one time (during rsync execution) 256K for the tag table and then 8 (or maybe 6 if your compiler doesn't pad the target struct) bytes per block of the file being worked on, which only exists for the duration of the file. This only occurs on the sender. There is also some fixed space for various things - I think the largest of which is up to 256K for the buffer used to map files. (b) Is this the same for the rsyncs on both ends, or is there some asymmetry there? There's asymmetry. Both sides need the memory to handle the lists of files involved. But while the receiver just constructs the checksums and sends them, and then waits for instructions on how to build the new file (either new data or pulling from the old file), the sender also constructs the hash of those checksums to use while walking through the new file. So in general on any given transfer, I think the sender will end up using a bit more memory. (c) Does it matter whether pushing or pulling? Yes, inasmuch as the asymmetry is based on who is sending and who is receiving a given file. It doesn't matter who initiates the contact, but the direction that the files are flowing. This is due to the algorithm (the sender is the component that has to construct the mapping from the new file using portions of the old file as transmitted by the receiver). - - - - - - - - - - - - - - - - - - - - - - - - - -- David /---\ \ David Bolen\ E-mail: [EMAIL PROTECTED] / | FitLinxx, Inc.\ Phone: (203) 708-5192| / 860 Canal Street, Stamford, CT 06902 \ Fax: (203) 316-5150 \ \---/
command= and arbitrary keys...
Hi all, I've asked this question before, but I was never able to fix the problem, and now it's back again and I'd like to try and resolve it. I have an authorized_keys file with about twenty keys, most of which are prefaced with command=/usr/bin/rsync If I put my host key at the top of the authorized_keys file, I can connect without a problem. If it gets put near the bottom, ssh seems to pick an arbitrary key, accepts my passphrase, and starts to run one of the rsync commands. One of the thoughts was that because many of the hosts are running openssh-2.3 and not the latest. If it's likely this is the problem, I'll upgrade them all, but there are a lot of hosts, and I'd rather not do it if it isn't necessary.. Other ideas, perhaps? Thanks, Dave
