Re: Rsync --daemon v 2.5.5 v26 causing kernel panic

[Edward King]

Just did a successful rsync -- problem was with reiser filesystem.

Moved all files off the raid, ran reiserfsck with --rebuild-tree (fix-fixable
was not enough), ran rsync after rebuild finished -- no problems.

There were a number of files that the rebuild-tree found that weren't attached


Edward King wrote:
 
  
  
 After switching much hardware (and getting some helpful suggestions)
I moved the specific machine's files on the backup server to a hard drive
outside the raid (still on the backup server, /dev/hdi1) and tried rsync
-- problem solved.
 
 It seems there's a problem with the journaled filesystems (reiserfs) and 
raid -- reading the files is ok (cp command worked fine), writing them is 
not. 
 
 I'm going to turn on reiser debugging and internal checks and re-run --
maybe I can send something to the reiser or kernel people of interest (don't
want to go back to ext2)
 
 
 jw schultz wrote:
 
 
On Fri, Jan 10, 2003 at 02:25:57PM -0600, Edward King wrote:
  
   
 
  Has anyone seen this?  Looking for past experiences / ideas.  Will post 
progress.

I'm tracking down a problem that seems to be caused by rsync.  When 
moving files from a remote server I get a kernel panic.

We have a number of servers that back up to a main box -- the panic only 
occurs when a specific client backs up.  It occurs on the box it is 
backing up to -- not the client.

The kernel is Linux 2.4.20 (just compiled -- no patches), files are 
stored on a 4 disk raid (80GB Western Digital drives, software raid) 
with Reiserfs.  This is being done over a vpn connection controlled by 
another machine (gateway machine) running tinc so we're not using ssh or 
any other shell on the rsync machine.

I have:

recompiled rsync at both locations
recompiled the kernel from new source code (panic in 2.4.19, system 
rebooted in 2.4.20)

I will:

try different hardware
run a file system check at the main system (the one that crashes)
exclude directories in the backup (rsync one directory at a time -- see 
where it crashes)

I did notice filenames on the client machine that contain control 
characters -- but they seem to have backed up before.

   
   

Just to confirm:  You are doing backup-server initiated pull
something like "rsync server::module destdir" and the
machine you execute this on (the receiver) panics.

Rsync will of course not be the culprit.  However, rsync is
very good at stressing a system and the kernel developers
have found it to be a common test case.

Most likely it is a hardware fault.  Probably timing
sensitive.  I'd check the logs for oopses and and disk
errors.  Also try downgrading the mode with hdparm.

If this isn't a obvious hardware fault you should report it
to the linux-kernel people.  Look on www.kernelnewbies.org 
for instructions.  This will be of interest to the
developers of md, reiserfs and the specific IDE driver.

  
 
 
 

Re: Rsync performance increase through buffering

[Dave Dykstra]

Craig, I'd like to get your patch into the 2.5.6 patches directory.
Could you please make sure it applies cleanly onto version 2.5.6pre1 (see
news on http://rsync.samba.org home page if you haven't been following
the mailing list) and repost it?

Thanks,

- Dave Dykstra

On Wed, Dec 11, 2002 at 03:20:57PM -0600, Dave Dykstra wrote:
 On Sun, Dec 08, 2002 at 11:48:57PM -0800, Craig Barratt wrote:
  I've been studying the read and write buffering in rsync and it turns
  out most I/O is done just a couple of bytes at a time.  This means there
  are lots of system calls, and also most network traffic comprises lots
  of small packets.  The behavior is most extreme when sending/receiving
  file deltas of identical files.
 ...
  Below is a patch to a few files that adds read and write buffering in
  the places where the I/O was unbuffered, adds buffering to write_file()
  and removes the unneeded gettimeofday() system call in show_progress().
 ...
  However, on a network test doing a send from cygwin/WinXP to rsyncd
  on rh-linux the running time improves from about 700 seconds to 215
  seconds (with a cpu load of around 17% versus 58%, if you believe
  cygwin's cpu stats).  This is probably an extreme case since the system
  call penalty in cygwin is high.  But I would suspect a significant
  improvement is possible with a slow network connection, since a lot
  less data is being sent.
 
 Looks like a good candidate for the patches directory now and for
 integration in 2.6.0.
 
 - Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

specifying a list of files to transfer

[Andrew J. Schorr]

Hi,

I don't want to start another --files-from war, but I am attaching
an updated version of my patch to allow you to specify a list
of files to transfer.  The normal rsync syntax allows you to specify
a list of SRC files to transfer on the command line.  This patch
adds some new options to allow you to instead supply a file that
contains a list of files to transfer.

The previous version of the patch was against rsync-2.4.6; this version
works for rsync-2.5.5.  The only real changes relate to the use of
the popt option parsing library in 2.5 (not used in 2.4).  This had
the minor effect of removing the possibility of using - to indicate
stdin since the new library seems to interpret this as an option and
barfs.  So instead I allow the use of /dev/stdin.

By the way, this patch should also work against rsync-2.5.6pre1 except
for a couple of changes relating to white space and comments.  So a couple
of patch hunks are rejected but are easy to fix by hand.  If there is a
need, I can post an updated patch.

Last time we discussed this, Dave Dykstra objected to this patch
for two reasons:

   1. This patch only works in a single direction: when sending from a local
  system to a remote system.  It does not handle the case where you
  are receiving from a remote system to a local system.
   
   2. This capability is possible to achieve by specifying a list
  of files with --include-from and then adding --exclude '*' to
  ignore other files.  While this is true, it turns out to be
  much slower.  I have finally run a performance test to demonstrate
  this.  Results are below.

The basic idea of the patch is to handle the case where you already know
a list of files that might need to be updated and don't want to use
rsync's recursive directory tree scanning logic to enumerate all files.
The patch adds the following options:

 --source-list   SRC arg will be a (local) file name containing a list of 
files, or /dev/stdin
 --null  used with --source-list to indicate that the file names 
will be separated by null (zero) bytes instead of linefeed characters; useful with 
gfind -print0
 --send-dirs send directory entries even though not in recursive mode
 --no-implicit-dirs  do not send implicit directories (parents of the file 
being sent)

The --source-list option allows you to supply an explicit list of filenames
to transport without using the --recursive feature and without playing
around with include and exclude files.  As discussed below, the same
thing can be done by combining --recursive with --include-from and --exclude,
but it's significantly slower and more arcane to do it that way.

The --null flag allows you to handle files with embedded linefeeds.  This
is in the style of gnu find's -print0 operator.

The --send-dirs overcomes a problem where rsync refuses to send directories
unless it's in recursive mode.  One needs this to make sure that even
empty directories get mirrored.

And the --no-implicit-dirs option turns off the default behavior in which
all the parent directories of a file are transmitted before sending the
file.  That default behavior is very inefficient in my scenario where I
am taking the responsibility for sending those directories myself.

And now for a performance test:

I have a directory tree containing 128219 files of which 16064 are
directories.

To start the test, I made a list of files that had changed in the
past day:

   find . -mtime -1 -print  /tmp/changed

(normally, my list of candidate files is generated by some other means,
this is just a test example).  There were 5059 entries in /tmp/changed.

I used my new options to sync up these files to another host
as follows:

  time rsync -RlHptgoD --numeric-ids --source-list \
--send-dirs --no-implicit-dirs -xz --stats /dev/stdin \
remotehost:/extra_disk/tmp/tree1  /tmp/changed

Here were the reported statistics:

 Number of files: 5059
 Number of files transferred: 5056
 Total file size: 355514100 bytes
 Total transferred file size: 355514100 bytes
 Literal data: 355514100 bytes
 Matched data: 0 bytes
 File list size: 139687
 Total bytes written: 154858363
 Total bytes read: 80916

 wrote 154858363 bytes  read 80916 bytes  364992.41 bytes/sec
 total size is 355514100  speedup is 2.29

And the time statistics:

  112.53u 8.82s 7:03.92 28.6%

I then ran the same command again (in which case there was nothing to
transfer).  Here's how long it took:

0.54u 0.62s 0:08.61 13.4%

Now to compare with the recursive method using --include-from.  First, we
must create the list of files.  In the case of include-from, we need to
include all the parent directories as include patterns.  The following
gawk seems to do the job:

  gawk '$0 != ./ {sub(/^\.\//,)} {while ((length  0)  !($0 in already)) 
{print /$0; already[$0] = 1; 

2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf

[Dave Dykstra]

2.5.6pre1 bombs on Sunos4 gcc with these errors
popt/popthelp.c: In function `singleOptionDefaultValue':
popt/popthelp.c:137: invalid operands to binary +
popt/popthelp.c:141: invalid operands to binary +
popt/popthelp.c:145: invalid operands to binary +
popt/popthelp.c:149: invalid operands to binary +
because it's depending on sprintf to return the number of bytes written
and that doesn't happen on Sunos4.

Wayne or anybody, do you have a suggested solution?  The popt maintainers
should also be told about that.  This foible of Sunos4 is mentioned in
the autoconf documentation of things to watch out for but I don't think
they suggest a solution.

We need to get one of these Sunos4 machines into the build farm.  I know
where to get one, but I had been hesitant to add machines to the build
farm because of security concerns.  I think the new rsync module over
ssh feature will eliminate those concerns.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

.rsync-/.rsync+ patch and --link-dest example

[John Bowman]

This is a patch to add an --rsync-exclude option to rsync-2.5.6cvs.  
File names in .rsync- (or .rsync+) are excluded (or included) from the file
lists associated with the current directory and all of its subdirectories.

This has advantages over --cvs-exclude for backing up large file systems
since the .cvsignore files only apply to the current directory:
unless the .cvsignore restrictions apply to the entire tree they must be
duplicated in each subdirectory. In any case, the --cvs-exclude option
isn't intended for general system backups (for example, unless the default
list is cleared with !, it automatically excludes *.a libraries).

The .rsync-/.rsync+ mechanism provided by this simple patch provides a more
general solution, since files may be added to or removed from the exclude
list at all levels in the directory hierarchy (similar to apache .htaccess
files).

This patch also adds an example for using --link-dest to the man page: 

This command creates a linked backup of /home named current on nimbus
relative to a previous backup named previous:

rsync -vax --delete --numeric-ids --link-dest=previous /home nimbus:current

diff -ru rsync-2.5.6cvs/flist.c rsync-2.5.6cvsJ/flist.c
--- rsync-2.5.6cvs/flist.c  Tue Dec 24 00:42:04 2002
+++ rsync-2.5.6cvsJ/flist.c Sat Jan 11 03:47:14 2003
@@ -39,6 +39,7 @@
 extern int always_checksum;
 
 extern int cvs_exclude;
+extern int rsync_exclude;
 
 extern int recurse;
 
@@ -64,6 +65,7 @@
 static char topsrcname[MAXPATHLEN];
 
 static struct exclude_struct **local_exclude_list;
+static struct exclude_struct **recur_local_exclude_list;
 
 static struct file_struct null_file;
 
@@ -260,6 +262,9 @@
if (check_exclude(fname, local_exclude_list, st)) {
return 1;
}
+   if (check_exclude(fname, recur_local_exclude_list, st)) {
+   return 1;
+   }
return 0;
 }
 
@@ -799,8 +804,11 @@
if (S_ISDIR(file-mode)  recursive) {
struct exclude_struct **last_exclude_list =
local_exclude_list;
+   struct exclude_struct **recur_last_exclude_list =
+   recur_local_exclude_list;
send_directory(f, flist, f_name(file));
local_exclude_list = last_exclude_list;
+   recur_local_exclude_list = recur_last_exclude_list;
return;
}
 }
@@ -840,6 +848,29 @@
 
local_exclude_list = NULL;
 
+   if (rsync_exclude) {
+   if (strlen(fname) + strlen(.rsync-) = MAXPATHLEN - 1) {
+   strcpy(p, .rsync-);
+   recur_local_exclude_list =
+   make_exclude_list(fname, recur_local_exclude_list, 0, 0);
+   } else {
+   io_error = 1;
+   rprintf(FINFO,
+   cannot cvs-exclude in long-named directory %s\n,
+   fname);
+   }
+   if (strlen(fname) + strlen(.rsync+) = MAXPATHLEN - 1) {
+   strcpy(p, .rsync+);
+   recur_local_exclude_list =
+   make_exclude_list(fname, recur_local_exclude_list, 0, 1);
+   } else {
+   io_error = 1;
+   rprintf(FINFO,
+   cannot cvs-exclude in long-named directory %s\n,
+   fname);
+   }
+   }
+
if (cvs_exclude) {
if (strlen(fname) + strlen(.cvsignore) = MAXPATHLEN - 1) {
strcpy(p, .cvsignore);
diff -ru rsync-2.5.6cvs/options.c rsync-2.5.6cvsJ/options.c
--- rsync-2.5.6cvs/options.cTue Jan 14 09:35:25 2003
+++ rsync-2.5.6cvsJ/options.c   Tue Jan 14 09:35:40 2003
@@ -45,6 +45,7 @@
 int preserve_times = 0;
 int update_only = 0;
 int cvs_exclude = 0;
+int rsync_exclude = 0;
 int dry_run=0;
 int local_server=0;
 int ignore_times=0;
@@ -237,6 +238,7 @@
   rprintf(F, -e, --rsh=COMMAND   specify the remote shell\n);
   rprintf(F, --rsync-path=PATH   specify path to rsync on the remote 
machine\n);
   rprintf(F, -C, --cvs-exclude   auto ignore files in the same way CVS 
does\n);
+  rprintf(F, --rsync-exclude exclude (include) files listed in .rsync- 
+(.rsync+)\n);
   rprintf(F, --existing  only update files that already exist\n);
   rprintf(F, --ignore-existing   ignore files that already exist on the 
receiving side\n);
   rprintf(F, --deletedelete files that don't exist on the 
sending side\n);
@@ -324,6 +326,7 @@
   {dry-run, 'n', POPT_ARG_NONE,   dry_run , 0, 0, 0 },
   {sparse,  'S', POPT_ARG_NONE,   sparse_files , 0, 0, 0 },
   {cvs-exclude, 'C', POPT_ARG_NONE,   cvs_exclude , 0, 0, 0 },
+  {rsync-exclude,   'C', POPT_ARG_NONE,   rsync_exclude , 0, 0, 0 },
   {update,  'u', POPT_ARG_NONE,   update_only , 0, 0, 0 },
   {links, 

Re: configure issue (ac_cv_lib_inet_connect) on DYNIX/ptx

[Dave Dykstra]

On Mon, Jan 13, 2003 at 09:03:23PM -0500, Michael Sterrett -Mr. Bones.- wrote:
 Greetings -
 
 In trying to get rsync-2.5.6pre1 working on DYNIX/ptx, I found I
 needed to pass ac_cv_lib_inet_connect=no to configure in order for
 it actually create a properly configured config.h file.
 
 I think this is because configure looks for connect in libinet,
 (and finds one there somehow).  But connect() is resolved by libsocket
 (but configure doesn't find it when it looks there).  I've never taken
 the time to delve into the guts of autoconf, so I'm not sure if this
 is a general autoconf problem or a problem that is specific to rsync.
 
 At any rate, hopefully, this will be useful to someone.  I can send
 a config.log file to anyone who's interested.
 
 After passing ac_cv_lib_inet_connect=no to configure, all tests either
 PASS or SKIP so it looks good.

Ok, we'll assume that you'll continue to do it that way unless you post
a patch to configure.in.  Really there should be a modification to 
autoconf, probably an AC_FUNC_CONNECT macro, but I'm not sure anybody
wants to do the work.  I do see a couple mentions of DYNIX in the autoconf
2.53 macros.



 I also get this warning, which is trivial, but it would be nice to
 clean it up:
 
   cleanup.c, line 36: portability warning: trigraph sequence replaced
 
 This is from the () in cleanup.c.  To get rid of the warning,
 just remove the () from line 36.

I don't want to remove it completely because apparently it means something
to the person who put it in.  Please suggest another form that conveys the
same meaning and doesn't cause a warning.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Question

[Jennifer Lu]

Hi,

I got the following error messages when ran rsync2.5.5 on
Solaris5.9, I have used this version for a long time 
until this week, I got the following errors:

ERROR: out of memory in string_area_new buffer
rsync error: error allocating core memory buffers (code 22) at util.c(232)
rsync: connection unexpectedly closed (8 bytes read so far)
rsync error: error in rsync protocol data stream (code 12) at io.c(150)

Is there not enough space on the partition?

Thanks for any help,
Jennifer

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Live DB backups with rsync?

[Steve Mallett]

Is it possible to do live backups of a running DB with rsync???

I was just talking to livevault.com  they (only on win boxes) say they 
can grab changes from a client's live DB  sync them remotely.

It sounded like BS to me, but I'm neither a DB guy or sync'ing expert.

Steve Mallett

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: 2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf

[Dave Dykstra]

On Tue, Jan 14, 2003 at 10:24:04AM -0600, Dave Dykstra wrote:
 2.5.6pre1 bombs on Sunos4 gcc with these errors
 popt/popthelp.c: In function `singleOptionDefaultValue':
 popt/popthelp.c:137: invalid operands to binary +
 popt/popthelp.c:141: invalid operands to binary +
 popt/popthelp.c:145: invalid operands to binary +
 popt/popthelp.c:149: invalid operands to binary +
 because it's depending on sprintf to return the number of bytes written
 and that doesn't happen on Sunos4.

If nobody objects to the following patch, I'll put it in.  It's a little
slower but this is definitely not time-critical code.

- Dave


--- popt/popthelp.c.O   Tue Jan 14 13:02:47 2003
+++ popt/popthelp.c Tue Jan 14 13:04:45 2003
@@ -134,19 +134,23 @@
 case POPT_ARG_VAL:
 case POPT_ARG_INT:
 {  long aLong = *((int *)opt-arg);
-   le += sprintf(le, %ld, aLong);
+   sprintf(le, %ld, aLong);
+   le += strlen(le);
 }  break;
 case POPT_ARG_LONG:
 {  long aLong = *((long *)opt-arg);
-   le += sprintf(le, %ld, aLong);
+   sprintf(le, %ld, aLong);
+   le += strlen(le);
 }  break;
 case POPT_ARG_FLOAT:
 {  double aDouble = *((float *)opt-arg);
-   le += sprintf(le, %g, aDouble);
+   sprintf(le, %g, aDouble);
+   le += strlen(le);
 }  break;
 case POPT_ARG_DOUBLE:
 {  double aDouble = *((double *)opt-arg);
-   le += sprintf(le, %g, aDouble);
+   sprintf(le, %g, aDouble);
+   le += strlen(le);
 }  break;
 case POPT_ARG_STRING:
 {  const char * s = *(const char **)opt-arg;
@@ -271,7 +275,8 @@
*le++ = '=';
if (negate) *le++ = '~';
/*@-formatconst@*/
-   le += sprintf(le, (ops ? 0x%lx : %ld), aLong);
+   sprintf(le, (ops ? 0x%lx : %ld), aLong);
+   le += strlen(le);
/*@=formatconst@*/
*le++ = ']';
}   break;
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Live DB backups with rsync?

[Jan-Benedict Glaw]

On Tue, 2003-01-14 15:44:26 -0400, Steve Mallett [EMAIL PROTECTED]
wrote in message [EMAIL PROTECTED]:
 Is it possible to do live backups of a running DB with rsync???
 
 I was just talking to livevault.com  they (only on win boxes) say they 
 can grab changes from a client's live DB  sync them remotely.
 
 It sounded like BS to me, but I'm neither a DB guy or sync'ing expert.

I think that's BS (or to name it for non-natice english speakers: bull
shit:-). Even if you had somethink like LVM to take a snapshot, the
database will be in some inconsistent state. There are only two (well,
three) sane ways of backing up a DB:

- Shutdown, then take a backup (or take a LVM snapshot and
  restart immediately afterwards).
- Use some backup functionality the database server itself
  offers
- Use a combination of both - some DB servers are IIRC capable
  of not touching some primary DB files/storage areas for some
  time but collecting any changes outside

MfG, JBG

-- 
   Jan-Benedict Glaw   [EMAIL PROTECTED]. +49-172-7608481
   Eine Freie Meinung in  einem Freien Kopf| Gegen Zensur
fuer einen Freien Staat voll Freier Bürger | im Internet!
   Shell Script APT-Proxy: http://lug-owl.de/~jbglaw/software/ap2/



msg06069/pgp0.pgp
Description: PGP signature

Re: The behavior of -u/--update option on directories

[Aaron W Morris]

On Mon, 2003-01-13 at 22:46, Florian-Daniel Otel wrote:
 
 Aaron,
 
 Aaron Morris writes:
  rsync is just doing what your are telling it to do.  Update if the file 
  is changed or does not exist on the remote side and delete if it no 
  longer exists on the local side.  
 
 Yes, I can RTFM, and, suprisingly!, I  even _did_ RTFM :)).
 
  The directory may have a newer 
  timestamp, but you are doing a recursive put so it has to check all the 
  files and dirs under each directory.  At least that is how I see it.
  
  solution:  --exclude /tmp/   or   --exclude tmp/
 
 Well, I think you missed the point. The tmp subdir was just an example. The
 same can happen in any subdir down the tree. And, if you re-read my
 post carefully (or even the subject), I said --update (mis)behavior
 __on_directories__ (and _not_ regular files).
 
 Truth to be told, after a bit of thinking I _might_ accept the fact
 that --update ignores the timestamps on (sub)directories. Because
 if it didn't, if any file was touch-ed remotely after the last rsync
 than all subdirs,  up to and including to the top dir, would have a
 newer timestamp remotely and thus no file putting would take place.

This is usually the case, however, not always true.  It is possible for
a file to have a newer timestamp then the directory it is contained
within.  If we used your logic below, then a file with a newer timestamp
(possibly changed data) on the source would not be transferred to the
remote machine if the source directory's timestamp did not change.

 
 However, I still think that --update (or, ideally,  another similar
 option or combination of options)  __should__ take into account
 (sub)directories timestamps and consider for processing only those
 (sub)directories that have remotely an older timestamp than
 locally. 
 
 Or, to put it in your words ...doing a recursive put should
 be restricted __only__ to the (sub)directories that remotely have a
 timestamp older than locally. If that means that _no_ update
 (i.e. put) will take place because __any__ file deep down the tree was
 touched remotely since the last put...well, than that's it.
 
 Again, thanks in advance to  anyone can suggest a way to have rsync
 __also__ in this way.
 
 Best regards,
 
 Florian
 
 
  Florian-Daniel Otel wrote:
   Dear all,
   
   I have the following problem: I use the following command to push
   files from local to a remote machine:
   
   [...]
   /usr/bin/rsync -avuz -e ssh -1 --exclude .Xauthority  --delete  
/user/home/directory/  [EMAIL PROTECTED]:/user/home/directory/ ;
   
   (The ssh is using RSA authentication btw. machines,  but that's beside the 
point).
   
   The problem:
   
   
   On both the local and remote machines there is a subdirectory
   /user/home/directory/tmp. If on the remote machine that subdirectory
   has a newer time stamp  -- e.g. due to a file that was created
   recently i.e. after the last time the local machine did a pull --  the
   --delete option in the command above makes it such that the said
   newer file is deleted on remote and the timestamp is reseted back to
   the old time stamp i.e. the one for the tmp/ subdir on the local machine.
   
   
   Putting it shortly, the --delete option has precendence over -u
   option w.r.t. directories. 
   
   Is it so that -u works only on files and not on directories ?
   IMHO the appropriate behaviour in a situation like the one described
   above directories on remote that have newer time stamp  should be ignored 
   and excluded from further processing (i.e. including updating and/or
   deleting any files that do not exist on the local machine).
   
   The question:
   
   Am I right and that is the way things should behave and I am doing
   smth wrong ? If not, and this is the way things are supposed to work,
   can anyone suggest a work-around that would simulate the desired behavior ?
   
   
   
   Many thanks in advance,
   
   Florian
   
   P.S. Please Cc: me on the replies, I'm not on the list. Thanks. 
   
   P.P.S: If it matters: In the above setup the local machine is a
   Linux running rsync  version 2.5.4  protocol version 26 whereas
   remote is a Solaris 2.8 running rsync  version 2.5.5  protocol version 26. 
   
  
  -- 
  Aaron W Morris
  decep
  PGP Key ID:  259978D1
  
  
-- 
Aaron W Morris
decep
PGP Key:  259978D1
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Wayne Davison]

On Tue, Jan 14, 2003 at 11:02:44AM -0500, Andrew J. Schorr wrote:
 I am attaching an updated version of my patch to allow you to specify
 a list of files to transfer.

Cool.  I'm looking into making this work when fetching files.  Towards
that end, I'd like to suggest an alternate command-line syntax to make
the --source-file option take a filename.  This will allow it to accept
- as stdin, and will make it easy to parse for the pull syntax.  This
means that we need to omit the SRC spec on a push or specify it as
empty.  E.g. these will all work:

  rsync --source-list=file remote:/path
  rsync --source-list file : remote:/path
  rsync --source-list=-  remote:/path file

A pull looks like this:

  rsync --source-list=file remote: /path
  rsync --source-list - remote::module /path file

What do people think?  Of course this is not for the rsync release we're
currently working on, but could be included as a patch, if desired.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Dave Dykstra]

On Tue, Jan 14, 2003 at 01:21:29PM -0800, Wayne Davison wrote:
 On Tue, Jan 14, 2003 at 11:02:44AM -0500, Andrew J. Schorr wrote:
  I am attaching an updated version of my patch to allow you to specify
  a list of files to transfer.
 
 Cool.  I'm looking into making this work when fetching files.  Towards
 that end, I'd like to suggest an alternate command-line syntax to make
 the --source-file option take a filename.  This will allow it to accept
 - as stdin, and will make it easy to parse for the pull syntax.  This
 means that we need to omit the SRC spec on a push or specify it as
 empty.  E.g. these will all work:
 
   rsync --source-list=file remote:/path
   rsync --source-list file : remote:/path
   rsync --source-list=-  remote:/path file
 
 A pull looks like this:
 
   rsync --source-list=file remote: /path
   rsync --source-list - remote::module /path file
 
 What do people think?  Of course this is not for the rsync release we're
 currently working on, but could be included as a patch, if desired.

I haven't looked at the implementation, but comments on the user
interface:
1. Yes it should take a filename or - as a parameter.
2. I don't like the idea of skipping the SRC spec.  Paths should be
relative to the SRC.  If somebody wants to use full paths they
can always have a SRC of /.
3. It should be called --files-from.
4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
they should be automatically turned on with the --files-from option.

- Dave
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Andrew J. Schorr]

On Tue, Jan 14, 2003 at 03:32:41PM -0600, Dave Dykstra wrote:
 I haven't looked at the implementation, but comments on the user
 interface:
 1. Yes it should take a filename or - as a parameter.
 2. I don't like the idea of skipping the SRC spec.  Paths should be
   relative to the SRC.  If somebody wants to use full paths they
   can always have a SRC of /.
 3. It should be called --files-from.
 4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
   they should be automatically turned on with the --files-from option.

Those comments all sound reasonable to me.  The only reason I broke
out the --send-dirs and --no-implicit-dirs options was because they
were orthogonal to what I was doing and could potentially also apply
to situations where the user was specifing various SRC filenames
on the command line.  But it's certainly fine to have --files-from turn
those on automatically.

-Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: The behavior of -u/--update option on directories

[Florian-Daniel Otel]

Aaron W Morris writes:
  
  Truth to be told, after a bit of thinking I _might_ accept the fact
  that --update ignores the timestamps on (sub)directories. Because
  if it didn't, if any file was touch-ed remotely after the last rsync
  than all subdirs,  up to and including to the top dir, would have a
  newer timestamp remotely and thus no file putting would take place.
 
 This is usually the case, however, not always true.  It is possible for
 a file to have a newer timestamp then the directory it is contained
 within.  If we used your logic below, then a file with a newer timestamp
 (possibly changed data) on the source would not be transferred to the
 remote machine if the source directory's timestamp did not change.

Hmm..well, while a possible scenario for that to happen doesn't come
right off the bat, I cannot disagree with you there. Still, I find
this as a rather shoddy justification for the present behavior of the
--update (i.e. the fact that in effect the timestamps on
(sub)directories are practically ignored). Frankly, I would rather
want to have the choice.

To get a bit more constructive: Any idea how can I simulate the
behavior I want ? Or am I simply left w/ the option (sic!) of voicing
this as an (exotic ?!?!?) wishlist item  :) ?

Thanks anyway,

Florian
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync hanging with openssh-2.9.9p2 as the transport

[Andrew J. Schorr]

Hi,

This is perhaps a stupid question.  I apologize in advance if it's already
been covered, but I'm stumped...

I'm using rsync to backup some file systems to a remote host.
The transport is openssh-2.9.9p2.

The problem does not occur when the transport is rsh.

I'm invoking rsync as follows:

  rsync -RlHptgoD --numeric-ids -e ssh -rzx --stats . remotehost:/nfs/mirror

But it hangs when there are large numbers of files in the . filesystem.
If I run with -vvv, I see lots of make_file entries as send_file_list
starts sending the list of files to the remote host, but then it freezes
after around 4500 to 6500 make_file messages.

I see the same problem with versions 2.4.6, 2.5.5, and 2.5.6pre1.
And I see it on linux 2.4.18 (red hat 8.0) and solaris 8/x86.

I can get it to work by breaking it up into chunks of 1000 files or
so.  It might work with more, I haven't tested exhaustively.

I have tried using the --blocking-io and --no-blocking-io options, but
neither one solves the problem.

I could provide more info about where it hangs, but I thought somebody
might know the answer, since this is clearly related to the
interaction with openssh (since there's no problem with rsh).

Is there a trick to make rsync work nicely with openssh?  I searched the
archives and haven't found anything...  Does upgrading to openssh 3 solve
the problem?

Thanks,
Andy
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync] Live DB backups with rsync?

[R P Herrold]

On Tue, 14 Jan 2003, Steve Mallett wrote:

 Is it possible to do live backups of a running DB with rsync???
 
 I was just talking to livevault.com  they (only on win boxes) say they 
 can grab changes from a client's live DB  sync them remotely.
 
 It sounded like BS to me, but I'm neither a DB guy or sync'ing expert.

I've done with unix DB, DB4, and MySQL backends [not of
choice, but sometimes, there's no other way, as when a hard
drive is dying, and you _know_ you will never be able to power
it on again ...] -- it works this way --

1.  Rsync the directory tree holding the database
2.  Re-run the command
3.  Keep re-running it, and eventually, you'll get an 
essentially immediate return with no content being moved.

... at this point stop.  You hit a time when the database was 
largely quiescent, and so was unchanged -- 

-- there will be some open locks and and open transactons will 
be lost -- but no worse than an unplanned reboot due to a 
power loss.

In a production environment, it seems to me that better ways 
usually exist.

-- Russ Herrold

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: .rsync-/.rsync+ patch and --link-dest example

[jw schultz]

On Tue, Jan 14, 2003 at 09:48:31AM -0700, John Bowman wrote:
 This is a patch to add an --rsync-exclude option to rsync-2.5.6cvs.  
 File names in .rsync- (or .rsync+) are excluded (or included) from the file
 lists associated with the current directory and all of its subdirectories.
 
 This has advantages over --cvs-exclude for backing up large file systems
 since the .cvsignore files only apply to the current directory:
 unless the .cvsignore restrictions apply to the entire tree they must be
 duplicated in each subdirectory. In any case, the --cvs-exclude option
 isn't intended for general system backups (for example, unless the default
 list is cleared with !, it automatically excludes *.a libraries).
 
 The .rsync-/.rsync+ mechanism provided by this simple patch provides a more
 general solution, since files may be added to or removed from the exclude
 list at all levels in the directory hierarchy (similar to apache .htaccess
 files).

An interesting idea.  Would be worthwhile putting in the
patches directory and solicit feedback to see how much use it gets.

You also bring up a good point about the --cvs-exclude
option.  In excluding *.a and *.so it limits its utility.
This should be noted in the manpage.

One wonders if perhaps a subset of --cvs-exclude's fixed
patterns might be worth having as an --bk-exclude option.
Perhaps *~ #* .#* ,* *.old *.bak *.BAK core
Purely as a convenience.

 This patch also adds an example for using --link-dest to the man page: 
 
 This command creates a linked backup of /home named current on nimbus
 relative to a previous backup named previous:
 
 rsync -vax --delete --numeric-ids --link-dest=previous /home nimbus:current

I don't know if that is needed.  Lacking context the example
has minimal meaning and compare-dest doesn't have an example
either.

Should be a separate patch.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: 2.5.6pre1 bombs on Sunos4 in popthelp.c on use of sprintf

[jw schultz]

On Tue, Jan 14, 2003 at 11:20:43AM -0600, Dave Dykstra wrote:
 On Tue, Jan 14, 2003 at 10:24:04AM -0600, Dave Dykstra wrote:
  2.5.6pre1 bombs on Sunos4 gcc with these errors
  popt/popthelp.c: In function `singleOptionDefaultValue':
  popt/popthelp.c:137: invalid operands to binary +
  popt/popthelp.c:141: invalid operands to binary +
  popt/popthelp.c:145: invalid operands to binary +
  popt/popthelp.c:149: invalid operands to binary +
  because it's depending on sprintf to return the number of bytes written
  and that doesn't happen on Sunos4.
 
 If nobody objects to the following patch, I'll put it in.  It's a little
 slower but this is definitely not time-critical code.

It is a bit ugly but i don't have a problem with the
performance here.

I'd be concerned about someone not knowing about the Sunos4
limitation messing with this though.  At a minimum this
needs comments.  Even better would be if we can manage a way
to conditionally have our own sprintf.  I've a vague nagging
that it can be done but can't recall how.

 
 --- popt/popthelp.c.O Tue Jan 14 13:02:47 2003
 +++ popt/popthelp.c   Tue Jan 14 13:04:45 2003
 @@ -134,19 +134,23 @@
  case POPT_ARG_VAL:
  case POPT_ARG_INT:
  {long aLong = *((int *)opt-arg);
 - le += sprintf(le, %ld, aLong);
 + sprintf(le, %ld, aLong);
 + le += strlen(le);
  }break;
  case POPT_ARG_LONG:
  {long aLong = *((long *)opt-arg);
 - le += sprintf(le, %ld, aLong);
 + sprintf(le, %ld, aLong);
 + le += strlen(le);
  }break;
  case POPT_ARG_FLOAT:
  {double aDouble = *((float *)opt-arg);
 - le += sprintf(le, %g, aDouble);
 + sprintf(le, %g, aDouble);
 + le += strlen(le);
  }break;
  case POPT_ARG_DOUBLE:
  {double aDouble = *((double *)opt-arg);
 - le += sprintf(le, %g, aDouble);
 + sprintf(le, %g, aDouble);
 + le += strlen(le);
  }break;
  case POPT_ARG_STRING:
  {const char * s = *(const char **)opt-arg;
 @@ -271,7 +275,8 @@
   *le++ = '=';
   if (negate) *le++ = '~';
   /*@-formatconst@*/
 - le += sprintf(le, (ops ? 0x%lx : %ld), aLong);
 + sprintf(le, (ops ? 0x%lx : %ld), aLong);
 + le += strlen(le);
   /*@=formatconst@*/
   *le++ = ']';
   }   break;
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Question

[jw schultz]

On Tue, Jan 14, 2003 at 11:11:20AM -0800, Jennifer Lu wrote:
 
 
 Hi,
 
 I got the following error messages when ran rsync2.5.5 on
 Solaris5.9, I have used this version for a long time 
 until this week, I got the following errors:
 
 ERROR: out of memory in string_area_new buffer
 rsync error: error allocating core memory buffers (code 22) at util.c(232)
 rsync: connection unexpectedly closed (8 bytes read so far)
 rsync error: error in rsync protocol data stream (code 12) at io.c(150)
 
 Is there not enough space on the partition?

You ran out of memory.  Malloc failed. 

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Wayne Davison]

On Tue, Jan 14, 2003 at 03:32:41PM -0600, Dave Dykstra wrote:
 1. Yes it should take a filename or - as a parameter.
 2. I don't like the idea of skipping the SRC spec.  Paths should be
   relative to the SRC.  If somebody wants to use full paths they
   can always have a SRC of /.
 3. It should be called --files-from.
 4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
   they should be automatically turned on with the --files-from option.

OK, I'm also fine with these points.  Note RE comment #2: even though
the relative path names now default to the SRC dir, the user can still
include absolute path names in the list and rsync will transfer them
without problem.  Also, I think the older implementation of --files-from
implied the -R (--relative) option, and this implementation does not.

So, here's a *VERY EARLY* implementation that can transfer files in
either direction.  It adds the option --files-from=FILE and the option
--null (for null-terminated names).  FILE can be - for stdin.  This
patch is relative to the CVS version, and is only for those that want to
assist in implementation, design, and/or testing.  **I have not tested
daemon mode at all yet, just simple ssh transfers in both directions.**

Compatibility note:  when pushing files, the --files-from mode will work
with any older version of rsync that we can transfer files with.  When
pulling files, the remote rsync must understand the --files-from=-
option (which tells it to read the file list over the stdin-socket since
it's combined with the --server option).

Aside:  there was a huge chunk of code in main.c that was not indented
correctly (due to the addition of some read_batch stuff).  I didn't want
to march the code off the edge of the screen any further, so I made the
read_batch code use a goto.  Those that have a weak stomach may wish to
avert their gaze from that portion of the patch.

..wayne..

Index: flist.c
--- flist.c 24 Dec 2002 07:42:04 -  1.127
+++ flist.c 14 Jan 2003 23:44:21 -
@@ -41,6 +41,8 @@
 extern int cvs_exclude;
 
 extern int recurse;
+extern char *files_from;
+extern int files_from_fd;
 
 extern int one_file_system;
 extern int make_backups;
@@ -680,7 +682,7 @@
if (noexcludes)
goto skip_excludes;
 
-   if (S_ISDIR(st.st_mode)  !recurse) {
+   if (S_ISDIR(st.st_mode)  !recurse  !files_from) {
rprintf(FINFO, skipping directory %s\n, fname);
return NULL;
}
@@ -876,12 +878,13 @@
  **/
 struct file_list *send_file_list(int f, int argc, char *argv[])
 {
-   int i, l;
+   int l;
STRUCT_STAT st;
char *p, *dir, *olddir;
char lastpath[MAXPATHLEN] = ;
struct file_list *flist;
int64 start_write;
+   int use_ff_fd = 0;
 
if (show_filelist_p()  f != -1)
start_filelist_progress(building file list);
@@ -890,16 +893,33 @@
 
flist = flist_new();
 
-   if (f != -1) {
+   if (f != -1)
io_start_buffering(f);
+
+   if (files_from  f != -1) {
+   if (!push_dir(argv[0], 0)) {
+   rprintf(FERROR, push_dir %s : %s\n,
+   argv[0], strerror(errno));
+   exit_cleanup(RERR_FILESELECT);
+   }
+   use_ff_fd = 1;
}
 
-   for (i = 0; i  argc; i++) {
+   while (1) {
char *fname = topsrcname;
 
-   strlcpy(fname, argv[i], MAXPATHLEN);
+   if (use_ff_fd) {
+   l = read_filesfrom_line(files_from_fd, fname);
+   if (!l)
+   break;
+   }
+   else {
+   if (argc-- == 0)
+   break;
+   strlcpy(fname, *argv++, MAXPATHLEN);
+   l = strlen(fname);
+   }
 
-   l = strlen(fname);
if (l != 1  fname[l - 1] == '/') {
if ((l == 2)  (fname[0] == '.')) {
/*  Turn ./ into just . rather than ./.
@@ -922,7 +942,7 @@
continue;
}
 
-   if (S_ISDIR(st.st_mode)  !recurse) {
+   if (S_ISDIR(st.st_mode)  !recurse  !files_from) {
rprintf(FINFO, skipping directory %s\n, fname);
continue;
}
@@ -940,7 +960,7 @@
dir = fname;
fname = p + 1;
}
-   } else if (f != -1  (p = strrchr(fname, '/'))) {
+   } else if (f != -1  !files_from  (p=strrchr(fname,'/'))) {
/* this ensures we send the intermediate directories,
   thus getting their permissions right */
*p = 0;
@@ -1034,6 +1054,9 @@
 
if 

Re: Live DB backups with rsync?

[jw schultz]

On Tue, Jan 14, 2003 at 03:44:26PM -0400, Steve Mallett wrote:
 Is it possible to do live backups of a running DB with rsync???
 
 I was just talking to livevault.com  they (only on win boxes) say they 
 can grab changes from a client's live DB  sync them remotely.
 
 It sounded like BS to me, but I'm neither a DB guy or sync'ing expert.

On it's face It sounds like worse the BS.  Depending on
which BS you can either grow roses or enjoy the way we
were.

Generally MS OSs you can't even rsync an open file.  As
such, to sync tablespaces you have to shut down the
database.

It could be they are using something like debase where you
use a set of libraries to access the data files directly and
there is no back end running.  The files are only accessed
during transactions so you could squeeze an rsync in
between accesses.  I believe the the MS Access DB[sic] works
this way.

A service oriented database (Oracle, SQLserver, pgsql,
mysql, etc) keeps the files open all the time and even if
you can read them the database will be inconsistent.

What you may be able to do is to rsync the commit log.  This
is a common practice in *IX systems.  On windows it may be
problematic doing this because of locking on open files.
This method does work and by replaying the logs whenever
they change can be used to keep a running copy of the
database almost up-to-date to allow for a limited (the most
recent transactions may be lost) fail-over capability.



-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[jw schultz]

On Tue, Jan 14, 2003 at 03:57:51PM -0800, Wayne Davison wrote:
 On Tue, Jan 14, 2003 at 03:32:41PM -0600, Dave Dykstra wrote:
  1. Yes it should take a filename or - as a parameter.
  2. I don't like the idea of skipping the SRC spec.  Paths should be
  relative to the SRC.  If somebody wants to use full paths they
  can always have a SRC of /.
  3. It should be called --files-from.
  4. --send-dirs and --no-implicit-dirs shouldn't be separate options,
  they should be automatically turned on with the --files-from option.
 
 OK, I'm also fine with these points.  Note RE comment #2: even though
 the relative path names now default to the SRC dir, the user can still
 include absolute path names in the list and rsync will transfer them

Absolute paths are bad news here.  Especially when
dealing with an rsync daemon.  This allows the user to
defeat any location restrictions.  Not only working outside
the module of an rsync daemon but also the kinds of
restrictions that someone might set up using command=
wrappers in ssh.

 without problem.  Also, I think the older implementation of --files-from
 implied the -R (--relative) option, and this implementation does not.
 
 So, here's a *VERY EARLY* implementation that can transfer files in
 either direction.  It adds the option --files-from=FILE and the option
 --null (for null-terminated names).  FILE can be - for stdin.  This
 patch is relative to the CVS version, and is only for those that want to
 assist in implementation, design, and/or testing.  **I have not tested
 daemon mode at all yet, just simple ssh transfers in both directions.**
 
 Compatibility note:  when pushing files, the --files-from mode will work
 with any older version of rsync that we can transfer files with.  When
 pulling files, the remote rsync must understand the --files-from=-
 option (which tells it to read the file list over the stdin-socket since
 it's combined with the --server option).

That's good at least temporarily, probably permanently.

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Newbie question - does it work with Sun Cobalt RaQ550?

[Majorosi.net - Stéphane]

Hi,

I'd like to know if RSYNC work on a Sun Cobalt 
RaQ550. Does someone trid it? If yes, please let me know if I can take contact 
with you.

Stephane

Re: specifying a list of files to transfer

[Wayne Davison]

On Tue, Jan 14, 2003 at 04:35:40PM -0800, jw schultz wrote:
 Absolute paths are bad news here.  Especially when dealing with an
 rsync daemon.

Yes, this is something that needs to be dealt with for daemon mode since
it does not appear to have been possible to specify multiple filenames
to pull before (unlike remote-shell mode).

For non-daemon mode, the code is the same as it always was in this
regard.  For example, this command:

rsync -av /tmp/one /foo/two /bar/three dest:

is no different than this command:

rsync -av --files-from=list /tmp dest:

where list contains:

one
/foo/two
/bar/three

In the patch I posted earlier, daemon mode did not work with the new
--from-files option.  My latest patch has this fixed:

http://www.clari.net/~wayne/rsync-files-from.patch

And it also runs the filenames through sanitize_path() in daemon mode
(when chroot is not specified, at least -- I haven't tested a chroot
version yet).

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[jw schultz]

On Tue, Jan 14, 2003 at 06:41:22PM -0800, Wayne Davison wrote:
 On Tue, Jan 14, 2003 at 04:35:40PM -0800, jw schultz wrote:
  Absolute paths are bad news here.  Especially when dealing with an
  rsync daemon.
 
 Yes, this is something that needs to be dealt with for daemon mode since
 it does not appear to have been possible to specify multiple filenames
 to pull before (unlike remote-shell mode).
 
 For non-daemon mode, the code is the same as it always was in this
 regard.  For example, this command:
 
 rsync -av /tmp/one /foo/two /bar/three dest:
 
 is no different than this command:
 
 rsync -av --files-from=list /tmp dest:
 
 where list contains:
 
 one
 /foo/two
 /bar/three

So in dest: you get
one
two
three

and if /foo/two and /bar/three are directories they are
recoursed due to -a ?

If so that would be OK, security wise for a push.

But we don't want
rsync -av --files-from=list source:dir /tmp
to allow pulling from source:/foo/two or source:/bar/three

Up till now rsync hasn't touched anything outside of the
paths specified on the command-line.  Changing that would
mean access to rsync via ssh would no longer be
restricted, just disabled.

Sanitizing the paths to force them to be relative on pulls
but not pushes would be too asymetrical for my liking.  I'd
rather just disallow or sanitize absolute paths.

 
 In the patch I posted earlier, daemon mode did not work with the new
 --from-files option.  My latest patch has this fixed:
 
 http://www.clari.net/~wayne/rsync-files-from.patch
 
 And it also runs the filenames through sanitize_path() in daemon mode
 (when chroot is not specified, at least -- I haven't tested a chroot
 version yet).

chroot changes the whole meaning of absolute paths anyway.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Lee Eakin]

FYI, pulling multiple files from a daemon currently supported (well, it
works).  Given a package of foo you can specify:

  rsync -av 'remote::foo/file1 foo/file5' /tmp

It appears the daemon does proper splitting based on either white-space,
or possibly the current value of $IFS in the daemon's environment?

One other note, I did not determine whether it was a Solaris issue, or
string length limit, or file argument limit, but in my tests I could only
specify about 20 files using this method.  When I went over the limit no
files were xfered.

I was testing this a while back (just to see if I could), so I don't
remember the exact limit, but I am fairly sure I experimented with shorter
pathnames and it did not effect the max filenames I could specify.

Oh, yes.  I did not have the same limitation over ssh.  The remote shell
seems to pass any number of filenames to the remote end (of course there
may be limits depending on what login shell is used on the remote server).
  -Lee

---begin quoted text---
 From: Wayne Davison [EMAIL PROTECTED]
 To: jw schultz [EMAIL PROTECTED]
 Cc: [EMAIL PROTECTED]
 Subject: Re: specifying a list of files to transfer
 User-Agent: Mutt/1.3.28i
 X-BeenThere: [EMAIL PROTECTED]
 X-Mailman-Version: 2.0.13
 X-Original-Date: Tue, 14 Jan 2003 18:41:22 -0800
 Date: Tue, 14 Jan 2003 18:41:22 -0800
 
 On Tue, Jan 14, 2003 at 04:35:40PM -0800, jw schultz wrote:
  Absolute paths are bad news here.  Especially when dealing with an
  rsync daemon.
 
 Yes, this is something that needs to be dealt with for daemon mode since
 it does not appear to have been possible to specify multiple filenames
 to pull before (unlike remote-shell mode).
 
 For non-daemon mode, the code is the same as it always was in this
 regard.  For example, this command:
 
 rsync -av /tmp/one /foo/two /bar/three dest:
 
 is no different than this command:
 
 rsync -av --files-from=list /tmp dest:
 
 where list contains:
 
 one
 /foo/two
 /bar/three
 
 In the patch I posted earlier, daemon mode did not work with the new
 --from-files option.  My latest patch has this fixed:
 
 http://www.clari.net/~wayne/rsync-files-from.patch
 
 And it also runs the filenames through sanitize_path() in daemon mode
 (when chroot is not specified, at least -- I haven't tested a chroot
 version yet).
 
 ..wayne..
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
---end quoted text---

-- 
Lee Eakin - [EMAIL PROTECTED]
 
With sufficient thrust, pigs fly just fine.  -- RFC 1925
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Wayne Davison]

On Tue, Jan 14, 2003 at 07:02:58PM -0800, jw schultz wrote:
 Up till now rsync hasn't touched anything outside of the paths
 specified on the command-line.  Changing that would mean access to
 rsync via ssh would no longer be restricted, just disabled.

Are you saying that some people have special ssh scripts that check
and/or tweak the file names on the command-line to ensure they fall with
certain bounds when running rsync commands?  I.e., if someone ran this
command:

rsync -av -e ssh source:dir /foo/two /bar/three /tmp

the remote ssh setup would handle the presence of the extra /foo/two,
/bar/three args?  If so, I hadn't realized that people were limiting
ssh access by more than the traditional user/group/permissions access.

 Sanitizing the paths to force them to be relative on pulls
 but not pushes would be too asymetrical for my liking.

I agree that if we find that we want to sanitize the paths in some cases
that we should just make it the default for files-from -- i.e. make it
where nothing can get beyond the root dir specified on the command-line.

 I'd rather just disallow or sanitize absolute paths.

Note that it's more pervasive than just absolute paths, since someone
can use args like ../../../etc/password or good_dir/../../bad_dir
(all of which the sanitize_path() call handles).

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Wayne Davison]

On Tue, Jan 14, 2003 at 09:15:02PM -0600, Lee Eakin wrote:
 FYI, pulling multiple files from a daemon currently supported (well, it
 works).  Given a package of foo you can specify:
 
   rsync -av 'remote::foo/file1 foo/file5' /tmp

Oh!  I had left off the repeat of the module name when I tried to cajole
the daemon using this kludge.  Thanks for the info.

..wayne..
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[jw schultz]

On Tue, Jan 14, 2003 at 07:39:49PM -0800, Wayne Davison wrote:
 On Tue, Jan 14, 2003 at 07:02:58PM -0800, jw schultz wrote:
  Up till now rsync hasn't touched anything outside of the paths
  specified on the command-line.  Changing that would mean access to
  rsync via ssh would no longer be restricted, just disabled.
 
 Are you saying that some people have special ssh scripts that check
 and/or tweak the file names on the command-line to ensure they fall with
 certain bounds when running rsync commands?  I.e., if someone ran this
 command:
 
 rsync -av -e ssh source:dir /foo/two /bar/three /tmp
 
 the remote ssh setup would handle the presence of the extra /foo/two,
 /bar/three args?  If so, I hadn't realized that people were limiting
 ssh access by more than the traditional user/group/permissions access.

I don't know if they can handle correctly multiple source
paths on the command line but there are certainly people
using the command= option in authorized_keys to invoke
special scripts to check and/or tweak the rsync command line
restrict rsync to pre-aproved paths.

  Sanitizing the paths to force them to be relative on pulls
  but not pushes would be too asymetrical for my liking.
 
 I agree that if we find that we want to sanitize the paths in some cases
 that we should just make it the default for files-from -- i.e. make it
 where nothing can get beyond the root dir specified on the command-line.
 
  I'd rather just disallow or sanitize absolute paths.
 
 Note that it's more pervasive than just absolute paths, since someone
 can use args like ../../../etc/password or good_dir/../../bad_dir
 (all of which the sanitize_path() call handles).

Yes, the relative ../../... paths slipped my mind but that
is a concern as well.

I'm aware that restricting --files-from to having relative
paths is somewhat limiting but i think it may be the better
approach.  You can always do

rsync --files-from=list / remote:/

If you need to.

I haven't had time yet to closely examine or try it but i
have two questions:

with the -r or -a options does this recourse on
directories in the --files-from list?

What happens when there are implied directories that are
missing on the destination?  For example
rsync -a --files-from=list src dest
with the list having
foo/bar/one
will dest/foo and dest/foo/bar be created with the source
directory attributes if they don't exist; will it fail; or
will the missing implied directories be created with umask
perms?

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Lee Eakin]

Please pardon my butting in again.  I am not a developer, but I am very
interested in this option because I've need it in the past and had to work
around the lack of it with include/exclude options (I wanted to sync only
a few files from a large directory, and needed it to work via the daemon
for anonymous access).  I also maintain the perl wrapper File::Rsync so I
do my best to understand all of the options so I can handle them properly
in the perl module.

---begin quoted text---
 From: Wayne Davison [EMAIL PROTECTED]
 Date: Tue, 14 Jan 2003 19:39:49 -0800
 
 On Tue, Jan 14, 2003 at 07:02:58PM -0800, jw schultz wrote:
  Up till now rsync hasn't touched anything outside of the paths
  specified on the command-line.  Changing that would mean access to
  rsync via ssh would no longer be restricted, just disabled.
 
 Are you saying that some people have special ssh scripts that check
 and/or tweak the file names on the command-line to ensure they fall with
 certain bounds when running rsync commands?  I.e., if someone ran this
 command:
 
 rsync -av -e ssh source:dir /foo/two /bar/three /tmp
 
 the remote ssh setup would handle the presence of the extra /foo/two,
 /bar/three args?  If so, I hadn't realized that people were limiting
 ssh access by more than the traditional user/group/permissions access.
 

Yes, people do restrict args via ssh key restrictions.  I have done this
myself on many occasions.  The environment variable SSH_ORIGINAL_COMMAND
is passed to the actual command called from the command= key option so I
write a small script to parse thru the variable checking each arg making
sure they are what I expect (and possibly modifying them).  I also check
pathnames to make sure they all fit my restrictions.  I then either exec
rsync, or email the offending command to root if I find an exception
(the mail also makes debugging easier).

I assume the remote end will get the expanded contents of files-from so
ssh command parsing would still work properly.

  Sanitizing the paths to force them to be relative on pulls
  but not pushes would be too asymetrical for my liking.
 
 I agree that if we find that we want to sanitize the paths in some cases
 that we should just make it the default for files-from -- i.e. make it
 where nothing can get beyond the root dir specified on the command-line.
 
  I'd rather just disallow or sanitize absolute paths.

If you try to pull a full pathname from a daemon 'rsync remote::/foo' it
errors out with:

  ERROR: The remote path must start with a module name not a /

so any sanitize code could first make sure all pathnames begin with a valid
module and then make sure the file or dir is really inside that module.

---end quoted text---

-- 
Lee Eakin - [EMAIL PROTECTED] - Internet/Naming Services, Texas Instruments
 
LAWS OF COMPUTER PROGRAMMING:
II. Any given program costs more and takes longer.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[jw schultz]

On Tue, Jan 14, 2003 at 10:01:47PM -0600, Lee Eakin wrote:
 Please pardon my butting in again.  I am not a developer, but I am very
 interested in this option because I've need it in the past and had to work
 around the lack of it with include/exclude options (I wanted to sync only
 a few files from a large directory, and needed it to work via the daemon
 for anonymous access).  I also maintain the perl wrapper File::Rsync so I
 do my best to understand all of the options so I can handle them properly
 in the perl module.
 
 ---begin quoted text---
  From: Wayne Davison [EMAIL PROTECTED]
  Date: Tue, 14 Jan 2003 19:39:49 -0800
  
  On Tue, Jan 14, 2003 at 07:02:58PM -0800, jw schultz wrote:
   Up till now rsync hasn't touched anything outside of the paths
   specified on the command-line.  Changing that would mean access to
   rsync via ssh would no longer be restricted, just disabled.
  
  Are you saying that some people have special ssh scripts that check
  and/or tweak the file names on the command-line to ensure they fall with
  certain bounds when running rsync commands?  I.e., if someone ran this
  command:
  
  rsync -av -e ssh source:dir /foo/two /bar/three /tmp
  
  the remote ssh setup would handle the presence of the extra /foo/two,
  /bar/three args?  If so, I hadn't realized that people were limiting
  ssh access by more than the traditional user/group/permissions access.
  
 
 Yes, people do restrict args via ssh key restrictions.  I have done this
 myself on many occasions.  The environment variable SSH_ORIGINAL_COMMAND
 is passed to the actual command called from the command= key option so I
 write a small script to parse thru the variable checking each arg making
 sure they are what I expect (and possibly modifying them).  I also check
 pathnames to make sure they all fit my restrictions.  I then either exec
 rsync, or email the offending command to root if I find an exception
 (the mail also makes debugging easier).
 
 I assume the remote end will get the expanded contents of files-from so
 ssh command parsing would still work properly.

Nope.  The files-from contents needs to passed on stdin otherwise
we would hit command-line length limits.  That is why i'm
stressing the fact that allowing paths not within the source
or destination trees specified on the command-line would
bypass your ssh command= wrapper restrictions.

 
   Sanitizing the paths to force them to be relative on pulls
   but not pushes would be too asymetrical for my liking.
  
  I agree that if we find that we want to sanitize the paths in some cases
  that we should just make it the default for files-from -- i.e. make it
  where nothing can get beyond the root dir specified on the command-line.
  
   I'd rather just disallow or sanitize absolute paths.
 
 If you try to pull a full pathname from a daemon 'rsync remote::/foo' it
 errors out with:
 
   ERROR: The remote path must start with a module name not a /
 
 so any sanitize code could first make sure all pathnames begin with a valid
 module and then make sure the file or dir is really inside that module.
 
 ---end quoted text---
 
 -- 
 Lee Eakin - [EMAIL PROTECTED] - Internet/Naming Services, Texas Instruments
  
 LAWS OF COMPUTER PROGRAMMING:
 II. Any given program costs more and takes longer.
 -- 
 To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
 Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Lee Eakin]

---begin quoted text---
 From: jw schultz [EMAIL PROTECTED]
 Date: Tue, 14 Jan 2003 20:07:58 -0800
 
 Nope.  The files-from contents needs to passed on stdin otherwise
 we would hit command-line length limits.  That is why i'm
 stressing the fact that allowing paths not within the source
 or destination trees specified on the command-line would
 bypass your ssh command= wrapper restrictions.
 

Oh, I see now.  Yes that could be a serious hole.  If the remote command
included an option (maybe a dummy --files-from) then the ssh wrapper could
at least abort and notify when it sees it.

  -Lee
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[jw schultz]

On Tue, Jan 14, 2003 at 10:15:42PM -0600, Lee Eakin wrote:
 ---begin quoted text---
  From: jw schultz [EMAIL PROTECTED]
  Date: Tue, 14 Jan 2003 20:07:58 -0800
  
  Nope.  The files-from contents needs to passed on stdin otherwise
  we would hit command-line length limits.  That is why i'm
  stressing the fact that allowing paths not within the source
  or destination trees specified on the command-line would
  bypass your ssh command= wrapper restrictions.
  
 
 Oh, I see now.  Yes that could be a serious hole.  If the remote command
 included an option (maybe a dummy --files-from) then the ssh wrapper could
 at least abort and notify when it sees it.

If you look at Wayne's description of the patch the remote
command does have a --files-from=- on it's command-line.
However it would be a shame to disable that performance
enhancing facility if we just need sanitize the contents of
the file-from list and require that it only specify paths
relative to the source and dest trees.

I suppose we could allow an option that would permit
unsanitized paths.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: specifying a list of files to transfer

[Lee Eakin]

 From: jw schultz [EMAIL PROTECTED]
 Date: Tue, 14 Jan 2003 20:33:46 -0800
 
 On Tue, Jan 14, 2003 at 10:15:42PM -0600, Lee Eakin wrote:
  ---begin quoted text---
   From: jw schultz [EMAIL PROTECTED]
   Date: Tue, 14 Jan 2003 20:07:58 -0800
   
   Nope.  The files-from contents needs to passed on stdin otherwise
   we would hit command-line length limits.  That is why i'm
   stressing the fact that allowing paths not within the source
   or destination trees specified on the command-line would
   bypass your ssh command= wrapper restrictions.
   
  
  Oh, I see now.  Yes that could be a serious hole.  If the remote command
  included an option (maybe a dummy --files-from) then the ssh wrapper could
  at least abort and notify when it sees it.
 
 If you look at Wayne's description of the patch the remote
 command does have a --files-from=- on it's command-line.
 However it would be a shame to disable that performance
 enhancing facility if we just need sanitize the contents of
 the file-from list and require that it only specify paths
 relative to the source and dest trees.
 
 I suppose we could allow an option that would permit
 unsanitized paths.
 

I would agree.  If the paths are known to be relative (forced to be by the
rsync running where the ssh restriction is) then (assuming the wrapper's
intent is to allow access to the whole sub-tree) it could allow the
files-from option.  If you only want to allow access to specific files,
then it would still have to disallow the option.

I can think of one possible way for the wrapper to find out what files are
being requested, but don't know enough about the interconnect between the
2 rsyncs to know if it would break it (probably).  If the wrapper could
run a modified version of the original command without a destination it
would print out a list of the files (remember that if you do not give a
destination it prints something similar to 'ls -l' with includes and
excludes applied) then it could walk the output and verify all the paths.
If it passed inspection, it could call the real command passing the file
list to stdin itself.  It would have to attach stdout and stderr properly,
and may even have to act as a pass-thru for further data coming on stdin.
It would be complicated, but might be possible if the dummy (no destination
run) did not close off the connection after reading the file list, and/or
the handshaking was clearly documented so non-developers could understand
it.

I only throw this idea out because I would really like files-from to work
even in a restricted-access mode.  It is a BIG win over parsing a large dir
for includes/excludes.

-- 
Lee Eakin - [EMAIL PROTECTED]
 
Murphy's Military Laws:
6. The buddy system is essential to your survival;
   it gives the enemy somebody else to shoot at.
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Please test rsync-2.5.6pre1

[Albert Chin]

On Mon, Jan 13, 2003 at 11:15:47AM -0600, Dave Dykstra wrote:
 The first rsync-2.5.6 pre-release version is now available at:

 ...
 
 * Updated included popt to the latest vendor drop, version 1.6.4.
   (Jos Backus)

Latest version of popt is 1.7:
  ftp://ftp.rpm.org/pub/rpm/dist/rpm-4.1.x/popt-1.7.tar.gz

-- 
albert chin ([EMAIL PROTECTED])
-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: [rsync] Re: [rsync] Please test rsync-2.5.6pre1

[Scott Evans]

  On Mon, Jan 13, 2003 at 01:59:47PM -0500, Scott Evans wrote:
Known issues that will probably need to be resolved before the final
release:
[...]
   
   Is there any interest in trying to hunt down the linux - cygwin
   rsync hanging problem I'm consistently seeing before this release?
   I'm more than happy to help track it down but I don't know enough about
   rsync to do it myself.
  
  Well, at a minimum it would be nice to know if it is still happening
  with version 2.5.6pre1.  If it's no worse than previous versions, we
  won't hold up a release of 2.5.6.  If a cygwin user tracks it down and
  provides a fix that doesn't break anybody else, we'll probably put it in.
 
 I'll try out 2.5.6pre1 and report back.  

Nope -- hangs up in the same fashion that 2.5.5 did.  
So it is indeed no worse than 2.5.5, but it's unusable for me
under Cygwin. :(



scott

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html