Re: Compressed backup

[Donovan Baarda]

On Thu, May 30, 2002 at 03:35:05PM -0700, jw schultz wrote:
[...]
> > There is a patch available to gzip to add an option --rsyncable that's
> > supposed to make it work better with rsync.  It's been put into the
> > "patches" directory for the next release of rsync, or you can get it at
> > 
> > http://rsync.samba.org/ftp/unpacked/rsync/patches/gzip-rsyncable.diff
> 
> I took a quick look at this patch and i think it does what i expected.
> It resets the compression algorithm after each 4KB of
> compresstext.  This means that if you change 1 byte early in
> the file it might or might not affect the blocks later on.
> The reason for the equivication is that if the change alters
> the compression ratio the savings are gone.

If that is how it works, and I think you are right, then it would only work
for the smallest of cases, rendering the gzip-rsyncable patch worse than
useless for the vast majority of cases.

Regular resets hurt the compression ratio. Resets must occur at the same
begin/end boundary points of an unchanged sequence of uncompresstext for the
resultant compresstext to be unchanged. The only changes that will result in
resets occuring at the same boundary points for any unchanged text following
the change _must_ result in compresstext that is an exact multiple of 4KB.
This means any insertion/deletion/replacement must not change the size of
the resulting compresstext unless it is by an exact multiple of 4KB.

I would guess that the number of changes meeting this criteria would be
almost non-existant. I suspect that the gzip-rsyncable patch does nearly
nothing except produce worse compression. It _might_ slightly increase the
rsyncability up to the point where the first change in the uncompresstext
occurs, but the chance of it re-syncing after that point would be extremely
low.

I tried to think of a way of doing this so that it would eventualy re-sync,
with things like resets every  bytes so that the reset window
moves, but the problem is the source and target reset windows must move
together for it to work, so any scheme that moves the reset window into sync
will also move the window _out_ of sync. 

I don't think it is possible to come up with a scheme where the reset
windows could re-sync after a change and then stay sync'ed until the next
change, unless you dynamiclly alter the compression at sync time... you may
as well rsync the decompressed files.

-- 
--
ABO: finger [EMAIL PROTECTED] for more info, including pgp key
--

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Compressed backup

[jw schultz]

This whole discussion on the efficiency of rsyncing
pre-compressed files is probably pointless for Matthias
Munnich.  He is trying to do backups.  Therefore, he doesn't
want the originals compressed.

On Thu, May 30, 2002 at 03:45:16PM -0500, Dave Dykstra wrote:
> On Thu, May 23, 2002 at 04:03:56PM -0400, David Bolen wrote:
> > Matthias Munnich [[EMAIL PROTECTED]] writes:
> > 
> > > No! Only the sender side has to compress the data. The comparison
> > > could be done in the compressed data format. With the -z option 
> > > the sender compresses the data anyway. The checksum test should
> > > be faster for the smaller compressed pieces.

Except that with the -z option only the changed blocks are
compressed.  The checksums are done at both ends on the
uncompressed files.

Storing the files blockwize compressed on the rsync server
would be major departure from current design.  I can see how
it would be done but i certainly don't want it.  With
--link-dest i'm getting 24 backups in the space of 3.6 in a
near-worst case, the equivalent of 73% compression that
even compounds on file compression.

> > 
> > Except that you'll probably end up retransmitting the whole thing due
> > to the change in compressed output.  Since a compression function is
> > essentially a data randomizer (the better the compression the better
> > the randomization of the output), tiny changes in input can result in
> > huge changes in output.  That's the traditional problem of trying to
> > use an algorithm like rsync's with compressed file formats.
> > 
> > You really need to apply the rsync algorithm to the uncompressed files
> > if you hope to gain any real efficiencies in terms of reduction of
> > traffic transmitted.

Myth: compression randomizes data.
Reality: gzip and other compression systems are
entirely deterministic solely based on the input data.
Unlike encryption the output is in no way randomized.

For gzip, bzip, pkzip and most other general purpose
compressors a change in the plaintext will only alter the
compresstext representing that point until the compression
algorithm resets.  If a file is altered near the end the
compresstext of the earlier blocks will remain the same.

> 
> There is a patch available to gzip to add an option --rsyncable that's
> supposed to make it work better with rsync.  It's been put into the
> "patches" directory for the next release of rsync, or you can get it at
> 
> http://rsync.samba.org/ftp/unpacked/rsync/patches/gzip-rsyncable.diff

I took a quick look at this patch and i think it does what i expected.
It resets the compression algorithm after each 4KB of
compresstext.  This means that if you change 1 byte early in
the file it might or might not affect the blocks later on.
The reason for the equivication is that if the change alters
the compression ratio the savings are gone.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

update on hung rsyncs

[Eric Whiting]

Some new data on my rsync hangs:

I run about 1500 rsync sessions over ssh daily. In the last 8 days that 
adds up to about 12k rsync sessions. Of those 12k sessions, 10 are right 
now sitting in a hung state. The rsync process on the destination has 
exited, but both rsync processes on the source are still 
running/waiting/hung. I use a timeout of 3600 (but this doesn't seem to 
work for this failure mode)

It is interesting to note that the 10 rsyncs that are hung are all 
solaris 8 destinations, yet about 1/2 of my solaris destinations work 
just fine.

Source is linux 2.4.18 rsync 2.5.5 + generator/timeout patch
(http://lists.samba.org/pipermail/rsync/2002-April/006976.html)

All suns are running solaris8 OpenSSH_2.9p2.
Suns that seem to fail are rsync 2.5.5 (no generator patch) and NFS 
destination for files.
Suns that work are rsync 2.5.3 and local file destination.

Is the NFS difference a clue? It seems like some delay/glitch/issue with 
NFS on the destination might be causing ocassional/random troubles for 
my rsync processes. It seems this NFS factor is something that people 
are bringing up more and more lately. Ideas? I'll try 2.5.5 with the 
generator patch on the destinations.

SendQ and RecvQ are 0 on the source sockets.
strace shows the parent rsync process on source is stuck in this endless 
loop:

gettimeofday({1022796482, 605543}, NULL) = 0
wait4(8783, 0xbc48, WNOHANG, NULL)  = 0
gettimeofday({1022796482, 605602}, NULL) = 0
gettimeofday({1022796482, 605626}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 2}) = 0 (Timeout)
gettimeofday({1022796482, 625224}, NULL) = 0
select(0, NULL, NULL, NULL, {0, 1000})  = 0 (Timeout)
gettimeofday({1022796482, 635262}, NULL) = 0
wait4(8783, 0xbc48, WNOHANG, NULL)  = 0
gettimeofday({1022796482, 635316}, NULL) = 0
gettimeofday({1022796482, 635350}, NULL) = 0

strace on the child rsync process triggers both to exit:

windriver:/home/tisadmin/bin # strace -p 8783
select(7, [3 4], [], NULL, NULL)= 1 (in [4])
read(4, "", 16384)  = 0
close(4)= 0
select(7, [3], [3], NULL, NULL) = 1 (out [3])
write(3, "\200\300G\223K\355\2322#\220~Y5\0\210x\206~1e\240M\250"..., 
32) = 32
select(7, [3], [], NULL, NULL)  = 1 (in [3])
read(3, "\226f\301\271\220\200\t6\\\177\"%\3477\336^\255\n\255I"..., 
8192) = 96
brk(0x809d000)  = 0x809d000
close(6)= 0
select(7, [3], [3], NULL, NULL) = 1 (out [3])
write(3, "\204\355\2058\330\360\242<\313\233QMc\311\307?\322\351"..., 
32) = 32
ioctl(0, TCGETS, 0xba18)= -1 EINVAL (Invalid argument)
fcntl64(0, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl64(0, F_SETFL, O_RDWR) = 0
ioctl(1, TCGETS, 0xba18)= -1 EINVAL (Invalid argument)
fcntl64(1, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)
fcntl64(1, F_SETFL, O_RDWR) = 0
ioctl(2, TCGETS, 0xba18)= -1 ENOTTY (Inappropriate ioctl 
for device)
fcntl64(2, F_GETFL) = 0x8801 (flags 
O_WRONLY|O_NONBLOCK|O_LARGEFILE)
fcntl64(2, F_SETFL, O_WRONLY|O_LARGEFILE) = 0
gettimeofday({1022796581, 661010}, NULL) = 0
shutdown(3, 2 /* send and receive */)   = 0
close(3)= 0
_exit(0)= ?




-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: I/O error when deleting files

[Dave Dykstra]

Are you on the latest version of rsync?  If so, it would help if you could
create a simple example that you could explain so we could reproduce it.

- Dave Dykstra

On Wed, May 29, 2002 at 01:43:11PM -0700, Bill Houle wrote:
> FWIW, I ran without -x and still get the same error
> 
> --bill
> 
> 
> 
> 
> > Does it happen if you don't use '-x'?  If both directories are on the
> > same filesystem, you shouldn't have any need for it.
> > 
> > I'm not sure what the specific problem is, but I do see that the
> > skip_filesystem() function calls link_stat so perhaps somehow the
> > empty string is getting passed in there.
> > 
> > - Dave Dykstra
> > 
> > 
> > On Fri, May 17, 2002 at 01:52:52PM -0700, Bill Houle wrote:
> > > OK, but I'm not exactly sure what I'm looking for... 
> > > 
> > > I don't think the link error is caused by my data (I have no symlinks).
> > > For whatever reason, it appears that a blank is leading the file list,
> > > and the 'stat' on NULL is what is causing the link_stat error.
> > > 
> > > write(1, " b u i l d i n g   f i l".., 23)  = 23
> > > brk(0x00058668) = 0
> > > brk(0x00060668) = 0
> > > lstat64("", 0xFFBEF8D8) Err#2 ENOENT
> > > write(2, " l i n k _ s t a t :".., 39)  = 39
> > > 
> > > The rsync child then just says an error has occured and tells the
> > > parent to not bother deleting files because of it.
> > > 
> > > waitid(P_PID, 10233, 0xFFBEF890, WEXITED|WTRAPPED|WNOHANG) = 0
> > > Received signal #18, SIGCLD, in poll() [caught]
> > >   siginfo: SIGCLD CLD_EXITED pid=10233 status=0x000C
> > > poll(0xFFBEF870, 0, 20) Err#4 EINTR
> > > waitid(P_ALL, 0, 0xFFBEF3A8, WEXITED|WTRAPPED|WNOHANG) = 0
> > > waitid(P_ALL, 0, 0xFFBEF3A8, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
> > > setcontext(0xFFBEF558)
> > > poll(0xFFBEF870, 0, 11) = 0
> > > waitid(P_PID, 10233, 0xFFBEF890, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
> > > time()  = 1021666775
> > > write(1, " w r o t e   7 9 8 5 5  ".., 54)  = 54
> > > write(1, " t o t a l   s i z e   i".., 43)  = 43
> > > sigaction(SIGUSR1, 0xFFBEF848, 0xFFBEF8C8)  = 0
> > > sigaction(SIGUSR2, 0xFFBEF848, 0xFFBEF8C8)  = 0
> > > waitid(P_PID, 10233, 0xFFBEF888, WEXITED|WTRAPPED|WNOHANG) Err#10 ECHILD
> > > write(2, " r s y n c   e r r o r :".., 55)  = 55
> > > 
> > > I see that I can turn on "ignore-errors" mode, but that just seems
> > > dangerous if there were ever anything other than the case I am seeing
> > > here. So I guess my question is what's with the lstat64("")??
> > > 
> > > --bill
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > > I suggest running truss -f -o trussfile before that command and look for
> > > > the error in the trussfile and see what's happening just before it.  Under
> > > > some circumstances rsync can print that message when a symlink goes
> > > > nowhere, but I'm not sure what the circumstances are.  Perhaps it does that
> > > > because you're using -x, I'm not sure.
> > > > 
> > > > - Dave 
> > > > 
> > > > 
> > > > On Wed, May 15, 2002 at 02:30:00PM -0700, Bill Houle wrote:
> > > > > /usr/local/bin/rsync \
> > > > > -axv --delete \
> > > > > "${MAINDIR}/${FSERVER}/" \
> > > > > "${MAINDIR}/${TSERVER}/"
> > > > > 
> > > > > Both locations are on the same (non-NFS) partition.
> > > > > 
> > > > > All old files are updated and new files copied, but the orphans
> > > > > are never deleted...
> > > > > 
> > > > > --bill
> > > > > 
> > > > > 
> > > > > 
> > > > > > What is the command you are running? 
> > > > > > 
> > > > > > > -Original Message-
> > > > > > > From: [EMAIL PROTECTED] 
> > > > > > > [mailto:[EMAIL PROTECTED]]On
> > > > > > > Behalf Of Bill Houle
> > > > > > > Sent: Monday, 1 January 1601 11:00 AM
> > > > > > > To: [EMAIL PROTECTED]
> > > > > > > Subject: I/O error when deleting files
> > > > > > > 
> > > > > > > 
> > > > > > > When I run rsync in "no-op" mode, all appears fine (it knows what to
> > > > > > > copy, what to delete). But when I run it "for real", I get:
> > > > > > > 
> > > > > > >   building file list ... link_stat  : No such file or 
> > > > > > > directory done
> > > > > > >   IO error encountered - skipping file deletion
> > > > > > > 
> > > > > > > followed by:
> > > > > > > 
> > > > > > >   total size is 98117172  speedup is 563.78
> > > > > > >   rsync error: partial transfer (code 23) at main.c(578)
> > > > > > > 
> > > > > > > Any help/pointers -- Solaris 8 on Ultra -- is appreciated.
> > > > > > > 
> > > > > > > --bill
> > > > > > > 
> > > > > > > 
> > > > > > > 
> > > > > > > -- 
> > > > > > > To unsubscribe or change options: 
> > > > > > > http://lists.samba.org/mailman/listinfo/rsync
> > > > > > > Before posting, read: 
> > > > > > http://www.tuxedo.org/~esr/faqs/smart-questions.html
> > > > > > 
> > > > > 
> > > > > 
> > > > > -- 

Re: Syncing open files

[jw schultz]

On Wed, May 22, 2002 at 11:57:17AM -0400, [EMAIL PROTECTED] wrote:
> Just a few quick questions.. 
> Does rsync copy open files? 
> If so, does file integrity remain intact?
> I’m looking to backup IIS log files and maybe Oracle and SQL Server DB files. If 
>rsync isn’t appropriate, what would you recommend?

rsync is perfectly appropriate.  You just need to prevent
corrupting changes from happening while rsync runs.

My guess is that IIS log files are, like apache's,
append-only and not machine read.  Therefore if copied
during a change the worst that can happen is the copy will
have a partial entry at the end.

Oracle is far more delicate.  _IF_ there is an update query
in progress the tablespaces will be inconsistant.  If you
flush and copy the redo logs _after_ the tablespaces you can
recover by replaying the redo logs just like you would after
a crash.  I don't reccomend that method but it will work.
Far better is to sync and lock the tablespaces while rsync
runs.  The Oracle documentation includes poorly designed
example code on how to do this in a section on on-line
backups.   The ultimate safe way is to shut down Oracle
for the backups but i recognize the shutdown+startup time
can be prohibitive.

As for MS SQL server, I can't say.  I expect it has
facilities to assist in backups.

In sum, what you need to do is bracket the rsync run with
commands to stop and then restart those processes that
may corrupt files.  Any good backup system will provide a
facility for doing so.

-- 

J.W. SchultzPegasystems Technologies
email address:  [EMAIL PROTECTED]

Remember Cernan and Schmitt

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: bug: copying unreadable file from remote host returns errorlevel 0

[Dave Dykstra]

Does the fix at
http://lists.samba.org/pipermail/rsync/2002-February/006371.html
help you?

- Dave Dykstra

On Mon, May 27, 2002 at 08:38:42PM +0400, [EMAIL PROTECTED] wrote:
> 
> [ Please keep me in Cc: as I am not subscribed to this list ]
> 
> Hello,
> 
> we have noticed the following bug in rsync (including latest CVS
> snapshot):
> 
> if I am copying from the remote host file that is unreadable by my
> account, the error is reported properly, but the errorlevel returned
> is 0. 
> 
> $ rsync distanza:/etc/shadow .
> send_files failed to open /etc/shadow: Permission denied
> rsync error: some files could not be transferred (code 23) at main.c(348)
> 
> $ echo $?
> 0
> 
> 
> If I'm trying to copy non-existent file, it returns proper errorlevel
> 
> $ rsync --rsync-path=/usr/local/rsync-cvs/bin/rsync distanza:/etc/shad .
> link_stat /etc/shad : No such file or directory
> rsync error: some files could not be transferred (code 23) at main.c(337)
> client: nothing to do
> $ echo $?
> 11
> 
> 
> 
> We tried to fix it by changing "continue" to "return":
> 
> --- sender.c.orig   Mon May 27 20:36:34 2002
> +++ sender.cMon May 27 20:35:33 2002
> @@ -184,7 +184,7 @@
> rprintf(FERROR,"send_files failed to open %s: %s\n",
> fname,strerror(errno));
> free_sums(s);
> -   continue;
> +   return;
> }
>   
> /* map the local file */
> 
> 
> and the errorlevel is returned properly, but there are additional
> messages:
> 
> send_files failed to open /etc/shadow: Permission denied
> rsync error: some files could not be transferred (code 23) at main.c(348)
> Invalid file index 24 in recv_files (count=1)
> unexpected EOF in read_timeout
> 
> and we're not sure if that's ok.
> 
> 
> Thanks a lot for your help,
> 
> --alexm
> 
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

XFS acls in rsync

[Olivier Tarnus]

Hi,

I'm actually building a big file server for my company and is testing rsync 
intensively since two weeks to synchronize a stand-by server every 15 
minutes. 
I've been really impressed by the performance of rsync :-)

The filesystem is actually 140GB on XFS under linux running samba with 
acl support... And you already know the problem :-)

Google told me you are planning full acl support (aka cross fs acls support)
in version 3, but i can't wait for it, so i want to code a patch to support 
XFS acls.

Assuming i'm a C programmer with some experience do you think it's 
doable (20-30 h)?

Could give me some hints about where to look first?

Many thanks to rsync developer... And SGI too :-)

--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: restricting rsync over ssh

[Dave Dykstra]

On Wed, May 22, 2002 at 02:39:00PM -0700, Mike Rubel wrote:
> 
> > > If so, I am trying to find the best way to restrict rsync -e ssh on the
> > > remote machine.  Prepending the authorized_keys entry with
> > > command='rsync ...' 1024... results in the 'Protocol mismatch - is your
> > > shell clean?' error.
> 
> This brings up an interesting question.  Does anyone use the server
> version of rsyncd with an ssh tunnel?  In other words:
> 
> On the server, bring up sshd listening on *:22, and rsyncd accepting
> connections from localhost only, port 873.
> 
> Then, on the client machine, set up an ssh tunnel:
> 
> ssh server -L 1873:server:873
> 
> (I just picked 1873 for convenience; you could use any open
> user-accessible port).
> 
> Then rsync to localhost:1873, letting ssh carry the connection over the
> tunnel to the server:
> 
> rsync rsync://user@localhost:1873/path/to/source dest
> 
> Does anyone use this approach?  Are there any advantages/disadvantages?


I didn't see any respones to your message so I guess it isn't very widely
done.  I haven't tried it but I don't see why it wouldn't work.  You could
probably use "hosts allow = localhost" to disallow people from using port
873 directly.

The solution we've been wanting to put in is JD Paul's patch to run 
rsync --daemon directly over ssh.  That would be easier to use because you
don't have to have to have anything listening on a port.

- Dave Dykstra

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Compressed backup

[Dave Dykstra]

On Thu, May 23, 2002 at 04:03:56PM -0400, David Bolen wrote:
> Matthias Munnich [[EMAIL PROTECTED]] writes:
> 
> > No! Only the sender side has to compress the data. The comparison
> > could be done in the compressed data format. With the -z option 
> > the sender compresses the data anyway. The checksum test should
> > be faster for the smaller compressed pieces.
> 
> Except that you'll probably end up retransmitting the whole thing due
> to the change in compressed output.  Since a compression function is
> essentially a data randomizer (the better the compression the better
> the randomization of the output), tiny changes in input can result in
> huge changes in output.  That's the traditional problem of trying to
> use an algorithm like rsync's with compressed file formats.
> 
> You really need to apply the rsync algorithm to the uncompressed files
> if you hope to gain any real efficiencies in terms of reduction of
> traffic transmitted.

There is a patch available to gzip to add an option --rsyncable that's
supposed to make it work better with rsync.  It's been put into the
"patches" directory for the next release of rsync, or you can get it at

http://rsync.samba.org/ftp/unpacked/rsync/patches/gzip-rsyncable.diff

- Dave Dykstra

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: Syncing open files

[Dave Dykstra]

On Wed, May 22, 2002 at 11:57:17AM -0400, [EMAIL PROTECTED] wrote:
> Just a few quick questions.. 
> Does rsync copy open files? 

Yes.

> If so, does file integrity remain intact?

No, not if it's being modified while rsync is copying it.

> I'm looking to backup IIS log files and maybe Oracle and SQL Server DB
> files. If rsync isn't appropriate, what would you recommend?

Something application-specific that can work with the locks used by
the products.  Ask the vendors.

> Thanks for your time,
> 
>  
> Mark J. de Jong

- Dave Dykstra

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsyncd listing of directories

[Dave Dykstra]

Can you figure out another way to do it (that's not a "complete hack")
without affecting the output of list_only mode but yet won't get in your
way?  The Rusty way doesn't actually recurse completely, it only goes one
level down.  I bet he did it that way to avoid having to change the remote
side.

- Dave Dykstra

On Wed, May 22, 2002 at 09:48:58AM -0400, Alberto Accomazzi wrote:
> 
> I just took a look at the 2.5.5 codebase to see how easy it would be to
> write a little driver script that downloads a big directory tree from an
> rsync daemon the chunky way (get a list of a module's subdirectories and
> do the transfer by subdirectory).  The reason for doing this is obvious
> when you have large directory trees, as is the case for many of us.
> Unfortunately the way list_only is currently implemented makes the whole
> idea useless since it forces the daemon to recurse the target directory
> tree anyways.  Here's the code:
> 
> in options.c:
> 
> /* this is a complete hack - blame Rusty 
> 
>this is a hack to make the list_only (remote file list)
>more useful */
> if (list_only && !recurse) 
> argstr[x++] = 'r';
> 
> 
> in exclude.c:
> 
> /* This is a complete hack - blame Rusty.
>  *
>  * FIXME: This pattern shows up in the output of
>  * report_exclude_result(), which is not ideal. */
> if (list_only && !recurse) {
> add_exclude("/*/*", 0);
> }
> 
> 
> So I'm going to bite and blame Rusty at this point and ask the question: 
> why was this implemented this way?  I can't think of a good reason why.  
> I'm happy to try and work on a patch if there is a consensus that this 
> is desireable.
> 
> -- Alberto
> 
> 
> 
> 
> Alberto Accomazzi  mailto:[EMAIL PROTECTED]
> NASA Astrophysics Data System  http://adsabs.harvard.edu
> Harvard-Smithsonian Center for Astrophysicshttp://cfawww.harvard.edu
> 60 Garden Street, MS 83, Cambridge, MA 02138 USA   
> 
> 
> -- 
> To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
> Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Recreating a different arrangement of the same files

[Elliot Lee]

Systems A & B both have copies of tree X. I want to synchronize tree Y
from A to B, tree Y consisting of hard links to various files in tree X.

--compare-dest is a good start (and the --link-dest patch goes a bit
further towards what I want), but both of those require the files on
system B to be arranged in the same layout as the target.

As one solution, if I rsync trees X & Y in one rsync -H (they are subdirs
on the same filesystem), will rsync figure out that only link operations
are necessary to put the files in their new locations? Or if not, would it
be possible for an rsync hacker to add an option for the type of
optimization I seek?

-- Elliot
Forget world domination. The Red Planet shall belong to Red Hat.


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync 2.5.5, HPUX, getting unexplained error at main.c(578)

[Paul Haas]

Since the process didn't exit, it must still be in the process table.
When you get to this state have your process do something like:
  fprintf(stderr, "The weird process is pid %d\n",pid);
  fflush(stderr);
  sleep(1000);

Then see what ps says about that process.  If that doesn't offer any
hints, attach to the process with gdb, vdb, wdb, dde or some other
debugger.

My HP-UX system doesn't have a signal 64, either.

On Thu, 30 May 2002, Charles F. Fisher wrote:

> On Thu, May 30, 2002 at 11:53:34AM -0400, Paul Haas wrote:

> Much stuff snipped..

> > The EINTR case doesn't make sense with WNOHANG, but HP-UX doesn't always
> > do things that make sense.  If you get the "Now what should I do?"
> > message, then add code to check WIFSIGNALED(), WIFSTOPPED() and
> > WIFCONTINUED() and do something appropriate.  If you reach the perror(),
> > hopefully the text there will offer some clues.
> >
> > I don't have an HP-UX 11.11 system, I'm looking at the manpages on
> > an HP-UX 11.00 system.
> >
> Thank you for the code snippit - checking WIFSIGNALED() showed a non-zero
> result, WTERMSIG() returned 64, which isn't in /usr/include/sys/signal.h.
> In any case WIFEXITED() was returning 0, so WEXITSTATUS was returning garbage.
> Any idea was signal 64 would be?  It's being returned consistently; if it's
> a normal termination signal I'll fix the routine to treat it as such.



-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: -c Option

[Eric Ziegast]

> Quick questioncan anyone explain to me when the data in a file
> might change without changing the mtime, ctime or size?  I'm not
> sure I've ever come across that before.  An example might help me
> determine if I can safely remove -c.

It's possible on Unix systems, but not practical.

An example script:

  #!/bin/sh
  # Run on a BSD Unix system, your touch(1) arguments may vary
  echo "foo" > File
  touch -t 200205300800 File
  ls -l File
  echo "foo" > File
  ls -l File
  touch -t 200205300800 File
  ls -l File

Output:

  -rw-r--r--  1 ziegast  ziegast  4 May 30 08:00 File
  -rw-r--r--  1 ziegast  ziegast  4 May 30 11:19 File
  -rw-r--r--  1 ziegast  ziegast  4 May 30 08:00 File


Here's one example where I might use -c:

  Hackers are not known for being practical.  They like to cover their
  tracks as best they can by setting the owner, group, permissions, size,
  mtime, and ctime of their fake programs to be the same as the original
  programs.  If you're distributing system software using rsync or rdist,
  you may want to force checksum comparisons just to be sure.

  One might also use rsync with "-n -c" to help compare a "gold copy"
  of OS files with an active system.  Then again, tripwire was designed
  to do this better.


Another example is when some network-based filesystems delay updating
their metadata even though the content has changed.  I remember once
using a client machine to update a file on a busy NFS server.  It took
several seconds for the change to be seen by another client machine.
If I were using rsync on the second client machine, its view of the
file might be inconsistent.  Then again, best practices would dictate
my wanting to run rsync on the NFS server and not on the clients that
might be inconsistent with the server (to keep network traffic down
and reduce overhead).

--
Eric Ziegast

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync 2.5.5, HPUX, getting unexplained error at main.c(578)

[Charles F. Fisher]

On Thu, May 30, 2002 at 11:53:34AM -0400, Paul Haas wrote:
> On Wed, 29 May 2002, Allen D. Winter wrote:
> 

Much stuff snipped..

> The EINTR case doesn't make sense with WNOHANG, but HP-UX doesn't always
> do things that make sense.  If you get the "Now what should I do?"
> message, then add code to check WIFSIGNALED(), WIFSTOPPED() and
> WIFCONTINUED() and do something appropriate.  If you reach the perror(),
> hopefully the text there will offer some clues.
> 
> I don't have an HP-UX 11.11 system, I'm looking at the manpages on
> an HP-UX 11.00 system.
> 
Thank you for the code snippit - checking WIFSIGNALED() showed a non-zero
result, WTERMSIG() returned 64, which isn't in /usr/include/sys/signal.h.
In any case WIFEXITED() was returning 0, so WEXITSTATUS was returning garbage.
Any idea was signal 64 would be?  It's being returned consistently; if it's
a normal termination signal I'll fix the routine to treat it as such.

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: -c Option

[Paul LaMadeleine]

Actually,

 We have the problem here.  We have an NFS server mounted on a mix 
of Solaris, AIX, HP, True64, NT and VMS.  What seems to be happening is 
that something on the VMS side updates the file which creates a new 
version.  On the unix side the plain files is somehow linked to the new 
version but does not always have it's inode info updated.

 For those of you who don't know VMS and VMS file versioning, 
whenever a file is modified by VMS it creates another copy of the file with 
a new extention number.  For example, the first time you create a file x.x 
it will actually be labeled x.x;1.  The next time you modify this file, 
there will be two copies, x.x;1 and x.x;2 where x.x;2 is the newer file and 
any reference to x.x will reference the newest file.

 Now multinet (the TCPIP stack) on vms creates another file on NFS 
mounted shares without a ;# that is somehow linked to the newest 
file.  This is what the unix servers have the inode linked to.

 Hope this is clear, but it's the only time I've seen it happen myself.

 Paul

At 12:43 PM 5/30/02 -0400, Michael Montero wrote:
> > The only reason to use -c is if you have data which may be changed without
> > changing mtime, ctime or size (is there any case where that happens?).  It
>
> That's great news.  I believe this applies to me just fine and I
>can turn off the checksum.  Quick questioncan anyone explain to me
>when the data in a file might change without changing the mtime, ctime or
>size?  I'm not sure I've ever come across that before.  An example might
>help me determine if I can safely remove -c.
>
>Tim, thanks for the reply!
>
>
>
>--
>To unsubscribe or change options: 
>http://lists.samba.org/mailman/listinfo/rsync
>Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: tty settings with rsync -e ssh interrupt

[James Manning]

[James Manning]
> best described here:
> 
> https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=64689
> 
> Confirmed also present with the rpm build at
> http://rsync.samba.org/ftp/rsync/binaries/redhat/rsync-2.4.6-1.i386.rpm

Sorry, also confirmed with rsync-2.5.5 built from source

jmm@jmm /tmp/rsync-2.5.5> rpm -qf $(which ssh)
openssh-clients-3.1p1-3
jmm@jmm /tmp/rsync-2.5.5> ./rsync -e ssh /tmp/foo root@angus:/tmp
root@angus's password: rsync error: received SIGUSR1 or SIGINT (code 20) at 
rsync.c(229)
jmm@jmm /tmp/rsync-2.5.5>

at that point, the terminal was still in "noecho" mode - simply typing
"stty" and hitting enter confirmed this (note that the command isn't
echoed, I only get the output)

jmm@jmm /tmp/rsync-2.5.5> speed 38400 baud; line = 0;
-echo

Since ssh getting interrupted acts fine (the terminal goes back
to a sane mode), I don't think it's the ssh client.  I confirmed
the behavior after building portable openssh 3.2.3p1 from source:

jmm@jmm /tmp/rsync-2.5.5> ./rsync -e /tmp/openssh-3.2.3p1/ssh /tmp/foo root@angus:/tmp
root@angus's password: rsync error: received SIGUSR1 or SIGINT (code 20) at 
rsync.c(229)
jmm@jmm /tmp/rsync-2.5.5> speed 38400 baud; line = 0;
-echo

Thanks!

James
-- 
James Manning 
GPG Key fingerprint = B913 2FBD 14A9 CE18 B2B7  9C8E A0BF B026 EEBB F6E4

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: -c Option

[Michael Montero]

> The only reason to use -c is if you have data which may be changed without 
> changing mtime, ctime or size (is there any case where that happens?).  It 

That's great news.  I believe this applies to me just fine and I
can turn off the checksum.  Quick questioncan anyone explain to me
when the data in a file might change without changing the mtime, ctime or
size?  I'm not sure I've ever come across that before.  An example might
help me determine if I can safely remove -c.

Tim, thanks for the reply!



-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

tty settings with rsync -e ssh interrupt

[James Manning]

best described here:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=64689

Confirmed also present with the rpm build at
http://rsync.samba.org/ftp/rsync/binaries/redhat/rsync-2.4.6-1.i386.rpm

Please cc: me on replies (I'm not on the list, yet - my procmailrc's
in a major state of flux as I'm switching machines) and/or add comments
to the above bugzilla entry

James
-- 
James Manning 
GPG Key fingerprint = B913 2FBD 14A9 CE18 B2B7  9C8E A0BF B026 EEBB F6E4

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: -c Option

[tim . conway]

The only reason to use -c is if you have data which may be changed without 
changing mtime, ctime or size (is there any case where that happens?).  It 
makes rsync read EVERY file, from beginning to end, and generate a 
checksum for comparison.   If the size/time assumption is valid, it's a 
heck of an optimization, as you've noticed.

Tim Conway
[EMAIL PROTECTED]
303.682.4917
Philips Semiconductor - Longmont TC
1880 Industrial Circle, Suite D
Longmont, CO 80501
Available via SameTime Connect within Philips, n9hmg on AIM
perl -e 'print pack(, 
19061,29556,8289,28271,29800,25970,8304,25970,27680,26721,25451,25970), 
".\n" '
"There are some who call me Tim?"




Michael Montero <[EMAIL PROTECTED]>
Sent by: [EMAIL PROTECTED]
05/30/2002 08:27 AM

 
To: [EMAIL PROTECTED]
cc: (bcc: Tim Conway/LMT/SC/PHILIPS)
Subject:-c Option
Classification: 



Hello all.  I've been using rsync for quite some time and when I
originally began using it I was religious about including the -c option.
I've read the man page and noticed the significant performance hit I take
in using it.  I was wondering if someone would be able to clarify for me
what the benefits are in using this option.  Why would someone want to?

I can't quite remember why I decided to use itthe scripts using rsync
were written nearly 3 years ago.  However, I must have read somewhere on a
mailing list that the option was important for one reason or another.

Any insight would be great.  I can cut my sync times down by almost 80%.

Thanks.



-- 
To unsubscribe or change options: 
http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html




-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: restricting rsync over ssh

[Bennett Todd]

2002-05-29-15:55:09 Brian D. Hamm:
> No harsh programming statements please this was my first crack at Perl
> :)

I'll avoid any harsh statements:-).

Your code is not particularly idiomatic perl, but it's exceptionally
clear, and as far as I can see correct. You note in a comment one
spot where you have a weakness:

> # ARG[5] Complain if this arg does not begin with /var/log/
> # SECURITY ISSUE: need to lock down further, /var/log/../../otherdir
> would succeed
> $log_substr = substr ("$rsync_argv[5]", 0, 16);
> unless ($log_substr eq '/var/log/rotate/') {
>print SSHOUT ("ARG[5] Failure\n");
>$ok = $FALSE;
> }

If you wrote:

if ($rsync_argv[5] =~ m#/\.\./# or $rsync_argv[5] !~ m#^/var/log/rotate/#) {
print SSHOUT ("ARG[5] Failure\n");
$ok = $FALSE;
}

that would address the security problem; and doing a pattern match
is more idiomatic, and perhaps clearer, than doing a string compare
against a counted substr.

Later you have problems getting output flushed to your SSHOUT handle
before you run a command; you can fix that by going

use IO::File;

and somewhere before where you want that data flushed --- perhaps
right after you open the file, since performance isn't an issue here
--- going

SSHOUT->autoflush(1);

to ensure that the handle is flushed after each write.

You'll still need to use system() instead of exec() if you want to
be able to log the success or failure of the command afterwards.

-Bennett



msg04246/pgp0.pgp
Description: PGP signature

Re: rsync 2.5.5, HPUX, getting unexplained error at main.c(578)

[Paul Haas]

On Wed, 29 May 2002, Allen D. Winter wrote:

> I finally found that in the wait_process() function in main.c, the status
> value returned from WEXITSTATUS() was apparently returning
> the pid of the child instead of the exit status of the child... or something
> like that.

Or something not like that, since the manpage for waitpid() says the
value of status is undefined if waitpd() returns  (pid_t)-1.

Here's the current version of the function, note how it isn't checking for
-1:

void wait_process(pid_t pid, int *status)
{
while (waitpid(pid, status, WNOHANG) == 0) {
msleep(20);
io_flush();
}

/* TODO: If the child exited on a signal, then log an
 * appropriate error message.  Perhaps we should also accept a
 * message describing the purpose of the child.  Also indicate
 * this to the caller so that thhey know something went
 * wrong.  */
*status = WEXITSTATUS(*status);
}

Elsewhere in the HP UX manpage for waitpid() it says that
WEXITSTATUS(*status) is only valid if WIFEXITED(*status) is nonzero.

So an HP-UX user could try the following code and see what lands in
standard error.

void wait_process(pid_t pid, int *status)
{
int notdone = 1;
int waitPidRet;
while ( notdone ) {
waitPidRet = waitpid(pid, status, WNOHANG);
if ( waitPidRet == 0 ||
( waitPidRet == (pid_t)-1 && errno == EINTR ) ) {
msleep(20);
io_flush();
if ( waitPidRet == (pid_t)-1 ) {
 fprintf(stderr,
  "A mysterious failure averted.\n");
}
} else if ( waitPidRet == (pid_t) -1 ) {
perror("waitpid() did something I don't understand.");
exit(1);
}
if ( WIFEXITED(*status) ){
*status = WEXITSTATUS(*status);
notdone = 0;
} else {
fprintf(stderr,"Now what should I do?\n");
exit(2);
}
}
}

The EINTR case doesn't make sense with WNOHANG, but HP-UX doesn't always
do things that make sense.  If you get the "Now what should I do?"
message, then add code to check WIFSIGNALED(), WIFSTOPPED() and
WIFCONTINUED() and do something appropriate.  If you reach the perror(),
hopefully the text there will offer some clues.

I don't have an HP-UX 11.11 system, I'm looking at the manpages on
an HP-UX 11.00 system.

--
Paul Haas
Just because I sometimes use HP-UX, doesn't mean I like it.


-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

rsync 2.5.5 hanging

[Michael Montero]

I've been through the archive and found nothing definitive on this.
However, I am definitely seeing rsync hanging.  I'm rsyncing from a
Solaris 2.8 box running rsync 2.5.5  protocol version 26 and SSH version
2.1.0 (non-commercial version) to a Linux RedHat box running rsync 2.5.5
protocol version 26 and SSH version OpenSSH_3.1p1, SSH protocols 1.5/2.0,
OpenSSL 0x0090603f.  Wondering if anyone has seen anything like this or
have any clues?

The one thing I did notice was that the temporary file written on the
receiving end (in /tmp) actually grows to be bigger then the actual file
size.  I did not include a strace/truss but I can if anyone thinks that
will help.

Thanks in advance.



-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

-c Option

[Michael Montero]

Hello all.  I've been using rsync for quite some time and when I
originally began using it I was religious about including the -c option.
I've read the man page and noticed the significant performance hit I take
in using it.  I was wondering if someone would be able to clarify for me
what the benefits are in using this option.  Why would someone want to?

I can't quite remember why I decided to use itthe scripts using rsync
were written nearly 3 years ago.  However, I must have read somewhere on a
mailing list that the option was important for one reason or another.

Any insight would be great.  I can cut my sync times down by almost 80%.

Thanks.



-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html

Re: rsync 2.5.5, HPUX, getting unexplained error at main.c(578)

[Charles F. Fisher]

On Wed, May 29, 2002 at 05:47:10PM -0400, Allen D. Winter wrote:
> Once upon a time I threw printfs all over the source looking to track down
> the cause of this problem.  Seemingly a random error code is generated
> each time rsync is run *succesfully* on HPUX.
> 
> I finally found that in the wait_process() function in main.c, the status 
> value returned from WEXITSTATUS() was apparently returning
> the pid of the child instead of the exit status of the child... or something
> like that.
> 
> I've learned to ignore the error codes and live with it.
> Love to see this fixed.
> 
It doesn't seem to be a simple function of the child pid. I put printfs in
wait_process and ran a few tests; the status was being returned by rsync
as the unknown error, but did not seem to be related to the pid:

pid status   unknown error reported
10899   49   49
11086   62   62
11192   221  221 

-- 
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html