Re: what kind attrs are going to be transfered by rsync?
It seems there is a discussion long time ago. But little info. http://fixunix.com/tools/540757-do-not-reply-bug-3245-improve-rsyncs-error-messages-make-themclearer.html I noticed that it's status is assigned, Has it been improved? https://bugzilla.samba.org/show_bug.cgi?id=3245 On Thu, 2009-06-04 at 14:44 +0800, Daniel.Li wrote: > Dear list, > > I have met this error. I wanna know what attrs are transfer by rsync > program? > > sent 5045 bytes received 262853 bytes 107159.20 bytes/sec > total size is 130560 speedup is 0.49 > rsync error: some files/attrs were not transferred (see previous errors) > (code 23) at main.c(1535) [generator=3.0.5] > > Any help is appreciated. Thanks. > > > -- > Daniel Li > > -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: what kind attrs are going to be transfered by rsync?
Each different OS has a different idea of what attributes exist for whatever they call files. The exact combination of attributes will not map exactly from any OS to any other OS, although it is possible to preserve some crude distinctions in many (most?) cases. You look at the error messages to see what did not go through. You read the man pages to get some idea of what to tell it to attempt. Critical information is what attributes exist in what files system on which system. That information is really outside the scope of documentation about rsync. You might have some luck with some kind of universal file system, but I'd bet on snake oil. If you can limit yourself to what is in common, you can make it look like it works. -Original Message- From: rsync-bounces+tony=servacorp@lists.samba.org [mailto:rsync-bounces+tony=servacorp@lists.samba.org] On Behalf Of Daniel.Li Sent: Thursday, June 04, 2009 2:53 AM To: Rsync Subject: Re: what kind attrs are going to be transfered by rsync? It seems there is a discussion long time ago. But little info. http://fixunix.com/tools/540757-do-not-reply-bug-3245-improve-rsyncs-error-m essages-make-themclearer.html I noticed that it's status is assigned, Has it been improved? https://bugzilla.samba.org/show_bug.cgi?id=3245 On Thu, 2009-06-04 at 14:44 +0800, Daniel.Li wrote: > Dear list, > > I have met this error. I wanna know what attrs are transfer by rsync > program? > > sent 5045 bytes received 262853 bytes 107159.20 bytes/sec > total size is 130560 speedup is 0.49 > rsync error: some files/attrs were not transferred (see previous errors) > (code 23) at main.c(1535) [generator=3.0.5] > > Any help is appreciated. Thanks. > > > -- > Daniel Li > > -- Daniel Li -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: WARNING: . . . .failed verification -- update discarded (will try again).
Well, when you backup a running vm disk image 3 cases may happen: 1) From the start of the copy until the end of the copy (or syncing, doesn't matter) the guest OS did not write anything on disk -> copy will be OK. 2) The guest OS write something to disk but luckily the copy process got all modifications (data block + catalog mods) or none of the modification -> the integrity of the copy will be OK, it may reflet the past state of the vm instead of the actual one. 3) The copy process get part of the modification by the guest, like only catalog info but not data blocks or inverse. Here you have no integrity of the disk image. And there is another thing: data in RAM and not in disk will not be saved. I think in a scenario of a database always open in the guest OS. You will never have a working copy of those files. You can backup from the guest OS like it was a "normal" computer. A basic approach that I use at the moment (lack of time to search a more sophisticated solution) or you can use snapshots as suggested by henri. I don't know what would exactly a snapshot do. A snapshot by the virtualization program or by the filesystem? I think if the snapshot is made by the virtualization program it may be safe for ram recovering issues, but filesystem snapshot is not. You will still need to save regularly open files from the guest OS. I have absolutely no idea about the "will try again" part. Maybe Wayne or Matt could give you the answer. Best regards, Vitorio Le 3 juin 09 à 23:14, JW a écrit : I actually have done it, and it does produce a useable copy. The backup script does not always fail. Still, even if I am completely wrong about that, I still want to know more details about the "will try again" part. When does it try again, right away? How do I know if the re-try failed? JW On Wednesday 03 June 2009 15:40:44 you wrote: You can't copy the disk image of a vm while the guest OS is running. Best regards, Vitorio Le 3 juin 09 à 18:58, JW a écrit : A nightly scripted rsync backup job is giving me this error WARNING: vm/escDebLenny14G-flat.vmdk failed verification -- update discarded (will try again). I've searched the web and I think probably the problem is the source file changed during the copy. However, I haven't seen any explanations for the "will try again" part. Does it try again immediately? If so, does a lack of further errors mean that it succeeded on the retry? Thanks, JW -- -- System Administrator - Cedar Creek Software http://www.cedarcreeksoftware.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- -- System Administrator - Cedar Creek Software http://www.cedarcreeksoftware.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: WARNING: . . . .failed verification -- update discarded (will try again).
JW (j...@mailsw.com) wrote on 3 June 2009 11:58: >A nightly scripted rsync backup job is giving me this error > >WARNING: vm/escDebLenny14G-flat.vmdk failed verification -- update discarded >(will try again). > >I've searched the web and I think probably the problem is the source file >changed during the copy. > >However, I haven't seen any explanations for the "will try again" part. > >Does it try again immediately? > >If so, does a lack of further errors mean that it succeeded on the retry? After pulling a file rsync verifies its checksum; if it fails rsync tries once more, and if it still fails rsync gives up and reports the error. Therefore if there is no final error message the file was successfully pulled. This is explained in http://rsync.samba.org/how-rsync-works.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Which function in generator trigger the file transfer?
On Tue, Jun 02, 2009 at 03:31:02PM +0800, Daniel.Li wrote: > As receiver will use "read_ndx_and_attrs" to read iflags to judge if > it's need to be transfered? You can get a good overview of how rsync works here: http://rsync.samba.org/how-rsync-works.html After reading through that, you should discover that it's the generator that decides what work gets done. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --remove-source-files
On Tue, Jun 02, 2009 at 04:08:26PM +, C?sar Tejeda Hern?ndez wrote:
> rsync -tai --remove-source-files --ignore-existing --no-g --no-o --no-p
> u...@source:operativo/almacen/{ESP05/recorte/*.tar,EU/recorte/*.tar}
> /media/BACKUP3_PROMES10/PROMES10km/
> [...]
> ¿Is there any way to specify that rsync should not report an error if it
> finds no files to backup?
Your shell is probably including the raw wildcards when they don't match
anything, so you should be using include/exclude options instead of the
wildcards. For instance:
rsync -rti --remove-source-files --ignore-existing --include=*.tar --exclude=* \
u...@source:operativo/almacen/{ESP05,EU}/recorte/
/media/BACKUP3_PROMES10/PROMES10km/
..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[adej...@debian.org: Bug#531639: rsync can end up deleting unrelated files when a directory is replaced by a symlink]
Please see the included debian bug report, where during a local transfer rsync (3.0.3) follows a symlink in the destination to delete a file in the source tree. At least 3.0.5 works correctly in this regard; however I can't find a changelog entry that's directly related to this. Would it be possible to extract the diff that fixes this so that the 3.0.3 version in debian stable might be fixed for this? Getting 3.0.5 into the stable archive won't happen... Thanks, Paul (Please Cc: 531639-forwar...@bugs.debian.org on followups) - Subject: Bug#531639: rsync can end up deleting unrelated files when a directory is replaced by a symlink From: Arthur de Jong Date: Wed, 03 Jun 2009 00:10:04 +0200 Subject: rsync can end up deleting unrelated files when a directory is replaced by a symlink Package: rsync Version: 3.0.3-2 Severity: important I ran into this when reorganising some directories and putting compatibility symlinks in place. The end result that the whole directory went missing. I think the best way to explain is a demonstration. We set up simple source and destination directories: % cd /tmp % mkdir -p src/foo dst % touch src/foo/some_file % rsync -a -v --delete --delete-delay src/ dst/ sending incremental file list foo/ foo/some_file sent 123 bytes received 35 bytes 316.00 bytes/sec total size is 0 speedup is 0.00 % find src dst -ls 116737 1 drwx-- 3 arthur users 1024 Jun 2 23:48 src 116738 1 drwx-- 2 arthur users 1024 Jun 2 23:48 src/foo 116739 0 -rw--- 1 arthur users 0 Jun 2 23:48 src/foo/some_file 120833 1 drwx-- 3 arthur users 1024 Jun 2 23:48 dst 120834 1 drwx-- 2 arthur users 1024 Jun 2 23:48 dst/foo 120835 0 -rw--- 1 arthur users 0 Jun 2 23:48 dst/foo/some_file So far so good. Now let's reorganise our src directory (note the absolute symlink): % mv src/foo src/bar % ln -s /tmp/src/bar src/foo % find src -ls 116737 1 drwx-- 3 arthur users 1024 Jun 2 23:49 src 116738 1 drwx-- 2 arthur users 1024 Jun 2 23:48 src/bar 116739 0 -rw--- 1 arthur users 0 Jun 2 23:48 src/bar/some_file 116740 0 lrwxrwxrwx 1 arthur users12 Jun 2 23:49 src/foo -> /tmp/src/bar % rsync -a -v --delete --delete-delay src/ dst/ sending incremental file list ./ deleting foo/some_file foo -> /tmp/src/bar bar/ bar/some_file deleting foo/some_file rsync: delete_file: rmdir(foo) failed: Not a directory (20) sent 164 bytes received 41 bytes 410.00 bytes/sec total size is 12 speedup is 0.06 % find src dst -ls 116737 1 drwx-- 3 arthur users 1024 Jun 2 23:49 src 116738 1 drwx-- 2 arthur users 1024 Jun 2 23:50 src/bar 116740 0 lrwxrwxrwx 1 arthur users12 Jun 2 23:49 src/foo -> /tmp/src/bar 120833 1 drwx-- 3 arthur users 1024 Jun 2 23:49 dst 120835 1 drwx-- 2 arthur users 1024 Jun 2 23:48 dst/bar 120836 0 -rw--- 1 arthur users 0 Jun 2 23:48 dst/bar/some_file 120834 0 lrwxrwxrwx 1 arthur users12 Jun 2 23:49 dst/foo -> /tmp/src/bar The rsync errored out (which is anoying) but the real problem is that some_file was removed from the src directory altogether. This can have more grave consequences if the symlink points to some other place. This could be considered a security bug because the source end can trick the destination end to remove arbitrary files on the destination (I only tested this in this local setup though). Note that this only seems to happen with --delete-delay (at least not with --delete-after and --delete-during). Also note that this does not occur with the version from sid (3.0.5-1). -- System Information: Debian Release: 5.0.1 APT prefers stable APT policy: (500, 'stable') Architecture: i386 (x86_64) Kernel: Linux 2.6.26-1-amd64 (SMP w/2 CPU cores) Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968) Shell: /bin/sh linked to /bin/bash Versions of packages rsync depends on: ii base-files5lenny2Debian base system miscellaneous f ii libacl1 2.2.47-2 Access control list shared library ii libc6 2.7-18 GNU C Library: Shared libraries ii libpopt0 1.14-4 lib for parsing cmdline parameters ii lsb-base 3.2-20 Linux Standard Base 3.2 init scrip rsync recommends no packages. Versions of packages rsync suggests: ii openssh-client1:5.1p1-5 secure shell client, an rlogin/rsh ii openssh-server1:5.1p1-5 secure shell server, an rshd repla -- no debconf information -- -- arthur - adej...@debian.org - http://people.debian.org/~adejong -- - End forwarded message - -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: WARNING: . . . .failed verification -- update discarded (will try again).
On Thursday 04 June 2009 10:28:00 you wrote: > JW (j...@mailsw.com) wrote on 3 June 2009 11:58: > >A nightly scripted rsync backup job is giving me this error > > > >WARNING: vm/escDebLenny14G-flat.vmdk failed verification -- update > > discarded (will try again). > >However, I haven't seen any explanations for the "will try again" part. > After pulling a file rsync verifies its checksum; if it fails rsync > tries once more, and if it still fails rsync gives up and reports the > error. Therefore if there is no final error message the file was > successfully pulled. Thank you for answering that question. > This is explained in http://rsync.samba.org/how-rsync-works.html It so happens that that page does not contain the of the phrases "will try again", "failed verification", "update discarded", -- no wonder I couldn't find that via a web search. Thank you, JW -- -- System Administrator - Cedar Creek Software http://www.cedarcreeksoftware.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem syncing large dataset
Hi Matt, Sorry for my late reply. Here is the output of dbx on the process. I cannot replicate this problem using non-optimized code. Only optimized code seems to hit this problem. I will try using gcc instead of Sun Studio CC and see if I can replicate this issue. The binary was compiled with "cc -xarch=v9" stopped in map_ptr (optimized) at line 188 in file "fileio.c" 188 rprintf(FERROR, "invalid len passed to map_ptr: %ld\n", (dbx) where =>[1] map_ptr(map = ???, offset = ???, len = ???) (optimized), at 0x100069a54 (line ~188) in "fileio.c" [2] send_deflated_token(f = ???, token = ???, buf = ???, offset = ???, nb = ???, toklen = ???) (optimized), at 0x100064584 (line ~353) in token.c" [3] matched(f = ???, s = ???, buf = ???, offset = ???, i = ???) (optimized), at 0x10004c15c (line ~118) in "match.c" [4] hash_search(f = ???, s = ???, buf = ???, len = ???) (optimized), at 0x10004cac4 (line ~270) in "match.c" [5] match_sums(f = ???, s = ???, buf = ???, len = ???) (optimized), at 0x10004d0ac (line ~370) in "match.c" [6] send_files(f_in = ???, f_out = ???) (optimized), at 0x100039224 (line ~337) in "sender.c" [7] start_client(argc = ???, argv = ???) (optimized), at 0x100049c8c (line ~1027) in "main.c" [8] main(argc = ???, argv = ???) (optimized), at 0x10004ad90 (line ~1502) in "main.c" cheers, maarten On 1/15/09 6:33 PM, Matt McCutchen wrote: > On Thu, 2009-01-15 at 17:33 +0100, Maarten Thibaut wrote: > >> When using rsync-3.0.2 through 3.0.5, I get this error on a large >> dataset syncing from machine-a to machine-b: >> >> $ /bin/rsync -aHSz /local/. machine-b:/local/. >> invalid len passed to map_ptr: -1737287498 >> rsync error: error in file IO (code 11) at fileio.c(188) [sender=3.0.5] >> > > It looks like you have a large file that is overflowing a 32-bit length > variable. This would be considered a bug in rsync. It would be a great > help if you would run the sending rsync under gdb: > > gdb --args /bin/rsync -aHSz /local/. machine-b:/local/. > > put a breakpoint on the line that prints that error message > (fileio.c:186 in rsync 3.0.5) and post the stack trace so we can see > which code path is failing to prevent overflow. > > -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
DO NOT REPLY [Bug 6438] New: rsync fails to understand IPv6 on Solaris
https://bugzilla.samba.org/show_bug.cgi?id=6438 Summary: rsync fails to understand IPv6 on Solaris Product: rsync Version: 3.0.6 Platform: x64 URL: http://bink.tajinc.org/tmp2/solaris_rsync_ipv6.patch OS/Version: Solaris Status: NEW Severity: major Priority: P3 Component: core AssignedTo: way...@samba.org ReportedBy: t...@tajinc.org QAContact: rsync...@samba.org running rsync as a daemon on Solaris fails to listen for IPv6 connections... even with the -6 flag. I traced this down to the configure script not correctly detecting that IPv6 is supported on Solaris/OpenSolaris. I've built rsync with a patch to configure.in which allows rsync to listen for IPv6 connections (see URL) -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: WARNING: . . . .failed verification -- update discarded (will try again).
Here is the mail from henri. Best regards, Vitorio Début du message réexpédié : De : henri Date : 4 juin 2009 00:30:32 HAEC À : Mac User FR Objet : Rép : WARNING: . . . .failed verification -- update discarded (will try again). I was going to write a daemon to manage suspend and resume of VM images during backups. After designing the system (on paper - not in code) I realized that this system was probably not required for dealing with VM backups. The reason I came to this conclusion was because I realized I could just enable auto-protect (automated snapshots of the VM) and then backup the live VM. Is this not the case? Any information or experiences would be very helpful. I would be particularly interested to know if anyone backed up running VM and then had issues rolling back to a known good snap-shot? Maybe the design (still in my draw) will actually be of some use? Thanks. Le 5 juin 09 à 00:42, JW a écrit : On Thursday 04 June 2009 03:29:49 you wrote: r you can use snapshots as suggested by henri. I don't know what would exactly a snapshot do. A snapshot by the virtualization program or by the filesystem? I think if the snapshot is made by the virtualization program it may be safe for ram recovering issues, but filesystem snapshot is not. You will still need to save regularly open files from the guest OS. Thank you for taking the time to answer. As an aside, I did not receive any emails from "henri" and i don't see any on the list - what did he say? I'm sure he's talking about VM snapshots, not FS. I am aware of the issues you raised and I do, in fact, backup the internal contents of the Guest OS. But I also backup the actual VM disk-image-files because recreating VMs from scratch is not fun - I can usually use the backup, and then "restore" whatever data might be missing "inside" the VM guest OS. At least the main system setup is preserved. In other words, it's a third-level backup just to save me time in the event of catastrophic failure. I also have an original copy of the VM from when it was first created (and that backup was made with the VM guest powered off). I have absolutely no idea about the "will try again" part. Maybe Wayne or Matt could give you the answer. Carlos' answer was pretty good. Thank you. JW -- -- System Administrator - Cedar Creek Software http://www.cedarcreeksoftware.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
