Re: Updating of access times
Karl - thanks heaps!
( I think I was looking at an old man page, that doesn't show that option).
Just what we need - thanks.
Rob.
On 10/2/20, 15:51, "Karl O. Pinc" wrote:
On Mon, 10 Feb 2020 03:27:39 +
"Bell, Robert \(IM, Clayton\) via rsync"
wrote:
> We have a scenario where we need to take a copy of a filesystem, and
> use rsync to do so, but would like to retain the original access
> times on the source files. I did not find an option to do this, and
> would be grateful for advice if this is possible.
From the man page:
--noatime do not alter atime when opening source files
Regards,
Karl
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Updating of access times
rsync Folks, Firstly, thanks heaps to the maintainers of rsync - such a valuable utility. We have a scenario where we need to take a copy of a filesystem, and use rsync to do so, but would like to retain the original access times on the source files. I did not find an option to do this, and would be grateful for advice if this is possible. If not, we have a case where an enhancement (--restore-access-times ?) would be valuable, to us at least. --restore-access-times rsync restores the pre-existing access time after accessing a file (System dump utilities like xfsdump do restore (or not update) access times.) Thanks Rob. Dr Robert C. Bell Retirement Fellow CSIRO Australia Information Management and Technology Scientific Computing -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Segmentation fault using rsync client on Android device
So I went through the effort of setting up gdbserver on my Android device and using gdb to break when a segfault happens. The issue was because if iconv. Apparently when cross compiling rsync, it automatically picks up the iconv on the *HOST* system instead of throwing an error. There's also apparently no explicit method of telling the configure script where to find a cross-compiled iconv library, so I had to manipulate CFLAGS / LDFLAGS to get it to work. Here's the script I ran: ``` RSYNC_VERSION=rsync-3.1.3 download_and_untar $RSYNC_VERSION \ https://rsync.samba.org/ftp/rsync/$RSYNC_VERSION.tar.gz cd $RSYNC_VERSION ./prepare-source ./configure \ --prefix=$INSTALL_ROOT/rsync \ --host=$AUTOCONF_PLATFORM \ CFLAGS="-I$INSTALL_ROOT/iconv/include $CFLAGS" \ LDFLAGS="-L$INSTALL_ROOT/iconv/lib $LDFLAGS" make -j8 && make install cd - ``` Prior to running the build for rsync, I cross-compiled iconv library and installed it to `$INSTALL_ROOT/iconv`. From there, all I had to do was tell rsync (via compiler flags) where to find it. I think an option to autoconf would be better, something like `--with-iconv=$INSTALL_ROOT/iconv` so it knows where to find the includes & libs. I'm able to run rsync on my device now without crashes. On Tue, Nov 5, 2019 at 3:29 PM Robert Dailey wrote: > > I've compiled rsync for my rooted Android (armeabi-v7a, 32-bit) device > using these instructions: > > ``` > clone_or_update rsync master git://git.samba.org/rsync.git > cd rsync > ./prepare-source > ./configure \ >--prefix=$INSTALL_ROOT/rsync \ >--host=$AUTOCONF_PLATFORM \ >CFLAGS="$CFLAGS -static" > make -j8 && make install > cd - > ``` > > Best place for instructions to build I could find were here: > https://stackoverflow.com/a/18032833/157971 > > I didn't get any compiler or linker errors. It seemed to work. > > I copied the `bin/rsync` executable to the device in the `/data` > directory. I then executed this command: > > ``` > ./rsync - --stats --progress -rdt rsync://192.168.1.205:12000 > ``` > > I get this output: > > ``` > opening tcp connection to 192.168.1.205 port 12000 > Connected to 192.168.1.205 (192.168.1.205) > Segmentation fault > ``` > > Not much to go on. I tried running the same command via rsync in > cygwin and it worked fine: > > ``` > FILE_STRUCT_LEN=24, EXTRA_LEN=4 > opening tcp connection to 192.168.1.205 port 12000 > Connected to 192.168.1.205 (192.168.1.205) > msg checking charset: UTF-8 > sending daemon args: --server --sender -tre.iLsfxC . / (5 args) > ziosk Ziosk Content Files > ``` > > Can anyone help me understand why rsync is crashing on Android? Note > that I compiled rsync using the Android NDK (r20). I set up the > environment (CC, LD, etc) to point to the correct toolchain in the > NDK. It's how I build all of my third party libraries as well, so I > know at least as far as the environment goes, it's all set up > correctly. > > Are there dependencies missing that could cause this problem? I did > not provide any dependencies to rsync when I compiled it. > > Thanks in advance for any help. I'm stuck. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Segmentation fault using rsync client on Android device
I've compiled rsync for my rooted Android (armeabi-v7a, 32-bit) device using these instructions: ``` clone_or_update rsync master git://git.samba.org/rsync.git cd rsync ./prepare-source ./configure \ --prefix=$INSTALL_ROOT/rsync \ --host=$AUTOCONF_PLATFORM \ CFLAGS="$CFLAGS -static" make -j8 && make install cd - ``` Best place for instructions to build I could find were here: https://stackoverflow.com/a/18032833/157971 I didn't get any compiler or linker errors. It seemed to work. I copied the `bin/rsync` executable to the device in the `/data` directory. I then executed this command: ``` ./rsync - --stats --progress -rdt rsync://192.168.1.205:12000 ``` I get this output: ``` opening tcp connection to 192.168.1.205 port 12000 Connected to 192.168.1.205 (192.168.1.205) Segmentation fault ``` Not much to go on. I tried running the same command via rsync in cygwin and it worked fine: ``` FILE_STRUCT_LEN=24, EXTRA_LEN=4 opening tcp connection to 192.168.1.205 port 12000 Connected to 192.168.1.205 (192.168.1.205) msg checking charset: UTF-8 sending daemon args: --server --sender -tre.iLsfxC . / (5 args) ziosk Ziosk Content Files ``` Can anyone help me understand why rsync is crashing on Android? Note that I compiled rsync using the Android NDK (r20). I set up the environment (CC, LD, etc) to point to the correct toolchain in the NDK. It's how I build all of my third party libraries as well, so I know at least as far as the environment goes, it's all set up correctly. Are there dependencies missing that could cause this problem? I did not provide any dependencies to rsync when I compiled it. Thanks in advance for any help. I'm stuck. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Multiple errors from rsync - later 'soft' error raised rather than earlier 'hard' error.
Peter, I was about to post the below to the rsync mailing list, but wondered whether your code which captures rsync output may have allowed rsync to continue past what I think would normally be a fatal error? Thanks Rob. P.S. Should I put the final backups of the old pearcey /home back into their correct slots - 5 backups were recycled, but nothing was added to them? The .Snapshots area is pointing to the old locations. Here are the recycles that need to be reversed: Recycling /backup/pearcey-user-dumps/home_intel.20190606.seq.1687.recycle to /backup/pearcey-user-dumps/home_intel.20190611.seq.1692 Recycling /backup/pearcey-user-dumps/home_intel.20190608.seq.1689.recycle to /backup/pearcey-user-dumps/home_intel.20190612.seq.1694 Recycling /backup/pearcey-user-dumps/home_intel.20190605.seq.1686.recycle to /backup/pearcey-user-dumps/home_intel.20190612.seq.1693 Recycling /backup/pearcey-user-dumps/home_intel.20190609.seq.1690.recycle to /backup/pearcey-user-dumps/home_intel.20190613.seq.1696 Recycling /backup/pearcey-user-dumps/home_intel.20190603.seq.1684.recycle to /backup/pearcey-user-dumps/home_intel.20190613.seq.1695 Rsync Folks, We recently encountered a case with rsync hitting what I regard as a 'hard' error (unable to change_dir to the source), but it did not immediately abort, but continued and went on to hit other errors (which I regard as soft) and returned the error code 23. Our backup suite continues after error 23, because it is common and usually not a problem. Here is selected output. rsync: change_dir "/home_intel" (in backups) failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1637) [Receiver=3.1.1+csiro-sc.4] rsync: read error: Connection reset by peer (104) Should rsync have aborted after encountering error 2? Regards Rob. Dr Robert C. Bell Retirement Fellow CSIRO Information Management and Technology Scientific Computing Mobile +61 428 108 333 robert.b...@csiro.au<mailto:robert.b...@csiro.au> | www.csiro.au https://confluence.csiro.au/display/SC/Scientific+Computing+Homepage T +61 3 9545 2368 (Clayton) Street: CSIRO IMT, Rivett Bldg 001 room RNW 2.12, Clayton Vic 3168, Australia Office: w3w: http://w3w.co/depend.jumpy.faster Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
3.1.3 not building on OS 10.14
Hi all, I have been using v 3.1.1 for some time which builds fine. Just tried v3.1.3 and getting this error at make: rsync.c:623:40: error: too few arguments to function call, expected 4, have 2 if (cmp_time(sxp->crtime, file_crtime) != 0 ^ ./proto.h:402:1: note: 'cmp_time' declared here int cmp_time(time_t f1_sec, unsigned long f1_nsec, time_t f2_sec, unsign... ^ 1 error generated. make: *** [rsync.o] Error 1 I am using the standard Mac OS patches on OS 10.14.4 patch -p1 https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
very slow performance OS X xattrs
Hi All, I am have been cloning drives with several versions of xcode and am seeing very long copy times. Perhaps just because of xcode I am noticing the change. Xcode of course has many files (400,000+) many xattrs and compressed bits, decmpfs… so only natural it will be slow but sometimes upwards of 1/2 hour to a fast drive for xcode. If I turn off Xattrs (leave out the X) it flies through in three minutes or so. I don’t know if there is anything to be done but wonder if anyone has any ideas or tweaks to speed up those xattrs? Am using rsync 3.1.1 but would assume no new changes in 3.1.2 to affect any of this. usual OS X build (on Sierra) patch -p1 https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync Digest, Vol 162, Issue 18
Folks, Am 20.06.2016 um 22:01 schrieb Larry Irwin (gmail): > The scripts I use analyze the rsync log after it completes and then sftp's a summary to the root of the just completed rsync. > If no summary is found or the summary is that it failed, the folder rotation for that set is skipped and that folder is re-used on the subsequent rsync. > The key here is that the folder rotation script runs separately from the rsync script(s). That is what we found to be important some years ago - do most of the management outside of the scripts that do rsync. In particular, our scripts prepare a target backup directory with a name of the form: slash.20151122.seq.1379.current - the name of the area being backed-up is obvious, as is the date. We use Tower of Hanoi management (see the update at https://en.wikipedia.org/wiki/Backup_rotation_scheme#Tower_of_Hanoi), so embed a sequence number in the backup directory name, and the suffix .current to indicate a current backup target. we can keep on trying the rsync backups until we get success, and the scripts then remove the .current suffix. This works because the rsync includes the --delete option, used since the backup target can be a recycled hard-linked directory: also recommended. > For each entity I want to rsync, I create a named folder to identify it and the rsync'd data is held in sub-folders: > daily.[1-7] and monthly.[1-3] > When I rsync, I rsync into daily.0 using daily.1 as the link-dest. > Then the rotation script checks daily.0/rsync.summary - and if it worked, it removes daily.7 and renames the daily folders. > On the first of the month, the rotation script removes monthly.3, renames the other 2 and makes a complete hard-link copy of daily.1 to monthly.1 > It's been running now for about 4 years and, in my environment, the 10 copies take about 4 times the space of a single copy. > (we do complete copies of linux servers - starting from /) > If there's a good spot to post the scripts, I'd be glad to put them up. I'd recommend Tower of Hanoi, as referenced above, for managing sets of backups - far simpler than dealing with days and dates, etc. For example, the above mentions special actions on the first of the month - what if that is missed because of some failure - more special case scripting? Tower of Hanoi is self-healing Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing CSIRO Information Management and Technology T +61 3 9545 2368 (Docklands and Clayton) T +61 3 9669 8102 (Docklands only) Mob +61 428 108 333 robert.b...@csiro.au<mailto:robert.b...@csiro.au> | www.csiro.au https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage Also: Bureau of Meteorology Scientific Computing Services Street: CSIRO SC Lvl 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO SC Lvl 11, GPO Box 1289, Melbourne Vic 3001, Australia Street: CSIRO IMT, Rivett Room RNW 2.03, Clayton Vic 3168, Australia Postal: CSIRO IMT, Private Bag 10, Clayton South Vic 3169, Australia Parcels: CSIRO IMT, Gate 3 Normanby Road, Clayton Vic 3168, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. On 22/06/2016 10:00 pm, rsync-requ...@lists.samba.org wrote: Send rsync mailing list submissions to rsync@lists.samba.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.samba.org/mailman/listinfo/rsync or, via email, send a message with subject or body 'help' to rsync-requ...@lists.samba.org You can reach the person managing the list at rsync-ow...@lists.samba.org When replying, please edit your Subject line so it is more specific than "Re: Contents of rsync digest..." To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html --- Today's Topics: 1. Re: rsync script for snapshot backups (Dennis Steinkamp) 2. Re: rsync script for snapshot backups (Petros Angelatos) ___ Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html rsync mailing list rsync@lists.samba.org https://lists.samba.org/mailman/listinfo/rsync -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before p
Re: Why is my rsync transfer slow?
Just chiming in here, I haven’t read all the previous posts so may be repeating… Mike Bombich has a good piece on benchmarks for various source/destination scenarios with rsync. https://bombich.com/kb/ccc3/how-long-should-clone-or-backup-take Note that copying to sparsebundle on local media is as fast as to local disk. Sparsebundle is a mac filesystem whereas NAS is not so certain file metadata are not covered like Hard links,Acls and groups and owners. So it is important to turn off those rsync options to scan those file parts on NAS. But sparsebundle on NAS is ok for HFS options. Network backups are always slower but you can see that sparseimage on network volume (AFP) is better according to Mike’s chart. I also develop an rsync based backup app (backupList+) and some users have reported very slow times even to sparse images on NAS but I am not the expert Mike is on these matters. Network issues vary for sure…. Cheers, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync - detecting change/no-change without an itemised list
rsync watchers, I've been using rsync version 3.1.1+, and do like the improved --stats output: e.g. Number of files: 2480403 (reg: 2034290, dir: 233487, link: 212174, special: 452) Number of created files: 356 (reg: 6, dir: 3, link: 347) Number of regular files transferred: 16 ... - thanks to the developers. However, I want to be able to look at the --stats output and make a decision about whether anything changed or not, and would rather not have to turn on -i and count the entries. Here's an example: - setting up source and dest directories - setting up a file in source - rsyncing source to dest - changing the permissions on the source file - redoing rsync - looking at the --stats output, and finding no clue in the --stats output that anything changed, whereas the -i output shows a change. (The files are local, so --whole-file is in effect). == prompt> mkdir source dest prompt> echo a > source/a prompt> /usr/local/bin/rsync -a -i --stats source/ dest .d..t.. ./ f+ a Number of files: 2 (reg: 1, dir: 1) Number of created files: 1 (reg: 1) Number of deleted files: 0 Number of regular files transferred: 1 Total file size: 2 bytes Total transferred file size: 2 bytes Literal data: 2 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 122 Total bytes received: 38 sent 122 bytes received 38 bytes 320.00 bytes/sec total size is 2 speedup is 0.01 prompt> chmod g+rw source/a prompt> /usr/local/bin/rsync -a -i --stats source/ dest .f...p. a Number of files: 2 (reg: 1, dir: 1) Number of created files: 0 Number of deleted files: 0 Number of regular files transferred: 0 Total file size: 2 bytes Total transferred file size: 0 bytes Literal data: 0 bytes Matched data: 0 bytes File list size: 0 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 77 Total bytes received: 19 sent 77 bytes received 19 bytes 192.00 bytes/sec total size is 2 speedup is 0.02 prompt> == Any clues please? Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing CSIRO Information Management and Technology T +61 3 9545 2368 (Docklands and Clayton) T +61 3 9669 8102 (Docklands only) Mob +61 428 108 333 robert.b...@csiro.au<mailto:robert.b...@csiro.au> | www.csiro.au https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage Street: CSIRO SC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO SC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia Street: CSIRO IM, Rivett Room RNW 2.03, Clayton Vic 3168, Australia Postal: CSIRO IM, Private Bag 10, Clayton South Vic 3169, Australia Parcels: CSIRO IM, Gate 3 Normanby Road, Clayton Vic 3168, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Forwarded message -- Date: Wed, 18 Nov 2015 12:00:02 + From: rsync-requ...@lists.samba.org Reply-To: rsync@lists.samba.org To: rsync@lists.samba.org Subject: rsync Digest, Vol 155, Issue 4 Resent-From: <robert.b...@csiro.au> Send rsync mailing list submissions to rsync@lists.samba.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.samba.org/mailman/listinfo/rsync or, via email, send a message with subject or body 'help' to rsync-requ...@lists.samba.org You can reach the person managing the list at rsync-ow...@lists.samba.org When replying, please edit your Subject line so it is more specific than "Re: Contents of rsync digest..." -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 11523] New: Request: Add option to unlink hard links when permissions change
Folks, b...@stupidkitties.com wrote: " I was wondering if it'd be feasible to add an option (or multiple options, depending on the desired granularity) to cause rsync to treat changes in file attributes the same way it treats changes to the file's contents when it encounters hard links at the destination. Maybe something like --relink-p or something?" One of colleagues has produced a patch to do just what you want, but we have not promulgated it yet. Sorry. Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing CSIRO Information Management and Technology T +61 3 9545 2368 (Docklands and Clayton) T +61 3 9669 8102 (Docklands only) Mob +61 428 108 333 robert.b...@csiro.au<mailto:robert.b...@csiro.au> | www.csiro.au https://wiki.csiro.au/display/ASC/Scientific+Computing+Homepage Street: CSIRO SC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO SC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia Street: CSIRO IM, Rivett Room RNW 2.03, Clayton Vic 3168, Australia Postal: CSIRO IM, Private Bag 10, Clayton South Vic 3169, Australia Parcels: CSIRO IM, Gate 3 Normanby Road, Clayton Vic 3168, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Recycling directories and backup performance. Was: Re: rsync --link-dest won't link even if existing file is out of date (fwd)
rsync folks, Henri Shustak henri.shus...@gmail.com wrote: LBackup always starts a new backup snapshot with an empty directory. I have been looking at extending --link-dest options to scan beyond just the previous successful backup to (failed backups / older backups). However, there are all kinds of edge cases which are worth considering with such a changes. At present LBackup is focused on reliability as such, this RD is quite slow given limited resources. The current version of LBackup offers IMHO reliable backups of user data and the scripting sub-system offers a high degree of flxibility. We recycle directories in our backup scheme, and on tests it is 3 to 6 times faster than creating a new directory tree and then deleting an old one. Your timing will be different - the speed depends on the relative numbers of files and directories I'd imagine. Half our recycled directories are 5 to 6 days old, and our churn rate is typically only 0.5% of files and 1% of data each day. So, the recycled directory is usually about 95% right. Our backup procudures have provision for looking back at previous directories, but there is not much to be gained with recycled directories. Without recycling, and after a failure, the latest available backup may not have much in it, and won't be a good place to link-dest from - you need to go further back, as Henry is considering. Yes, every time you start a backup snapshot, a directory is re-populated from scratch and this takes time with LBackup. However, if you are seeking reliability then you may wish to check out the following URL : http://www.lbackup.org We rarely have a failure with our backups. If we do, our procedure just re-labels the unfinished directory, and re-syncs as normal on the next attempt. And, in another post, Henri Shustak henri.shus...@gmail.com gave good advice on splitting up backups, etc to get performance, for someone whose original post I didn't find. Ill take a look but I imagine I cant backup the 80 Million files I need to in under the 5 hours i have for nightly maintenance/backups. Currently it's possible by recycling directories... Here are some performance figures from our backup yesterday. (We have multiple streams to several filesystems). The backups completed in 41 minutes. They transferred: 80533 files out of 18127734 files available (0.4%) 57871616693 bytes out of 6827716377557 bytes available (0.8%) - so, with this low churn rate, we could backup 80 million files in about 3 hours. Our backup target filesystems include SSD and are managed by SGI's DMF. Hope this helps. Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Patch for rsync --link-dest won't link even if existing file is out of date (fwd)
Folks, We faced a similar situation to that which Ken described - we recycle backup directories, for good reason. There is a patch to solve the problem. Our systems administrator provided the following description of the patches we use: 1. rsync_link_dest improvement by Bryant Hansen Normally, existing files in destination are never updated from link-dest but are transferred over the wire. This patch changes that behaviour to use link-dest instead, which is a major performance enhancement in our environment. 2. Warnings for --max-size ignored files are displayed if -w/--warning is specified by Rowan McKenzie (CSIRO SC) Warnings for -max-size ignored files are displayed if -w/-warning is specified. Normally, -max-size causes files to be silently ignored! 3. Only output '=' notifications when -v/--verbose specified by Rowan McKenzie (CSIRO SC) Only output '#' notifications when -v/-verbose specified (it's a patch to the rsync_link_dest_from_bryant patch). This reduces clutter by suppressing a large class of false positives.: Hope you can find these. (All we need now for rsync perfection for our backups is a solution to the problem of metadata changes being propagated across all directories for hard-linked files - we would rather new copies be made than lose the old metadata.) Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Forwarded message -- Date: Mon, 6 Apr 2015 01:51:21 -0400 From: Ken Chase rsync-list-m...@sizone.org To: rsync@lists.samba.org Subject: rsync --link-dest won't link even if existing file is out of date Feature request: allow --link-dest dir to be linked to even if file exists in target. This statement from the man page is adhered to too strongly IMHO: This option works best when copying into an empty destination hierarchy, as rsync treats existing files as definitive (so it never looks in the link-dest dirs when a destination file already exists). I was suprised by this behaviour as generally the scheme is to be efficient/save space with rsync. When the file is out of date but exists in the --l-d target, it would be great if it could be removed and linked. If an option was supplied to request this behaviour, I'd actually throw some money at making it happen. (And a further option to retain a copy if inode permissions/ownership would otherwise be changed.) Reasoning: I backup many servers with --link-dest that have filesystems of 10+M files on them. I do not delete old backups - which take 60min per tree or more just so rsync can recreate them all in an empty target dir when 1% of files change per day (takes 3-5 hrs per backup!). Instead, I cycle them in with mv $olddate $today then rsync --del --link-dest over them - takes 30-60 min depending. (Yes, some malleability of permissions risk there, mostly interested in contents tho). Problem is, if a file exists AT ALL, even out of date, a new copy is put overtop of it per the above man page decree. Thus much more disk space is used. Running this scheme with moving old backups to be written overtop of accumulates many copies of the exact same file over time. Running pax -rpl over the copies before rsyncing to them works (and saves much space!), but takes a very long time as it traverses and compares 2 large backup trees thrashing the same device (in the order of 3-5x the rsync's time, 3-5 hrs for pax - hardlink(1) is far worse, I suspect a some non-linear algorithm therein - it ran 3-5x slower than pax again). I have detailed an example of this scenario at http://unix.stackexchange.com/questions/193308/rsyncs-link-dest-option-does-not-link-identical-files-if-an-old-file-exists which also indicates --delete-before and --whole-file do not help at all. /kc -- Ken Chase - k...@heavycomputing.ca skype:kenchase23 +1 416 897 6284 Toronto Canada Heavy Computing - Clued bandwidth, colocation and managed linux VPS @151 Front St. W
Possible Error in rsync 3.0.9
Hi I was using rsync with the following options rsync -aPH --exclude-from=/var/tmp/ignorelist /home/me/ /run/media/me/ADATA\ #2/rsync-me/ and I was creating a copy of a home directory on my Linux machine. It went along for sometime (some 30,000 files) when it stopped with the following message. --- Desktop/OSme/Web-Site-Backups/backup-5.4.2014_15-11-43_linux014.tar.gz 4288446464 78% 32.15MB/s0:00:36 rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: connection unexpectedly closed (1964386 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9] [root@puter ~]# --- I suspect, because the 4,288,446,464 is so close to 2^32 that it is 32 bit limit error of some kind. It happened twice on the same file. The version I am using is rsync.x86_64 version on Fedora 21. It appears that although it says it is a 64 bit version something in the code must be limited by 32 bit values. I could be off track here but I don't think so. I do not know how to raise a Possible Bug Report so I post my concerns here. Thanks... Robert -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.1.1 wrong transfer of app packages using --backup
I have noticed another glitch,
On OSX with the 3 usual patches fileflags, crtimes, hfs_compression and using
—backup and —backup-dir=PATH
Whenever source has a trailing slash, the backup dir is doubled in depth.
So when backup_Dir = “BL_Archive and source is “some/folder/path” the usual
correct result at root of backup folder is
“BL_Archive/some/file/path”
but when trailing slash = “some/folder/path/“
result = “BL_Archive/BL_Archive/some/file/path”
I tried this copying to different locations, root, nested etc and all the same
result. ON 3.0.9 the slash does not do this - you get the normal backup_Dir.
backup_dir_buf before stringjoin(…) in copy_valid_path is normal and after
stringjoin(…) it becomes the “doubleDir.” which goes to do_mkdir.
Again, have tried for days to sort this out but can’t follow the trail right.
Thanks, Rob
On Jan 22, 2015, at 7:16 PM, Robert DuToit rdut...@comcast.net wrote:
Dear All,
I found a workaround for the corrupted backup app package files though there
is probably a more intelligent way.
When the destination is empty or missing (rather than an update)
rsync.c calls make_backup and returns 1 when it encounters
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
the calling code in rsync.c:
if ( make_backups 0 overwriting_basis) {
int ok = make_backup(fname, False);
if (!ok)
return 1;
if (ok == 1 fnamecmp == fname)
fnamecmp = get_backup_name(fname);
}
in 3.0.9 get_backup_name just returns the backup_Dir path but in 3.1.1 it
proceeds to copy_valid_path which then creates the folder structure for the
DIR, which in this case is an APP package or rtfd file which should not be
copied at all. I don’t think any dirs should be created if there are no
backups to be made so probably the older simple get_backup_name is what is
needed here. I don’t know if it is possible to differentiate app packages
from regular directories and it doesn’t matter if the app file is really
being copied to the backup_Dir. But the backup Dirs are being created in this
case though there are no backups to be made.
As a workaround I moved
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
to the begiining in make_backup and added it also to copy_valid_path so it
would return the call from rsync instead of making screwy directories.
This is my diff, assuming one runs fileflags, crtimes, and hfs_compression
diffs first for OSX and rsync 3.1.1.
Rob
diff --git a/backup.c b/backup.c
--- a/backup.c
+++ b/--- a/backup.c
@@ -65,6 +65,9 @@
BOOL ret = True;
stat_x sx;
char *b, *rel = backup_dir_buf + backup_dir_len, *name = rel;
+
+if (x_lstat(fname, sx.st, NULL) 0)
+ return 1;
for (f = fname, b = rel; *f *f == *b; f++, b++) {
if (*b == '/')
@@ -215,17 +218,18 @@
stat_x sx;
struct file_struct *file;
int save_preserve_xattrs;
- char *buf = get_backup_name(fname);
- int ret = 0;
-
- if (!buf)
- return 0;
-
- init_stat_x(sx);
+int ret = 0;
+
+init_stat_x(sx);
/* Return success if no file to keep. */
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
-
+
+char *buf = get_backup_name(fname);
+
+ if (!buf)
+ return 0;
+
/* Try a hard-link or a rename first. Using rename is not atomic, but
* is more efficient than forcing a copy for larger files when no hard-
* linking is possible. */
On Jan 16, 2015, at 9:41 AM, Robert DuToit rdut...@comcast.net wrote:
Dear All,
I have been seeing some strange behavior using rsync 3.1.1 on OSX with the 3
standard patches applied when using the --backup and --backup-Dir options.
This works as expected on 3.0.9, moving any files on dest that are not on
source to the backup folder. If no changes then the backup folder is not
even created.
-aHAXN --fileflags --force-change --protect-decmpfs —delete --backup
--backup-dir=BL_Archive --stats —progress -vvv
on 3.1.1 , where dest and source are already in sync, any apps and rtfd
files (both are “packages” of other files) on the dest are replicated in the
backup folder. They aren’t removed from the dest. They have a cross through
them (corrupted) and I noticed that not all the files within these corrupted
apps are present, mostly nib files, frameworks etc..
Anyone else experience this?
I ran with verbose but the only references to the backup folder are setting
uids on that backup folder. It is created early on in the output.
Created backup_dir BL_Archive/
set uid of BL_Archive/testfolder from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
set uid of BL_Archive/testfolder
Re: rsync 3.1.1 wrong transfer of app packages using --backup
Dear All,
I found a workaround for the corrupted backup app package files though there is
probably a more intelligent way.
When the destination is empty or missing (rather than an update)
rsync.c calls make_backup and returns 1 when it encounters
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
the calling code in rsync.c:
if ( make_backups 0 overwriting_basis) {
int ok = make_backup(fname, False);
if (!ok)
return 1;
if (ok == 1 fnamecmp == fname)
fnamecmp = get_backup_name(fname);
}
in 3.0.9 get_backup_name just returns the backup_Dir path but in 3.1.1 it
proceeds to copy_valid_path which then creates the folder structure for the
DIR, which in this case is an APP package or rtfd file which should not be
copied at all. I don’t think any dirs should be created if there are no backups
to be made so probably the older simple get_backup_name is what is needed here.
I don’t know if it is possible to differentiate app packages from regular
directories and it doesn’t matter if the app file is really being copied to the
backup_Dir. But the backup Dirs are being created in this case though there are
no backups to be made.
As a workaround I moved
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
to the begiining in make_backup and added it also to copy_valid_path so it
would return the call from rsync instead of making screwy directories.
This is my diff, assuming one runs fileflags, crtimes, and hfs_compression
diffs first for OSX and rsync 3.1.1.
Rob
diff --git a/backup.c b/backup.c
--- a/backup.c
+++ b/--- a/backup.c
@@ -65,6 +65,9 @@
BOOL ret = True;
stat_x sx;
char *b, *rel = backup_dir_buf + backup_dir_len, *name = rel;
+
+if (x_lstat(fname, sx.st, NULL) 0)
+ return 1;
for (f = fname, b = rel; *f *f == *b; f++, b++) {
if (*b == '/')
@@ -215,17 +218,18 @@
stat_x sx;
struct file_struct *file;
int save_preserve_xattrs;
- char *buf = get_backup_name(fname);
- int ret = 0;
-
- if (!buf)
- return 0;
-
- init_stat_x(sx);
+int ret = 0;
+
+init_stat_x(sx);
/* Return success if no file to keep. */
if (x_lstat(fname, sx.st, NULL) 0)
return 1;
-
+
+char *buf = get_backup_name(fname);
+
+ if (!buf)
+ return 0;
+
/* Try a hard-link or a rename first. Using rename is not atomic, but
* is more efficient than forcing a copy for larger files when no hard-
* linking is possible. */
On Jan 16, 2015, at 9:41 AM, Robert DuToit rdut...@comcast.net wrote:
Dear All,
I have been seeing some strange behavior using rsync 3.1.1 on OSX with the 3
standard patches applied when using the --backup and --backup-Dir options.
This works as expected on 3.0.9, moving any files on dest that are not on
source to the backup folder. If no changes then the backup folder is not even
created.
-aHAXN --fileflags --force-change --protect-decmpfs —delete --backup
--backup-dir=BL_Archive --stats —progress -vvv
on 3.1.1 , where dest and source are already in sync, any apps and rtfd
files (both are “packages” of other files) on the dest are replicated in the
backup folder. They aren’t removed from the dest. They have a cross through
them (corrupted) and I noticed that not all the files within these corrupted
apps are present, mostly nib files, frameworks etc..
Anyone else experience this?
I ran with verbose but the only references to the backup folder are setting
uids on that backup folder. It is created early on in the output.
Created backup_dir BL_Archive/
set uid of BL_Archive/testfolder from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501
set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501
Thanks, Rob
smime.p7s
Description: S/MIME cryptographic signature
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 3.1.1 wrong transfer of app packages using --backup
Hi Again, Update : I have spent some hours trying to track down where this happens, mostly in backup.c. This just seems to concern any files which are .app or .nib or .rtfd as they are packages and act like directories. As below, 3.0.9 does not try to copy these at all. This only happens when there is no source file present on destination. It then sends the app package folder to the backup_Dir which is corrupted since it contains nothing inside. It seems that it is making the backup directories for the .app package and also the .nib files inside. This means it is corrupted on first pass and if copying many files there can be many of these corrupts in backup_Dir which is not great. So I am trying to find where in backup.c (or maybe rsync.c) this decision is being made. Probably somewhere between validate_backup_dir and copy_valid_path. My C skills aren’t that great so following this path is difficult. Thanks, Rob On Jan 16, 2015, at 9:41 AM, Robert DuToit rdut...@comcast.net wrote: Dear All, I have been seeing some strange behavior using rsync 3.1.1 on OSX with the 3 standard patches applied when using the --backup and --backup-Dir options. This works as expected on 3.0.9, moving any files on dest that are not on source to the backup folder. If no changes then the backup folder is not even created. -aHAXN --fileflags --force-change --protect-decmpfs —delete --backup --backup-dir=BL_Archive --stats —progress -vvv on 3.1.1 , where dest and source are already in sync, any apps and rtfd files (both are “packages” of other files) on the dest are replicated in the backup folder. They aren’t removed from the dest. They have a cross through them (corrupted) and I noticed that not all the files within these corrupted apps are present, mostly nib files, frameworks etc.. Anyone else experience this? I ran with verbose but the only references to the backup folder are setting uids on that backup folder. It is created early on in the output. Created backup_dir BL_Archive/ set uid of BL_Archive/testfolder from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501 Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html smime.p7s Description: S/MIME cryptographic signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
wrong transfer of app packages using --backup
Dear All, I have been seeing some strange behavior using rsync 3.1.1 on OSX with the 3 standard patches applied when using the --backup and --backup-Dir options. This works as expected on 3.0.9, moving any files on dest that are not on source to the backup folder. If no changes then the backup folder is not even created. -aHAXN --fileflags --force-change --protect-decmpfs —delete --backup --backup-dir=BL_Archive --stats —progress -vvv on 3.1.1 , where dest and source are already in sync, any apps and rtfd files (both are “packages” of other files) on the dest are replicated in the backup folder. They aren’t removed from the dest. They have a cross through them (corrupted) and I noticed that not all the files within these corrupted apps are present, mostly nib files, frameworks etc.. Anyone else experience this? I ran with verbose but the only references to the backup folder are setting uids on that backup folder. It is created early on in the output. Created backup_dir BL_Archive/ set uid of BL_Archive/testfolder from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app/Contents from 0 to 501 set uid of BL_Archive/testfolder/AppKiDo.app/Contents/MacOS from 0 to 501 Thanks, Rob smime.p7s Description: S/MIME cryptographic signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Backup scripts - recycling old backup directories (Kevin Korb)
Kevin, Thanks for the reply and interest in this topic. Comments below. Regards Rob. I did consider that but rejected it for 2 reasons... 1. Backup run time. We have a 4 hour window to run backups at night. Using recycled directories significantly extended the backup run time. The deletion time is eliminated but frankly, we have the other 20 hours of the day to do deletions. We had to give up using - --link-dest when the deletions started to actually take that long even though the backups still ran in under 4 hours. For us, the recycling of old directories significantly shortened the time to do backups, since the recycled backups have typically 95% of the files/directories correct (with daily backups and Tower of Hanoi, half of our recycled backups are only 5 to 6 days old). I've just done some tests with a fairly pathological case, all on one host. I set up a source tree 's' with 1 sub-directories and 1 files, and then two destinations: cp -a s d1 cp -afl d1 d2 I then did the first test: # rsync to a new directory, followed by a remove of an old directory. time rsync -a --link-dest=../d2 s/ d3 time /bin/rm -rf d1 I then scrubbed the lot, set it up again, and did the second test: mv d1 d3 # rsync to a recycled directory time rsync -a --link-dest=../d2 --delete s/ d3 I hope I got this right! I've made no effort to circumvent caching. Anyway, here is a table of the average times (seconds) over 5 runs of each test. RealUserSys (User+Sys) test 1 2.454s 0.150s 2.196s 2.346s test 2 0.392s 0.100s 0.572s 0.672s ratio 6.3 1.5 3.8 3.5 (The User+Sys time is pretty much invariant, even though in earlier tests the real time suffered major blowouts owing to contention.) So, the big difference is that in test 1, the 1 sub-directories and 1 files were created in the destination d3, and then the same numbers were deleted from the old directory d1. In test 2, rsync does none of that, but only has to check for differences. ~40,000 metadata operations avoided on the filesystem in this case. 2. Metadata history. If there is an existing file in the target dir that differs only by metadata (permissions, ownership, timestamp) then rsync will simply change that metadata. That change affects all instances of that file. Of course this is better for storage space as the alternative is storing another copy of the file with the different metadata but we decided it was better to have that information saved. Yes. I would love to see someone make a patched version of rsync to allow callers to select a different behaviour in this case! So, if a file has identical content on source and destination but different metadata, then if --link-dest is in use and the link count on the destination is 1, then take a new copy from source rather than just updating the metadata (the file could be copied on the destination and then the copy updated with the new metadata and the old version removed, but this would not be essential - just perhaps an efficiency gain.) Thanks in anticipation! Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Recycling and keeping backups - Tower of Hanoi management of backups using rsync
Thanks to Kevin and Paul for responses. We use a modified Tower of Hanoi scheme (on top of rsync and --link-dest and recycling) for deciding which backups to keep. Here is a sample of our holdings for one area: home.2024.seq.0 set 0 home.20130512.seq.512 set 10 home.20140203.seq.768 set 9 home.20140414.seq.832 set 7 home.20140708.seq.896 set 8 home.20140815.seq.928 set 6 home.20140831.seq.944 set 5 home.20140904.seq.948 set 3 home.20140908.seq.950 set 2 home.20140909.seq.951 set 1 home.20140910.seq.952 set 4 home.20140911.seq.953 set 1 home.20140912.seq.954 set 2 Found 13 backups as expected up to sequence number 954 Marking for recycling home.20140908.seq.950, set number 2 The coverage matches the likelihood of restorations being required - the coverage tails off exponentially over time. I found the key to running a Tower of Hanoi scheme is to assign a sequence number to each backup, from which you can derive a set number. The dates are for humans - scripts don't have to deal with days of week, days of months, etc We keep one of each set number, except we keep two set ones, and a 'set 0' as a base set. With daily backups, any file that has existed for a period of more than about 1.5*n days in the last 2*n days will be covered, with better coverage than that for recent files. A strategy to keep two of every set will easily provide cover for every file that existed for n days in the last 2*n days. Kevin: you could use Tower of Hanoi for managing your snapshots... :-) Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Backup scripts - recycling old backup directories
Folks, Kevin Korb wrote: Have you considered more advanced methods such as subvolume snapshots provided by ZFS and BTRFS? At work we were forced to abandon rsync - --link-dest because of the amount of time it takes to delete old backups when the data is primarily many millions of small files (shared web hosting company). We don't have more advanced methods like subvolume snapshots available to us. However, we can recycle backup directories. When we started using rsync with --link-dest back in about 2007, we deleted old backups, but realised soon after that we could recycle old backups. With daily backups, we find about 1.5% of the data and 0.5% of the files change from one day to the next, so a directory from about 5 days ago will typically be only 5-10% wrong and can be recycled to be the target of the latest directory - that's a lot better than recreating the whole directory tree for a new backup, and then deleting a whole old directory tree. We use --delete of course. Hope this helps someone. Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delete not working with multiple source directories
HI kevin, No luck with —relative. I’ll try and see where in the code —delete gets bypassed. Rob On Aug 16, 2014, at 9:27 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Does --relative fix it? You almost always want --relative with multiple sources. On 08/16/2014 09:10 PM, Robert DuToit wrote: Hello All, I have lately heard from a couple of people that delete option was not deleting extraneous files on destination. This only happens when there are several source directories and at destination some show delete working and others not. I can not reproduce the issue so it is hard to pinpoint. one example: This is on OSX Mavericks running rsync 3.0.9 with the usual osx patches and options rsync -aHAXN --fileflags --force-change --protect-decmpfs --exclude-from=/tmp/BLexclusions.txt --stats —delete —progress -vvv /Applications /Library /Users /Volumes/Backup So with /Applications /Library /Users being source, only /Applications shows files being deleted. Removing /Applications and the two other directories work with delete fine. Even changing the order has same result. The exclude file shows only one directory in /Applications. I asked user to try different (new) external drive - same results. I’m stumped as there are no rsync errors. Rsync progress shows everything fine and mentions files deleted in /Applications and nowhere else. I am not very good with rsync source but perhaps there are areas I could look where --delete might be disabled for whatever reasons? There are usually ACL's on both /Applications and /Library in OSX 0: group:everyone deny delete but this is overridden by rsync as root any ways… usual regular permissions on all directories as well. Any thoughts appreciated Thanks, Rob - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlPwBRsACgkQVKC1jlbQAQcg4QCg20zvWlxWMlKwVZ0VFyiYPFQ0 tB0AoJ6M15dVXVyCbdnSweNLfwKMpAC2 =PY5O -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
delete not working with multiple source directories
Hello All, I have lately heard from a couple of people that delete option was not deleting extraneous files on destination. This only happens when there are several source directories and at destination some show delete working and others not. I can not reproduce the issue so it is hard to pinpoint. one example: This is on OSX Mavericks running rsync 3.0.9 with the usual osx patches and options rsync -aHAXN --fileflags --force-change --protect-decmpfs --exclude-from=/tmp/BLexclusions.txt --stats —delete —progress -vvv /Applications /Library /Users /Volumes/Backup So with /Applications /Library /Users being source, only /Applications shows files being deleted. Removing /Applications and the two other directories work with delete fine. Even changing the order has same result. The exclude file shows only one directory in /Applications. I asked user to try different (new) external drive - same results. I’m stumped as there are no rsync errors. Rsync progress shows everything fine and mentions files deleted in /Applications and nowhere else. I am not very good with rsync source but perhaps there are areas I could look where --delete might be disabled for whatever reasons? There are usually ACL's on both /Applications and /Library in OSX 0: group:everyone deny delete but this is overridden by rsync as root any ways… usual regular permissions on all directories as well. Any thoughts appreciated Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: delete not working with multiple source directories
Kevin, I don’t usually use -R unless users choose it -wanting the full directory structure on destination. In this case of course - it doesn’t matter as the three are all at root level. Worth a shot. Will let you know. Thanks, Rob On Aug 16, 2014, at 9:27 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Does --relative fix it? You almost always want --relative with multiple sources. On 08/16/2014 09:10 PM, Robert DuToit wrote: Hello All, I have lately heard from a couple of people that delete option was not deleting extraneous files on destination. This only happens when there are several source directories and at destination some show delete working and others not. I can not reproduce the issue so it is hard to pinpoint. one example: This is on OSX Mavericks running rsync 3.0.9 with the usual osx patches and options rsync -aHAXN --fileflags --force-change --protect-decmpfs --exclude-from=/tmp/BLexclusions.txt --stats —delete —progress -vvv /Applications /Library /Users /Volumes/Backup So with /Applications /Library /Users being source, only /Applications shows files being deleted. Removing /Applications and the two other directories work with delete fine. Even changing the order has same result. The exclude file shows only one directory in /Applications. I asked user to try different (new) external drive - same results. I’m stumped as there are no rsync errors. Rsync progress shows everything fine and mentions files deleted in /Applications and nowhere else. I am not very good with rsync source but perhaps there are areas I could look where --delete might be disabled for whatever reasons? There are usually ACL's on both /Applications and /Library in OSX 0: group:everyone deny delete but this is overridden by rsync as root any ways… usual regular permissions on all directories as well. Any thoughts appreciated Thanks, Rob - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2 iEYEARECAAYFAlPwBRsACgkQVKC1jlbQAQcg4QCg20zvWlxWMlKwVZ0VFyiYPFQ0 tB0AoJ6M15dVXVyCbdnSweNLfwKMpAC2 =PY5O -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 10738] New: report --stats output when termination signal arrives (fwd)
Folks, I would find the output of -stats information to be useful when rsync fails. Typically, we use rsync to do backups, and if an rsync fails, we'd like to know how much data has been transferred in the failed run, to report on our internal web page for the backups, and to make estimates of how much work a resumed run might have to do. At present, we have no idea. Thanks Regards Rob. Dr Robert C. Bell HPC National Partnerships | Scientific Computing Information Management and Technology CSIRO T +61 3 9669 8102 Alt +61 3 8601 3810 Mob +61 428 108 333 robert.b...@csiro.aumailto:robert.b...@csiro.au | www.csiro.au | wiki.csiro.au/display/ASC/ Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --log-file=File output
On Mar 12, 2014, at 12:21 AM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -v would be %n%L You probably don't want --progress in a log file. It works great. The one thing that it doesn’t seem to offer is any file count for those uploaded as in progress. If possible I may try to channel the progress to the logfile- I havn’t been able to find the place in the source that displays the (xfer#8, to-check=1115/1125) or if that is even possible. But this is a big improvement at least. Rob On 03/12/2014 12:20 AM, Robert DuToit wrote: Thanks Kevin, On Mar 12, 2014, at 12:10 AM, Kevin Korb k...@sanitarium.net wrote: See --log-file-format Most of the info for it is in man rsyncd.conf since it is mostly used by servers. I looked in there - a bit daunting. Ideally I would just want the stdout format with —progress -v. Maybe there are some simple examples out there. Rob On 03/12/2014 12:02 AM, Robert DuToit wrote: Hi All, I have a situations where I need output to a file and normally would just append “output.log” to the rsync command line but can’t do that in this situation and need to run rsync with just the args via nstask. I tried the internal --log-file=File option which works except it outputs very verbosely and not the usual (xfer#8, to-check=1115/1125) output I need to parse. I don’t knoiw if there is any way to tweak that to the normal output in the code and couldn’t see any settings relating to that. On OSX 10.9 using rsync 3.0.9 and the standard osx options Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMf4NoACgkQVKC1jlbQAQchsgCfcBG5uD7R5AVkd6lQ0B22duH2 qgcAn34kqThvbSK8acwGWjCawwEH+Mqh =XHo/ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --log-file=File output
On Mar 12, 2014, at 9:11 AM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I doubt it. Only the first of the three numbers is even accurate (unless you disable incremental indexing or rsync is almost done). Yes I use the file list (non incremental) and then the total count is always there. wish there was way to send the regular progress to the log file, though I know that isn’t the idea really. I can get a file count at the beginning and then I guess count the lines in the logfile but that’s a bit intensive. Great if there was a formt specifier for file count. thanks, Rob On 03/12/2014 08:04 AM, Robert DuToit wrote: On Mar 12, 2014, at 12:21 AM, Kevin Korb k...@sanitarium.net wrote: -v would be %n%L You probably don't want --progress in a log file. It works great. The one thing that it doesn’t seem to offer is any file count for those uploaded as in progress. If possible I may try to channel the progress to the logfile- I havn’t been able to find the place in the source that displays the (xfer#8, to-check=1115/1125) or if that is even possible. But this is a big improvement at least. Rob On 03/12/2014 12:20 AM, Robert DuToit wrote: Thanks Kevin, On Mar 12, 2014, at 12:10 AM, Kevin Korb k...@sanitarium.net wrote: See --log-file-format Most of the info for it is in man rsyncd.conf since it is mostly used by servers. I looked in there - a bit daunting. Ideally I would just want the stdout format with —progress -v. Maybe there are some simple examples out there. Rob On 03/12/2014 12:02 AM, Robert DuToit wrote: Hi All, I have a situations where I need output to a file and normally would just append “output.log” to the rsync command line but can’t do that in this situation and need to run rsync with just the args via nstask. I tried the internal --log-file=File option which works except it outputs very verbosely and not the usual (xfer#8, to-check=1115/1125) output I need to parse. I don’t knoiw if there is any way to tweak that to the normal output in the code and couldn’t see any settings relating to that. On OSX 10.9 using rsync 3.0.9 and the standard osx options Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMgXQMACgkQVKC1jlbQAQdkSACfVXVmouzDwH14Hpx1+TeCgW7B ZzcAn3ZkxCap7IDKsxu6Zrwh2KGIFbty =GRJJ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--log-file=File output
Hi All, I have a situations where I need output to a file and normally would just append “output.log” to the rsync command line but can’t do that in this situation and need to run rsync with just the args via nstask. I tried the internal --log-file=File option which works except it outputs very verbosely and not the usual (xfer#8, to-check=1115/1125) output I need to parse. I don’t knoiw if there is any way to tweak that to the normal output in the code and couldn’t see any settings relating to that. On OSX 10.9 using rsync 3.0.9 and the standard osx options Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --log-file=File output
Thanks Kevin, On Mar 12, 2014, at 12:10 AM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 See --log-file-format Most of the info for it is in man rsyncd.conf since it is mostly used by servers. I looked in there - a bit daunting. Ideally I would just want the stdout format with —progress -v. Maybe there are some simple examples out there. Rob On 03/12/2014 12:02 AM, Robert DuToit wrote: Hi All, I have a situations where I need output to a file and normally would just append “output.log” to the rsync command line but can’t do that in this situation and need to run rsync with just the args via nstask. I tried the internal --log-file=File option which works except it outputs very verbosely and not the usual (xfer#8, to-check=1115/1125) output I need to parse. I don’t knoiw if there is any way to tweak that to the normal output in the code and couldn’t see any settings relating to that. On OSX 10.9 using rsync 3.0.9 and the standard osx options Thanks, Rob - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMf3ikACgkQVKC1jlbQAQcdOQCdGIOb8gWe66cJM+Fu/RlKJKPW KPYAoNAY+RyVr3T8MhEroQ96NNw2r5eI =FqF4 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --log-file=File output
On Mar 12, 2014, at 12:21 AM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 - -v would be %n%L You probably don't want --progress in a log file. Ah yes - I completely missed the formatting options - a long scroll down the man page! I’ll experiment with those. Thanks Kevin, rob On 03/12/2014 12:20 AM, Robert DuToit wrote: Thanks Kevin, On Mar 12, 2014, at 12:10 AM, Kevin Korb k...@sanitarium.net wrote: See --log-file-format Most of the info for it is in man rsyncd.conf since it is mostly used by servers. I looked in there - a bit daunting. Ideally I would just want the stdout format with —progress -v. Maybe there are some simple examples out there. Rob On 03/12/2014 12:02 AM, Robert DuToit wrote: Hi All, I have a situations where I need output to a file and normally would just append “output.log” to the rsync command line but can’t do that in this situation and need to run rsync with just the args via nstask. I tried the internal --log-file=File option which works except it outputs very verbosely and not the usual (xfer#8, to-check=1115/1125) output I need to parse. I don’t knoiw if there is any way to tweak that to the normal output in the code and couldn’t see any settings relating to that. On OSX 10.9 using rsync 3.0.9 and the standard osx options Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlMf4NoACgkQVKC1jlbQAQchsgCfcBG5uD7R5AVkd6lQ0B22duH2 qgcAn34kqThvbSK8acwGWjCawwEH+Mqh =XHo/ -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 3.1.0 failing [iobuf] on MacOSX 10.8.5 (was Re: Cannot build rsync 3.1.0 on MacOSX 10.8.5 with fileflags patch)
On Oct 12, 2013, at 12:36 AM, Anthony Morton amor...@fastmail.fm wrote: I got 3.1 to build all right on OS X but it won't run without stalling and lots of errors. Is it working for you or anyone else yet or have the problems come to light? Yes, now I'm using it in anger I'm seeing several fatal errors like the following (I'm syncing with a local mount point, no network access involved): need to write 129836 bytes, iobuf.out.buf is only 65532 bytes. rsync error: protocol incompatibility (code 2) at io.c(599) [sender=3.1.0] rsync: [sender] write error: Broken pipe (32) Yes this is the error. I assume it has to do with the changes to the output buffer code in io.c 3.0.9 works great so I'm staying there for now and thankful for it. Would be nice if someone with a bit more rsync code experience could take a look too. Cheers, Rob need to write 401925 bytes, iobuf.out.buf is only 65532 bytes. rsync error: protocol incompatibility (code 2) at io.c(599) [sender=3.1.0] rsync: [sender] write error: Broken pipe (32) I've rolled back to version 3.0.5 (the most recent I had installed previously) and it seems to be doing the job. Tony M. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync 3.1.0 failing [iobuf] on MacOSX 10.8.5 (was Re: Cannot build rsync 3.1.0 on MacOSX 10.8.5 with fileflags patch)
On Oct 12, 2013, at 7:11 AM, Dmitry Yu Okunev dyoku...@ut.mephi.ru wrote: Hello, people. On 10/12/2013 02:56 PM, Robert DuToit wrote: On Oct 12, 2013, at 12:36 AM, Anthony Morton amor...@fastmail.fm wrote: I got 3.1 to build all right on OS X but it won't run without stalling and lots of errors. Is it working for you or anyone else yet or have the problems come to light? Yes, now I'm using it in anger I'm seeing several fatal errors like the following (I'm syncing with a local mount point, no network access involved): need to write 129836 bytes, iobuf.out.buf is only 65532 bytes. rsync error: protocol incompatibility (code 2) at io.c(599) [sender=3.1.0] rsync: [sender] write error: Broken pipe (32) Yes this is the error. I assume it has to do with the changes to the output buffer code in io.c 3.0.9 works great so I'm staying there for now and thankful for it. Would be nice if someone with a bit more rsync code experience could take a look too. Yeah, I've seen that IO_BUFFER_SIZE was increased from 4092 to 32*1024 in version 3.1.0. However it works well on my Linux systems.. And I'm using it on my production servers already. Just a hypothesis, can you show ulimit -a? it shows: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited max locked memory (kbytes, -l) unlimited max memory size (kbytes, -m) unlimited open files (-n) 256 pipe size(512 bytes, -p) 1 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 709 virtual memory (kbytes, -v) unlimited Rob Cheers, Rob need to write 401925 bytes, iobuf.out.buf is only 65532 bytes. rsync error: protocol incompatibility (code 2) at io.c(599) [sender=3.1.0] rsync: [sender] write error: Broken pipe (32) I've rolled back to version 3.0.5 (the most recent I had installed previously) and it seems to be doing the job. Tony M. -- С уважением, Дмитрий, начальник отдела UNIX-технологий, тел. 8 (495) 788-56-99, доб. 8255 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: pre 1 OSX errors
On Aug 3, 2013, at 12:31 PM, Wayne Davison way...@samba.org wrote: On Wed, Jul 31, 2013 at 7:46 AM, Robert DuToit rdut...@comcast.net wrote: Hi Wayne, Trying out 3.1 pre 1 on OSX 10.8.4 I like the new extended stats and the ir-chunk file numbers in log. Also getting a lot of errors and most copies not completing to local disk. rsync just stalls. Standard osx build: patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 patches/hfs-compression.diff ./configure make rsync --fileflags --force-change --protect-decmpfs --numeric-ids --protect-args --stats --progress -vvv when the stderr shows it is: need to write 71730 bytes, iobuf.out.buf is only 65532 bytes. rsync error: protocol incompatibility (code 2) at io.c(599) [sender=3.1.0pre1] rsync: [sender] write error: Broken pipe (32) seems to stall on several different files in output: sender finished /Users/astrid/Desktop/robdutoit/me2.png send_files(6, /Users/astrid/Desktop/robdutoit/photo_28.jpg) send_files mapped /Users/astrid/Desktop/robdutoit/photo_28.jpg of size 31740 [sender] _exit_cleanup(code=2, file=io.c, line=599): about to call exit(2) and often with calling match sums... sender finished /Users/astrid/Desktop/ccc/CHase_05_22_13.html send_files(6, /Users/astrid/Desktop/ccc/GetPID.c.html) send_files mapped /Users/astrid/Desktop/ccc/GetPID.c.html of size 96214 calling match_sums /Users/astrid/Desktop/ccc/GetPID.c.html ccc/GetPID.c.html I can restart and then it stalls again on same line or a few files down. Sometimes it stalls and no error is generated in stderr. The OS X code is essentially as is untested at this point, since I don't use it nor have a good system to test it with. If folks have fixes for it, let me know and I'll work on getting them into the patches. Thanks Wayne, Any takers? I'm not that versant in C but can start looking if anyone has ideas what is breaking. Looks like calling match sums is a good start. Best, Rob ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync: unpack_smb_acl: sys_acl_get_info(): Unknown error: 0 (0)
Dear All, I have been getting reports and noticing these acl errors again recently. rsync 3.0.7 thru 3.0.9. Someone reported that files were being skipped but we don't know if it is the acl issue since it doesn't state the file name. In the archives I found Henry's post about this and saw that Matt was going to post a patch. I haven't been able to find the patch and wonder if one exists or anyone can point me to the code where I can perhaps add at least the file path to the error thrown. Also I would probably opt myself for keeping the file without the acls, knowing the possible risks. Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Changed attributes for a file in destination that is hard linked get propagated to --link-dest directories
Folks, Thinking again about the issue of changed attributes for a file in source that is identical in content to a file in destination that is hard linked get propagated to the --link-dest directories I would like to suggest a new flag, (perhaps called something like --preserve-linked-attributes), to allow callers to get the behaviour that is needed to allow the use of recycled destination directories and have attributes preserved back in time. So, if destination file is unchanged but destination attrs are changed if --new-flag is set and if the link-count of the destination file is 1 create a new copy of the file and attrs # from the source, but could copy the file in the destination # and update attrs, and remove the old file else # current behaviour update just the attributes endif endif Or, in the terms used in previous posts: if ( destination unchanged_file() is true and unchanged_attrs() is false) The test on the link-count is done, since teh new behaviour is needed only when the target file has a link-count greater than 1. Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ Addresses: Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. This means that recycling directories, which as Teodor Milkov noted: Such a behaviour (unlink changed files and then hard link to dest dir) would be very handy, because rotating large directory trees (e.g. 10 milion files, 10k files changed) is sooo much more efficient than deleting them and then repopulating from scratch. is an issue as Wayne noted: A pre-existing hard-linked copy of the files causes rsync to just change the attributes on the file in-place (without breaking the hard-link). This can be a minor point for some people (if historical permissions/ACLs/xattrs don't need to be accurate), but could be a deal breaker for some. I can see the need for another rsync option here to allow users to select the making of a fresh copy of the file in this case. That would restore the behaviour I implicitly assumed we had, but didn't. I've updated the documentation for our backups, and prepared a note for users. I'm also thinking about ways around this issue, none of which are particularly appealing: - drop the recycling of old directories (parameterised in our set-up) - break the linking at regular intervals (parameterised in our set-up) - do a dry run to identify changed files, delete those on the destination, and then do a non-dry run (there are timing issues here, but there always will be for a non-quiet filesystem). Thanks again Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ Addresses: Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia Please see earlier postings for the disclaimer. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync - using a --files-from list to cut out scanning. How to handle deletions? (fwd)
Kevin, Thanks for your response. Some observations are inter-lined below. Rob. Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ Addresses: Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Forwarded message -- Date: Tue, 15 Jan 2013 09:25:05 -0500 From: Kevin Korb k...@sanitarium.net To: rsync@lists.samba.org Subject: Re: rsync - using a --files-from list to cut out scanning. How to handle deletions? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 If you are going to do it this way please be aware of: https://bugzilla.samba.org/show_bug.cgi?id=8712 and https://bugzilla.samba.org/show_bug.cgi?id=5644 If a file exists in the target directory when using --link-dest rsync modifies the link rather than replacing it which means you don't have history for files that have been replaced rather than added or deleted. Thanks for your astute observation about updating hard-linked files: you had me worried for a while. Fortunately, we are using the --whole-file option in our production backups, since the target of our backups is an HSM system (SGI's DMF), and we don't want rsync to start comparing files (and thus triggering a recall). With this option, if a file is changed between the source and a target which contains a hard-linked version of the file, then the rsync update replaces the file in the target, not overwrites it and all its hard-linked cousins. Whew! If you are dealing with backing up many millions of files then I suggest looking into a more advanced filesystem that can handle this functionality internally rather than using --link-dest. Currently that is limited to ZFS or BTRFS (if you are brave). Both of these filesystems have subvolumes and subvolume snapshot capabilities. This means you can do something similar to an lvm2 snapshot at the directory level instead of the whole filesystem. You can rsync with the same target directory each run and do a snapshot of that target between runs. The recycling concept is not needed because deleting an old snapshot is much faster than doing an rm -rf on a huge tree of hard links. This is especially true on ZFS which usually does the job in 1 second regardless of size. Unfortunately BTRFS usually completes the command quickly but the space is then slowly reclaimed by a kernel thread in the background. We are restricted in our use of filesystems to what is chosen for particular hosts, so smarter backups using advanced filesystems is a long way off. Here is something I wrote up about it a while back: http://sanitarium.net/golug/rsync+btrfs_backups_2011.html Thanks - good stuff. It parallels some of the work we have done - I should have looked up your papers earlier. Our recycling of old backup directories gets around the performance issue of having to delete old backups - deletes can certainly take a long time, and we do it only for old systems progressively over a year or so until we finally remove the last backups. We have added Tower of Hanoi management of the backups - great for automatically deciding which backups to keep in a rational way, and not having to mess with dates. Rob. It is a little out of date now and since I wrote it for a LUG it only covers BTRFS. A FreeBSD 9 system with at least 8GB of RAM running ZFS will outperform pretty much any Linux system running BTRFS (currently) which will outperform any Linux system running ext4 and --link-dest. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.19 (GNU/Linux) iEYEARECAAYFAlD1ZsEACgkQVKC1jlbQAQcqBwCg7AEnzQQj9vFV9WWnpIYfQS2W EvoAoIFjtx8
rsync - using a --files-from list to cut out scanning. How to handle deletions?
Folks, We use rsync extensively for protecting data by making backups. Thank you to the authors and maintainers. Like many others, we use the --link-dest option to cut down on the space occupied by the backups. Unlike many others, we re-cycle old backup directories. Since most file systems change only slowly (ours average about 0.5% of files and about 1.5% of data being churned each day), a recycled directory is a good start for the next backup. Our most common case is that a directory from 5 days ago becomes the target for the current backup, with the yesterday's backup being provided by a --link-dest= setting. Since the source file system changes only slowly, I have been thinking about ways to speed up the backups in the future. One way is to have the backups deal only with files that have changed on the source since the last backup. This would save having to scan the whole source and destination areas each time a backup is done. The Linux inotify capability looks like it might be useful for collecting a list of changed files. Has anyone done this? However, there is one case that I have not been able to get to work in a test of rsync. This is the case where a file exists in the destination, does not exist in the source, but is named in the --files-from= list. This would be the case if a file had been deleted from the source. We would want rsync in this case to delete the file on the destination. However, with a test command like: rsync -a -i --delete --files-from=list --link-dest=../linked source/ dest I was unable to get rsync to delete on the destination a file which did not exist in the source but was named in the list. rsync baulked at a file being listed that was not in the source. For example: rsync: link_stat /data/flush/inter/bel107.80527/source/0yyy failed: No such file or directory (2) [The test file 0yyy existed in the destination, the link-dest area and in the list, but not in the source.] Thanks to those who have read down to here. :-) Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ Addresses: Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync Digest, Vol 116, Issue 19
Wayne, Thanks for the information - that looks useful for the future, and would replace our option. Rob. Date: Fri, 24 Aug 2012 23:10:43 -0700 From: Wayne Davison way...@samba.org To: Robert Bell robert.b...@csiro.au Cc: rsync@lists.samba.org Subject: Re: [patch] link-dest messages and max-size warnings (fwd) On Thu, Aug 23, 2012 at 6:29 PM, Robert Bell robert.b...@csiro.au wrote: This patch adds another parameter, --warn, to select the output of a warning message when files bigger than the selected --max-size are skipped. Note that 3.1.0dev has the --info option for finer-grained verbosity. If you specify --info=skip then it will tell you about any skipped files, such as those skipped via --max-size. ..wayne.. Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[patch] link-dest messages and max-size warnings (fwd)
rsync Folks, The following explanatory text is by me and the patches are by Rowan McKenzie for use by the Advanced Scientific Computing group at CSIRO. This patch builds upon the --link-dest patch by Bryant Hansen (Thanks heaps!). 1. The original patch provided an alternate behaviour for rsync when using the --link-dest option. When there are identical files in the source and link-dest areas, but a different file in the destination area, standard rsync will update the destination by copying the file from source to destination: the patch updates the destination by hard-linking from the link-dest area. Why do we want this behaviour? In our backup system which we have been running since 2007 using rsync, we recycle old backup destinations to be the target for new backups. This is an efficiency gain - we don't want to create a new area with, in several cases, millions of files, when we have an older area which is nearly up to date. (With mature backups of user areas, we typically find that our daily backups have a churn of 0.5% of files and 1% of data.) The original patch ensures we get the maximum amount of hard-linking available. However, the original patch unconditionally outputs messages for every file hard-linked under the scenario outlined above. Our modified patch makes output of the diagnostic message controlled by the -v option. 2. In addition, the patch adds one more feature and option. In our backups some time ago, we wished to avoid repeated daily backups of large files that were being appended to each day - they were the outputs of computational models. We used the --max-size parameter to skip these files: however, we did not like the lack of warning about skipped files. This patch adds another parameter, --warn, to select the output of a warning message when files bigger than the selected --max-size are skipped. The message is of the form: big_file is over max-size 3. For information: our backups are controlled from the destination of the backups (pull rather than push as Kevin Korb recently advised). We use the rsync daemon capability. The destinations of our backups are file systems subject to HSM (Hierarchical Storage Management), using SGI's Data Migration Facility (DMF). A typical command we use is the following (but I have shortened the paths and addresses). rsync --password-file=not_for_your_eyes --numeric-ids -a --stats --one-file-system --max-size=8.0GB --warn --whole-file --link-dest=previous --delete root@source_host::backups/source_dir current --password-file=not_for_your_eyes . for the daemon --numeric-ids . since the userids on the source are not always available on the destination -a . archive mode --stats . statistics --one-file-system . stops the backup of everything when backing up / --max-size=8.0GB . to skip large files --warn . NEW parameter - warn of skipped files because of --max-size --whole-file . essential when the destination is subject to HSM: otherwise, files will be recalled to use the rsync comparison algorithm --link-dest=previous . pointer to previous backup: to provide a source of files for hard-linking --delete . essential when the destination is a recycled directory, to ensure superseded files are deleted root@source_host::backups/source_dir . the source specification: username @source_host, module specification, and source directory current . the destination directory. We use an extended Tower of Hanoi scheme to manage the keeping of backups: - highly recommended for its ability to provide sensible keeping of backups matched to the likelihood of restores, and because it avoids messy management using dates and times. Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell, BSc (Hons) PhD Technical Services Manager Advanced Scientific Computing CSIRO IMT Phone: +61 3 9669 8102 | Mobile: +61 428 108 333 | CSIRO 93 3810 robert.b...@csiro.au | http://www.csiro.au/ | http://www.hpsc.csiro.au/ Addresses: Street: CSIRO ASC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: CSIRO ASC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia # CSIRO-ASC patches for rsync. # # Note: name this so that it's called after rsync_link_dest_from_bryant # # Rowan McKenzie 27/7/2012 # Warnings for --max-size ignored files are displayed if -w/--warning is specified diff -Naur orig/options.c new/options.c --- orig/options.c 2012-07-26 16:36:59.201899312 +1000 +++ new/options.c 2012-07-26 16:49:40.065898039 +1000 @@ -173,6 +173,7 @@ char *dest_option = NULL; int verbose = 0; +int warn = 0; int quiet = 0; int output_motd = 1; int log_before_transfer = 0; @@ -313,6 +314,7 @@ rprintf(F,\n); rprintf(F,Options\n); rprintf(F, -v, --verbose increase verbosity\n); + rprintf(F, -w, --warn display warnings\n); rprintf(F, -q, --quiet
Problem with excludes and includes
Hi Folks, I can't get the syntax of --include-from right, nothing at all gets transferred. The rsync command: # do the backup nice -n 15 rsync -av \ --include-from=/home/bob/backup_includes \ --exclude-from=/home/bob/backup_excludes \ $fromdir $todir The excludes file: .* *~ Videos/ Downloads/ Dropbox/ The includes file: .vimrc .gitconfig .bash_aliases .bashrc Videos/Music/Playlist/ Videos/Music/Originals/ Downloads/Iso/ I've spent hours on the man page getting nowhere. Thanks Bob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with excludes and includes
Oops I'm now getting all the dot files and all the rubbish from Downloads. I put the lines you gave me into 'backup_excludes'. Should I have used some other file? Bob On Mon, Feb 27, 2012 at 12:17 AM, Robert Parker rlp1...@gmail.com wrote: Thanks I'll try that right away. Bob On Sun, Feb 26, 2012 at 11:04 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Things get much easier if you only use one file and use the +/- syntax. Here is a quick rewrite of your file using that syntax and some changes and comments: + .vimrc + .gitconfig + .bash_aliases + .bashrc # Exclude all .files except the ones commented above - - .* - - *~ # Have to include this or rsync will never see Playlist + Videos/Music/ + Videos/Music/Playlist/ # Can't exclude Videos/ because then rsync will never look inside it - - Videos/* + Downloads/Iso/ - - Downloads/* - - Dropbox On 02/26/12 09:54, Robert Parker wrote: Hi Folks, I can't get the syntax of --include-from right, nothing at all gets transferred. The rsync command: # do the backup nice -n 15 rsync -av \ --include-from=/home/bob/backup_includes \ --exclude-from=/home/bob/backup_excludes \ $fromdir $todir The excludes file: .* *~ Videos/ Downloads/ Dropbox/ The includes file: .vimrc .gitconfig .bash_aliases .bashrc Videos/Music/Playlist/ Videos/Music/Originals/ Downloads/Iso/ I've spent hours on the man page getting nowhere. Thanks Bob - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9KWBAACgkQVKC1jlbQAQfHggCeL0dRbjClnOakbMIeLTjH/SIr ZLgAoOiB3NTq73rVTYbvtVzSAdCapxqG =/uH4 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- The healthy eating pyramid as published by the USFDA and it's satellites all over the world is purposely designed to bring about an epidemic of obesity, hypertension and diabetes. It is wildly successful in this aim. -- The healthy eating pyramid as published by the USFDA and it's satellites all over the world is purposely designed to bring about an epidemic of obesity, hypertension and diabetes. It is wildly successful in this aim. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Problem with excludes and includes
On Mon, Feb 27, 2012 at 12:34 AM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 In your replies I am seeing doubled - signs. I am unsure if that is an artifact of one of our email clients or if that is what you were actually using. There should only be one - on those lines. The line that says - .* should exclude any file that starts with a . other than the specific examples previously listed as + .file. Thanks. That will be the problem because I just copy/ pasted the content of the email as it was. I made the single '-' and I'm running it now and Downloads/Iso is copying but none the rubbish that was going before. I think that this is now solved. On 02/26/12 12:31, Robert Parker wrote: Oops I'm now getting all the dot files and all the rubbish from Downloads. I put the lines you gave me into 'backup_excludes'. Should I have used some other file? Bob On Mon, Feb 27, 2012 at 12:17 AM, Robert Parker rlp1...@gmail.com mailto:rlp1...@gmail.com wrote: Thanks I'll try that right away. Bob On Sun, Feb 26, 2012 at 11:04 PM, Kevin Korb k...@sanitarium.net mailto:k...@sanitarium.net wrote: Things get much easier if you only use one file and use the +/- syntax. Here is a quick rewrite of your file using that syntax and some changes and comments: + .vimrc + .gitconfig + .bash_aliases + .bashrc # Exclude all .files except the ones commented above - .* - *~ # Have to include this or rsync will never see Playlist + Videos/Music/ + Videos/Music/Playlist/ # Can't exclude Videos/ because then rsync will never look inside it - Videos/* + Downloads/Iso/ - Downloads/* - Dropbox On 02/26/12 09:54, Robert Parker wrote: Hi Folks, I can't get the syntax of --include-from right, nothing at all gets transferred. The rsync command: # do the backup nice -n 15 rsync -av \ --include-from=/home/bob/backup_includes \ --exclude-from=/home/bob/backup_excludes \ $fromdir $todir The excludes file: .* *~ Videos/ Downloads/ Dropbox/ The includes file: .vimrc .gitconfig .bash_aliases .bashrc Videos/Music/Playlist/ Videos/Music/Originals/ Downloads/Iso/ I've spent hours on the man page getting nowhere. Thanks Bob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- The healthy eating pyramid as published by the USFDA and it's satellites all over the world is purposely designed to bring about an epidemic of obesity, hypertension and diabetes. It is wildly successful in this aim. -- The healthy eating pyramid as published by the USFDA and it's satellites all over the world is purposely designed to bring about an epidemic of obesity, hypertension and diabetes. It is wildly successful in this aim. - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone:(407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Floridak...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9KbRwACgkQVKC1jlbQAQd4SQCg+ToKjHKnXPhLopLsbozdjBW3 pzQAn0XQ9kD6cPGmgjt/uI52M9x7n3S6 =6/BC -END PGP SIGNATURE- -- The healthy eating pyramid as published by the USFDA and it's satellites all over the world is purposely designed to bring about an epidemic of obesity, hypertension and diabetes. It is wildly successful in this aim. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
unpack_smb_acl() errors
Hi All, I have been noticing this in stderr recently. Is this a problem? Standard OSX patches on 3.0.7 - everything working perfectly otherwise. Best, Rob Calling unpack_smb_acl() on /Users/astrid Calling unpack_smb_acl() on /Users/astrid/Desktop Calling unpack_smb_acl() on /Users/astrid/Documents Calling unpack_smb_acl() on /Users/astrid/Downloads Calling unpack_smb_acl() on /Users/astrid/Library Calling unpack_smb_acl() on /Users/astrid/Movies Calling unpack_smb_acl() on /Users/astrid/Music Calling unpack_smb_acl() on /Users/astrid/Pictures Calling unpack_smb_acl() on /Users/astrid/Public Calling unpack_smb_acl() on /Users/astrid/Sites etc etc etc….. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cannot get the syntax of --include-from right
On Fri, Sep 16, 2011 at 10:47 PM, Steven Levine stev...@earthlink.net wrote: In CAAykW6mMRiTVqt0KrrcgYiwNvz7kjHNQC76=rJc+c=+qpg3...@mail.gmail.com, on 09/16/11 at 07:34 PM, Robert Parker rlp1...@gmail.com said: HI Robert, snipped The paths in your --include-from file must be relative to $fromdir. In general, this is true of all paths other than paths specified on the command line. The paths on the command line effectively create logic roots and the other paths will be relative to these paths. Thank you Steven, The revised script part: nice -n 15 rsync -av --include-from=/home/bob/backup_includes \ --exclude=.* $fromdir $todir and backup_files: .vimrc .gitconfig .bash_aliases .bashrc And my backup script is now doing exactly what I want it to do. Bob Parker Steven -- -- Steven Levine stev...@earthlink.net eCS/Warp/DIY etc. www.scoug.com www.ecomstation.com -- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Cannot get the syntax of --include-from right
On Fri, Sep 16, 2011 at 11:40 PM, Kevin Korb k...@sanitarium.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Also, includes only override excludes (which I see you are doing) but they must be specified BEFORE the exclude they override. IOW, your include of .vimrc must be before your exclude of .* or it will not do anything. Thanks Kevin, It's all good now as per my reply to Steven. Bob Parker On 09/16/11 11:47, Steven Levine wrote: In CAAykW6mMRiTVqt0KrrcgYiwNvz7kjHNQC76=rJc+c=+qpg3...@mail.gmail.com, on 09/16/11 at 07:34 PM, Robert Parker rlp1...@gmail.com said: HI Robert, nice -n 15 rsync -avl --exclude=.* \ --include-from=/home/bob/backup_includes \ $fromdir $todir Here is the content of /home/bob/backup_includes /home/bob/.vimrc /home/bob/.gitconfig The paths in your --include-from file must be relative to $fromdir. In general, this is true of all paths other than paths specified on the command line. The paths on the command line effectively create logic roots and the other paths will be relative to these paths. Steven - -- ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ Kevin Korb Phone: (407) 252-6853 Systems Administrator Internet: FutureQuest, Inc. ke...@futurequest.net (work) Orlando, Florida k...@sanitarium.net (personal) Web page: http://www.sanitarium.net/ PGP public key available on web site. ~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~'`^`'~*-,._.,-*~ -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk5zfBgACgkQVKC1jlbQAQdcIwCfZKIJk9pixLywS91plqveWy/X lQMAnR3klzsRwX26esWe8YfEuzLGp+gH =OeR3 -END PGP SIGNATURE- -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Broaden your kids education! Send them to Christian Schools so they can include the experience of being raped in their education. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Post held in moderation Rsync chokes on this file
On Tue, Sep 13, 2011 at 02:47:40PM +0100, Frank Church wrote: I sent a message with the above subject:Rsync chokes on this file, which as a 7Mb file attached to it and got a message that the post was held in moderation because the attached file is too large. What is the procedure for sending a file to the developers for checking? Can I upload the file somewhere and create a link to it in my email Pastebin -- Bob Holtzman If you think you're getting free lunch, check the price of the beer. Key ID: 8D549279 signature.asc Description: Digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Is it possible to be on the rsync list without getting all the samba bugs?
On Wed, Sep 07, 2011 at 10:09:02PM -0700, Michael_google gmail_Gersten wrote: When I signed up for the rsync mailing list, I was not expecting to get all the samba bugs as well. Is it possible to be on just the rsync mailing list, without being on the bug report list as well? Your best bet is to configure your email client to filter out the samba bugs messages. -- Bob Holtzman If you think you're getting free lunch, check the price of the beer. Key ID: 8D549279 signature.asc Description: Digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude hidden files
On Fri, Aug 19, 2011 at 5:47 PM, Ken Gillett email...@ukgb.net wrote: I want to exclude from the sync ALL files that start with a dot, IOW hidden files. But using --exclude '.*' does funny things with directories (or at least I think it does). Is there a recognised correct way to do this? I thought --exclude '.???*' would avoid the directory issue by only excluding any dot file(dir) with a name longer than 2 characters. Not tried it yet, but would this be a good solution? I think you need to specifically name the dot files you want to exclude, eg '.*.swp' will ignore vi/vim swap files. Bob Parker Ken G i l l e t t _/_/_/_/_/_/_/_/ -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Scientific studies prove that scientists are whores just like the rest of us wage slaves. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
thank you wayne, andrew, paul and everyone else
For me, rsync is an indispensable tool. I don't know what I'd do without it. Rsync and OpenSSH make our open systems possible. Of course, we wouldn't NEED them so much if we had better distributed file systems, but let's keep this an upbeat message. :-) I'd like to thank Andrew and Paul for giving birth to it, Wayne for nurturing it, and everyone else for providing support through their testing patches, and suggestions. -- Bob Bagwill -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
restoring system
I have restored my home directory in the past using rsync but never the entire system. If I rsync / (on my usb drive containing the backup) to / on the computer with a fresh install of the OS, what problems will I run into? What about links? I am presently triple booting 3 linux distros and want to delete one to install something else. The distro I'm installing will be on 3 partitions (/, /boot, and /home). Will what I'm proposing populate the directories in these partitions or will it dump everything into /? I realize that I probably be better off creating a snapshot and restoring from that but I have never had good luck with MondoRescue or Clonezilla. Thought I'd better ask now before I stuck my foot in it. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 3.0.8 OSX build hfs+ patch failing
Thanks Wayne, That's great - I didn't even think to look in patches for those! I'll build and send it around for testing. best, Rob On Apr 9, 2011, at 12:00 PM, Wayne Davison wrote: On Tue, Apr 5, 2011 at 6:11 PM, Robert DuToit rdut...@comcast.net wrote: I am just building 3.0.8 on osx 10.6.6 and mike's hfs_compression.diff is failing. Am I missing something? Those instructions are for 3.0.7, and include applying patches for things that are already fixed in 3.0.8. You can try using the official hfs+ patch for 3.0.8 (the one that is in the 3.0.8 patches tar file) -- read the patches/hfs-compression.diff patch for the sequence of patch commands it expects (you don't need to curl anything for it other than the 3.0.8 source and patches tar files). This new patch has only been lightly tested, though, and may still need some work for full functioning, so if you or anyone else tries it out, let me know if there are things that need to be fixed. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
3.0.8 OSX build hfs+ patch failing
Hi all, hi Mike, I am just building 3.0.8 on osx 10.6.6 and mike's hfs_compression.diff is failing. Am I missing something? Thanks, Rob patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 patches/crtimes-64bit.diff ( this says previously applied…) patch -p1 patches/crtimes-hfs+.diff but for patch -p1 patches/hfs_compression.diff patching file rsync.h Hunk #1 succeeded at 509 (offset 19 lines). patching file sender.c patching file xattrs.c Hunk #1 FAILED at 32. Hunk #2 succeeded at 52 (offset 3 lines). Hunk #3 succeeded at 210 (offset 3 lines). Hunk #4 succeeded at 220 (offset 3 lines). Hunk #5 succeeded at 251 (offset 3 lines). Hunk #6 succeeded at 323 with fuzz 2 (offset 19 lines). Hunk #7 FAILED at 361. Hunk #8 succeeded at 372 (offset 19 lines). Hunk #9 FAILED at 422. Hunk #10 succeeded at 487 (offset 19 lines). Hunk #11 succeeded at 521 (offset 19 lines). Hunk #12 succeeded at 573 (offset 19 lines). Hunk #13 succeeded at 602 (offset 19 lines). Hunk #14 succeeded at 649 with fuzz 2 (offset 19 lines). Hunk #15 succeeded at 792 (offset 19 lines). Hunk #16 succeeded at 849 (offset 19 lines). Hunk #17 FAILED at 889. Hunk #18 succeeded at 927 (offset 19 lines). 4 out of 18 hunks FAILED -- saving rejects to file xattrs.c.rej -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Error when filters exclude all files from transfer
Hi I am using rsync in combination with lsyncd to do incremental backups. The following rsync command is being executed when lsyncd detects that files have changed: /usr/bin/rsync -lptgoDsz --filter=.\ /etc/rsync.filter -r --delete --force --from0 --files-from=- / rsynchead::school The file /etc/rsync.filter contains the following: + /home/*/.kde/share/apps/kmail/mail/** + /home/*/.kde/share/config/emailidentities + /home/*/.kde/share/config/kmailrc + /home/*/.kde/share/config/mailtransports - /home/git/** - /home/*/.* - /home/*/.*/** + /etc/passwd + /etc/shadow + /etc/group - /etc/** + /var/spool/mail/* The list of files from lsyncd is piped to stdin. On the rsynchead server I am seeing a lot of entries like: Mar 1 11:23:12 rsynchead1 rsyncd[18773]: connect from fw.school158.gp-online.net (41.161.29.109) Mar 1 11:23:13 rsynchead1 rsyncd[18773]: rsync to school158/ from fw.school158.gp-online.net (41.161.29.109) Mar 1 11:23:14 rsynchead1 rsyncd[18773]: rsync: connection unexpectedly closed (24 bytes received so far) [Receiver] Mar 1 11:23:14 rsynchead1 rsyncd[18773]: rsync error: error in rsync protocol data stream (code 12) at io.c(601) [Receiver=3.0.7] What I suspect is happening is that lsyncd is calling rsync with a list of files all of which get excluded by the filters. Rsync establishes the connection with the server before it realizes that it has nothing to do and then disconnects a bit ungracefully. So, is there some way to prevent this from happening, ie if rsync has no files to process that it exits without bothering to contact the remote server? Robert Crida rcr...@clue.co.za -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Trying to preserve UID and GID when backing up with rsync
Awesome, thanks for the feedback. I had actually managed to work it out in the meantime... On 14 Feb 2011, at 3:54 PM, Paul Slootman wrote: On Mon 14 Feb 2011, Robert Crida wrote: I am in early prototyping phase now and have the following rsyncd.conf file: max connections = 4 [school1] path = /backup/school/./1 read only = false use chroot = true I ran the following command (on the same machine as the server): rsync -az ./.viminfo fw.school1::school1 The result on the server is: root@fw.school1:/etc# ls -nal /backup/school/1/ total 12 drwxrwxrwx 2 0 0 4096 2011-02-14 14:08 . drwxr-xr-x 3 0 0 4096 2011-02-14 11:30 .. -rw--- 1 65534 65534 590 2011-02-14 10:52 .viminfo Check the manual page for rsyncd.conf; you'll see that the default uid under which rsyncd runs for such modules is nobody, which generally equates to 65534. Add a line uid = 0 to enable the daemon to correctly set the owner and groups. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html Robert Crida rcr...@clue.co.za -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: �ظ� : How to sync an exact list of files, Including deletes!?
On Wed, Nov 24, 2010 at 08:42:16AM +0800, (New Plaza F9) Lee, Sam wrote: 2010/11/23 23:36:52 [444] rsync: connection unexpectedly closed (255 bytes received so far) [generator] 2010/11/23 23:36:52 [444] rsync error: error in rsync protocol data stream (code 12) at io.c(641) [generator=3.0.4] Hello，My Friends,what's mean that? Thanks! Why did you hijack the thread? Start a new one. This does *not* mean simply changing the subject. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx remote backup wrong permissions
Hi All, On Oct 31, 2010, at 5:12 PM, Wayne Davison wrote: On Sun, Oct 31, 2010 at 10:57 AM, Robert DuToit rdut...@comcast.net wrote: Does the server (or other computer in this case) always need a root account established for this to work? I tried everything else and the owner always became that of the remote user. The user running the receiving rsync needs to be able to chown things (change a file's ownership). Rsync only attempts to chown files if (1) the running user ID is 0 (typically root) or (2) the --super option was used. See the manpage for what --super affects. I've been wrestling with this all week and basically. It is fairly easy to run non-attended backups with rsync using the PKA dsa keys with various security measures, either with passwords and ssh-agent or, without password with limits on the receiver for the allowed users, command etc… But doing that and preserving ownership ( chown ) on the receiver is another matter. No problem with enabling the root account but that is definitely not a good thing to do on a permanent basis. So: So far transferring a test folder with system privileges to preserve, I found using sudo on the remote rsync path to work but you have to edit the sudoers file (not for the faint of heart) to allow noPasswd for the admin user if you want to not be asked for password on the remote side. the remoteUserName = my admin account on the remote side. sudo /rsync -aNHAXx --protect-args --fileflags --protect-decmpfs --force-change --stats --progress -v --rsync-path=sudo /rsync /var/audit remoteusern...@192.168.11.2:/Users/remoteUserName/Desktop again I can set this up but wouldn't suggest someone else modifying sudoers with visudo. So I tried Mike's suggested method and set up the keys in the root .ssh folder and the authorized_keys file in the remote root .ssh folder: http://www.afp548.com/netboot/mactips/rsync.html sudo ssh-keygen -t dsa -f /private/var/root/.ssh/id_dsa -C your comment sudo cat /private/var/root/.ssh/id_dsa.pub | ssh r...@192.168.11.2 'cat - ~/.ssh/authorized_keys' local/Path/To/rsync -aNHAXx --protect-args --fileflags --force-change --rsync-path=/usr/local/bin/rsync /var/audit r...@192.168.11.2:/Users/remoteUserName/Desktop all permissions set appropriately etc. But every time I run this it prompts for password three times and fails with Permission denied ~ $ ssh rem...@192.168.11.2 The authenticity of host '192.168.15.2 (192.168.15.2)' can't be established. RSA key fingerprint is 0c:aa:76:ca:89:54:dd:49:13:c7:dd:09:ee:19:7d:8c. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.15.2' (RSA) to the list of known hosts. Password: Password: Password: Permission denied (publickey,keyboard-interactive). I tried specifying the -e ssh -i /private/var/root/.ssh/id_dsa in the rsync line but that throws same error- Actually it says access denied for /private/var/root/.ssh/id_dsa - no such file etc.. I have been using CCC with Mike's packaged keys for some time and that works great but my own setup doesn't. Not sure what the dif is here. If anyone, or Mike if you are out there, has some input I would be grateful. I have my own rsync wrapper app and would like to be able to get this working for it. Thanks, Rob ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx remote backup wrong permissions
Hi All, FWIW, I got my test to work. I realized the permissions were actually ok- the owner/group bits were off. I set up a root account on the other laptop and logged out of it then ran /rsync -aNHAXx --protect-args --fileflags --protect-decmpfs --force-change --stats --progress -v --rsync-path=/rsync /private/var/audit r...@ipaddress:/Users/myUser/backup and the resulting owner (System) and Group were preserved. Does the server (or other computer in this case) always need a root account established for this to work? I tried everything else and the owner always became that of the remote user. Cheers, Rob On Oct 30, 2010, at 9:35 PM, Robert DuToit wrote: On Oct 30, 2010, at 8:27 PM, Mike Bombich wrote: On Oct 30, 2010, at 9:02 AM, Robert DuToit wrote: Hi All, I'm trying a remote backup for the first time. It is between two laptops, I installed rsync 3.0.7 on both compiled patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 patches/crtimes-64bit.diff patch -p1 patches/crtimes-hfs+.dif f patch -p1 patches/hfs_compression.diff ./configure make this all works perfectly for local backups, clones etc. Everything gets copied as is to the volume. I have tried without ssh keys, with keys, keys with and without passwords (via ssh-agent) and it all works except that it won't copy the correct permissions on System files example run with locked system folder var/audit going to servern...@100.102.15.4 (not my real address here of course): sudo /rsync -aNHAXx --protect-args --fileflags --protect-decmpfs --force-change --stats --progress -vvv --rsync-path=/rsync /private/var/audit servern...@100.102.15.4:/Volumes/Extra/eee What's this servername account? You should be using root if you want to preserve root ownership. Mike Hi Mike, That's the admin account name/address on the other laptop. Should it be r...@192.168.15.4? I'm afraid this is all new to me - the remote stuff…. Rob Ignore ownership unchecked on dest. The permissions are changed from readwrite only on System to readwrite on my user account on the receiver. I copied the whole System but the results were the same though some files had permission denied errors. The output on my test folder show nothing remarkable, no errors, etc. Am I missing something here? Any thoughts appreciated or directions to look in. Thanks, Rob delta-transmission enabled recv_files(1) starting recv_generator(audit,1) recv_generator(audit,2) send_files(2, /private/var/audit) audit/ set modtime of audit to (1288417603) Sat Oct 30 01:46:43 2010 recv_generator(audit/20100701144405.20100702030039,3) send_files(3, /private/var/audit/20100701144405.20100702030039) send_files mapped /private/var/audit/20100701144405.20100702030039 of size 21066 calling match_sums /private/var/audit/20100701144405.20100702030039 audit/20100701144405.20100702030039 21066 100%0.00kB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 21066 100%0.00kB/s0:00:00 (xfer#1, to-check=240/242) sender finished /private/var/audit/20100701144405.20100702030039 recv_generator(audit/20100702030127.20100702052825,4) send_files(4, /private/var/audit/20100702030127.20100702052825) send_files mapped /private/var/audit/20100702030127.20100702052825 of size 15417 calling match_sums /private/var/audit/20100702030127.20100702052825 audit/20100702030127.20100702052825 15417 100%7.35MB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 15417 100%7.35MB/s0:00:00 (xfer#2, to-check=239/242) sender finished /private/var/audit/20100702030127.20100702052825 recv_generator(audit/20100702111829.20100703030341,5) send_files(5, /private/var/audit/20100702111829.20100703030341) send_files mapped /private/var/audit/20100702111829.20100703030341 of size 64184 calling match_sums /private/var/audit/20100702111829.20100703030341 audit/20100702111829.20100703030341 ##and later on down the page### generate_files phase=1 recv_files(audit) recv_files(audit/20100701144405.20100702030039) got file_sum set modtime of audit/.20100701144405.20100702030039.U7yt17 to (1278039638) Thu Jul 1 23:00:38 2010 renaming audit/.20100701144405.20100702030039.U7yt17 to audit/20100701144405.20100702030039 recv_files(audit/20100702030127.20100702052825) got file_sum set modtime of audit/.20100702030127.20100702052825.hzHr5M to (1278048505) Fri Jul 2 01:28:25 2010 renaming audit/.20100702030127.20100702052825.hzHr5M to audit/20100702030127.20100702052825 recv_files(audit/20100702111829.20100703030341) got file_sum set modtime of audit/.20100702111829.20100703030341.sSoTwl to (1278126221) Fri Jul 2 23:03:41 2010 renaming audit/.20100702111829.20100703030341.sSoTwl to audit/20100702111829.20100703030341 recv_files(audit/20100703030442.20100703045821) got file_sum
Re: rsync errors[SOLVED]
On Tue, Sep 21, 2010 at 04:59:39AM -0400, Tony Abernethy wrote: Robert Holtzman wrote: Running Ubuntu 10.04.1 and rsync 3.0.7-1. Used rsync successfully for over a year for backups. Recently there was a kernel upgrade to 2.6.32-25. Don't know if that has any bearing. When I tried to run a backup today rsync failed with the error message: sending incremental file list rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: mkdir /media/disk/10.04laptop failed: No such file or directory What does ls -l /media/disk on the target give? rsync must be able to create (ie write) in that directory. Sorry to be late with this reply. I didn't think it was a permission problem and it turned out not to be. I had renamed the target directory and forgot to change the backup script. In addition, prior to the upgrade to 10.04, the usb hd showed up as disk. After the upgrade it had changed to the UUID and, of course the script still reflected disk. Once this was corrected it ran perfectly. Many thanks. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer signature.asc Description: Digital signature -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync errors
Running Ubuntu 10.04.1 and rsync 3.0.7-1. Used rsync successfully for over a year for backups. Recently there was a kernel upgrade to 2.6.32-25. Don't know if that has any bearing. When I tried to run a backup today rsync failed with the error message: sending incremental file list rsync: writefd_unbuffered failed to write 4 bytes to socket [sender]: Broken pipe (32) rsync: mkdir /media/disk/10.04laptop failed: No such file or directory (2) rsync error: error in file IO (code 11) at main.c(595) [Receiver=3.0.7] rsync: connection unexpectedly closed (9 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(601) [sender=3.0.7] /var/log/syslog just showed the above message. Searched on the message and got one hit that said it was because a FAT32 partition had a size limit. I doubt that I've come close to that. Couldn't find anything in this list's archives that matched the error message. Did I miss something? The command that has been working until now is: sudo rsync -vaHz --exclude '/proc' --exclude '/sys' --exclude '*.iso' --exclude '/home/holtzm/Documents/*.iso' --exclude '/media' /. /media/disk/10.04laptop I'm at a loss as to how to proceed. any pointers appreciated. -- Bob Holtzman Key ID: 8D549279 If you think you're getting free lunch, check the price of the beer -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
OSX 10.6.4 error with -R option
Hi All, I have had reports of problems with the -R option on OSX 10.6.4. Just tested it myself and found this odd result: When I run this dtruss -f path/to/rsync -aHAXNR --fileflags --force-change --protect-decmpfs --stats -v /Users/astrid/Documents/main.m /Users/astrid/Desktop/rrr it produces the expected results with the relative folder paths in place /Users/astrid/Desktop/rrr/Users/astrid/Desktop/main.m but copying file from Desktop itself dtruss -f path/to/rsync -aHAXNR --fileflags --force-change --protect-decmpfs --stats -v /Users/astrid/Desktop/main.m /Users/astrid/Desktop/rrr /Users/astrid/Desktop/rrr/Users/astrid/ empty! Oddly, it works fine from a clone of this system (cloned a few weeks ago) and on OSX 10.5. Any thoughts appreciated. Thanks, Rob Here are the first few lines from running dtruss on rsync rsync: unpack_smb_acl: sys_acl_get_info(): Unknown error: 0 (0) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1044) [sender=3.0.7] PID/THRD SYSCALL(args) = return 609/0x11a0: sigaction(0x1E, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigaction(0x1F, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigaction(0x14, 0xB5A0, 0x0) = 0 0 609/0x11a0: geteuid(0x14, 0xB5A0, 0x0)= 0 0 609/0x11a0: umask(0x0, 0xB5A0, 0x0) = 18 0 609/0x11a0: open(/etc/popt\0, 0x0, 0x8FE44840) = -1 Err#2 609/0x11a0: open(/Users/astrid/.popt\0, 0x0, 0x0) = -1 Err#2 note: (It cuts off at /astrid/ and no files are copied after that point) 609/0x11a0: sigaction(0x2, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigaction(0x1, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigaction(0xF, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigprocmask(0x2, 0xB5FC, 0x0) = 0x0 0 609/0x11a0: sigaction(0xD, 0xB5A0, 0x0) = 0 0 609/0x11a0: sigaction(0x19, 0xB5A0, 0x0) = 0 0 -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
re: OSX 10.6.4 error with -R option
Hi again, Problem solved Sorry. I didn't even think to check permissions on the Desktop folder, the first logical thing to check. Somehow they had been changed. Fixed them and -R works fine now. I believe other users are likely having permissions issues too. Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: Rsync sequence
I have one script that has multiple rsync to a backup server. I do not want them all running at the same time but I want to be sure that they all complete. What I do is use the sh wait command link rsync . rsync . Wait rsync . rsync . wait And so on It will run the rsyncs in the background but will wait until both have finished until it runs the next two rsync. Robert -Original Message- From: rsync-boun...@lists.samba.org [mailto:rsync- boun...@lists.samba.org] On Behalf Of Paul Slootman Sent: Tuesday, July 20, 2010 5:19 AM To: dschuett Cc: rsync@lists.samba.org Subject: Re: Rsync sequence On Mon 19 Jul 2010, dschuett wrote: allow for a more efficient backup. I know I could just run them at different times in the same night, but I don't want to chance one not completing before the other starts. Can I just list them one after the other like so in crontab -e: # m h dom mon dow command 00 4 * * 7 rsync -auv --delete /Godfather/Music /Backup rsync -auv --delete /Godfather/Documents /Backup rsync -auv --delete /Godfather/Setups /Backup rsync -auv --delete /Godfather/Pictures /Backup rsync -auv --delete /Godfather/Backups /Backup rsync -auv --delete /Godfather/Videos /Backup This is hardly an rsync question... I would put the commands in a separate shell script file: #!/bin/sh rsync -auv --delete /Godfather/Music /Backup rsync -auv --delete /Godfather/Documents /Backup rsync -auv --delete /Godfather/Setups /Backup rsync -auv --delete /Godfather/Pictures /Backup rsync -auv --delete /Godfather/Backups /Backup rsync -auv --delete /Godfather/Videos /Backup Then make the file executable (chmod +x) and then put the name of the script in the crontab. Paul -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart- questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
Hi All, I reported a ways back about strange file replication via rsync copying to an OSX 10.6.3 local network volume. Hidden dot files such as .DS-Store and other dot files on OSX were copied to the local volume with appended names indicating that the temp files created by rsync were ending up on the destination instead of the originals, often by the thousands. Ok, so I am back on the case but now can not recreate this issue. I am now running OSX 10.6.4. Has anyone else noticed that this problem has ceased with 10.6.4 or is it still happening? I tried builds of rsync 3.0.6 and 3.0.7 and both seem ok now…. If it is fixed with the latest OS release - so much the better. News? Thanks, rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and hfs+ compressed files damaged
Hi Mike and All, I have been doing some research on the damaged HFS compressed files and found some interesting clues. Looking at mike's patch so far I can't find any reason for the damage (I have to learn more C language ) but what still seems odd is that the damage only happens when copying the whole /usr directory or /usr/bin /usr/share. Individually copying one of the original files does not create the damaged file. Pertaining to the use of Mike's patch on OSX 10.6.4 copying /usr/bin and resulting damaged files In /usr/bin/ I found that the originals of the damaged files are different from the other normal exec files. In this case I am looking at the find exec. If I copy find from /usr/bin/ to the desktop via finder copy, it is changed to resemble the normal execs. And indeed if I replace the original with this copy in /usr/bin and copy /usr/bin with patched rsync, the copied find is no longer damaged! So it appears that when the patched rsync encounters one of these odd files ( and only when copying the whole /usr/bin/ directory containing them ) it causes the problem. Here are results using hfsdebug (sorry about the amount of reading!) original file exec - works ok but different from other execs in /usr/bin. ( It shows the com.apple.decmpfs flag and data is stored in Resource Fork) flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 101755 bytes totalBlocks = 25 # Attributes Attributes B-Tree node = 6754 (sector 0x294b0) # Attribute Key keyLength= 46 pad = 0 fileID = 3904426 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 16 bytes attrData = 66 70 6d 63 04 00 00 00 b0 a0 03 00 00 00 00 00 f p m c compression magic= cmpf compression type = 4 (resource fork has compressed data) uncompressed size= 237744 bytes A normal exec find in /usr/bin/ ( data stored in # Data Fork so it is showing as uncompressed in finder) flags= 0010 # BSD Info ownerID = 0 (root) groupID = 0 (wheel) = 0 # Data Fork logicalSize = 147680 bytes totalBlocks = 37 clumpSize= 2445 # Resource Fork logicalSize = 0 bytes the original file exec now copied via patched rsync within the whole /usr/bin directory (Note: no data in Data Fork or Resource Fork) First note that the xattrs show up ( not supposed to when seen from 10.6) robert-dutoits-macbook-pro:~ astrid$ xattr -l /Users/astrid/Desktop/copied_bin_test/bin/file com.apple.ResourceFork: com.apple.decmpfs: and the hfsdebug report: flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 1 bytes totalBlocks = 1 # Attributes Attributes B-Tree node = 5258 (sector 0x23730) # Attribute Key keyLength= 46 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 1 bytes attrData = 04 compression magic= . compression type = 4400 (?) uncompressed size= 1224979098644812920 bytes I can't put this all together yet but if anyone can see what is going on I would much appreciate it. What throws me though it that the damage only happens when copying the whole /usr/bin/ directory… Again - this is rsync 3.0.6 patched with fileflags, crtimes and Mike's hfs patch. Thanks, Rob On Jun 25, 2010, at 4:03 PM, Mike Bombich wrote: Can you share the steps required to reproduce this? I've seen two reports of this but have not been able to reproduce it myself, and the users that reported it also could not reproduce it. Mike On Jun 25, 2010, at 5:16 AM, Robert DuToit wrote: Hi All, I have been using rysnc 3.0.6 with Mike's rsync_3.0.6-hfs-compression_20091027.diff patch (as well as the standard osx patches) with good results. I discovered that clones done on 10.6 using this rsync cause some executable files ( the ones that are mostly compressed this way on osx ) to be damaged. I did a test twice and indeed a handful of them are damaged. They all appear to be in /Volume/Snowy/usr/bin and/or /Volume/Snowy/usr/share. I ran xattr
Re: rsync and hfs+ compressed files damaged
Thanks Mike once again, I'll test this out tomorrow. Rob On Jul 1, 2010, at 6:13 PM, Mike Bombich wrote: Well, it looks like the hfs_compression patch that I posted was the problem, and I've solved it since then. I can reproduce the issue with the patch, but not with my subsequent changes (my changes are posted in their entirety, vs. breaking out the hfs_compression- specific changes). Here is an updated patch, and I have verified that it resolves this issue. As an aside, the patch contains a couple bug fixes to the xattr code as well. I'll break those out and submit them separately in the near future. Mike rsync_3.0.7-hfs_compression_20100701.diff On Jul 1, 2010, at 6:16 AM, Robert DuToit wrote: Hi Mike and All, I have been doing some research on the damaged HFS compressed files and found some interesting clues. Looking at mike's patch so far I can't find any reason for the damage (I have to learn more C language ) but what still seems odd is that the damage only happens when copying the whole /usr directory or /usr/bin /usr/share. Individually copying one of the original files does not create the damaged file. Pertaining to the use of Mike's patch on OSX 10.6.4 copying /usr/ bin and resulting damaged files In /usr/bin/ I found that the originals of the damaged files are different from the other normal exec files. In this case I am looking at the find exec. If I copy find from /usr/bin/ to the desktop via finder copy, it is changed to resemble the normal execs. And indeed if I replace the original with this copy in /usr/bin and copy /usr/bin with patched rsync, the copied find is no longer damaged! So it appears that when the patched rsync encounters one of these odd files ( and only when copying the whole /usr/bin/ directory containing them ) it causes the problem. Here are results using hfsdebug (sorry about the amount of reading!) original file exec - works ok but different from other execs in / usr/bin. ( It shows the com.apple.decmpfs flag and data is stored in Resource Fork) flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 101755 bytes totalBlocks = 25 # Attributes Attributes B-Tree node = 6754 (sector 0x294b0) # Attribute Key keyLength= 46 pad = 0 fileID = 3904426 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 16 bytes attrData = 66 70 6d 63 04 00 00 00 b0 a0 03 00 00 00 00 00 f p m c compression magic= cmpf compression type = 4 (resource fork has compressed data) uncompressed size= 237744 bytes A normal exec find in /usr/bin/ ( data stored in # Data Fork so it is showing as uncompressed in finder) flags= 0010 # BSD Info ownerID = 0 (root) groupID = 0 (wheel) = 0 # Data Fork logicalSize = 147680 bytes totalBlocks = 37 clumpSize= 2445 # Resource Fork logicalSize = 0 bytes the original file exec now copied via patched rsync within the whole /usr/bin directory (Note: no data in Data Fork or Resource Fork) First note that the xattrs show up ( not supposed to when seen from 10.6) robert-dutoits-macbook-pro:~ astrid$ xattr -l /Users/astrid/Desktop/ copied_bin_test/bin/file com.apple.ResourceFork: com.apple.decmpfs: and the hfsdebug report: flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 1 bytes totalBlocks = 1 # Attributes Attributes B-Tree node = 5258 (sector 0x23730) # Attribute Key keyLength= 46 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 1 bytes attrData = 04 compression magic= . compression type = 4400 (?) uncompressed size= 1224979098644812920 bytes I can't put this all together yet but if anyone can see what is going on I would much appreciate it. What throws me though it that the damage only happens when copying the whole /usr/bin/ directory… Again - this is rsync 3.0.6 patched with fileflags, crtimes and Mike's hfs patch. Thanks, Rob On Jun 25, 2010, at 4:03 PM, Mike Bombich wrote: Can you share the steps required to reproduce this? I've seen two reports of this but have not been able to reproduce
Re: rsync and hfs+ compressed files damaged
Hi Mike, on the first build try I got a make error: patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 patches/rsync_3.0.7-hfs_compression_20100701.diff ./configure make compat.c: In function ‘setup_protocol’: compat.c:293: error: ‘CF_HFS_COMPRESSION’ undeclared (first use in this function) compat.c:293: error: (Each undeclared identifier is reported only once compat.c:293: error: for each function it appears in.) make: *** [compat.o] Error 1 Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and hfs+ compressed files damaged
Mike, Sorry about that - it seems to be working perfectly now. I hadn't realized it was built on 3.0.7 (I tried on 3.0.6). Even better! Thanks, Rob On Jul 1, 2010, at 6:13 PM, Mike Bombich wrote: Well, it looks like the hfs_compression patch that I posted was the problem, and I've solved it since then. I can reproduce the issue with the patch, but not with my subsequent changes (my changes are posted in their entirety, vs. breaking out the hfs_compression-specific changes). Here is an updated patch, and I have verified that it resolves this issue. As an aside, the patch contains a couple bug fixes to the xattr code as well. I'll break those out and submit them separately in the near future. Mike rsync_3.0.7-hfs_compression_20100701.diff On Jul 1, 2010, at 6:16 AM, Robert DuToit wrote: Hi Mike and All, I have been doing some research on the damaged HFS compressed files and found some interesting clues. Looking at mike's patch so far I can't find any reason for the damage (I have to learn more C language ) but what still seems odd is that the damage only happens when copying the whole /usr directory or /usr/bin /usr/share. Individually copying one of the original files does not create the damaged file. Pertaining to the use of Mike's patch on OSX 10.6.4 copying /usr/bin and resulting damaged files In /usr/bin/ I found that the originals of the damaged files are different from the other normal exec files. In this case I am looking at the find exec. If I copy find from /usr/bin/ to the desktop via finder copy, it is changed to resemble the normal execs. And indeed if I replace the original with this copy in /usr/bin and copy /usr/bin with patched rsync, the copied find is no longer damaged! So it appears that when the patched rsync encounters one of these odd files ( and only when copying the whole /usr/bin/ directory containing them ) it causes the problem. Here are results using hfsdebug (sorry about the amount of reading!) original file exec - works ok but different from other execs in /usr/bin. ( It shows the com.apple.decmpfs flag and data is stored in Resource Fork) flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 101755 bytes totalBlocks = 25 # Attributes Attributes B-Tree node = 6754 (sector 0x294b0) # Attribute Key keyLength= 46 pad = 0 fileID = 3904426 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 16 bytes attrData = 66 70 6d 63 04 00 00 00 b0 a0 03 00 00 00 00 00 f p m c compression magic= cmpf compression type = 4 (resource fork has compressed data) uncompressed size= 237744 bytes A normal exec find in /usr/bin/ ( data stored in # Data Fork so it is showing as uncompressed in finder) flags= 0010 # BSD Info ownerID = 0 (root) groupID = 0 (wheel) = 0 # Data Fork logicalSize = 147680 bytes totalBlocks = 37 clumpSize= 2445 # Resource Fork logicalSize = 0 bytes the original file exec now copied via patched rsync within the whole /usr/bin directory (Note: no data in Data Fork or Resource Fork) First note that the xattrs show up ( not supposed to when seen from 10.6) robert-dutoits-macbook-pro:~ astrid$ xattr -l /Users/astrid/Desktop/copied_bin_test/bin/file com.apple.ResourceFork: com.apple.decmpfs: and the hfsdebug report: flags= 0110 . File has extended attributes. # Data Fork logicalSize = 0 bytes # Resource Fork logicalSize = 1 bytes totalBlocks = 1 # Attributes Attributes B-Tree node = 5258 (sector 0x23730) # Attribute Key keyLength= 46 startBlock = 0 attrNameLen = 17 attrName = com.apple.decmpfs # Inline Data recordType = 0x10 reserved[0] = 0 reserved[1] = 0 attrSize = 1 bytes attrData = 04 compression magic= . compression type = 4400 (?) uncompressed size= 1224979098644812920 bytes I can't put this all together yet but if anyone can see what is going on I would much appreciate it. What throws me though it that the damage only happens when copying the whole /usr/bin/ directory… Again - this is rsync 3.0.6 patched with fileflags, crtimes and Mike's hfs patch
log format question
USING: rsync version 3.0.6 protocol version 30 on a Sun Solaris 10 x86. This
is a precompiled version from opencsw.
/opt/csw/bin/rsync -n -axzH -v --delete-after
--log-file=$RSYNCOUTPUT/export_-${DAILYDT}X --rsync-path=/opt/csw/bin/rsync -e
ssh -i /root/.ssh/id_hertz --max-delete=100 /export/.zfs/snapshot/$DAILYDT/
hertz3:/zvol/backup/gauss/export
When I added the --log-file= line I get log file messages like
2010/06/25 19:43:38 [16789] f..t.. rmcgraw/Library/Caches/Adobe/Flash
Player/AssetCache/5GUH56X5/1C04C61346A1FA3139A37D860ED92632AA13DECF.heu
2010/06/25 19:43:38 [16789] .d..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/
2010/06/25 19:43:38 [16789] f..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/._Cache
2010/06/25 19:43:41 [16789] f..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/urlclassifier3.sqlite
2010/06/25 19:43:41 [16789] .d..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._01D9AD9Ed01
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._07D1636Ad01
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._0AF1538Dd01
I am trying to find information on the string of character that follow the PID.
I have search high ad low with now luck.
I have looked in the rsyncd.conf file but what is talked about log-format does
not seem to give the information that I am looking for.
Can someone point me to a documentation that will give me the information.
Thanks
Robert
_
Robert P. McGraw, Jr.
Manager, Computer SystemEMAIL: rmcg...@purdue.edu
Purdue UniversityROOM: MATH-807
Department of Mathematics PHONE: (765) 494-6055
150 N. University Street
West Lafayette, IN 47907-2067
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
log format question: resolved
It was pointed out that this information is in the --itemize-changes parameter
in the man page.
I looked at the man page but missed this part. I guess it was the forest for
the trees.
Thanks and it is resolved.
Robert
USING: rsync version 3.0.6 protocol version 30 on a Sun Solaris 10 x86. This
is a precompiled version from opencsw.
/opt/csw/bin/rsync -n -axzH -v --delete-after
--log-file=$RSYNCOUTPUT/export_-${DAILYDT}X --rsync-path=/opt/csw/bin/rsync -e
ssh -i /root/.ssh/id_hertz --max-delete=100 /export/.zfs/snapshot/$DAILYDT/
hertz3:/zvol/backup/gauss/export
When I added the --log-file= line I get log file messages like
2010/06/25 19:43:38 [16789] f..t.. rmcgraw/Library/Caches/Adobe/Flash
Player/AssetCache/5GUH56X5/1C04C61346A1FA3139A37D860ED92632AA13DECF.heu
2010/06/25 19:43:38 [16789] .d..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/
2010/06/25 19:43:38 [16789] f..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/._Cache
2010/06/25 19:43:41 [16789] f..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/urlclassifier3.sqlite
2010/06/25 19:43:41 [16789] .d..t..
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._01D9AD9Ed01
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._07D1636Ad01
2010/06/25 19:43:41 [16789] f+
rmcgraw/Library/Caches/Firefox/Profiles/a66wvzw3.default/Cache/._0AF1538Dd01
I am trying to find information on the string of character that follow the PID.
I have search high ad low with now luck.
I have looked in the rsyncd.conf file but what is talked about log-format does
not seem to give the information that I am looking for.
Can someone point me to a documentation that will give me the information.
Thanks
Robert
_
Robert P. McGraw, Jr.
Manager, Computer SystemEMAIL: rmcg...@purdue.edu
Purdue UniversityROOM: MATH-807
Department of Mathematics PHONE: (765) 494-6055
150 N. University Street
West Lafayette, IN 47907-2067
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and hfs+ compressed files damaged
Hi Mike and All, Pertaining to the corrupted files via rsync: Build on OSX 10.6.4 rsync 3.0.6 patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 patches/rsync_3.0.6-hfs-compression_20091027.diff ./configure make (I have tried also using backup-dir-dels.diff and your extended_stats.diff with same results) this is my own basic clone line pathToPatchedRsync -aHAXNx --fileflags --force-change --protect-decmpfs --stats -v --progress / /Volumes/Snowy output file if I just copy the /usr folder to some other directory it has the same effect. pathToPatchedRsync -aHAXNx --fileflags --force-change --protect-decmpfs --stats -v --progress /usr /Volumes/Snowy/testFolder output file I ran find_sl_damaged_files tool which checks the file size and the decmpfs and resource fork flags and it shows this: /Volumes/Snowy/usr/bin/bzip2recover appears to be damaged /Volumes/Snowy/usr/bin/file appears to be damaged /Volumes/Snowy/usr/bin/chudRemoteCtrl appears to be damaged /Volumes/Snowy/usr/bin/chumAddRights appears to be damaged /Volumes/Snowy/usr/bin/ci appears to be damaged /Volumes/Snowy/usr/bin/xed appears to be damaged /Volumes/Snowy/usr/share/man/man1/ibtool.1 appears to be damaged /Volumes/Snowy/usr/share/zoneinfo/America/Santa_Isabel appears to be damaged I tested /Volumes/Snowy/usr/bin/file and indeed it is broken on the that clone It copies everything fine - seems a good clone but there are these several files. Someone has reported more corrupted files, or none, and say sometimes it changes from clone to clone. My experience produces the same corrupted files every time. I have done maybe 20 tests now. I'll ask others to check it out. Now here's the strange part. If I copy just those problem files themselves, and not the entire /usr directory they aren't corrupted. Also if I copy a handful of exec files (including the problem ones) to another folder and copy those with rsync, they are fine too! As I said before I have read the Ars Technica article and others about hfs+ compression and seen reports of these same problems on a fresh install of 10.6 As a test I tried using CCC making a clone and then just the /usr folder and both times it comes up clean in the test - no corrupted files in there. I also tried ditto with the new −-hfsCompression option and that seems ok. So you must have done something right though I think we are using the same hfs+ patch that you posted. Or maybe you didn't use the patch but patched rsync directly…. Let me know if I can supply any more details on this. On Jun 25, 2010, at 4:03 PM, Mike Bombich wrote: Can you share the steps required to reproduce this? I've seen two reports of this but have not been able to reproduce it myself, and the users that reported it also could not reproduce it. Mike On Jun 25, 2010, at 5:16 AM, Robert DuToit wrote: Hi All, I have been using rysnc 3.0.6 with Mike's rsync_3.0.6-hfs-compression_20091027.diff patch (as well as the standard osx patches) with good results. I discovered that clones done on 10.6 using this rsync cause some executable files ( the ones that are mostly compressed this way on osx ) to be damaged. I did a test twice and indeed a handful of them are damaged. They all appear to be in /Volume/Snowy/usr/bin and/or /Volume/Snowy/usr/share. I ran xattr on it and robert-dutoits-macbook-pro:~ astrid$ xattr -l /Volume/Snowy/usr/bin/file com.apple.ResourceFork: com.apple.decmpfs: These attributes are supposed to be hidden when on 10.6. The files have 0 bytes to appear to be compressed but can not be un-compresed. I tried file on the clone and it doesn't work. Someone else reported du not working so there are some pretty basic files damaged. This appears to be the same issue that happens with some people when installing Snow Leopard fresh - some execs are damaged in this way. There are some threads about this, specifically bugs in the Snow Leopard causing this very thing to happen. There is a tool someone created for scanning for these damaged files at: http://www.charlessoft.com/find_sl_damaged_files.zip After running rsync 3.0.6 , with patches for hfs+, it reports /Volume/Snowy/usr/bin/chudRemoteCtrl appears to be damaged /Volume/Snowy/usr/bin/chumAddRights appears to be damaged /Volume/Snowy/usr/bin/ci appears to be damaged /Volume/Snowy/usr/bin/bzip2recover appears to be damaged /Volume/Snowy/usr/bin/file appears to be damaged /Volume/Snowy/usr/bin/xed appears to be damaged These same files are fine on the source volume. Anyone know why this is happening? The hfs+ compressed files are otherwise handled perfectly using Mike's patch. Thanks Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
RE: How can I make rsync not overwrite files that has not changed?
Subject: How can I make rsync not overwrite files that has not changed? Sorry for asking this but I find the manpage to be a little confusing on this. I want rsync to only replace files that has acctually a different content, it should completely ignore any timestamps on files. -c, --checksum skip based on checksum, not mod-time size -- Bob Bagwill -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OSX and HFS+ filesystem compression
Hi Mike, Just got in I did discover the --protect-decmpfs and got it working! Great news. Thanks for keeping this alive and well for us OSX folks. I'll rebuild with the new diff you sent today. Thanks again, Rob On May 19, 2010, at 12:19 PM, Mike Bombich wrote: Hi Robert: My apologies, that if (fnamecmp) should be commented out as you discovered. Did you read the updated man page to see the new arguments available for supporting HFS+ compression? It isn't on by default, you need to use --protect-decmpfs, for example, to preserve those xattrs. I crafted a new diff and tested it again against 3.0.6. These steps work for me: cd ~/Desktop curl -O http://rsync.samba.org/ftp/rsync/src/rsync-3.0.6.tar.gz tar -xzvf rsync-3.0.6.tar.gz rm rsync-3.0.6.tar.gz curl -O http://rsync.samba.org/ftp/rsync/src/rsync-patches-3.0.6.tar.gz tar -xzvf rsync-patches-3.0.6.tar.gz rm rsync-patches-3.0.6.tar.gz cd rsync-3.0.6 patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff patch -p1 (path to the attached file) echo #define EXTRA_ROUNDING 0 rounding.h ./prepare-source ./configure make sudo ./rsync -aHANXx --force-change --protect-decmpfs --fileflags /Volumes/Source/ /Volumes/Target/ sudo bbouncer verify -d /Volumes/Source /Volumes/Target Verifying: hfs-compression ... Sub-test:decmpfs xattr ... preserved Sub-test: UF_COMPRESSED flag ... set Sub-test:file contents ... match Sub-test:creation time ... ok Sub-test:modification time ... ok Sub-test: hard link inode ... ok Sub-test: hard link decmpfs xattr ... preserved Sub-test: hard link UF_COMPRESSED flag ... set Sub-test: hard link modification time ... ok ok Verifying: hfs-compression_large ... Sub-test:decmpfs xattr ... preserved Sub-test: UF_COMPRESSED flag ... set Sub-test:file contents ... match Sub-test:creation time ... ok Sub-test:modification time ... ok Sub-test: hard link inode ... ok Sub-test: hard link decmpfs xattr ... preserved Sub-test: hard link UF_COMPRESSED flag ... set Sub-test: hard link modification time ... ok ok Mike rsync_3.0.6-hfs-compression_20091027.diff On May 18, 2010, at 10:04 PM, Robert DuToit wrote: Hi Mike, I guess I did something wrong or removing that line 283 in proto.h was critical I ran fileflags and crtimes patches first and then rsync_3.0.6-hfs-compression_20091027.diff but the test shows no go. Sub-test: decmpfs xattr ... not preserved Sub-test: UF_COMPRESSED flag ... not set Thanks, rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
OSX and HFS+ filesystem compression
Hi All, I have been reading about the HFS+ filesystem compression on Snow Leopard and how copying or cloning over system files with rsync and other tools results in them being expanded on destination. I was wondering if there was any thoughts on updating rsync to accommodate this? It is kind of a shame to the lose the benefits of the new compression. Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OSX and HFS+ filesystem compression
Hi Mike, Thanks so much After some searching I just found the patch and built on 3.0.6 but got an error on make after running fileflags.diff, crtimes.diff and your patch perl ./mkproto.pl ./*.c ./lib/compat.c In file included from ./rsync.h:971, from ./rounding.c:20: ./proto.h:283: error: expected identifier or ‘(’ before ‘if’ Failed to create rounding.h! make: *** [rounding.h] Error 1 I wasn't sure what to do with line 283 if (fnamecmp); seems to be stuck in there so I removed it and it builds but is this ok now? Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: OSX and HFS+ filesystem compression
Hi Mike, I guess I did something wrong or removing that line 283 in proto.h was critical I ran fileflags and crtimes patches first and then rsync_3.0.6-hfs-compression_20091027.diff but the test shows no go. Sub-test: decmpfs xattr ... not preserved Sub-test: UF_COMPRESSED flag ... not set Thanks, rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Count of files to be transferred
Hi All, On Apr 30, 2010, at 5:29 PM, Matt McCutchen wrote: On Fri, 2010-04-30 at 14:21 +0700, Ryan Joseph wrote: On Apr 30, 2010, at 12:54 PM, Matt McCutchen wrote: Rsync determines which of the files in the file list need a data transfer as it goes, so the only way to get the count in advance is to do a separate dry run. Then you could take the Number of files transferred from --stats output. What do you mean dry run and wouldn't that take as much time as the real backup thus effectively doubling the time? If so that's probably not a good option. Rsync has a --dry-run option to run without actually making the changes to the destination. The source and destination are scanned as normal, but transmission of the contents of files is skipped. See the man page for more information. Alternatively, you could base the progress bar on the total number of entries in the file list. The --progress option does this, but it won't be accurate if incremental recursion is enabled. I tried the --progress option but it only seemed to show the progress for each file which really isn't helpful for my case. See the to-check values at the end of each progress line, for example: $ rsync -r --progress src2/ dest2/ sending incremental file list created directory dest2 ./ filea 0 100%0.00kB/s0:00:00 (xfer#1, to-check=1/3) fileb 0 100%0.00kB/s0:00:00 (xfer#2, to-check=0/3) sent 126 bytes received 53 bytes 358.00 bytes/sec total size is 0 speedup is 0.00 Alternatively, if you use a development snapshot of rsync 3.1.0, the --info=progress2 option will give a size and percentage that refer to the entire set of source files, not the current file. How do I get the file list then? If I could read a list of files that will be transferred (rsync says it builds this list so it must be somewhere) I don't know if this is what you are looking for but I have a GUI for rsync (Backuplist+) and now use unix find directoryPath | wc -l and it is very fast and gives you the total file count which reflects the rsync count in stats at the end of the run, almost exactly. I then just use the total count and the to check number to easily get the # of files transferred of total, for updating a progress bar. You could get fancy and use find' with options to reflect any options for rsync that may exclude certain files etc... This is on OSX but would apply anywhere I would think. For 1 million files it takes about 2.5 minutes for the initial find count, on my old Macbook Pro laptop, and then rsync goes about it's business. This is much faster then running rsync non-recursively. FWIW... yours, Rob What rsync is building is the file list, which contains /all/ source files. As I said, rsync determines as it goes which of those files need a data transfer. -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Count of files to be transferred
On Apr 30, 2010, at 8:26 PM, Ryan Joseph wrote: On May 1, 2010, at 6:58 AM, Robert DuToit wrote: I don't know if this is what you are looking for but I have a GUI for rsync (Backuplist+) and now use unix find directoryPath | wc -l and it is very fast and gives you the total file count which reflects the rsync count in stats at the end of the run, almost exactly. I then just use the total count and the to check number to easily get the # of files transferred of total, for updating a progress bar. You could get fancy and use find' with options to reflect any options for rsync that may exclude certain files etc... This is on OSX but would apply anywhere I would think. For 1 million files it takes about 2.5 minutes for the initial find count, on my old Macbook Pro laptop, and then rsync goes about it's business. This is much faster then running rsync non-recursively. I actually already tried this approach first until I realized the files that will be transferred are often less than the total count returned by find so the progress bar was never able to complete totally. In the instance the backup was transferring all files it worked fine but often the transfers just update changed files so the numbers were off. I know - I couldn't find a way to get the count of changes only but it doesn't bother me to see the percentage of changes copied vs. the total count. Let me know if you do though. Rob Regards, Josef -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
Hi Carson, OK no differences: Build 3.0.7 with or without patches Build 3.0.6 with or without patches Apple rsync (as far as Users report all have same issues with hidden files on OSX 10.6.3 This is transferring over local ethernet network to another Mac OSX 10.6.3 ( Again I had to use my old 10.5 box to send but all users are reporting issues with all 10.6.3 machines) basic Mac options path/to/rsync -aHAXN --fileflags --force-change -stats - src dst output Rob On Apr 7, 2010, at 5:59 AM, carson+rs...@taltos.org wrote: Robert DuToit wrote: On Apr 6, 2010, at 9:14 AM, Benjamin R. Haskell wrote: On Mon, 5 Apr 2010, Robert DuToit wrote: Hi Matt, I set up a simple test with a nest of directories ( aa bb cc dd ee) with 1 file in each. running rsync from OS 10.6 to another Mac with OS10.5 there seems to be no problem. When doing the reverse I am seeing the odd behavior. Below is the log from running the options -aHAXN --fileflags --force-change -stats -vvv Please specify _exactly_ how you're building rsync. Those options don't exist without applying some extra patches. I can't replicate the problem with stock rsync, which means it's either a problem with the patches, or I'm not triggering the bug properly. Can you replicate this without a 10.5 box? Can you replicate it with Apple's rsync? Unpatched 3.0.7 rsync? Patched 3.0.7 rsync? Nightly snapshot? I really don't want to spend time investigating an old release, and I need to be able to reproduce the problem before I can help fix it. I have 10.6 and OpenSolaris easily at hand to test with, but no 10.5 boxes. I checked out dtrace (OSX version of strace) but it seems a bit complicated and I don't know how to use it yet. If anyone has maybe they can let me know. sudo dtruss -f /path/to/rsync --whatever src dest 2/tmp/mydtrusslog should dump all the syscalls into /tmp/mydtrusslog -- Carson -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
Matt, I forgot to mention, If I add an exclude for .DS_Store* it doesn't generate the odd .DS_Store files. However, if I copy directories with other types of hidden files like .bash_history then it generates odd versions of those too. So I added an exclude for every hidden (dot prefix) file .* and it seems to work ok. This might be ok as an emergency measure of course but not desirable since a lot of hidden files are needed and at root level indispensable. Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
On Apr 6, 2010, at 9:14 AM, Benjamin R. Haskell wrote: On Mon, 5 Apr 2010, Robert DuToit wrote: Hi Matt, I set up a simple test with a nest of directories ( aa bb cc dd ee) with 1 file in each. running rsync from OS 10.6 to another Mac with OS10.5 there seems to be no problem. When doing the reverse I am seeing the odd behavior. Below is the log from running the options -aHAXN --fileflags --force-change -stats -vvv note that the lines rsync: mkstemp /Volumes/xxx/testaaa/aa/..DS_Store.gnG29l failed: No such file or directory (2) are correct in that there are no such files but why does it think there are? Those files are rsync's temporary files, which it transfers first, then atomically moves into place. Note the many other entries like: Yes I see now that something in that process of temp files goes awry. /path/to/.ORIGINALNAME.XX e.g.: ORIGINALNAME = splittersize.h and XX = hVRihb in the following: set modtime of aa/bb/cc/dd/.splittersize.h.hVRihb to (1236212797) Wed Mar 4 19:26:37 2009 renaming aa/bb/cc/dd/.splittersize.h.hVRihb to aa/bb/cc/dd/splittersize.h So, it appears something has changed between 10.5 and 10.6 with how the annoying-to-non-OSX users .DS_Store file is handled. I just ran into __MACOSX/.DS_Store files in a ZIP file the other day. Annoying because they force me to have to filter out the garbage files: unzip file.zip -x '__MACOSX/*' A possibly interesting (probably not) experiment would be to create ZIP files on each machine, then open them on a non-OSX machine to see whether 10.6 creates a file without the .DS_Store files. See: http://en.wikipedia.org/wiki/.DS_Store for the purpose of the files. (basically, per-folder Finder preferences) They don't exist on source or dest. They do however get created on the destination. It seems in my test that there is one created for every directory. However, the original .DS_Store files are not copied over just the new ones with the odd extra dot prefix and the cryptic extension on the end. The new odd .DS_Store files take the place of the normal ones that should be created by the finder. I don't know if this sheds any light at all on this. Others have reported that these files get created by the thousands on the destination. Not only .DS_Store files but other hidden files. My guess is that one ..DS_Store.XX file is created per-directory per-transfer. Other hidden files are likely created during failed transfers. If something changed between 10.5 and 10.6 regarding default permissions on .DS_Store files, that might explain it. And maybe the files are created automatically at an earlier time (so can't be overwritten by the rename). But I'm not sure what the best solution would be. The .DS_Store files may or may not be something each user would want to keep. You can live without the .DS_Store files though they do hold directory info but the many root level hidden files are part of the OS. It seems any hidden files (with a dot prefix) cause this to happen - though I haven't tested yet at root level. I checked out dtrace (OSX version of strace) but it seems a bit complicated and I don't know how to use it yet. If anyone has maybe they can let me know. Thanks, Rob -- Best, Ben -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
On Apr 5, 2010, at 1:28 PM, Matt McCutchen wrote: On Sat, 2010-04-03 at 18:12 -0400, Robert DuToit wrote: I have been getting reports from users of backuplist+, my wrapper application for rsync (currently with build of 3.0.6), about odd behavior after updating to OS 10.6 Snow Leopard. Basically: the problem occurs backing up a directory to a local mounted network volume. Previously all worked fine but after updating to 10.6 there are reports that no files get copied and the destination fills up with thousands (yes thousands) of .DS-Store files which on OSX are invisible and carry metadata about the directory that contains them (usually just one per directory...) In order to investigate this phenomenon from the rsync side, we'll need strace output (or whatever the Mac equivalent is). Matt Not sure what Strace is... would that be the Mac console and crash reporter logs? I am trying to get some more verbose rsync error reporting from users. So far it seems rsync is reporting missing files (the hidden dot files on the mac) and then mysteriously creating thousands of hidden files into directories. These are files that don't exist - they are being created somehow! It gets worse and worse apparently. I'll rig up a network myself and get back to you with some details of what is happening. Rob -- Matt -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: osx 10.6 strange rsync errors
Hi Matt, I set up a simple test with a nest of directories ( aa bb cc dd ee) with 1 file in each. running rsync from OS 10.6 to another Mac with OS10.5 there seems to be no problem. When doing the reverse I am seeing the odd behavior. Below is the log from running the options -aHAXN --fileflags --force-change -stats -vvv note that the lines rsync: mkstemp /Volumes/xxx/testaaa/aa/..DS_Store.gnG29l failed: No such file or directory (2) are correct in that there are no such files but why does it think there are? They don't exist on source or dest. They do however get created on the destination. It seems in my test that there is one created for every directory. However, the original .DS_Store files are not copied over just the new ones with the odd extra dot prefix and the cryptic extension on the end. The new odd .DS_Store files take the place of the normal ones that should be created by the finder. I don't know if this sheds any light at all on this. Others have reported that these files get created by the thousands on the destination. Not only .DS_Store files but other hidden files. Rob sending incremental file list [sender] make_file(aa,*,2) send_file_list done [sender] make_file(aa/.DS_Store,*,2) [sender] make_file(aa/bb,*,2) [sender] make_file(aa/zwhite.icns,*,2) send_files starting [sender] make_file(aa/bb/.DS_Store,*,2) [sender] make_file(aa/bb/cc,*,2) [sender] make_file(aa/bb/excludes.txt,*,2) [sender] make_file(aa/bb/cc/.DS_Store,*,2) [sender] make_file(aa/bb/cc/dd,*,2) [sender] make_file(aa/bb/cc/ee,*,2) [sender] make_file(aa/bb/cc/size.txt,*,2) [sender] make_file(aa/bb/cc/dd/.DS_Store,*,2) [sender] make_file(aa/bb/cc/dd/splittersize.h,*,2) server_recv(2) starting pid=939 recv_file_name(aa) received 1 names recv_file_list done recv_file_name(aa/.DS_Store) recv_file_name(aa/bb) recv_file_name(aa/zwhite.icns) received 3 names recv_file_list done get_local_name count=4 /Volumes/xxx/testaaa/ generator starting pid=939 delta-transmission disabled for local transfer or --whole-file recv_files(1) starting recv_file_name(aa/bb/.DS_Store) recv_file_name(aa/bb/cc) recv_file_name(aa/bb/excludes.txt) received 3 names recv_file_list done recv_file_name(aa/bb/cc/.DS_Store) recv_file_name(aa/bb/cc/dd) recv_file_name(aa/bb/cc/ee) recv_file_name(aa/bb/cc/size.txt) received 4 names recv_file_list done recv_file_name(aa/bb/cc/dd/.DS_Store) recv_file_name(aa/bb/cc/dd/splittersize.h) received 2 names recv_file_list done received 0 names recv_file_list done recv_generator(aa,1) recv_generator(aa,2) send_files(2, /Users/astrid/Desktop/aa) aa/ set modtime of aa to (1270518365) Mon Apr 5 21:46:05 2010 recv_generator(aa/.DS_Store,3) recv_files(aa) send_files(3, /Users/astrid/Desktop/aa/.DS_Store) send_files mapped /Users/astrid/Desktop/aa/.DS_Store of size 6148 calling match_sums /Users/astrid/Desktop/aa/.DS_Store aa/.DS_Store 6148 100%0.00kB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 6148 100%0.00kB/s0:00:00 (xfer#1, to-check=11/13) sender finished /Users/astrid/Desktop/aa/.DS_Store recv_generator(aa/zwhite.icns,4) send_files(4, /Users/astrid/Desktop/aa/zwhite.icns) send_files mapped /Users/astrid/Desktop/aa/zwhite.icns of size 54611 calling match_sums /Users/astrid/Desktop/aa/zwhite.icns aa/zwhite.icns 32768 60% 820.51kB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 54611 100%1.21MB/s0:00:00 (xfer#2, to-check=10/13) sender finished /Users/astrid/Desktop/aa/zwhite.icns recv_generator(aa/bb,5) recv_generator(aa/bb,6) recv_files(aa/.DS_Store) send_files(6, /Users/astrid/Desktop/aa/bb) aa/bb/ set modtime of aa/bb to (1270518373) Mon Apr 5 21:46:13 2010 recv_generator(aa/bb/.DS_Store,7) rsync: mkstemp /Volumes/xxx/testaaa/aa/..DS_Store.gnG29l failed: No such file or directory (2) send_files(7, /Users/astrid/Desktop/aa/bb/.DS_Store) send_files mapped /Users/astrid/Desktop/aa/bb/.DS_Store of size 6148 calling match_sums /Users/astrid/Desktop/aa/bb/.DS_Store aa/bb/.DS_Store 6148 100% 103.52kB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 6148 100% 101.76kB/s0:00:00 (xfer#3, to-check=8/13) sender finished /Users/astrid/Desktop/aa/bb/.DS_Store recv_generator(aa/bb/excludes.txt,8) send_files(8, /Users/astrid/Desktop/aa/bb/excludes.txt) send_files mapped /Users/astrid/Desktop/aa/bb/excludes.txt of size 285 calling match_sums /Users/astrid/Desktop/aa/bb/excludes.txt aa/bb/excludes.txt 285 100%4.42kB/s0:00:00 sending file_sum false_alarms=0 hash_hits=0 matches=0 285 100%4.35kB/s0:00:00 (xfer#4, to-check=7/13) sender finished /Users/astrid/Desktop/aa/bb/excludes.txt got file_sum recv_generator(aa/bb/cc,9) recv_generator(aa/bb/cc,10) send_files(10, /Users/astrid/Desktop/aa/bb/cc) aa/bb/cc/ recv_files(aa/zwhite.icns) set modtime of aa/bb/cc to (1270518390) Mon Apr 5 21:46:30 2010 recv_generator(aa/bb/cc/.DS_Store,11)
Re: osx 10.6 strange rsync errors
Hi Eric et al, It seems this is not an rsync problem though I am not sure The same thing happens with the Apple supplied rsync as with 3.0.7 (someone emailed me about the same problem using the old apple supplied rsync) and I wonder if other file transfer options might have similar results. I am still mystified about the cause of this except that it started with the latest OS update. Perhaps having to do with security enhancements? This is a big problem as administrators have been using rysnc to sync their network machines and everything is failing suddenly. Any one else have a clue? Rob On Apr 4, 2010, at 3:34 PM, Eric Jan Pot wrote: With rsync 3.0.7 I have exactly the same problem. I am using rSync to sync my Userdirectory to another Mac in a home network. I am not 100% sure, but the problem occurred after the latest update of the MacOS (Mac OS X 10.6.3 (10D573)) of the Source. My macmini (the Dest of rSync) is also running (Mac OS X 10.6.3 (10D573)). I tried several options but till now without any results. Eric -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
osx 10.6 strange rsync errors
Hi All, I have been getting reports from users of backuplist+, my wrapper application for rsync (currently with build of 3.0.6), about odd behavior after updating to OS 10.6 Snow Leopard. Basically: the problem occurs backing up a directory to a local mounted network volume. Previously all worked fine but after updating to 10.6 there are reports that no files get copied and the destination fills up with thousands (yes thousands) of .DS-Store files which on OSX are invisible and carry metadata about the directory that contains them (usually just one per directory...) The destination files are visible and labeled ..DS_Store.0aecRT ..DS_Store.0afcRT ..DS_Store.0cecRT ..DS_Store.0cfcRT (Note the two dots prefix?) on an on and on. One user reported 97,000 of these with seemingly random names attached. rsync log shows file has vanished: /Volumes/MacMini/Users/rg/..DS_Store.ZxgavW file has vanished: /Volumes/MacMini/Users/rg/..DS_Store.ZygavW file has vanished: /Volumes/MacMini/Users/rg/..DS_Store.ZzgavW So before any files are copied it fails on the missing files that aren't there in the first place! The basic command line is pathtoRsync -aHAXN --fileflags --force-change --stats -v -u --progress pathttoSourceDirectory pathToDestinationNetworkDirectory pathToOutputLogFile echo $! This works fine when going to an external mounted Volume or a partition, just not to a local mounted Network Volume. I haven't been able to get any more detailed information from users so far. Has anyone seen this or have a clue to what's happening? Something to do with the Network and DS_Store files it seems. I'm not sure where to even begin. Thanks, Rob DuToit -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
File list and find_fuzzy() insights for algo fiddling
Hi! I read the source around find_fuzzy(), but got stuck in get_dirlist(). I thought the latter returns the subset of the current “global” file list of the receiver side (which is complete, or an incremental part(?)), reduced to the “same directory” restriction. Can someone enlighten me, where the “global file list” is stored? Is that actually used for fuzzy-match? If not, why not? When is recv_generator() invoked? It seems to store some static fuzzy-related data, but I don’t get the big picture when this data is updated. My intention is to augment the fuzzy-search to search in the whole receiver target subtree. Thanks, Robert -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
--fuzzy enhancements: size match in all directories
Hello everyone, Image you rsynced your mp3 archive. Later you do some cleanup renaming and start splitting up the directory into a hierarchy and do some file move around. Data-wise you did nothing, meta-data-wise you did a lot. --fuzzy comes into mind for the next rsync. Unfortunately fuzzy matching does not include other (sub-)directories and cares a little too much about modification times for this case. I was thinking about introducing a superset of the current fuzzy matching (works initially like the original, but tries more base files if nothing matched so far), and/or two new threshold values with e.g. --fuzzy-thresholds 1000:2 where the numbers refer to the file size on the sender-side, the first meaning “below this size, don’t even consider fuzzy matching” and the second number meaning “above this size try harder to find a base file”. This could default to --fuzzy-thresholds 0:unlimited, the old behaviour. In case of the more aggressive search: when running out of base files with the original algorithm, try _all_ files in the destination hierarchy with just the same size, possibly sorted by Levenshtein-distance for the file name with full path. The idea is to catch simple copy/move arounds, while still keeping unreasonable base files away. Especially with bigger files, the likeliness of exact same size collisions is pretty small. The risk is: unnecessary checksum calculations with a wrong base file. If you think that risk is too high, don’t use that option... Is there a good reason why this functionality is not in rsync yet? Regards, Robert -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
build on osx 10.6
Hi All, I've been off list for a while and just got around to building rsync 3.0.6 on Mac OSX 10.6 intel core 2 duo machine. It runs great on similar machines but someone just said it won't for their OSX 10.6 intel core duo and returns an error: rsync Bad CPU type in executable. I check the file and it says: rsync: Mach-O 64-bit executable x86_64 my older builds all automatically came up i386. I just ran the usual patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff ./configure make Is this right or should it be i386? and how to I get that back or build so it runs on the other maqchine? Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: build on osx 10.6
Hi Vitorio, Yes -I kind of figured default was that now. What would be the way to get i386 back: would ./configure -arch i386 -arch x86_64 work? Rob On Oct 25, 2009, at 1:30 PM, Mac User FR wrote: For me this is a problem of gcc config on X.6. It's targetting x64 by default instead of x86. Cheers, Vitorio Le 25 oct. 09 à 18:19, Robert DuToit a écrit : Hi All, I've been off list for a while and just got around to building rsync 3.0.6 on Mac OSX 10.6 intel core 2 duo machine. It runs great on similar machines but someone just said it won't for their OSX 10.6 intel core duo and returns an error: rsync Bad CPU type in executable. I check the file and it says: rsync: Mach-O 64-bit executable x86_64 my older builds all automatically came up i386. I just ran the usual patch -p1 patches/fileflags.diff patch -p1 patches/crtimes.diff ./configure make Is this right or should it be i386? and how to I get that back or build so it runs on the other maqchine? Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Backups Directory Timestamps Not Preserved
Hello, I've been testing out using rsync for nightly incremental backups through the '--backup' '--backup-dir' options. So far, I've noticed two issues. 1) First, if an empty directory is removed from the source, rsync will remove it from the destination but not create the directory in the backup-dir location. $ mkdir -p /tmp/src/foo $ rsync -a /tmp/src/ /tmp/dest/ $ rmdir /tmp/src/foo $ rsync -a --delete -P -b --backup-dir=/tmp/desti/ /tmp/src/ /tmp/dest/ 2) Second, when the contents of a non-empty directory is modified, the original timestamp is not preserved in the backup-dir location. $ mkdir -p /tmp/src/foo $ touch -t 200904011200 /tmp/src/foo/bar /tmp/src/foo /tmp/src $ rsync -a /tmp/src/ /tmp/dest/ $ rm -rf /tmp/src/foo/bar $ rsync -a --delete -P -b --backup-dir=/tmp/desti/ /tmp/src/ /tmp/dest/ $ ls -ald /tmp/desti/foo /tmp/desti/foo/bar I'm using version 3.0.6. I've played around with various options but did not find a combination where this worked. I scanned through the source, and it appears as though rsync only creates directories as required by files when they are moved over and doesn't update the timestamps after the directory's contents are processed. Is this accurate or is there some combination which will allow this to work? -Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Using rsync for a backup program but having trouble getting --exclude-from to work properly
On Thu, Apr 16, 2009 at 4:33 PM, Paul Slootman paul+rs...@wurtel.net wrote: On Thu 16 Apr 2009, Robert Parker wrote: fromdir=/home/bob/ rsync --exclude-from=$excl -a -b -vv --suffix=$suffix $fromdir $todir Here is the content of the exclude file /home/bob/.mozilla/firefox/ If you're passing /home/bob/ as the source dir, then you have to remove /home/bob from the exclude patterns as those patterns are relative to the transfer source. I.e. /home/bob/.mozilla/firefox/ etc. becomes /.mozilla/firefox/ /Ebooks.desktop /Office.desktop /Pictures.desktop /Programs.desktop Thank you for that. I reckon I can fix it now. Bob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Using rsync for a backup program but having trouble getting --exclude-from to work properly
Reading and re-reading the man page has got me nowhere with this issue and searching your site on '--exclude-from' got me to the same place. Here is the script I am using: #/bin/bash suffix=`date +%Y%m%d` echo Suffix is $suffix fromdir=/home/bob/ echo Source dir is $fromdir todir=$maxtor6BackupOfHome excl=$fromdir.dobackup.excl echo Exclusions file is $excl echo Destination dir is $todir rsync --exclude-from=$excl -a -b -vv --suffix=$suffix $fromdir $todir exit 0 Here is the content of the exclude file # .dobackup.excl - list of exclusions from rsync backup /home/bob/.mozilla/firefox/ /home/bob/Ebooks.desktop /home/bob/Office.desktop /home/bob/Pictures.desktop /home/bob/Programs.desktop The command I use: ./dobackup.sh list Partial content of list Suffix is 20090416 Source dir is /home/bob/ Exclusions file is /home/bob/.dobackup.excl Destination dir is /media/disk-4/BackupOfHome sending incremental file list delta-transmission disabled for local transfer or --whole-file snippage .xsession-errors Bookmarks 2009-04-11.json is uptodate Ebooks.desktop # rubbish that I don't want in the backup Index.pdf is uptodate Names.pdf is uptodate Office.desktop # rubbish that I don't want in the backup ParrotSecrets.pdf is uptodate Pictures.desktop # rubbish that I don't want in the backup Programs.desktop # rubbish that I don't want in the backup Qantas.wmv is uptodate ThaiAlphabet.ods is uptodate snippage .mozilla/firefox/# and I want none of this directory at all .mozilla/firefox/profiles.ini .mozilla/firefox/28l3cgx4.default/ .mozilla/firefox/28l3cgx4.default/.parentlock .mozilla/firefox/28l3cgx4.default/XPC.mfasl .mozilla/firefox/28l3cgx4.default/XUL.mfasl .mozilla/firefox/28l3cgx4.default/blocklist.xml .mozilla/firefox/28l3cgx4.default/bookmarks.html .mozilla/firefox/28l3cgx4.default/cert8.db snipped the rest The system I am using is eeebuntu 8.10 on a eeePc 701 if that has any bearing on anything. I'm probably just misreading something. Thanks, Bob Parker -- In a world without walls who needs Windows (or Gates)? Try Linux instead! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync is failiing
Hi list, I am copying about 200GB of data from Windows to UNIX, I have setup Cygwin/rsync on Windows 2003R2 and when rsync was run on Windows it copied about 40GB of data and after that it is failing with the following errors: *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir /tempfs/238c/80/12/59 failed: Permission deni ed (13) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir /tempfs/238c/80/12/5a failed: Permission deni ed (13) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir /tempfs/238c/80/12/5b failed: Permission deni ed (13) *** Skipping any contents from this failed directory *** rsync: recv_generator: mkdir /tempfs/238c/80/12/5c failed: Permission deni ed (13) It created whole bunch of directories not sure why it is failing to create some, I gave permissions 777 for the entire file system to just make this work but no luck. Any one know why it is failing? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync 3.0.5 - ERROR: buffer overflow in recv_file_entry [generator]
Hello dragy, Anyone? Monday, February 16, 2009, 1:54:36 PM, you wrote: dr...@wp.pl Hi, dr...@wp.pl Rsync 3.0.5 on both sides. receiver is Open Solaris and sender is Linux. dr...@wp.pl I got a nested directory - relatively depth but still it should work. It dr...@wp.pl fails instead. dr...@wp.pl sending daemon args: --server --sender -vvlWHogDtpre.is --ignore-errors dr...@wp.pl --numeric-ids --inplace . TEST/ dr...@wp.pl receiving incremental file list dr...@wp.pl delta-transmission disabled for local transfer or --whole-file dr...@wp.pl [...] dr...@wp.pl overflow: xflags=0x7a l1=255 l2=779 dr...@wp.pl lastname=/Z/c/OOO/www/sigs dr...@wp.pl etc/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations dr...@wp.pl [receiver] dr...@wp.pl ERROR: buffer overflow in recv_file_entry [receiver] dr...@wp.pl rsync error: error allocating core memory buffers (code 22) at dr...@wp.pl util.c(123) [receiver=3.0.5] dr...@wp.pl I also tried rsync 2.6.9 on client side and I got: dr...@wp.pl opening tcp connection to saturn.nildram.co.uk port 873 dr...@wp.pl opening connection using --server --sender -vvlWHogDtpr --ignore-errors dr...@wp.pl --numeric-ids --inplace . ALL/ dr...@wp.pl receiving file list ... dr...@wp.pl [receiver] expand file_list to 262144 bytes, did move dr...@wp.pl [receiver] expand file_list to 524288 bytes, did move dr...@wp.pl [receiver] expand file_list to 1048576 bytes, did move dr...@wp.pl [receiver] expand file_list to 2097152 bytes, did move dr...@wp.pl [receiver] expand file_list to 4194304 bytes, did move dr...@wp.pl [receiver] expand file_list to 8388608 bytes, did move dr...@wp.pl overflow: flags=0x7a l1=255 l2=779 dr...@wp.pl lastname=/Y/c/000/www/sigs dr...@wp.pl etc/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations/animations dr...@wp.pl ERROR: buffer overflow in receive_file_entry [receiver] dr...@wp.pl rsync error: error allocating core memory buffers (code 22) at dr...@wp.pl util.c(121) [receiver=2.6.9] dr...@wp.pl cd'ing locally to that directory is fine. dr...@wp.pl -- dr...@wp.pl Robert Milkowski dr...@wp.pl http://milek.blogspot.com -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync --link-dest option with the destination directory containing old files.
Way back: -- Forwarded message -- Date: Mon, 2 Jun 2003 09:56:50 -0700 (PDT) From: Michael Rubel mrubel at galcit.caltech.edu To: rsync at lists.samba.org Subject: --link-dest when target and compare_dir both have file Hi J.W. et al, Kevin Everets was kind enough to inform me about some strange behavior in his backup script, which seems to be the result of --link-dest behaving unexpectedly in the case where target/ is already populated with older versions of the same file. Here's the situation: We want to do: $ rsync -a --link-dest=../backup.1 source/ backup.0/ There is a file present under all three directories. Suppose that it the version in backup.1/ is identical to the version in source/, and that backup.0/ contains an older version. In this case, I would expect: 1. the version in backup.0 should be unlinked 2. a new hard link should be created in backup.0/ to the copy in backup.1/ In fact, rsync (at least as of 2.5.6) seems to copy the full file: --link-dest is built on --compare-dest and behaves like --compare-dest as indicated in the manpage. Observe the description of --compare-dest (ALLCAPS added for emphasis): This option instructs rsync to use DIR on the des?? tination machine as an additional directory to com?? pare destination files against when doing transfers IF THE FILES ARE MISSING IN THE DESTINATION DIREC?? TORY. In other words, the files in the --link-dest location will only be used if there is no existing file in the destination. The best way to use --link-dest is to have an empty destination. For a link-dest based rotating backup that reuses directory names the best bet is to do a rm -r $dest; mkdir $dest or to rsync -a --link-dest=../backup.1 $source temp; mv temp backup.0 -- J.W. SchultzPegasystems Technologies email address: jw at pegasys.ws rsync folks, Can this behaviour be circumvented, or can an option be provided to change this behaviour? When doing precisely the type of backups listed earlier in the post: rsync -a --link-dest=../backup.1 source/ backup.0/ then we would like rsync to use a hard-link from ../backup.1 in preference to a new copy from source/ to backup.0/ . In particular, the removal suggested can be a very slow operation - we have a backup area containing about 4.5M inodes. Currently, a removal is the only way we have to stop the hard-links from leaking when we re-use directories. Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell CSIRO Advanced Scientific Computing, Technical Services Manager Chief Technology Officer, Bureau of Meteorology / CSIRO High Performance Computing and Communications Centre (HPCCC) Street: HPCCC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: HPCCC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia Phone +61 3 9669 8102, fax +61 3 9669 8112, mobile 0428 108 333, CSIRO 93 3810 http://www.hpsc.csiro.au/ http://www.hpccc.gov.au/ -- Forwarded message -- Date: Mon, 22 Dec 2008 12:00:32 + From: rsync-requ...@lists.samba.org Reply-To: rsync@lists.samba.org To: rsync@lists.samba.org Subject: rsync Digest, Vol 72, Issue 20 Resent-Date: Mon, 22 Dec 2008 23:00:48 +1100 (EST) Resent-From: robert.b...@csiro.au Send rsync mailing list submissions to rsync@lists.samba.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.samba.org/mailman/listinfo/rsync or, via email, send a message with subject or body 'help' to rsync-requ...@lists.samba.org You can reach the person managing the list at rsync-ow...@lists.samba.org When replying, please edit your Subject line so it is more specific than Re: Contents of rsync digest...-- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync --link-dest option with the destination directory containing old files.
Wayne, Thanks for the prompt response. The issue is around performance. We have a backup area containing 4.5M inodes in each backup, of which around 0.5M to 1M are directories. Backing up to a new directory each time is far slower than backing up to an almost completely filled directory. Deleting a directory with 4.5M inodes is non-trivial too. (We are running on an SGI Altix 4700 with 128 cores, 500 Gbyte of memory, with XFS file systems running the Data Migration Facility product. The target file systems for holding backups are on cheap and slow SATA disc - we see at best 10,000 inode deletions per second, and this can drop to fewer than 100 per second when the file systems are busy with just a few housekeeping tasks.). Starting with a new directory structure each time, so building 4.5M inodes, and then destroying a similar directory and file structure which is almost correct is very wasteful. We do have a workaround, which keeps the hard-links numbers at nearly their maximum level, but this involves a scan of the directory prior to receiving each rsync and hence a performance penalty. We would rather avoid this. It seems to us that there is a good case for the enhanced functionality. When a file is found in the destination which should be replaced by one in the source, look in the --link-dest directory first for a candidate, and hard-link that in preference to doing a copy from source to destination. Thanks Regards Rob. Bell e-mail: robert.b...@csiro.au -- Dr Robert C. Bell CSIRO Advanced Scientific Computing, Technical Services Manager Chief Technology Officer, Bureau of Meteorology / CSIRO High Performance Computing and Communications Centre (HPCCC) Street: HPCCC Level 11, 700 Collins Street, Docklands Vic 3008, Australia Postal: HPCCC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia Phone +61 3 9669 8102, fax +61 3 9669 8112, mobile 0428 108 333, CSIRO 93 3810 http://www.hpsc.csiro.au/ http://www.hpccc.gov.au/ On Mon, 22 Dec 2008, Wayne Davison wrote: Date: Mon, 22 Dec 2008 20:56:09 -0800 From: Wayne Davison way...@samba.org To: Robert Bell robert.b...@csiro.au Cc: rsync@lists.samba.org Subject: Re: rsync --link-dest option with the destination directory containing old files. Resent-Date: Tue, 23 Dec 2008 15:56:17 +1100 (EST) Resent-From: robert.b...@csiro.au On Tue, Dec 23, 2008 at 10:22:42AM +1100, Robert Bell wrote: we would like rsync to use a hard-link from ../backup.1 in preference to a new copy from source/ to backup.0/ . That's not something that I plan to implement. I'd suggest changing your naming to always use a new directory, which will allow you to remove the oldest backup directory at your leisure. ..wayne.. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync Log change Protocol
Hi, I am an German so sorry for my bad Engrish ^^ I need to know what : .d..t cd+++ gt;f+++ gt;f.st .f...p... gt;f..t and so on in the Log describe. Is there a Manual, Forum or a FAQ? Robert Obremski -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync hang at end
Hi All, My wrapper application for rsync 3.0.2 on osx has been working smoothly except for reports of it hanging on the last file of the transfer to a local firewire drive. We seem to have solved the mystery of the slow transfers but now, every four or five runs, it hangs at the last file to transfer and refuses to quit. Activity Monitor shows three rsync processes all at 0%CPU at that point (and virtual memory allocated of 3.74GB, 2.6GB and 2.6GB!). The last file incidentally is / usr/standalone/bootx.xcoff My friend is using the standard options -aHAXN --fileflags --force-change --stats -v --progress on an Intel mac copying his system of 100GB, or 1 million files, to the external drive. As I said, it works well and always copies all of the files perfectly. When the hang occurs, the output shows nothing unusual - it just stops at the last file transferred and does not proceed to shut itself off. We havn't managed to run with -vvv when this happens but will try to catch it with that now. I should also mention that rsync is launched via do shell script and the output sent to a log file in /tmp which the app reads progress from... I googled around and found references to an rsync end hang that said the child process wasn't getting the quit signal from parent rsync (on rsync 2.5.5) Don't know if that is relevant at all... Any thoughts, anyone? thanks so much for this great tool! Rob D -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync hang at end
Hi larry, This rsync was compiled on an intel running 10.4.11 so the architecture is right. I am wondering though if it would make a difference to compile it on the 10.5.2 machine it seems to hang on I have assumed all along that compiling it on the previous OS would be upward compatible. Thanks, rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
large backups taking longer with 3.0.2
Hi All, I havn't compiled 3.0.3 pre1 yet but have been seeing considerable longer backup times on OSX 5.2, using 3.0.2 over 3.0.1. Has anyone else noticed this? The progress doesn't seem to show any problems or differences from 3.0.1 either. backing up 100+GB internal drive, with 1 million files, takes over 5 hours compared to 4 hours or less on 3.0.1. I can't seem to find any reason for this. Using options: -aHAXN --progress. Thanks, Rob -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
osx custom icons
Hi All, This is a simple question but I have noticed that using rsync 3.0.2 on a local copy on OSX with basic options rsync -aHAXN --fileflags --force-change --stats -v --progress the custom file/folder icons do not always show up in destination. Sometimes they do and sometimes they don't. This doesn't bother much me but others have complained. Any thoughts appreciated. Thanks, Rob D -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
