Re: Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)
It is just the performance issue. However, this issue is unacceptable because of the number of files I am synching.. millions.. sending all of these resource forks is just asking for trouble... using RsyncX I only transfer a few thousand files each session but this error just transfers resource forks 24 hours a day.. and this is insane. I would gladly keep using RsyncX except it has a tendency to get Stuck on Stupid and worry over a missing file or some such nonsence rather than moving on. I have found another project to fix rsync, but it still suffers from the same problem but he/they have implemented some novel solutions to the problem. Still the binary at onthenet is easy to install and seems to work pretty well. http://www.onthenet.com.au/~q/rsync/ Q. wrote to me.. I know in your notes you say this should only occur when the last change time was more recent than the file's modification date, but the net effect is that every resource copies every sync. Is this what you are talking about when you are referencing getattrlist? This seems more like spotlight data.. but perhaps I am wrong. Unfortunately yes. When you use the unix stat() function call it will return an access time, a change time and a modification time. When you modify the extended attributes of a file it will changes the change time without touching the modification time. Unfortunately the unix function call for setting these times doesn't allow you to set the change time, and will instead set it to the current time, hence the problem. It appears that setattrlist() will allow the setting of a files change time on a HFS+ volume which would eliminate this issue, however it's syntax is somewhat complex, and I have not yet had a chance to work out how to use it as my powerbook is currently in for repairs. So it might be a little while before I have this problem fixed. --- Appearantly the RsyncX people uses Apple specific Carbon library calls to read and write data to accomplish not copying the resources.. David J.D. Bakker wrote: At 09:54 -0500 01-12-2005, David Powell wrote: ... ._resource forks being synced over and over despite the target file itself not moving.. Is this Problem #4 mentioned in http://www.lartmaker.nl/rsync/ , or something else ? Missing fact... I am attempting a link-destination based multiple backup that creates hard links to files that are duplicates and only moves the changed files.. This implementation of rsync is not noticing the existing resource fork associated with the previous copy of the file so it moves the resource fork again. I hope this makes sense. It would help to know if you're doing a local copy or a networked one. Could you show us the full rsync command used ? Are you experiencing data loss, or is this simply a performance issue ? Regards, JDB. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)
This patch seems to work nicely except for the problem that if you resync a directory that has been synced using this.. Every resource fork is resynced because OS 10.4 accepts the ._filename and stores it properly in the resource fork but the rsync doesn't notice this.. and decides the file is missing.. and tries to move it.. or conversely .. in some instances it is saying this file has vanished.. This all adds up to alot of wasted time.. and file pieces being moved.. with the resulting possibility of corrupting the whole. Otherwise.. way cool. This needs fixed immediately though. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)
... ._resource forks being synced over and over despite the target file itself not moving.. Missing fact... I am attempting a link-destination based multiple backup that creates hard links to files that are duplicates and only moves the changed files.. This implementation of rsync is not noticing the existing resource fork associated with the previous copy of the file so it moves the resource fork again. I hope this makes sense. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)
At 09:54 -0500 01-12-2005, David Powell wrote: ... ._resource forks being synced over and over despite the target file itself not moving.. Is this Problem #4 mentioned in http://www.lartmaker.nl/rsync/ , or something else ? Missing fact... I am attempting a link-destination based multiple backup that creates hard links to files that are duplicates and only moves the changed files.. This implementation of rsync is not noticing the existing resource fork associated with the previous copy of the file so it moves the resource fork again. I hope this makes sense. It would help to know if you're doing a local copy or a networked one. Could you show us the full rsync command used ? Are you experiencing data loss, or is this simply a performance issue ? Regards, JDB. -- LART. 250 MIPS under one Watt. Free hardware design files. http://www.lartmaker.nl/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Mini-HOWTO: Fixing rsync on Tiger (Mac OS X 10.4.x)
My solution is to use a separate utility called SplitForks on OSX, then a unpatched rsync is used on a Mac to transfer files to a linux box for backup. SplitForks is a tool in OSX's Developer toolkit, which will make those ._xyz files on HFS+ volume. I developed a shell script to use SplitForks on a volume, rsync it, then a find command to delete those ._xyz files, coz, I suspect, those ._xyz resource forks will not be updated by OSX automatically on HFS+ volumes. OSX might see the (internal) resource fork in HFS+ file, and use it. So, I run the SplitForks routine every backup cycle to regenerate a fresh copy. My 2 cents. Stephen Wong @ Hong Kong From J.D. Bakker: [Now the horse has left the barn, I decided to finally implement that backup system I'd been thinking about for ages. Disk crashes can be great motivators] [Web-location for the living version of this document: http://www.lartmaker.nl/rsync/] Goal: Producing a working network backup / cloning system for Mac OS X systems. The system can be used for local backups as well, for example to FireWire disks. Problems: Many files on HFS+, the Mac's most common file system, have metadata. This is partly a leftover from the past (resource forks), and partly a new development (ACLs, extended attributes). Plain rsync doesn't (yet) cope with this metadata. Since OS X 10.4 (aka Tiger) the MacOS ships with a modified version of rsync. An added option, '-E', enables the transfer of extended attributes. This is done by encapsulating the resource fork, Finder data et al in a synthetic file which is added to the rsync transfer list. The name of this file is formed by prepending '._' to the name of the original file, a technique which is also used when copying data from HFS+ partitions to non-Apple file systems such as NFS mounts. It may not be pretty or foolproof (what happens when both foo and ._foo exist?), but at least it's documented by Apple and lot likely to change in the very near future. This rsync derivative is based on rsync-2.6.3. However, Googling and testing have revealed four problems with Apple's rsync. In order of severity, worst first: 1) The rsync sender will frequently crash with a Bus Error / Segmentation Fault after generating the file list, but before transferring any files. This turns out to be caused by a buffer overrun. 2) When used with the --delete option, the rsync receiver will try to unlink the (fake) synthetic files, flooding the syslog with failure reports, possibly filling the entire boot disk. 3) When files with extended attributes are transferred, the modification time will be set to the time of the transfer, even when the user has specified that modification times be preserved. As a result, using mtime to determine whether a file has changed is broken. 4) Extended attributes have no modification time of themselves. Since a file's mtime is not updated when its attributes are changed, only checksumming can be used to determine whether attribute data needs to be transferred. With default settings, this means that ALL extended attributes are ALWAYS copied. The patch: Problems 1-3 are fixed by the patch at http://www.lartmaker.nl/rsync/rsync-tiger-fixes.diff . This patch is released under version 2 of the GNU GPL. I know of no fix for problem 4, but consider it mostly an annoyance. Putting it all together: NOTE: This requires familiarity with the Terminal. I have no .dmg or whatnot, since I wouldn't know how to create one (and there are licensing issues, see below). Following these steps should get you a working rsync, though. 0) Update Tiger to 10.4.2. Install XCode, the Apple developer tools. If you don't have the disc (it's shipped with the Tiger install media), you can get the latest version from Apple's developer website (free registration required): http://developer.apple.com/tools/download/ 1) Get the sources. Open the terminal, and type: mkdir rsync-build cd rsync-build curl -O http://www.opensource.apple.com/darwinsource/10.4.3/rsync-20/rsync-2.6.3.tar.gz curl -O http://www.opensource.apple.com/darwinsource/10.4.3/rsync-20/patches/EA.diff curl -O http://www.opensource.apple.com/darwinsource/10.4.3/rsync-20/patches/PR-3945747-endian.diff curl -O http://www.lartmaker.nl/rsync/rsync-tiger-fixes.diff 2) If you don't already have it, install copyfile.h in /usr/include . Get it from Apple's developer website http://www.opensource.apple.com/darwinsource/10.4.3/Libc-391.2.3/darwin/copyfile.h (again, free registration required). In the Terminal: sudo mv -n copyfile.h /usr/include Copying to /usr/include requires root privileges; enter your password when prompted. The '-n' option to mv makes sure that you don't overwrite a (newer) installed version. NOTE: copyfile.h is *NOT* licensed under the GPL, but rather under the Apple Public Source Licence (http://www.opensource.apple.com/apsl).
