Re: [gentoo-user] How to exclude a directory from rsync
On Fri, 26 Nov 2010 09:02:49 +0100 Helmut Jarausch jarau...@igpm.rwth-aachen.de wrote: On 11/25/10 22:51:36, Renat Golubchyk wrote: The man page is actually pretty clear on this issue. Quote: if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus /foo would match a name of foo at either the root of the transfer (for a global rule) or in the merge-file’s directory (for a per-directory rule). An unqualified foo would match a name of foo anywhere in the tree because the algorithm is applied recursively from the top down; it behaves as if each path component gets a turn at being the end of the filename. Even the unanchored sub/foo would match at any point in the hierarchy where a foo was found within a directory named sub. Root of the transfer is the directory you want to sync. Thus, if you run e.g. rsync /var/log/ /mnt/backups/ --exclude=/portage/ then root of the transfer is /var/log, and therefore the directory /var/log/portage will be excluded. If on the other hand you write --exclude=portage/ then a directory named portage anywhere in the tree under /var/log will be excluded. Without the trailing slash, i.e. just --exclude=portage any file (regular file, directory, link, whatever) named portage anywhere in the tree gets excluded. And finally --exclude=/portage would exclude a file only at the top of the tree that is going to be synchronsed. Let me add some caveat which has trapped me recently. I had (in your terms) rsync /var/log/ /mnt/backups/ --delete --exclude=/portage/ and /var/log/portage was just a symlink to some other directory while /mnt/backups/portage was a real directory. In that case rsync deletes /mnt/backups/portage ! It looks as if the directory property is check in the source tree only. Quite an unpleasant surprise. The behavior is logical and consistent with the command you provided. If you tell rsync to synchronise two directories and delete everything in the destination that is not in the source then rsync does just that. And a symbolic link is not a directory. One might think that exclude rule applies to both source and destination, but it doesn't. That's a common pitfall. But the documentation is clear on that too. It says, This option allows you to add rules to selectively exclude certain files from the list of files to be transferred. Since destination directory is not going to be transferred it is clear that this option doesn't apply there. In order to avoid the problem of files being accidentally deleted I just never run rsync without running it with --dry-run first if I specify a --delete option. Thus I have an opportunity to review the file list before anything unpleasant is done. Cheers, Renat -- Probleme kann man niemals mit derselben Denkweise loesen, durch die sie entstanden sind. (Einstein) signature.asc Description: PGP signature
Re: [gentoo-user] How to exclude a directory from rsync
On 11/25/10 22:51:36, Renat Golubchyk wrote: On Tue, 16 Nov 2010 23:01:51 + Mick michaelkintz...@gmail.com wrote: On Tuesday 16 November 2010 22:26:28 Stefan G. Weichinger wrote: Am 2010-11-16 22:24, schrieb Alan McKinnon: Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) ;-) I would like to know if my suggestion also works ;-) Yeah, include/exclude-patterns are rather hard to figure out sometimes ... nearly like regexes - write once, read never Ha, ha! True! Stefan, I tried escaping the spaces (even tried \\ double and \\\ triple escapes in case it makes a difference because of using ssh) but still did not work. In my head I couldn't see how the full path would not work, but the relative path would, but I tried it out all the same. I still don't understand why Alan's recommendation works ;-) I'm probably late with my reply, but I'll post it so it will be in the archives for future reference. The man page is actually pretty clear on this issue. Quote: if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus /foo would match a name of foo at either the root of the transfer (for a global rule) or in the merge-file’s directory (for a per-directory rule). An unqualified foo would match a name of foo anywhere in the tree because the algorithm is applied recursively from the top down; it behaves as if each path component gets a turn at being the end of the filename. Even the unanchored sub/foo would match at any point in the hierarchy where a foo was found within a directory named sub. Root of the transfer is the directory you want to sync. Thus, if you run e.g. rsync /var/log/ /mnt/backups/ --exclude=/portage/ then root of the transfer is /var/log, and therefore the directory /var/log/portage will be excluded. If on the other hand you write --exclude=portage/ then a directory named portage anywhere in the tree under /var/log will be excluded. Without the trailing slash, i.e. just --exclude=portage any file (regular file, directory, link, whatever) named portage anywhere in the tree gets excluded. And finally --exclude=/portage would exclude a file only at the top of the tree that is going to be synchronsed. Let me add some caveat which has trapped me recently. I had (in your terms) rsync /var/log/ /mnt/backups/ --delete --exclude=/portage/ and /var/log/portage was just a symlink to some other directory while /mnt/backups/portage was a real directory. In that case rsync deletes /mnt/backups/portage ! It looks as if the directory property is check in the source tree only. Quite an unpleasant surprise. Helmut.
Re: [gentoo-user] How to exclude a directory from rsync
On Thursday 25 November 2010 21:51:36 Renat Golubchyk wrote: On Tue, 16 Nov 2010 23:01:51 + Mick michaelkintz...@gmail.com wrote: On Tuesday 16 November 2010 22:26:28 Stefan G. Weichinger wrote: Am 2010-11-16 22:24, schrieb Alan McKinnon: Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) ;-) I would like to know if my suggestion also works ;-) Yeah, include/exclude-patterns are rather hard to figure out sometimes ... nearly like regexes - write once, read never Ha, ha! True! Stefan, I tried escaping the spaces (even tried \\ double and \\\ triple escapes in case it makes a difference because of using ssh) but still did not work. In my head I couldn't see how the full path would not work, but the relative path would, but I tried it out all the same. I still don't understand why Alan's recommendation works ;-) I'm probably late with my reply, but I'll post it so it will be in the archives for future reference. The man page is actually pretty clear on this issue. Quote: if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus /foo would match a name of foo at either the root of the transfer (for a global rule) or in the merge-file’s directory (for a per-directory rule). An unqualified foo would match a name of foo anywhere in the tree because the algorithm is applied recursively from the top down; it behaves as if each path component gets a turn at being the end of the filename. Even the unanchored sub/foo would match at any point in the hierarchy where a foo was found within a directory named sub. Root of the transfer is the directory you want to sync. Thus, if you run e.g. rsync /var/log/ /mnt/backups/ --exclude=/portage/ then root of the transfer is /var/log, and therefore the directory /var/log/portage will be excluded. If on the other hand you write --exclude=portage/ then a directory named portage anywhere in the tree under /var/log will be excluded. Without the trailing slash, i.e. just --exclude=portage any file (regular file, directory, link, whatever) named portage anywhere in the tree gets excluded. And finally --exclude=/portage would exclude a file only at the top of the tree that is going to be synchronsed. Hope it helps. Yes it does! Thank you, it's clear to me now why it behaved so. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] How to exclude a directory from rsync
On Tue, 16 Nov 2010 23:01:51 + Mick michaelkintz...@gmail.com wrote: On Tuesday 16 November 2010 22:26:28 Stefan G. Weichinger wrote: Am 2010-11-16 22:24, schrieb Alan McKinnon: Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) ;-) I would like to know if my suggestion also works ;-) Yeah, include/exclude-patterns are rather hard to figure out sometimes ... nearly like regexes - write once, read never Ha, ha! True! Stefan, I tried escaping the spaces (even tried \\ double and \\\ triple escapes in case it makes a difference because of using ssh) but still did not work. In my head I couldn't see how the full path would not work, but the relative path would, but I tried it out all the same. I still don't understand why Alan's recommendation works ;-) I'm probably late with my reply, but I'll post it so it will be in the archives for future reference. The man page is actually pretty clear on this issue. Quote: if the pattern starts with a / then it is anchored to a particular spot in the hierarchy of files, otherwise it is matched against the end of the pathname. This is similar to a leading ^ in regular expressions. Thus /foo would match a name of foo at either the root of the transfer (for a global rule) or in the merge-file’s directory (for a per-directory rule). An unqualified foo would match a name of foo anywhere in the tree because the algorithm is applied recursively from the top down; it behaves as if each path component gets a turn at being the end of the filename. Even the unanchored sub/foo would match at any point in the hierarchy where a foo was found within a directory named sub. Root of the transfer is the directory you want to sync. Thus, if you run e.g. rsync /var/log/ /mnt/backups/ --exclude=/portage/ then root of the transfer is /var/log, and therefore the directory /var/log/portage will be excluded. If on the other hand you write --exclude=portage/ then a directory named portage anywhere in the tree under /var/log will be excluded. Without the trailing slash, i.e. just --exclude=portage any file (regular file, directory, link, whatever) named portage anywhere in the tree gets excluded. And finally --exclude=/portage would exclude a file only at the top of the tree that is going to be synchronsed. Hope it helps. Cheers, Renat -- Probleme kann man niemals mit derselben Denkweise loesen, durch die sie entstanden sind. (Einstein) signature.asc Description: PGP signature
Re: [gentoo-user] How to exclude a directory from rsync
Apparently, though unproven, at 15:28 on Tuesday 16 November 2010, Mick did opine thusly: On 16 November 2010 09:00, Stefan G. Weichinger li...@xunil.at wrote: Am 15.11.2010 23:50, schrieb Mick: Thanks Stefan, I'm afraid I'm still getting the same problem: rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) Why is rsync trying to open this directory, when I thought I've asked it to exclude it? Maybe you did it wrong? ;-) You don't show us what you did ... I ran the same as before but changed the path to the one you suggested: === 'rsync -a -l -v --exclude ./System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' sending incremental file list rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) [snip ...] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] === The System Volume is shown as 0700 (mounted with default permissions) and is owned by root when viewed from Gentoo. Don't think of --exclude as being a file path match, think of it as more a regex (usually just a literal one). It specifies a pattern that if found if the full pathname, results in the file not being synced. The string ./System Volume Information of course appears nowhere in the rsync output list of files. I think you want: --exclude /System Volume Information Then rsync will not attempt to open the directory at all -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] How to exclude a directory from rsync
Am 16.11.2010 14:28, schrieb Mick: I ran the same as before but changed the path to the one you suggested: === 'rsync -a -l -v --exclude ./System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' Try it with escaped spaces as I mentioned also: --exclude ./System\ Volume\ Information or even --exclude ./System* if there is nothing else starting with System ;-) S
Re: [gentoo-user] How to exclude a directory from rsync
Am 15.11.2010 23:50, schrieb Mick: Thanks Stefan, I'm afraid I'm still getting the same problem: rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) Why is rsync trying to open this directory, when I thought I've asked it to exclude it? Maybe you did it wrong? ;-) You don't show us what you did ...
Re: [gentoo-user] How to exclude a directory from rsync
On 16 November 2010 09:00, Stefan G. Weichinger li...@xunil.at wrote: Am 15.11.2010 23:50, schrieb Mick: Thanks Stefan, I'm afraid I'm still getting the same problem: rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) Why is rsync trying to open this directory, when I thought I've asked it to exclude it? Maybe you did it wrong? ;-) You don't show us what you did ... I ran the same as before but changed the path to the one you suggested: === 'rsync -a -l -v --exclude ./System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' sending incremental file list rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) [snip ...] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] === The System Volume is shown as 0700 (mounted with default permissions) and is owned by root when viewed from Gentoo. -- Regards, Mick
Re: [gentoo-user] How to exclude a directory from rsync
On Tuesday 16 November 2010, Mick wrote: Try: 'rsync -a -l -v --exclude */System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' From man rsync Note that if you don’t specify --backup-dir, (1) the --omit-dir-times option will be implied, and (2) if --delete is also in effect (without --delete- excluded), rsync will add a protect filter-rule for the backup suffix to the end of all your existing excludes (e.g. -f P *~). This will prevent previously backed-up files from being deleted. Note that if you are supplying your own filter rules, you may need to manually insert your own exclude/protect rule somewhere higher up in the list so that it has a high enough priority to be effective (e.g., if your rules specify a trailing inclusion/exclusion of ’*’, the auto-added rule would never be reached). HTH -Robin -- -- Robin Atwood. Ship me somewheres east of Suez, where the best is like the worst, Where there ain't no Ten Commandments an' a man can raise a thirst from Mandalay by Rudyard Kipling --
Re: [gentoo-user] How to exclude a directory from rsync
On Tuesday 16 November 2010 14:15:00 Alan McKinnon wrote: Apparently, though unproven, at 15:28 on Tuesday 16 November 2010, Mick did opine thusly: On 16 November 2010 09:00, Stefan G. Weichinger li...@xunil.at wrote: Am 15.11.2010 23:50, schrieb Mick: You don't show us what you did ... I ran the same as before but changed the path to the one you suggested: === 'rsync -a -l -v --exclude ./System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' sending incremental file list rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) [snip ...] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] === The System Volume is shown as 0700 (mounted with default permissions) and is owned by root when viewed from Gentoo. Don't think of --exclude as being a file path match, think of it as more a regex (usually just a literal one). It specifies a pattern that if found if the full pathname, results in the file not being synced. The string ./System Volume Information of course appears nowhere in the rsync output list of files. I think you want: --exclude /System Volume Information Then rsync will not attempt to open the directory at all Excellent, it worked! :-) Thank you all for your help. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] How to exclude a directory from rsync
Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Don't think of --exclude as being a file path match, think of it as more a regex (usually just a literal one). It specifies a pattern that if found if the full pathname, results in the file not being synced. The string ./System Volume Information of course appears nowhere in the rsync output list of files. I think you want: --exclude /System Volume Information Then rsync will not attempt to open the directory at all Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) -- alan dot mckinnon at gmail dot com
Re: [gentoo-user] How to exclude a directory from rsync
On Tuesday 16 November 2010 22:26:28 Stefan G. Weichinger wrote: Am 2010-11-16 22:24, schrieb Alan McKinnon: Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) ;-) I would like to know if my suggestion also works ;-) Yeah, include/exclude-patterns are rather hard to figure out sometimes ... nearly like regexes - write once, read never Ha, ha! True! Stefan, I tried escaping the spaces (even tried \\ double and \\\ triple escapes in case it makes a difference because of using ssh) but still did not work. In my head I couldn't see how the full path would not work, but the relative path would, but I tried it out all the same. I still don't understand why Alan's recommendation works ;-) -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] How to exclude a directory from rsync
Am 2010-11-16 22:24, schrieb Alan McKinnon: Apparently, though unproven, at 23:12 on Tuesday 16 November 2010, Mick did opine thusly: Excellent, it worked! :-) Glad to hear it. I could help because part of my job is running a rather big public ftp mirror that management graciously pay for. And I went down this rsync road a long time ago myself. You have no idea how many brain cells died in agony to figure out this specific piece of rsync behaviour :-) ;-) I would like to know if my suggestion also works ;-) Yeah, include/exclude-patterns are rather hard to figure out sometimes ... nearly like regexes - write once, read never S
Re: [gentoo-user] How to exclude a directory from rsync
On Sunday 14 November 2010 22:47:02 you wrote: Am 14.11.2010 22:03, schrieb Mick: I am not sure how to exclude a directory on an ntfs partition from being accessed during rsync. The attributes do not seem to be right and it comes up with this error: === 'rsync -a -l -v --exclude /mnt/User_WinXP/System Volume Information try something like --exclude ./System Volume Information (relative paths) or even --exclude ./System\ Volume\ Information (escaping the spaces) Thanks Stefan, I'm afraid I'm still getting the same problem: rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) Why is rsync trying to open this directory, when I thought I've asked it to exclude it? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
[gentoo-user] How to exclude a directory from rsync
I am not sure how to exclude a directory on an ntfs partition from being accessed during rsync. The attributes do not seem to be right and it comes up with this error: === 'rsync -a -l -v --exclude /mnt/User_WinXP/System Volume Information -e ssh -c blowfish -l root /mnt/User_WinXP/ 10.10.10.25:/home/httpd/backup' sending incremental file list rsync: opendir /mnt/User_WinXP/System Volume Information failed: Permission denied (13) sent 975792 bytes received 3760 bytes 23048.28 bytes/sec total size is 1195724 speedup is 12207.28 rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1042) [sender=3.0.7] === Is my syntax wrong? This what mount shows for the partition (I use ntfs-3g): /dev/sda7 on /mnt/User_WinXP type fuseblk (rw,nosuid,nodev,noatime,allow_other,blksize=4096) -- Regards, Mick
Re: [gentoo-user] How to exclude a directory from rsync
Am 14.11.2010 22:03, schrieb Mick: I am not sure how to exclude a directory on an ntfs partition from being accessed during rsync. The attributes do not seem to be right and it comes up with this error: === 'rsync -a -l -v --exclude /mnt/User_WinXP/System Volume Information try something like --exclude ./System Volume Information (relative paths) or even --exclude ./System\ Volume\ Information (escaping the spaces) S
