--include vs. --exclude
It guess I still haven't figured out the entire sematics of the --include
and --exclude options. From reading the man page, it seems to say that
what happens is that each file being checked is tested against each pattern
in order, and when one matches the tests end, and whether it is --include
or --exclude determines if that file is included or excluded.
So I have on my server a big file tree. I want to use rsync to download
only the PDF files, which make up a small portion of that tree. So I try
it this way:
rsync -aHPvz --include '*.pdf' --exclude '**' [EMAIL PROTECTED]:
which gives me nothing. For reference, I try:
rsync -aHPvz --include '*.pdf' [EMAIL PROTECTED]:
which starts downloading other files. That confirms that the default final
action is equivalent to --include '**' or something like that.
So it seems the include pattern isn't matching. So I try variations:
rsync -aHPvz --include '*.pdf' --exclude '*' [EMAIL PROTECTED]:
rsync -aHPvz --include '**.pdf' --exclude '**' [EMAIL PROTECTED]:
rsync -aHPvz --include '**.pdf' --exclude '*' [EMAIL PROTECTED]:
rsync -aHPvz --include '**/*.pdf' --exclude '**' [EMAIL PROTECTED]:
rsync -aHPvz --include '**/*.pdf' --exclude '*' [EMAIL PROTECTED]:
rsync -aHPvz --include '/**.pdf' --exclude '**' [EMAIL PROTECTED]:
rsync -aHPvz --include '/**.pdf' --exclude '*' [EMAIL PROTECTED]:
rsync -aHPvz --include '/**/*.pdf' --exclude '**' [EMAIL PROTECTED]:
rsync -aHPvz --include '/**/*.pdf' --exclude '*' [EMAIL PROTECTED]:
None of these work.
So finally, I replicate the file tree on the server with:
cp -al
And proceed to remove all non-PDF files:
find -type f ! -name '*.pdf' -exec rm -f {} ';'
Then I do:
rsync -aHPvz --include '*.pdf' --exclude '**' [EMAIL PROTECTED]:
which now works.
Can rsync do this by itself? Is there a way to tell rsync "only download
this particular extension"? How SHOULD I have done this?
I generally understand things best by knowing what sequence of steps is
performed. I thought I understood this for rsync based on what the man
page said. I guess one of us is wrong.
I'm running: rsync version 2.6.0 protocol version 27
--
-
| Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ |
| (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ |
-
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude confusion: what does it match against?
On Fri, 26 Mar 2004, Wayne Davison <[EMAIL PROTECTED]> wrote: > On Thu, Mar 25, 2004 at 11:03:31PM -0500, Andrew Pimlott wrote: >> The exact filename tested against include/exclude patterns is not >> always obvious, and the documentation is somewhat ambiguous about it. >> This bites particularly when using patterns starting with '/'. > > Yes, this could certainly use some improvement to make things clearer. > I've never liked the description of the patterns being relative to the > destination directory, though, since the patterns also affect files on > the source side. I prefer the phrase "root of the transfer", which > would have to be defined in the documentation. > > What about something like this? > > The filenames matched against the exclude/include patterns are > relative to the "root of the transfer". If you think of the > transfer as a subtree of names that are being sent from sender > to receiver, the root is where the tree starts to be duplicated > in the destination directory. This root governs where patterns > that start with a / match (see below). > > Because the matching is relative to the transfer root, changing > the trailing slash on the source path or changing your use of > the --relative option affects the names you need to use in your > matching. For example, using these rsync commands: > > rsync -a /home/me /dest > rsync -a /home/me/ /dest > rsync -a --relative /home/me /dest > > the file /home/me/foo/bar would be matched using, respectively: > > /me/foo/bar > /foo/bar > /home/me/foo/bar > > The easiest way to see what name you should include/exclude is > to just look at the output when using --verbose and put a / in > front of the name (use the --dry-run option if you're not yet > ready to copy any files). This is another opportunity to (try to) make more clear the effects of using a trailing slash on the source and using the --relative option. They not only affect the absolute-path pattern but also where the source file ends up on the target. I'd like to see something like this included in the explanation. (And I'm not even 100% sure these details are correct...) -- The following examples demonstrate the effect that various invocations of rsync have on the Include and Exclude pattern needed to match a source filename of "/home/me/foo/bar", and also show the resulting file path on the target: Example cmd: rsync -a /home/me /dest Source root: /home (NOT /home/me) Target root: /dest I/E pattern: /me/foo/bar Source file: /home/me/foo/bar Target file: /dest/me/foo/bar Example cmd: rsync -a /home/me/ /dest Source root: /home/me (due to trailing /) Target root: /dest I/E pattern: /foo/bar (note missing 'me') Source file: /home/me/foo/bar Target file: /dest/foo/bar (note missing 'me') Example cmd: rsync -a --relative /home/me /dest Source root: /home Target root: /dest I/E pattern: /home/me/foo/bar (note full path) Source file: /home/me/foo/bar Target file: /dest/home/me/foo/bar (note full path) -- John Van Essen Univ of MN Alumnus <[EMAIL PROTECTED]> -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync doesn't log properly when in standalone daemon mode
On Fri, Mar 26, 2004 at 01:32:14PM -0800, Joseph Tam wrote: > I was under the impression that "[global]" sets up the defaults, which > all the other modules inherit, if they don't specify something else. The global defaults are those that occur outside all module sections. Just to be clear, I mean something like this: uid = nobody gid = nogroup use chroot = yes max connections = 4 syslog facility = user pid file = /var/run/rsyncd.pid refuse options = compress delete checksum link-dest backup-dir read only = yes [foo] path = /some/path/foo comment = foo area read only = no [bar] path = /some/path/bar comment = bar area ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync doesn't log properly when in standalone daemon mode
On Fri, 26 Mar 2004, Wayne Davison wrote: > On Fri, Mar 26, 2004 at 01:17:27AM -0800, Joseph Tam wrote: > > [global] > > transfer logging= false > > syslog facility = uucp > > If you get rid of the [global] heading and just put those items at the > start of the file, it should work: > > [...] > transfer logging= false > syslog facility = uucp > [...] > > Your config file was specifying a module named "global", and those > settings didn't affect other modules. Thanks, Wayne. I'll try this out. I was under the impression that "[global]" sets up the defaults, which all the other modules inherit, if they don't specify something else. That's why my "[global]" section is restrictive in terms of access -- individual modules then turn access back on. A sort of default deny stance. Now that you tell me otherwise, are there any other parameters I should take out of the "[global]" section and into indiv. sections? > > On another tangential topic, why does logging on the server get controlled > > by the client? I'd like to have the client determine the verboseness of > > their rsync output without having it also filling up server log files with > > useless details. > > Improving this is on the list of future enhancements/fixes. The current > code caps the user-supplied verboseness at 1 (so you can't get - > amounts of logging at the user's whim), but it would be better to have > the server-side logging totally under the control of the config file. I'll look forward to that. I couldn't configure the behaviour I wanted out of "refuse options = verbose" since it adversely affects the client. Once again, thanks for your help. Joseph Tam <[EMAIL PROTECTED]> -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsync/cwsync dir spaces on the path
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi: First let me say that rsync/cwrsync is amazing ! And sorry ! if I'm bothering you with this question... I need to sync my work files from the WinXP Pro machine to our Solaris one... so far so fine.. My problem is when I specify the path on the rsyncd.conf (cwrsync) I have tried many combinations from: /cygdrive/c/Documents*/rbadilla/My*/clients /cygdrive/c/Documents*/rbadilla/ "/cygdrive/c/Documents and Settings/rbadilla" /cygdrive/c/Documents\ and\ Settings /cygdrive/c/Documents*tings /cygdrive/c/"Documents and Settings/rbadilla" Until now I always get the " chdir /cygdrive/c/XXX failed : No such file or directory " Can you enlight me??? Thanks a lot! -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.1 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQD1AwUBQGSSkULyKFRWGHddAQKRUwcApYjlGxv48l11BYQAYIM6x5v3GsEyH+9q EZXCs6GkeOLb0+7+Sx5fKSTS7l+nrogM/B+csJXHebuZO+xxoBfrQTHr45Z0qt/V Sv6v1vXGYT+fEYE3LOO9XeM26kwnAUoDvvekNcT7KYC3VOeNoFPOF4AOVX/1Td8V F3hKc5ZdA0m2El2xpPC2Z9U0nabGUEc3AJtibLduYYK+v4dVMzfFjWJ9ssiKr0ID E8fJHadc11LIK8cDBqum39bq+7+VpEcyyK0hc+YiPZfybkZbH/aKlh93LBgOQsxr 25o0pZ2a++I= =QZJv -END PGP SIGNATURE- begin:vcard fn:Randall Badilla n:Badilla;Randall org:Control Electronico S.A. (CESA);Soporte y Servicio adr:Norteamericano, Barrio Dent;;100 m. norte, 75 m. este del Centro Cultural Costarricense;San Jose;San Jose;;Costa Rica email;internet:[EMAIL PROTECTED] tel;work:+506-224- tel;fax:+506-253-5450 x-mozilla-html:FALSE url:http://www.cesa.co.cr version:2.1 end:vcard -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: exclude confusion: what does it match against?
On Thu, Mar 25, 2004 at 11:03:31PM -0500, Andrew Pimlott wrote: > The exact filename tested against include/exclude patterns is not > always obvious, and the documentation is somewhat ambiguous about it. > This bites particularly when using patterns starting with '/'. Yes, this could certainly use some improvement to make things clearer. I've never liked the description of the patterns being relative to the destination directory, though, since the patterns also affect files on the source side. I prefer the phrase "root of the transfer", which would have to be defined in the documentation. What about something like this? The filenames matched against the exclude/include patterns are relative to the "root of the transfer". If you think of the transfer as a subtree of names that are being sent from sender to receiver, the root is where the tree starts to be duplicated in the destination directory. This root governs where patterns that start with a / match (see below). Because the matching is relative to the transfer root, changing the trailing slash on the source path or changing your use of the --relative option affects the names you need to use in your matching. For example, using these rsync commands: rsync -a /home/me /dest rsync -a /home/me/ /dest rsync -a --relative /home/me /dest the file /home/me/foo/bar would be matched using, respectively: /me/foo/bar /foo/bar /home/me/foo/bar The easiest way to see what name you should include/exclude is to just look at the output when using --verbose and put a / in front of the name (use the --dry-run option if you're not yet ready to copy any files). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Root access over ssh?
On Fri, Mar 26, 2004 at 02:30:48PM -0500, Paul Galbraith wrote: > My impression is that when run as a server over a remote ssh > connection, rsync is being started up as a process under the ssh user's > account, instead of connecting to the root rsync server that I've got > running out of inetd. That's correct. If you want to connect to a pre-running server, you need to use the :: syntax without a remote shell. Check out the tunnel support in ssh or in software such as stunnel for how to tunnel a secure connection from one system to another. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Root access over ssh?
Tim Conway wrote: You have to have a "uid = 0" in the rsyncd.conf file for that module, since only root can give away files, and also bypass all (except over NFS, etc.) file protections. For (insert diety's name here)'s sake, don't use "/" for a path, and don't leave it un-chrooted, unless you're putting up a honeypot or something. You might want to password-protect that module, too, to inhibit casual unwanted file modification. I know it sounds sick. I've got a remote box that I want to back up /etc, /var, and /home. I wanted to use rsync to do this because my current method (tar, encrypt with openssl and leave on the ftp server) is taking too long over the remote connection. I was trying to run rsync as a server on the remote box, but with port 873 blocked by a packet filter, so that ssh was the only way to get remote access to rsync. I DO have a root module, protected by an rsync secret, but I share your discomfort...I just can't think of a better solution. At any rate, when I try to run rsync from my workstation across the 'net, rsync complains that it can't find rsyncd.conf (which is in /etc). My impression is that when run as a server over a remote ssh connection, rsync is being started up as a process under the ssh user's account, instead of connecting to the root rsync server that I've got running out of inetd. Am I missing something? you might as well also add "opendoorstream tcp nowait root /bin/sh sh" to your inetd.conf and "opendoor666" to services. Hmmm, thanks for the advice, I tried this, but my rsync still isn't working. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync doesn't log properly when in standalone daemon mode
On Fri, Mar 26, 2004 at 01:17:27AM -0800, Joseph Tam wrote: > [global] > transfer logging= false > syslog facility = uucp If you get rid of the [global] heading and just put those items at the start of the file, it should work: [...] transfer logging= false syslog facility = uucp [...] Your config file was specifying a module named "global", and those settings didn't affect other modules. > On another tangential topic, why does logging on the server get controlled > by the client? I'd like to have the client determine the verboseness of > their rsync output without having it also filling up server log files with > useless details. Improving this is on the list of future enhancements/fixes. The current code caps the user-supplied verboseness at 1 (so you can't get - amounts of logging at the user's whim), but it would be better to have the server-side logging totally under the control of the config file. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: --link-dest mostly useless on a server without chroot
On Fri, Mar 26, 2004 at 12:09:08AM -0500, Andrew Pimlott wrote: > When using the rsync server without chroot, the --link-dest and > --compare-dest options are almost useless. This is because > sanitize_paths is called on the link-dest directory, stripping out > leading dots and slashes. Yes, those options should really replace "/" with the root dir of the module when chroot isn't specified (rather than just stripping the slash off). ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: 2.6.1pre-1 Segmentation fault & other fun
On Fri, Mar 26, 2004 at 10:03:54AM -0500, Erik Jan Tromp wrote:
> $ rsync -aHPv rsync://host/module/single.file .
>
> receiving file list ...
> 1 file to consider
> Segmentation fault
Thanks for the report! I've fixed this bug in CVS as follows:
--- hlink.c 11 Feb 2004 05:03:04 - 1.38
+++ hlink.c 26 Mar 2004 16:41:28 -
@@ -135,7 +135,7 @@
int hard_link_check(struct file_struct *file, int skip)
{
- if (!file->link_u.links)
+ if (!hlink_list || !file->link_u.links)
return 0;
if (skip && !(file->flags & FLAG_HLINK_EOL))
hlink_list[file->F_HLINDEX] = file->F_NEXT;
..wayne..
--
To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: write only in rsyncd.conf
On Fri, Mar 26, 2004 at 12:45:43AM -0500, Andrew Pimlott wrote: > Is there any interest in reviving this patch? The diff has been in the "patches" dir of CVS for a while now, and is being considered for inclusion in a future release. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fwd: RSync Progress Report Request
On Thu, Mar 25, 2004 at 10:21:39PM -0500, Jim Salter wrote: > Whether you did the progress report as "file [x] of [y]" or "megabyte > [x] of [y]" or whatever would barely even matter; Please try 2.6.1pre-1. From the NEWS file (taken from CVS, which was just improved to make this entry better): * The --progress output now mentions how far along in the transfer we are, including both a count of files transferred and a percentage of the total file-count that we've processed. It also shows better current-rate-of-transfer and remaining- transfer-time values. ..wayne.. -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
2.6.1pre-1 Segmentation fault & other fun
Oops :) -- quote -- $ rsync -aHPv rsync://host/module/single.file . receiving file list ... 1 file to consider Segmentation fault -- endquote -- The rsync process remains running until explicitly 'kill'ed. At which point, the following is printed: -- quote -- rsync error: received SIGUSR1 or SIGINT (code 20) at rsync.c(225) -- endquote -- -- quote -- $ rsync -aHPv /path/to/single.file . building file list ... done rsync: connection unexpectedly closed (8 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(342) -- endquote -- rsync 2.6.0 transfers the file with no errors when called as above. I realize it's silly to attempt preserving hardlinks when transferring a single file, btw. This behaviour manifested when I was testing some scripts using '-aHPv' as the default options. Test machines: slackware 8.1 & 9.1, stock build chain for each. (I hope I gave enough info.. not exactly clear headed at the moment) -- Registered Linux User Number 140066 http://counter.li.org/cgi-bin/runscript/display-person.cgi?user=140066 "Failure is not an option. (It comes bundled with Windows.)" "If it ain't broke, overclock it." -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to getting clean list of transferred files
On Fri, 2004-03-26 14:38:22 +0100, Markus Marquardt <[EMAIL PROTECTED]> wrote in message <[EMAIL PROTECTED]>: > Now rsync sometimes generates messages like "file has vanished: ..." on > STDOUT (_not_ STDERR) which breaks this processing. Sounds at least surprising. Just prepare a patch to output this on stderr if your scripts break. > Is there any way to get a "clean" list of the transferred files (without > doing things like grep or sed on the output stream)? Well, "grep -v" would IMHO be the "natural" way to clean the output. Long-term solution is more like patching rsync to use stderr, though... MfG, JBG -- Jan-Benedict Glaw [EMAIL PROTECTED]. +49-172-7608481 "Eine Freie Meinung in einem Freien Kopf| Gegen Zensur | Gegen Krieg fuer einen Freien Staat voll Freier Bürger" | im Internet! | im Irak! ret = do_actions((curr | FREE_SPEECH) & ~(NEW_COPYRIGHT_LAW | DRM | TCPA)); signature.asc Description: Digital signature -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
How to getting clean list of transferred files
Hello, i am using currently rsync-2.6.0 with option --log-format=%f to produce a list of the transferred files. This list is then used to generate hard-links for backup-purposes. Now rsync sometimes generates messages like "file has vanished: ..." on STDOUT (_not_ STDERR) which breaks this processing. When using option -q there is no output at all. Is there any way to get a "clean" list of the transferred files (without doing things like grep or sed on the output stream)? Regards, Markus -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync doesn't log properly when in standalone daemon mode
I'm trying to get over an exasperating problem with rsync 2.5.7, but I'm not making any headway. The nut of the problem is that the rsync daemon seemingly ignores the "syslog facility" configuration when running in standalone daemon mode, as opposed to being started with inetd. So, for example, if I have my configuration file with [global] pid file= /etc/rsync/rsyncd.pid lock file = /etc/rsync/rsyncd.lock use chroot = true max connections = 1 hosts deny = 0.0.0.0/0 read only = true list= false uid = nobody gid = nobody secrets file= /etc/rsync/secrets ignore errors = true transfer logging= false syslog facility = uucp # log file= /tmp/rsyncd.log timeout = 30 [archive] ... blah blah blah ... I get log entries in the uucp log files when running out of inetd. But running as a standalone daemon with the same config file, log entries get dumped into the default syslog facility: daemon. I've tried fiddling with file permissions on the configuration file and these configuration variables, use chroot = false transfer logging = true syslog facility = different facilities and even logging to a file, but it still logs to the daemon syslog. Is there some subtlety I'm missing here? Any light shed here would be appreciated. Othwerwise I guess I patch the source. On another tangential topic, why does logging on the server get controlled by the client? I'd like to have the client determine the verboseness of their rsync output without having it also filling up server log files with useless details. Joseph Tam <[EMAIL PROTECTED]> -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
