Re: Rsync over NFS mount sending whole files
On Thu, Oct 27, 2005 at 07:27:09PM +0200, Anban Mestry wrote: > rsync -avtz --no-whole-file \test1\ \mnt\test2\ You don't want to do that, because --no-whole-file optimizes rsync's socket I/O at the expense of disk I/O, which means that you're making things less efficient when the "connection" between the sender and the receiver is a local pipe. The use of -z for a "local" copy is also wasteful because you're using CPU to optimize the transfer of data over a connection that is faster than the disk I/O, even when uncompressed. Your best configuration is to avoid updating via NFS and instead connect to the NFS server directly so that rsync can update the files on a local disk. That allows rsync to optimize the network traffic. rsync -avtz /test1/ remoteNFShost:/test2/ If that is not possible, the method that uses the least disk I/O for a local copy is --whole-file and (to a much smaller extent) --inplace. That still writes out the entire file over NFS for each update, though, but it does at least avoid having rsync do a full-file read followed by a full-file write (which is what occurs with --no-whole-file). ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
On Thu, Oct 27, 2005 at 07:35:47PM +0100, Manuel L?pez-Ib??ez wrote: > Could this also be added to the FAQ ? Please! You must not have read the FAQ recently, because this idea has been there for a week now: after Matt mentioned the HostKeyAlias option back on the 19th, I replied saying what a nice solution it was, and that I had added it to the FAQ. ..wayne.. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 3220] list of error messages
https://bugzilla.samba.org/show_bug.cgi?id=3220 [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||INVALID --- Additional Comments From [EMAIL PROTECTED] 2005-10-27 17:29 --- This is a request for information, not a bug report. The best place to ask questions is on the mailing list. As long as I'm here, I might as well answer: A usage error means that you got the options wrong or didn't use any at all. See the manpage for the full details of how to use rsync, including a brief list of all the "EXIT VALUES". Also, running "rsync --help" gives you a quick command/option summary. -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Converting OS X UTF-8 NFD to UTF-8 NFC
To make rsync work better when rsyncing between OS X (whose filenames
are encoded in UTF-8 NFD form) and just about every other OS (whose
filenames are encoded in UTF-8 NFC form), I wrote a little patch for
rsync that converts the filenames to NFC before sending them to the
other side.
It uses libidn for that, and currently has no option to enable/disable
this behaviour, but since on OS X all filenames are UTF-8 no matter
what, this shouldn't matter much. When creating files, OS X handles
UTF-8 NFC fine, so a reverse conversion isn't needed.
I'm posting this patch here and not on sourceforge because it probably
isn't production quality, but it's useful for me and perhaps for the
countless other people who've reported this problem before as well.
If the rsync maintainers are interested in integrating this patch, I'll
be happy to polish it up a little. I'm not sure if it should be a
configure option (autoenabled on OS X maybe) or a command line option,
so in that case some input here would be nice. Or else, feel free to
adapt it.
After applying it, "-lidn" needs to be added to LIBS in the Makefile.
And without further ado, here is the patch:
diff -u rsync-2.6.6/flist.c rsync-2.6.6-nfc/flist.c
--- rsync-2.6.6/flist.c 2005-07-07 15:49:14.0 -0400
+++ rsync-2.6.6-nfc/flist.c 2005-10-26 15:18:01.0 -0400
@@ -1748,8 +1748,11 @@
* buffer. No size-checking is done because we checked the size when creating
* the file_struct entry.
*/
+#include
char *f_name_to(struct file_struct *f, char *fbuf)
{
+ char *norm;
+
if (!f || !f->basename)
return NULL;
@@ -1760,6 +1763,11 @@
strcpy(fbuf + len + 1, f->basename);
} else
strcpy(fbuf, f->basename);
+
+ norm = stringprep_utf8_nfkc_normalize(fbuf, -1);
+ strcpy(fbuf, norm);
+ free(norm);
+
return fbuf;
}
--
Josef Drexler | http://jdrexler.com/home/
-+---
Please help Conserve Gravity| Email address is *valid*.
Don't do push ups | Don't remove the "nospam" part.
--
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
Yes, your explanations are better. I will promote #2 to become #1, since it is more likely the one that will work better in all situations. Another thing is... if you can use "Method #2 Rsync SSH-es to target using a proxy command that first SSH-es to middle" also with rsync daemon servers, then: is there any possible advantage on the other three methods? If not, can we remove everything but just Method #2 possible with another example using a rsync daemon server? You know, I remember some say about keeping things simple "something". Cheers, Manuel. Matt McCutchen wrote: On Thu, 2005-10-27 at 21:48 +0100, Manuel López-Ibáñez wrote: Method 1: no rsync daemon server, passwordless authentication in middle machine. Method 2: no rsync daemon server, using SSH proxy Method 3: no rsync daemon server, using SSH port forwarding Method 4: no rsync daemon server, using SSH tunnel The difference between #3 and #4 is not port forwarding vs. tunnel (we've been using the terms synonymously) but SSH port vs. rsync daemon port. Here's how I would summarize all the methods: #1: Rsync runs a chained SSH command as transport; authentication on middle must be passwordless #2: Rsync SSH-es to target using a proxy command that first SSH-es to middle #3: Forward target's SSH port to a local port; rsync SSH-es to that port #4: Run rsync daemon on target and forward its port to a local port; rsync accesses the daemon using that port I dislike #1 because the middle machine can subvert the connection. I dislike #3 and #4 because (a) one must remember to set up and take down the tunnel and (b) others can take advantage of the tunnel. (If, as many hope, SSH learns to forward filesystem sockets, (b) will go away.) Except for some technicalities in how the proxy connection closes, #2 is the ideal technique, and that's what I use to access my school's firewalled backup machine. The updated FAQ is very nice, but perhaps the "rsync through a firewall" section should be factored out into another page because it occupies more than half of the FAQ page. Incidentally, I set up SSH on my machine to prefer password authentication to keyboard-interactive authentication; now the password prompt shows the target user and host. Thanks, Carson! __ Renovamos el Correo Yahoo! Nuevos servicios, más seguridad http://correo.yahoo.es -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
On Thu, 2005-10-27 at 21:48 +0100, Manuel López-Ibáñez wrote: > Method 1: no rsync daemon server, passwordless authentication in middle > machine. > > Method 2: no rsync daemon server, using SSH proxy > > Method 3: no rsync daemon server, using SSH port forwarding > > Method 4: no rsync daemon server, using SSH tunnel The difference between #3 and #4 is not port forwarding vs. tunnel (we've been using the terms synonymously) but SSH port vs. rsync daemon port. Here's how I would summarize all the methods: #1: Rsync runs a chained SSH command as transport; authentication on middle must be passwordless #2: Rsync SSH-es to target using a proxy command that first SSH-es to middle #3: Forward target's SSH port to a local port; rsync SSH-es to that port #4: Run rsync daemon on target and forward its port to a local port; rsync accesses the daemon using that port I dislike #1 because the middle machine can subvert the connection. I dislike #3 and #4 because (a) one must remember to set up and take down the tunnel and (b) others can take advantage of the tunnel. (If, as many hope, SSH learns to forward filesystem sockets, (b) will go away.) Except for some technicalities in how the proxy connection closes, #2 is the ideal technique, and that's what I use to access my school's firewalled backup machine. The updated FAQ is very nice, but perhaps the "rsync through a firewall" section should be factored out into another page because it occupies more than half of the FAQ page. Incidentally, I set up SSH on my machine to prefer password authentication to keyboard-interactive authentication; now the password prompt shows the target user and host. Thanks, Carson! -- Matt McCutchen, ``hashproduct'' [EMAIL PROTECTED] -- http://mysite.verizon.net/hashproduct/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
Oh, yes! I am sorry, I should have checked that twice. Wayne did really a great work on the FAQ. I would add something instead (of added to) the Method #. For example, Method 1: no rsync daemon server, passwordless authentication in middle machine. Method 2: no rsync daemon server, using SSH proxy Method 3: no rsync daemon server, using SSH port forwarding Method 4: no rsync daemon server, using SSH tunnel Only just to give a quick summary of the differences of the methods. Cheers, Manuel. Matt McCutchen wrote: On Thu, 2005-10-27 at 09:46 -0400, Carson Gaspar wrote: Hot keys aren't a problem if you use HostKeyAlias - see ssh_config(4). e.g.: On Thu, 2005-10-27 at 19:35 +0100, Manuel López-Ibáñez wrote: Could this also be added to the FAQ ? Please! My original explanation included the HostKeyAlias technique, and hence so does Method 3 of the FAQ. No change needs to be made to the FAQ. Thanks to whoever it was (Wayne?) for adding the SSH proxy technique. Now anyone who happens upon the FAQ page should be able to find at least one way of getting through a firewall that works in their setup. :) __ Renovamos el Correo Yahoo! Nuevos servicios, más seguridad http://correo.yahoo.es -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
On Thu, 2005-10-27 at 09:46 -0400, Carson Gaspar wrote: > Hot keys aren't a problem if you use HostKeyAlias - see ssh_config(4). e.g.: On Thu, 2005-10-27 at 19:35 +0100, Manuel López-Ibáñez wrote: > Could this also be added to the FAQ ? Please! My original explanation included the HostKeyAlias technique, and hence so does Method 3 of the FAQ. No change needs to be made to the FAQ. Thanks to whoever it was (Wayne?) for adding the SSH proxy technique. Now anyone who happens upon the FAQ page should be able to find at least one way of getting through a firewall that works in their setup. :) -- Matt McCutchen, ``hashproduct'' [EMAIL PROTECTED] -- http://mysite.verizon.net/hashproduct/ -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Rsync over NFS mount sending whole files
Hi, On Thu, 27 Oct 2005, Anban Mestry wrote: I'm not sure if anyone has experienced this, and I have searched for it online, with no conclusive, err.. conclusions. Basically, when rsyncing two \test1(local) and \mnt\test2\ (NFS mount) it seems that when using rsync with --no-whole-file entire files (instead of just updated blocks) are sent through. I am using the following command rsync -avtz --no-whole-file \test1\ \mnt\test2\ For example, if say a particular file was 30KB, IPTRAF reports just over 31KB transferred while rsync itself reports just a few bytes (i.e. 200 or so). One of your "local only" rsync instances has to read the entire file in order to find out that only 200 bytes are to transfer. ;-)) Cheers -e -- Eberhard Moenkeberg ([EMAIL PROTECTED], [EMAIL PROTECTED]) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
Carson Gaspar wrote: > > > --On Saturday, October 22, 2005 1:56 AM +0100 Manuel López-Ibáñez > <[EMAIL PROTECTED]> wrote: > >>> This setup seems to work well--perhaps it could be added to the rsync >>> FAQ page as Method 2b. The only annoyance is that one might still get >>> two indistinguishable "Password:" prompts; could someone tell me how to >>> configure SSH so the prompt reveals the target >> >> >> Well, actually, I get "Password: " prompts when a ssh into linux machines >> and "user (at) hostname's password: " when I log into OpenBSD. I have not >> been able to find any option to configure the prompt in man pages >> ssh_config and sshd_config, so I would assume that it is an >> implementation issue. > > > It's probably a difference between password (shows hostname) and > keyboard-interactive (doesn't unless the remote server sends it). > Yes, it is. I have checked it with the people of openssh (see below) Manuel López-Ibáñez wrote: Darren Tucker wrote: As long as the server supports it, the easy way to get it to do what you want is is to tell your client to try "password" authentication first (see PreferredAuthentications in ssh_config(5). Yes, you are right, I get the "[EMAIL PROTECTED]'s password:" prompt when using 'ssh -o "PreferredAuthentications=password" target'. However, apart from using PAM, what is the difference between password and keyboard-interactive authentications? In OpenSSH 3.9 and up (and 3.6x and below), both use PAM. The difference is complexity: the "password" authentication allows the client to provide a password (and, optionally, change it) but that's it. "keyboard-interactive" allows conversations of arbitrary complexity. The classic use for this is a "challenge-response" token: it supplies a challenge which you punch into a little hand-held authenticator then type in what it displays. It could do more than this and more (as can PAM, which is why the two are often used together). And, what is the difference from the point of view of security? Are both equally secure? I theory, they're both equally secure. Maybe there should be an FAQ entry for this. Yeah, the question would be: "How can I configure the password prompt?", wouldn't? Unfortunately, I don't know the answer. Right now, the answers are a) configure PAM to do it (if possible), and b) modify the ssh client. __ Renovamos el Correo Yahoo! Nuevos servicios, más seguridad http://correo.yahoo.es -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
Could this also be added to the FAQ ? Please! Thanks, Manuel. Carson Gaspar wrote: --On Thursday, October 20, 2005 12:53 PM -0700 Wayne Davison <[EMAIL PROTECTED]> wrote: On Wed, Oct 19, 2005 at 10:29:21PM -0400, Matt McCutchen wrote: $ ssh -L :target:22 -N -f [EMAIL PROTECTED] Password: middlepass Port-forwarding 22 is a great idea as long as ssh is configured not to complain about the host conflict -- nicely done. I've added this to the FAQ in the "rsync through a firewall" section. Host keys aren't a problem if you use HostKeyAlias - see ssh_config(4). e.g.: ssh -L 2022:remote-host.example.com:22 -N -d middle-host.example.com Host remote-tunnel.example.com User remuser HostName 127.0.0.1 Port 2022 HostKeyAlias remote-host.example.com rsync -aH /foo/ remote-tunnel.example.com:/bar/ __ Renovamos el Correo Yahoo! Nuevos servicios, más seguridad http://correo.yahoo.es -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 3220] New: list of error messages
https://bugzilla.samba.org/show_bug.cgi?id=3220 Summary: list of error messages Product: rsync Version: 2.6.4 Platform: All OS/Version: HP-UX Status: NEW Severity: normal Priority: P3 Component: core AssignedTo: [EMAIL PROTECTED] ReportedBy: [EMAIL PROTECTED] QAContact: [EMAIL PROTECTED] I get the following error when running rsync on my backup box (HP L2000) hpux 11i. rsync error: syntax or usage error (code 1) at main.c(1151) I was wondering where i can find an area that list errors specifically.) -- Configure bugmail: https://bugzilla.samba.org/userprefs.cgi?tab=email --- You are receiving this mail because: --- You are the QA contact for the bug, or are watching the QA contact. -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync over NFS mount sending whole files
Hey all, I'm not sure if anyone has experienced this, and I have searched for it online, with no conclusive, err.. conclusions. Basically, when rsyncing two \test1(local) and \mnt\test2\ (NFS mount) it seems that when using rsync with --no-whole-file entire files (instead of just updated blocks) are sent through. I am using the following command rsync -avtz --no-whole-file \test1\ \mnt\test2\ For example, if say a particular file was 30KB, IPTRAF reports just over 31KB transferred while rsync itself reports just a few bytes (i.e. 200 or so). I am thinking that either NFS is to blame, or that rsync is trying to rebuild the file on the other side and requires additional reads and commands to do this. --in-place doesn't seem to alleviate the problem much either. Any help would be appreciated. Thanks -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
--On Saturday, October 22, 2005 1:56 AM +0100 Manuel López-Ibáñez <[EMAIL PROTECTED]> wrote: This setup seems to work well--perhaps it could be added to the rsync FAQ page as Method 2b. The only annoyance is that one might still get two indistinguishable "Password:" prompts; could someone tell me how to configure SSH so the prompt reveals the target Well, actually, I get "Password: " prompts when a ssh into linux machines and "user (at) hostname's password: " when I log into OpenBSD. I have not been able to find any option to configure the prompt in man pages ssh_config and sshd_config, so I would assume that it is an implementation issue. It's probably a difference between password (shows hostname) and keyboard-interactive (doesn't unless the remote server sends it). -- Carson -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync through multiple ssh hops with password authentication prompt
--On Thursday, October 20, 2005 12:53 PM -0700 Wayne Davison <[EMAIL PROTECTED]> wrote: On Wed, Oct 19, 2005 at 10:29:21PM -0400, Matt McCutchen wrote: $ ssh -L :target:22 -N -f [EMAIL PROTECTED] Password: middlepass Port-forwarding 22 is a great idea as long as ssh is configured not to complain about the host conflict -- nicely done. I've added this to the FAQ in the "rsync through a firewall" section. Hot keys aren't a problem if you use HostKeyAlias - see ssh_config(4). e.g.: ssh -L 2022:remote-host.example.com:22 -N -d middle-host.example.com Host remote-tunnel.example.com User remuser HostName 127.0.0.1 Port 2022 HostKeyAlias remote-host.example.com rsync -aH /foo/ remote-tunnel.example.com:/bar/ -- Carson -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Banned file: .exe in mail from you
BANNED FILENAME ALERT Your message to: [EMAIL PROTECTED] was blocked by our Spam Firewall. The email you sent with the following subject has NOT BEEN DELIVERED: Subject: Status An attachment in that mail was of a file type that the Spam Firewall is set to block. Reporting-MTA: dns; mx1.101m3.com Received-From-MTA: smtp; mx1.101m3.com ([127.0.0.1]) Arrival-Date: Thu, 27 Oct 2005 05:02:14 -0700 (PDT) Final-Recipient: rfc822; helen@certcities.com Action: failed Status: 5.7.1 Diagnostic-Code: smtp; 550 5.7.1 Message content rejected, id=09828-01-27 - BANNED: .exe Last-Attempt-Date: Thu, 27 Oct 2005 05:02:15 -0700 (PDT) Received: from lists.samba.org (unknown [203.200.177.249]) by mx1.101m3.com (Spam Firewall) with ESMTP id A090634F3F0 for <[EMAIL PROTECTED]>; Thu, 27 Oct 2005 05:02:09 -0700 (PDT) From: rsync@lists.samba.org To: [EMAIL PROTECTED] Subject: Status Date: Thu, 27 Oct 2005 17:35:18 -0700 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=_NextPart_000_0009_B9C6B9C6.73BE73BE" X-Priority: 3 X-MSMail-Priority: Normal Message-Id: <[EMAIL PROTECTED]> -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync and o_direct
Just in case anyone is interested, I've just got confirmation from Oracle that it is valid to use OCFS without O_DIRECT for reads, it's only writes that are an issue. So it is perfectly valid to use rsync to clone an OCFS Oracle archive log area to another server on standard filesystem. Simon > On Wed, Oct 19, 2005 at 11:59:57AM +0100, Simon Hargrave wrote: >> My question is, is there a facility to ensure rsync reads files in O_DIRECT mode so that we can use it for this purpose? > > You'd need to modify the code yourself to add O_DIRECT to the various system calls. I haven't heard about this being done before. > > ..wayne.. > > -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Fast question...
Hi Brad, Yes, I tried, the problem, as Wayne told, comes with ":", that notation conflicts with the sintax's hostname ... Thus, I think that, the only way to use rsync with windows, implies Cygwin. Thank u! Judith pd: Thanks Wayne!! Brad Rigden wrote: Hi Judith, have you tried replacing your forward slashes with back slashes? Brad -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync failed: Too many links
On Wed 26 Oct 2005, Kent Miller wrote: > $ mkdir ~/briefcase/1205275 > mkdir: cannot create directory `1205275': Too many links > > Therefore, the limitation appears to be on the side of the ext3 file > system, and unrelated to rsync. If you have so many subdirectories that ext3 is complaining about "too many links", then the performance will also be impacted badly. > Of course, this means that I am really stuck. In a push to standardize, I > deployed debian sarge throughout my entire network. All partitions are > either ext3 or swap. I use reiserfs everywhere (and Debian of course), but perhaps a better directory layout would be more fruitful in your case. Paul Slootman -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
