Re: What is the different between these rsync command
On 9/9/07, Matt McCutchen [EMAIL PROTECTED] wrote: The first two commands do exactly the same thing because the destination has a single colon, indicating rsync over a remote shell, and the default remote shell is ssh. All -e ssh does in this situation is explicitly restate the default. (If the source or destination is an rsync daemon, -e ssh does make a difference: it indicates a single-use daemon invoked over ssh.) Matt Unless it is a really old version of rsync where the default shell was rsh. Yue, what version of rsync are you using? -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Win32 port?
On 9/5/07, Matt McCutchen [EMAIL PROTECTED] wrote: It might be worth a try to build rsync on mingw first. If that works, I think a mingw version would be easier to maintain than a complete port in the long term. Matt I am just fishing for opinions here and by no means am I asking anybody to do something, but... :-) Could something like APR from the Apache project be used to make rsync more platform agnositic? Would APR even be ideal? -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Local disk to disk Rsync taking an hour longer than disk to remote
On 7/16/07, W Smith [EMAIL PROTECTED] wrote: Back in June I posted about the trouble I've been having backing up some local directories and I'm no further ahead than back then. Link for that discussion: http://lists.samba.org/archive/rsync/2007-June/017882.html In summary: I'm copying nearly a million small files from the main disk in a server to another disk in the same machine. Still on this server, but at a different time, I am backing up the same million files to a remote server which is completed massively quicker and I just can't get my head around why this should be. Here are some new stats I got whilst using time to measure the difference between the two processes. Local backup --- Number of files: 1060320 Number of files transferred: 2233 Total file size: 26814206753 bytes Total transferred file size: 58711290 bytes Literal data: 58711290 bytes Matched data: 0 bytes File list size: 47383393 Total bytes sent: 106196771 Total bytes received: 44680 sent 106196771 bytes received 44680 bytes 12914.54 bytes/sec total size is 26814206753 speedup is 252.39 real137m5.932s user0m16.843s sys 125m35.697s Remote backup --- Number of files: 1060823 Number of files transferred: 1758 Total file size: 28255934663 bytes Total transferred file size: 63027228 bytes Literal data: 63027228 bytes Matched data: 0 bytes File list size: 47404505 Total bytes sent: 110514244 Total bytes received: 35180 sent 110514244 bytes received 35180 bytes 89045.05 bytes/sec total size is 28255934663 speedup is 255.60 real20m40.908s user0m11.344s sys 0m22.644s Sure there are a few less files on the remote backup, due to the scripts running at different times but this can't explain the hour difference in runtime can it? I've studied the rsync man page for hours, trying to find some elusive option to speed up local performance to no avail. I've also tried Unison and rdiff-backup, hoping that they might be better for local copying, but nothing comes close to rsync remote backup speeds. At this stage I'm close to giving up and am considering forgetting local backups and getting another server to have two remote backups instead, which wouldn't be a bad thing. The difference could be --whole-file which is enabled by default when the source and destination are local disks. You could try to disable that with --no-whole-file . -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: need help
On 5/4/07, akahitz [EMAIL PROTECTED] wrote: hi there, i have a rsync server running in deamon mode, with a 1000 plus clients connecting to it to retrieve files. the problem is that when the clients connects the server has many rsync processes running on it, each of which is around 3.8mb each. Basically the server is running out of memory and grinding to a halt. is there anyway of controlling this or is adding more memory the only option? thanks in advance I believe the max connections option for rsyncd.conf is what you are looking for. More info can be found in the rsyncd.conf man page. -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync SSL 'for real'
On 4/18/07, Carson Gaspar [EMAIL PROTECTED] wrote: Lawrence D. Dunn wrote: Colleagues, If you do pursue SSL functionality directly in rsync, please be sure to take a look at Chris Rapier's work to fix standard ssh implementations, at: http://www.psc.edu/networking/projects/hpn-ssh/ Turns out -e ssh using most libraries puts a fixed-window-size ssh-windowing behavior on top of TCP - so for large bandwidth*delay product paths, even if you use large TCP buffers (which Wayne added for such paths), an un-fixed SSL library can clobber overall performance/throughput, even for a perfectly clean (no errors/loss) path. SSL != SSH. This still applies (depending on the ssl toolkit being used). The problem referenced here is the TCP window size is hard coded inside the openssl library. In order to change the window size, one must patch openssl. Of course, there is also the question of if openssl is the appropriate toolkit to use with rsync. I am not sure of the issues with a GPL binary linking against a BSD library. Perhaps GnuTLS is more appropriate... (I know... this is probably a whole different can of worms. :-) ). -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync SSL 'for real'
This still applies (depending on the ssl toolkit being used). The problem referenced here is the TCP window size is hard coded inside the openssl library. In order to change the window size, one must patch openssl. TCP window size is not, and can not, be hard coded inside openssl. Do you know what TCP window size is? Of course, there is also the question of if openssl is the appropriate toolkit to use with rsync. I am not sure of the issues with a GPL binary linking against a BSD library. Perhaps GnuTLS is more appropriate... (I know... this is probably a whole different can of worms. :-) ). There is no license issue. This is just a troll. that was a little harsh I'll say. Completely uncalled for and unprofessional. There is no need for name calling here. I can admit when I'm wrong. I apologize for any mis-information, I was just trying to help. -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync SSL 'for real'
On 4/18/07, Andreas Kotes [EMAIL PROTECTED] wrote: This is just a troll. just who exactly are you talking about? ;) Ahem... That would be me. That will be $10 for posting in this list. :-) -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HTTP encapsulated rsync??
On 4/17/07, Robert Denton [EMAIL PROTECTED] wrote: Hi folks, I am hoping someone here can offer some suggestions. Here is my situation: I am using rsync over the internet for several hundred clients to keep them in sync with a master repository of files. The rsync daemon is listening on port 80, because most of the clients are behind firewalls that only allow outbound port 80 (and other common ports). This works fine for most of them but not all. The trouble is that some of them are also behind devices that do not allow non-http traffic on port 80 and thus block the rsync. Therefore, I am looking for a way to use http encapsulation as a means to bypass the blocking devices. I have seen a few scattered queries around the net asking similar questions but I have not seen any good solutions offered. One person suggested that http-tunnel be used but this is not a very elegant solution and I would like to avoid it if at all possible. If the tunnel were to hang or sever then I would have a difficult time correcting it since I do not have direct access to many of the clients. Does anyone here have any ideas? Surely I am not the first person to ask this question. Thanks!! Robert You can use an HTTP proxy. Look at the RSYNC_PROXY environment variable in the man page. -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HTTP encapsulated rsync??
On 4/17/07, Robert Denton [EMAIL PROTECTED] wrote: Do you mean for example.. Instead of running rsync on port 80, have rsync listen for requests on 873 as usual, but also have squid running on the same server listening for port 80 connections, and then just configure squid to send all port 80 traffic to 873? Robert I was thinking more along the lines of creating a proxy for all of your rsync requests. You would not necessarily have to do any special configuration to squid (except allow CONNECT with port 873, something I know has to be done with mod_proxy), it should route the request itself. Apache with mod_proxy would work as well. -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: HTTP encapsulated rsync??
On 4/17/07, Robert Denton [EMAIL PROTECTED] wrote: This is an interesting idea. Here is what I am trying: I have set up squid to listen on port 81, since rsync on the same machine is already listening for requests on 80. I have set the RSYNC_PROXY env var to the hostname:81 and rand a quick test. The result is that I am getting this error: bad response from proxy -- HTTP/1.0 503 Service Unavailable rsync: failed to connect to 192.168.233.56: Success (0) I am thinking this is a problem with the squid config not passing the rsync request to the rsync daemon. Another though occurs to me: many of the clients are behind enforced proxies of their own, set via an env var: http_proxy. Is this going to work with the RSYNC_PROXY? Since the rsync proxy is pointing to an http port on a server, will the request automatically be channeled through the enforced http_proxy? Robert If some of the clients are already having to go through proxies, it does complicate matters. If that is the case, you may have to investigate setting up said proxies to proxy again using proxy chaining/cascading, but that's a bit out of scope for this mailing list. It's difficult for me to say what the problem is with the 503 error, but it sounds like you're on the right track. -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Redirect --stats to STDERR.
));/ /}/ But the problem is i have an error 23 when i use rsync. You could just pipe STDOUT and STDERR to the same file and email that to yourself. To prevent getting the list of files transfered, you could make rsync be quiet by removing -v and adding -q. rsync out.log 21 -- Aaron W Morris (decep) -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: question
[EMAIL PROTECTED] wrote: hello i looked in google , faq etc but didnt found a answer. sorry if i overseen something. how i can auth. via a hostkey without make a config in ~/.ssh normaly ssh has support with ssh -i /keyfile is there any way to combine it via rsync , with rsync -e ssh -i key ..etc etc does not work thx bye Have you tried the RSYNC_RSH environment variable yet? -- Aaron W Morris [EMAIL PROTECTED] (decep) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Shell or rsync issue?
I ran across something I did not expect in rsync the other day. If you want to sync the contents of a local directory to a remote directory, you would use something like this: rsync -rptv /local/directory/ [EMAIL PROTECTED]::module/remotedirectory/ The root of the sync is /local/directory and files are referenced as file1, file2, etc... However, the reverse is not true. This command: rsync -rptv [EMAIL PROTECTED]::module/remotedirectory/ /local/directory/ will sync the files using /local as the root and files are referenced as remotedirectory/file1, remotedirectory/file2, etc. You can fix it with: rsync -rptv [EMAIL PROTECTED]::module/remotedirectory/. /local/directory/ (notice the .) which is not a big problem, but not expected. What is the root cause behind this? Using rsync 2.5.7 on AIX 5.1. -- Aaron W Morris [EMAIL PROTECTED] (decep) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsyncing individual files
Mayuran Yogarajah wrote: Im trying the following command: rsync -ae ssh host.com:/creative/37295/7/file1.jpg /creative rsync fails saying 'no such file or directory' because the directories /creative/37295 and consequently /creative/37295/7 do not exist. What Im wondering is, is it possible to have rsyc make those directories? I want to be able to run that rsync command and have rsync create the necessary directories and put the file there. Thanks rsync -aRe ssh host.com:/creative/37295/7/file1.jpg / might do what you want. Your current command should be copying the file to the /creative/ directory (not the subdirectories) because you have not specified rsync to use relative paths (-R). Someone correct me if I am wrong. -- Aaron W Morris [EMAIL PROTECTED] (decep) -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Capturing output
It may be helpful to capture STDERR with STDOUT. my $message = `/usr/bin/rsync --rsh=/usr/local/openssh/bin/ssh -avu $directory $destination 21`; See: http://perldoc.com/perl5.8.0/pod/perlop.html#%60STRING%60 At 05:47 PM 3/4/2003 +, you wrote: Hi folks, I am running rsync via a perl script run in a cronjob. Here's the command i'm running my $message = `/usr/bin/rsync --rsh=/usr/local/openssh/bin/ssh -avu $directory $destination`; At the moment when I run the script it writes all output to the screen. I want to be able to write this data to a log file, so that if things don't work, I can find out what's wrong. I've tried writeLog($message); but this misses out some of the stuff that rsync writes to the screen, i.e. the useful stuff actually, like if the file is not found on the machine. Any ideas how I can achieve this? Or is this a perl type question rather than something I can do with rsync...? cheers, Tom -- Tom Freeman Web Developer NISS - EduServ +44 (0)1225 474371 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: rsync for DOS
Because rsync requires cygwin to run, other solutions (such as unison) immediately become more attractive. Also, someone correct me if I am wrong, any limitation that cygwin has (files 2GB, etc) rsync also inherits. At 11:49 AM 2/17/2003 +, you wrote: On Sunday 16 Feb 03, Chris Simmonds writes: I don't think it would be too difficult, but is there any point when there is a fully functional Cygwin port? Huh? You can already get it from http://cygwin.com/. Or did I misunderstand your question? (It's not so much a port. It simply builds and works under Cygwin.) David -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: The behavior of -u/--update option on directories
On Mon, 2003-01-13 at 22:46, Florian-Daniel Otel wrote: Aaron, Aaron Morris writes: rsync is just doing what your are telling it to do. Update if the file is changed or does not exist on the remote side and delete if it no longer exists on the local side. Yes, I can RTFM, and, suprisingly!, I even _did_ RTFM :)). The directory may have a newer timestamp, but you are doing a recursive put so it has to check all the files and dirs under each directory. At least that is how I see it. solution: --exclude /tmp/ or --exclude tmp/ Well, I think you missed the point. The tmp subdir was just an example. The same can happen in any subdir down the tree. And, if you re-read my post carefully (or even the subject), I said --update (mis)behavior __on_directories__ (and _not_ regular files). Truth to be told, after a bit of thinking I _might_ accept the fact that --update ignores the timestamps on (sub)directories. Because if it didn't, if any file was touch-ed remotely after the last rsync than all subdirs, up to and including to the top dir, would have a newer timestamp remotely and thus no file putting would take place. This is usually the case, however, not always true. It is possible for a file to have a newer timestamp then the directory it is contained within. If we used your logic below, then a file with a newer timestamp (possibly changed data) on the source would not be transferred to the remote machine if the source directory's timestamp did not change. However, I still think that --update (or, ideally, another similar option or combination of options) __should__ take into account (sub)directories timestamps and consider for processing only those (sub)directories that have remotely an older timestamp than locally. Or, to put it in your words ...doing a recursive put should be restricted __only__ to the (sub)directories that remotely have a timestamp older than locally. If that means that _no_ update (i.e. put) will take place because __any__ file deep down the tree was touched remotely since the last put...well, than that's it. Again, thanks in advance to anyone can suggest a way to have rsync __also__ in this way. Best regards, Florian Florian-Daniel Otel wrote: Dear all, I have the following problem: I use the following command to push files from local to a remote machine: [...] /usr/bin/rsync -avuz -e ssh -1 --exclude .Xauthority --delete /user/home/directory/ [EMAIL PROTECTED]:/user/home/directory/ ; (The ssh is using RSA authentication btw. machines, but that's beside the point). The problem: On both the local and remote machines there is a subdirectory /user/home/directory/tmp. If on the remote machine that subdirectory has a newer time stamp -- e.g. due to a file that was created recently i.e. after the last time the local machine did a pull -- the --delete option in the command above makes it such that the said newer file is deleted on remote and the timestamp is reseted back to the old time stamp i.e. the one for the tmp/ subdir on the local machine. Putting it shortly, the --delete option has precendence over -u option w.r.t. directories. Is it so that -u works only on files and not on directories ? IMHO the appropriate behaviour in a situation like the one described above directories on remote that have newer time stamp should be ignored and excluded from further processing (i.e. including updating and/or deleting any files that do not exist on the local machine). The question: Am I right and that is the way things should behave and I am doing smth wrong ? If not, and this is the way things are supposed to work, can anyone suggest a work-around that would simulate the desired behavior ? Many thanks in advance, Florian P.S. Please Cc: me on the replies, I'm not on the list. Thanks. P.P.S: If it matters: In the above setup the local machine is a Linux running rsync version 2.5.4 protocol version 26 whereas remote is a Solaris 2.8 running rsync version 2.5.5 protocol version 26. -- Aaron W Morris decep PGP Key ID: 259978D1 -- Aaron W Morris decep PGP Key: 259978D1 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
Re: filelist calculation algoritm
I do not think there would be a problem using ssh as the shell. The cat trick or using - is basically the same thing as specifying the file names manually. On Wed, 2003-01-08 at 03:42, Lorenzo Bettini wrote: jw schultz wrote: On Tue, Jan 07, 2003 at 07:15:27PM +0100, Lorenzo Bettini wrote: Aaron Morris wrote: You did not specifically mention it: compression (-z) would probably help more than anything. Otherwise, you could do something like: Have a file (ie filelist.txt) that contains the filename (with relative paths), one file per line. rsync -rRWz `cat filelist.txt` user@hostname::module as I understand this works only when connecting to a rsync daemon... what if I wanted to use ssh as a shell? Compression works fine over ssh. Don't know where you got the impression it doesn't given what is in the manpage. I did not refer to compression: I was wondering if there's a way for updating only a list of files with the `cat` trick... Lorenzo -- +-+ | Lorenzo Bettini ICQ# lbetto, 16080134 | | PhD student in Computer Science| | Dip. Sistemi e Informatica, Univ. di Firenze | | Tel +39 055 4796741, Fax +39 055 4796730 | | Florence - Italy (Linux User # 158233) | | Home Page: http://www.lorenzobettini.it| | E-Mail : [EMAIL PROTECTED]| | http://music.dsi.unifi.it XKlaim language | | http://www.lorenzobettini.it/purpleCover Band | | http://www.gnu.org/software/src-highlite | | http://www.gnu.org/software/gengetopt | +-+ -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html -- Aaron W Morris decep PGP Key: 259978D1 -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.tuxedo.org/~esr/faqs/smart-questions.html
