Re: [CentOS] Rsync/SSH automation problem?
-Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Kai Schaetzl Sent: Wednesday, 29 April 2009 9:09 PM To: centos@centos.org Subject: Re: [CentOS] Rsync/SSH automation problem? is a quote marker, don't write your own text after a quote marker! I totally understand how to quote messages when replying to posts in newsgroups. I've just checked my sent items folder and the message did not go out with the you talk about!!! I'm not sure where it got injected into the message before getting to the newsgroup! Tia, Tkb. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Gordon Messmer wrote on Thu, 30 Apr 2009 14:19:04 -0700: I have a win2k3 server that its backing up to a CentOS 5.3 server. On the Win2k3 machine I plan to have rsync back up nightly to the CentOS server through ssh. The command I am using is as follows: rsync -vrPtz -e ssh myu...@myserver:/remote/backup/folder/ /local/backup/folder/ The first thing I'm sure you noticed was that this syntax copies data from a remote system to the local one, which is the opposite of what you described doing, above. No. He just tells he wants to backup win2k to CentOS. He doesn't say on which site he starts the operation. So, going from the command line one assumes he's working on the CentoS box. That may be incorrect, but that's what fits to the whole paragraph nicely. The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. You mean *starting* an rsync operation on that side? Using rsync over ssh essentially uses rsync on *both* ends. So, it's running under Cygwin, anyway, which makes your statement a bit confusing. Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Les Mikesell wrote: You have always been able to initiate the command over ssh from a windows box with the answering copy under sshd on linux. Very recent versions may even work the other way. My understanding was that rsync will used non-blocking pipes to communicate with ssh, regardless of which system initiates the connections, and is subject to the bug. It may work for you, but that doesn't mean the bug is gone; it only means that your system doesn't trigger it. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Kai Schaetzl wrote: The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. You mean *starting* an rsync operation on that side? Using rsync over ssh essentially uses rsync on *both* ends. So, it's running under Cygwin, anyway, which makes your statement a bit confusing. What I mean is that if you launch rsync with something like: rsync -e ssh server:/path /path then rsync uses a non-blocking (I said blocking earlier, which was a mistake) socket pair to communicate with ssh. This may trigger a bug in cygwin which can cause the application to hang. If, instead, you run rsync as a daemon on Windows, you can reliably communicate with the daemon over TCP. This remains true if you use ssh to forward a port. Thus, I recommend that anyone running rsync on Windows set up rsync as a daemon that listens for connections on localhost only and use ssh port forwards to reach it from remote systems. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Gordon Messmer wrote: Kai Schaetzl wrote: The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. You mean *starting* an rsync operation on that side? Using rsync over ssh essentially uses rsync on *both* ends. So, it's running under Cygwin, anyway, which makes your statement a bit confusing. What I mean is that if you launch rsync with something like: rsync -e ssh server:/path /path then rsync uses a non-blocking (I said blocking earlier, which was a mistake) socket pair to communicate with ssh. This may trigger a bug in cygwin which can cause the application to hang. It's been 7 years since I use rsync over ssh to backup windows boxes though it worked pretty well for me back then. One thing to try if rsync hangs on you is the --timeout option, which should cause rsync to abort if no data is transferred within X seconds. I wrote up a fairly big rsync log retrieval system that has about 90 systems uploading more than a TB of data a day to a NFS cluster, sometimes the system is really busy, so rather then have rsync hang for a really long period of time I just have it abort after 10 minutes of no activity. Also put retry logic in the rsync scripts themselves, so they attempt to send data up to 20 times per hour per system(new data is made available to upload once an hour). Of course this is entirely linux based, and I am using rsync over HPN-SSH with encryption disabled for higher performance. Sample rsync command line that I use: rsync -ae /usr/bin/hpnssh -v -o TcpRcvBufPoll=yes -o NoneEnabled=yes -o NoneSwitch=yes --timeout=600 --log-format=[%p] %t %o %f (%l/%b) --files-from=/home/logrsync/conf/rsync_log_file_list.20090501_090201 /local_dir/ 10.254.213.203:/remote/dir/ 1/home/logrsync/logs/server_name_rsync_log_transfer_20090501_090201.log 21 Just finished another rsync deployment system that downloads data to those same servers, with built in parallelism for increased throughput over the WAN. I currently have 6 rsync/ssh systems that do the file serving which are load balanced behind a BigIP. Main bottleneck is the cisco firewall which can only do 1.2Gbps of throughput. nate ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Gordon Messmer wrote: The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. You mean *starting* an rsync operation on that side? Using rsync over ssh essentially uses rsync on *both* ends. So, it's running under Cygwin, anyway, which makes your statement a bit confusing. What I mean is that if you launch rsync with something like: rsync -e ssh server:/path /path then rsync uses a non-blocking (I said blocking earlier, which was a mistake) socket pair to communicate with ssh. This may trigger a bug in cygwin which can cause the application to hang. It always seemed to work when you execute the command on the windows side but had a bug that would hang when windows was on the answering side and started rsync under sshd. If, instead, you run rsync as a daemon on Windows, you can reliably communicate with the daemon over TCP. This remains true if you use ssh to forward a port. Thus, I recommend that anyone running rsync on Windows set up rsync as a daemon that listens for connections on localhost only and use ssh port forwards to reach it from remote systems. I think the sshd issue is fixed in the current cygwin but another bug in rsync can cause problems with certain windows paths unless both ends are newer than 2.6.9 and support protocol 30 (Centos ships a 2.6.8). -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
James Bensley wrote: I have a win2k3 server that its backing up to a CentOS 5.3 server. On the Win2k3 machine I plan to have rsync back up nightly to the CentOS server through ssh. The command I am using is as follows: rsync -vrPtz -e ssh myu...@myserver:/remote/backup/folder/ /local/backup/folder/ The first thing I'm sure you noticed was that this syntax copies data from a remote system to the local one, which is the opposite of what you described doing, above. The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. This has been the case for years and has something to do with Cygwin's emulation of blocking sockets which causes rsync to hang. Even if your backups are working now, I strongly recommend against continuing in this manner. If you want to use rsync on Windows, the best thing to do is run rsync and ssh daemons on the Windows host. Rsync should only listen for localhost connections. Run your backup jobs from Linux; ssh to the Windows machine and forward a port for rsync. A local rsync can then connect to the Windows machine over the port forward. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Gordon Messmer wrote: James Bensley wrote: I have a win2k3 server that its backing up to a CentOS 5.3 server. On the Win2k3 machine I plan to have rsync back up nightly to the CentOS server through ssh. The command I am using is as follows: rsync -vrPtz -e ssh myu...@myserver:/remote/backup/folder/ /local/backup/folder/ The first thing I'm sure you noticed was that this syntax copies data from a remote system to the local one, which is the opposite of what you described doing, above. The second thing you will notice, eventually, is that rsync over ssh under Cygwin is unreliable. This has been the case for years and has something to do with Cygwin's emulation of blocking sockets which causes rsync to hang. Even if your backups are working now, I strongly recommend against continuing in this manner. You have always been able to initiate the command over ssh from a windows box with the answering copy under sshd on linux. Very recent versions may even work the other way. If you want to use rsync on Windows, the best thing to do is run rsync and ssh daemons on the Windows host. Rsync should only listen for localhost connections. Run your backup jobs from Linux; ssh to the Windows machine and forward a port for rsync. A local rsync can then connect to the Windows machine over the port forward. Or, if you want it automated more nicely to keep a history of backups with a web interface to browse/restore them, look at backuppc. http://backuppc.sourceforge.net/ You can still use rsync as the copy mechanism if you want. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
[CentOS] Rsync/SSH automation problem?
Hi All, I have a win2k3 server that its backing up to a CentOS 5.3 server. On the Win2k3 machine I plan to have rsync back up nightly to the CentOS server through ssh. The command I am using is as follows: rsync -vrPtz -e ssh myu...@myserver:/remote/backup/folder/ /local/backup/folder/ This all works great for the inital test except its prompts me for a password for myuser (as you would except) but how can I embed the password for ssh into the command line? Or can I not? Is the following some how possible: rsync -vrPtz -e ssh myu...@myserver mypassword /remote/backup/folder/ /local/backup/folder/ Or something like that? I can't find anything that says it is possible so its probably isn't but I know that some of you out there have far superior searching skills than I or hopefully, you might already know? Assuming this isn't possible as I've got that gut feeling it won't be, does anyone else have a better idea? BTW; Rsync in running on the windows box through cygwin in case you wondered or hadn't guessed ;) Many thanks, James ;) -BEGIN GEEK CODE BLOCK- Version: 3.1 GIT/MU/U dpu s: a-- C++$ U+ L++ B- P+ E? W+++$ N K W++ O M++$ V- PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+ DI D+++ G+ e(+) h--(++) r++ z++ --END GEEK CODE BLOCK-- ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
James Bensley wrote on Wed, 29 Apr 2009 08:14:54 +0100: This all works great for the inital test except its prompts me for a password for myuser (as you would except) but how can I embed the password for ssh into the command line? Or can I not? You can't. You use a certificate. There are many tutorials about this. I found this one in the short time: http://www.ehow.com/how_4750765_rsync-over-ssh-password.html Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
hi Kai, You can't. Thats what I thought, I have been looking into using certificates and passphrases and that will have to be it! Thank for your help! James ;) -BEGIN GEEK CODE BLOCK- Version: 3.1 GIT/MU/U dpu s: a-- C++$ U+ L++ B- P+ E? W+++$ N K W++ O M++$ V- PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+ DI D+++ G+ e(+) h--(++) r++ z++ --END GEEK CODE BLOCK-- ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
-Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of James Bensley Sent: Wednesday, 29 April 2009 6:54 PM To: CentOS mailing list Subject: Re: [CentOS] Rsync/SSH automation problem? hi Kai, You can't. Thats what I thought, I have been looking into using certificates and passphrases and that will have to be it! Interesting - why not do a pull from the CentOS box? This would only work if the machines are local to each other! I mount Windows Server 2003 as follows: mount //[servername]/C$ /winmnts/[servername]/cdrive \ -o credentials=/root/sysaccess /root/sysaccess - contains just two lines: username=[administrative account] password=[administrative account password] Also, you'll need to have created the path /winmnts/[servername]/cdrive beforehand (as if you hadn't guessed that already)! From here, I run the rsync command without the need for the -e option. Of course, if the servers are in geographically different locations, then Kai's link is the way to go! Cheers, Tkb. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Thanks for your input, unfortunately the servers will be communicating via the inter-web-net-spagetthi. Its OK I have sorted this with ssh passphrases, the only reason I posted to the list instead of using them in the first place is that I was sceptical of the reliability of using them from a windows server but in actual fact it all went swimmingly! Thanks all xD -BEGIN GEEK CODE BLOCK- Version: 3.1 GIT/MU/U dpu s: a-- C++$ U+ L++ B- P+ E? W+++$ N K W++ O M++$ V- PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+ DI D+++ G+ e(+) h--(++) r++ z++ --END GEEK CODE BLOCK-- ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Anthony Kamau wrote on Wed, 29 Apr 2009 20:40:10 +1000: Interesting - why not do a pull from the CentOS box? He's doing that. I mount Windows Server 2003 as follows: Mounting is completely different from using rsync on two hosts - e.g. you loose the delta algorithm advantage of rsync the way you use it. What you do may actually take *double* the time of a normal copy. From here, I run the rsync command without the need for the -e option. is a quote marker, don't write your own text after a quote marker! Kai -- Kai Schätzl, Berlin, Germany Get your web at Conactive Internet Services: http://www.conactive.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
On Wed, Apr 29, 2009, James Bensley wrote: Hi All, I have a win2k3 server that its backing up to a CentOS 5.3 server. On the Win2k3 machine I plan to have rsync back up nightly to the CentOS server through ssh. The command I am using is as follows: rsync -vrPtz -e ssh myu...@myserver:/remote/backup/folder/ /local/backup/folder/ This all works great for the inital test except its prompts me for a password for myuser (as you would except) but how can I embed the password for ssh into the command line? Or can I not? We use rsync modules, hostname::modulename, extensively for things like this as they can restrict the remote host to specific directories, and IP address/CIDR block ranges without using ssh. For things like rsync dns updates of djbdns/tinydns data files, this can be done without encryption. Where the data is sensitive, doing the update over OpenVPN tunnels works nicely. Bill -- INTERNET: b...@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way Voice: (206) 236-1676 Mercer Island, WA 98040-0820 Fax:(206) 232-9186 Skype: jwccsllc (206) 855-5792 Common sense is genius dressed in its working clothes. -- Ralph Waldo Emerson ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
On Wed, Apr 29, 2009 at 08:13, Tony Mountifield t...@softins.clara.co.uk wrote: In article va.37fd.00e33...@news.conactive.com, Kai Schaetzl mailli...@conactive.com wrote: Anthony Kamau wrote on Wed, 29 Apr 2009 20:40:10 +1000: From here, I run the rsync command without the need for the -e option. is a quote marker, don't write your own text after a quote marker! He probably didn't do it himself. Some mail transports prepend a '' to any line that begins with From , because otherwise such a line could be interpreted as a message delimiter in a standard Unix mailbox. But escaping From to From should be done only by the program that writes to an mbox formatted file. See: http://qmail.org/qmail-manual-html/man5/mbox.html Looks like another thing that his MUA (header says X-Mailer: Microsoft Office Outlook 11) is doing wrong. [Unless it is an artifact of the mailing list software, in which case I would apologize to the makers of said MUA. But if I had to bet, I would say it's the MUA's fault.] Filipe ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
-Original Message- From: centos-boun...@centos.org [mailto:centos-boun...@centos.org] On Behalf Of Filipe Brandenburger Sent: Wednesday, April 29, 2009 12:51 PM To: CentOS mailing list Subject: Re: [CentOS] Rsync/SSH automation problem? On Wed, Apr 29, 2009 at 08:13, Tony Mountifield t...@softins.clara.co.uk wrote: In article va.37fd.00e33...@news.conactive.com, Kai Schaetzl mailli...@conactive.com wrote: Anthony Kamau wrote on Wed, 29 Apr 2009 20:40:10 +1000: From here, I run the rsync command without the need for the -e option. is a quote marker, don't write your own text after a quote marker! He probably didn't do it himself. Some mail transports prepend a '' to any line that begins with From , because otherwise such a line could be interpreted as a message delimiter in a standard Unix mailbox. But escaping From to From should be done only by the program that writes to an mbox formatted file. See: http://qmail.org/qmail-manual-html/man5/mbox.html Looks like another thing that his MUA (header says X-Mailer: Microsoft Office Outlook 11) is doing wrong. [Unless it is an artifact of the mailing list software, in which case I would apologize to the makers of said MUA. But if I had to bet, I would say it's the MUA's fault.] Filipe This isn't surprising. When I have to mail out to mailing lists for FOSS projects, I have to manually put my signature at the bottom, and turn off HTML mails, since Outlook encourages top posting, and HTML. Additionally, as you can see from the quoting of your message, the From: line is treated like normal quoted text instead of the norm for good mail clients. (No, I cannot switch off from Outlook since it's the standard here since we've an Exchange server.) -- Gary L. Greene, Jr. IT Operations Minerva Networks, Inc. Cell: (650) 704-6633 Phone: (408) 240-1239 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] Rsync/SSH automation problem?
Maybe you guys should start your own thread about how to reply to emails and how changing the subject isn't really appropriate? -BEGIN GEEK CODE BLOCK- Version: 3.1 GIT/MU/U dpu s: a-- C++$ U+ L++ B- P+ E? W+++$ N K W++ O M++$ V- PS+++ PE++ Y+ PGP t 5 X+ R- tv+ b+ DI D+++ G+ e(+) h--(++) r++ z++ --END GEEK CODE BLOCK-- ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
