Extremely slow on certain files
I'm trying to rsync a large data set (approx 140K files, 4TB) between machines directly connected on gigabit ethernet. Normally it runs around 20MB/sec, which is acceptable. However, it seems like there is one particular file that is giving rsync heartburn... whenever it encounters this file, the speed drops down to 150KB/sec speed within a few seconds. I don't know what's so special about this file. I've tried scp'ing it directly between boxes - 20MB/sec. I tried rsync in ssh and daemon modes, same deal - 150KB/sec. rsync 2.6.9 on Redhat EL4 (x86_64). /opt/bin/rsync -av --progress --port=8738 gsc free c15-gs-deliv1-data1::d4 building file list ... 52478 files to consider gsc/2007/04/ gsc/2007/04/indievelopment005_vf_040407.mpg 96436224 28% 154.31kB/s0:25:53 Any ideas? -jsd- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
need rsync.conf exclude example
on the client i am running the following command: rsync -Cav --exclude=templates_c --port=8738 GNE_FRAMEWORK gamespot_5 c14-gs-ww1::nfs/gamespot on the server i have the following module declaration: [nfs] path = /nfs/htdocs read only = no use chroot = no exclude = gamespot_5/promos/ gamespot_5/misc/ obviously my intent is to prevent the promos and misc subdirectories from being copied from client to server. this is not functioning though. what do i have to do to get it to work? my logfile shows this: 2005/10/07 09:38:45 [31991] rsync to nfs/gamespot from c10-gs-dev1.cnet.com (10.16.98.21) 2005/10/07 09:38:45 [31991] gamespot_5/ 2005/10/07 09:38:45 [31991] gamespot_5/promos/ 2005/10/07 09:38:45 [31992] wrote 32 bytes read 23896 bytes total size 3796806 clearly it's not paying any attention to the exclude. -jsd- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
starting daemon from ssh
This may seem odd, but I'm trying to start rsync in daemon mode from an ssh connection... ie, something like this: ssh host1 /usr/bin/rsync --daemon --port= --config=/path/to/config.txt the reason why is i need to start it on 40 machines and manually logging into each one is a pain. even with copying and pasting. i'd rather just write a shell script to do it in a loop. anyway it doesn't work. it just prints @RSYNCD: 26 i'm assuming since ssh host command doesn't create a tty that rsync gets confused and thinks it is already daemonized. is there any workaround or do i have to just bite the bullet and do this the nasty way. -jsd- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: starting daemon from ssh
Jon Drukman wrote: This may seem odd, but I'm trying to start rsync in daemon mode from an ssh connection... ie, something like this: ssh host1 /usr/bin/rsync --daemon --port= --config=/path/to/config.txt the reason why is i need to start it on 40 machines and manually logging into each one is a pain. even with copying and pasting. i'd rather just write a shell script to do it in a loop. anyway it doesn't work. it just prints @RSYNCD: 26 turned out to be an ssh question more than an rsync question. just do ssh -t host rsync --daemon forces ssh to allocate a pseudotty so rsync has something to detach from. hope this helps somebody out. -jsd- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Creating intermediate directories?
Forgive me if this is a FAQ, but Google hasn't helped me find what I want to know. If I have a local file at: /some/local/path/file.zip And I want it to arrive on the remote machine at: /other/foreign/directory/file.zip But only /other exists, is there any way to have rsync create the intermediate directories? I know that if I have /foreign/directory/file.zip locally I can rsync foreign with the -r option, but the path structure on the local machine does not match that of the remote machine. -jsd- -- To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: include directory and all files under
Tim Conway wrote: I personally have never used include/excludes, but I remember reading the man page a few years ago, and though it was very well written, I've never had a scenario where they were appropriate. From your specification, I don't think you do either. Simplify. the main use i've found for them is if you have tons and tons of directories, but similarly named files in those directories that need syncing. for instance i've got a web site with thousands of product directories, and there might be a file named review.html in all those directories. in that case i run the following include/exclude file: + / + review*.html - *.html - *.php -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: synchronizing 4617 files (out of 1538823)
Wayne Davison [EMAIL PROTECTED] wrote in message news:[EMAIL PROTECTED] On Fri, Aug 08, 2003 at 11:30:22AM -0700, Jon Drukman wrote: i have a list of files relative to a docroot that need to be synchronized. [...] is there a fast way to do this? The version is CVS has the new --files-from option. You'd do something like this: this worked: rsync -Rte ssh --files-from=/tmp/paths.txt docroot ww1:/var/httpd/htdocs/docroot thanks for the tip, very handy. -jsd- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
synchronizing 4617 files (out of 1538823)
i have a list of files relative to a docroot that need to be synchronized. example: pc/action/apocalyptica/wrapper_gslive.html pc/action/apocalyptica/wrapper_gslive_ice.html pc/action/apocalyptica/wrapper_ice.html pc/rpg/darkageofcamelot/buyit_unit.html pc/rpg/darkageofcamelot/buyit_unit_ice.html the actual docroot has 1538823 files. building the file list takes a very long time. i would like rsync to only consider the 4617 files listed in the file. i've tried: rsync -Re ssh --include-from=/tmp/paths.txt docroot ww1:/var/httpd/htdocs it says skipping directory docroot and nothing happens. i tried this: rsync -Re ssh `cat /tmp/paths.txt` ww1:/var/httpd/htdocs it returned: /usr/bin/rsync: Argument list too long. the only way i got it to work was to do this: rsync -ave ssh --include '*/' --include-from=/tmp/paths.txt --exclude '*' docroot ww1:/var/httpd/htdocs this takes a very very long time to run. is there a fast way to do this? -jsd- -- To unsubscribe or change options: http://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
