hello fellow apache mirrors!
since i'm running a quite-not-so-tiny site in europe i want to share you
some of my rsync experiences and config stuff.
just a few points to consider:
-z does not really speed up the transfer, in the server config it should
be disabled generally
-rpltv should be the standard singelchar option string since it nearly
does everything one needs including linking and permission stuff, see
rsync(1) for explanation
--delete is mandatory
--safe-links is also mandatory for mirror sites (!!!)
to mirror apache-site one should exclude:
--exclude incoming/
--exclude bugs/
notice the trailing slash wich says, that these are directories
so for a plain vanilla mirror it boils down to
-----BEGIN FILE apache-site.rsync.sh-----
#!/bin/sh
# apache-site.rsync.sh -- [EMAIL PROTECTED] -- 20000918
/usr/local/bin/rsync -rtvpl --safe-links --delete \
--exclude incoming/ \
--exclude bugs/ \
rsync://dev.apache.org/apache-site/ \
/home/service/ftp/mirrors/dev.apache.org
-----END FILE apache-site-rsync.sh-----
example output follows:
-----BEGIN EXAMPLE OUTPUT-----
[EMAIL PROTECTED]:datasink[~/bin/24h]23% ./apache-site.rsync
receiving file list ... done
wrote 121 bytes read 32264 bytes 5888.18 bytes/sec
total size is 397241652 speedup is 12266.22
-----END EXAMPLE OUTPUT-----
on the server side one's got to look at a few things, especially
quenching client options for
a) compression, since it means sysload
b) timeouts greater 5 minutes because of dos attacks
if your clients are fast, you can go below 30 secs with client timeouts,
i got a few russian peers which access my site as master, therefor it is
5 minutes.
-----BEGIN FILE rsyncd.conf-----
# rsyncd.conf -- [EMAIL PROTECTED] -- 20000320
# don't do any harm to the system
uid = nobody
gid = nogroup
use chroot = no
max connections = 40
pid file = /var/run/rsyncd.pid
motd file = /etc/rsyncd/motd
syslog facility = local7
read only = yes
timeout = 300
# quench clients' nasty options
dont compress = *
refuse options = timeout
# here go your [package] parameters...
[apache-ftp]
path = /ftp/pub/unix/apache
comment = [~0.3GB] Apache WWW Server FTP Archive
list = yes
[apache-www]
path = /meta/mirror/apache-www
comment = [~0.4GB] Apache WWW Server WWW Site
list = yes
#eof
-----END FILE rsyncd.conf-----
example output follows:
-----BEGIN EXAMPLE OUTPUT-----
[EMAIL PROTECTED]:datasink[~/bin/24h]24% rsync rsync://filepile.nacamar.net
###
### You reached
### ____________.____ ______________________.____ ___________
### \_ _____/_| | \_ _____/______ \_| | \_ _____/
### | __)| | | | __)_ | ___/ | | | __)_
### | \ | | |__| \| | | | |__| \
### \___ / |__|_______ \_____ /|____| |__|_______ \_____ /
### \/ \/ \/ .nacamar.net \/ \/
###
### ____ ____ __________ _____ ___ ____ ______
### / __ \/ __ `/ ___/ __ `/ __ `__ \/ __ `/ ___/
### / / / / /_/ / /__/ /_/ / / / / / / /_/ / /
### one world. one net. /_/ /_/\__,_/\___/\__,_/_/ /_/ /_/\__,_/_/ .
[...]
package [ size] description
=============== ======== ==============================================
apache-ftp [~0.3GB] Apache WWW Server FTP Archive
apache-www [~0.4GB] Apache WWW Server WWW Site
[...]
-----END EXAMPLE OUTPUT-----
i hope this answers some of the questions that came up in the last
months.
<shameless plug>
if you want to see a picture of the server carrying all rsync
im-/exports try
ftp://filepile.nacamar.net/info/
there you can see what i'm talking about, it has become quite
a big site blasting some gigs per day into the european countries:
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda1 3842376 521684 3125504 14% /
/dev/sda2 1968620 1049652 918968 53% /swap
/dev/sda3 11820088 1383248 9836412 12% /home
/dev/sdb1 89591724 6150600 83441124 7% /meta
/dev/sdc1 125431656 48284732 77146924 38% /ftp/.vol/vol0
/dev/sdd1 125431656 49909256 75522400 40% /ftp/.vol/vol1
/dev/sde1 125431656 68579468 56852188 55% /ftp/.vol/vol2
</shameless plug>
if you are interested in my collection of mirroring scripts including
all the lock/rsync/ftpmirror and log management, just drop me a mail
have fun & keep up the good work!
/k
--
> Coders do it with a routine.
karsten w. rohrbach - research engineer - dept. strategies & development
nacamar data communications - http://www.nacamar.de - http://www.nacamar.net
KR433/KR11-RIPE - PGP-KFP = F9 A0 DF 91 74 07 6A 1C 5F 0B E0 6B 4D CD 8C 44
