Kyle wrote:
Ok,
a couple of responses thus far. Some further info.
The software I can tune myself. I was more looking for Linux specific
tuning.
* Yes, I was/am concerned about I/O.
* But also ensuring the OS itself (system processes) is not hindering
anything otherwise.
* The RAID is the storage medium. (Hardware RAID)
* Incremental change analysis is done client side.
* Dual P4's / 1GB RAM
* Filesys is ext3 mounted with 'defaults'
You've chosen *the* application which most stresses the operating system :-)
Cut the problem into three
- tune the disk
- tune the network
- tune the backup software.
Disk:
- you are writing large files.
- RAID5 is not your friend, why not RAID10 since disk is
so cheap?
- some filesystems do big files better than others (xfs > ext3)
- you need all spindles running under the same load, so layout
your disks that that in mind. You'll probably need four spindles
running to ensure that the average write speed exceeds the maximum
read speed of the clients. Test this -- the client should not
stall.
- you are not reading
- caching gains you little, so adjust the weighting so caches are
cleared down more agressively
- discard metadata uselessness (such as atime).
- kill all low value disk-using processes (such as Beagle, slocate
and other such rubbish, typically run from cron).
- The stripe sizes used to build the RAID should be unusually large
and should mesh well with the filesystem's extents.
Network:
- set autotuning for the bandwidth-display product. A reasonable
reference is:
http://www.gdt.id.au/~gdt/presentations/2008-01-29-linuxconfau-tcptune/
- use jumbo frames (9000B packet > 8KB disk block, so very efficient)
- avoid firewalls and other bogusness
- check every counter on every host/switch/router for errors. You
need zero errors.
Note the contention between network and disk I/O buffers. These both
need low memory. A 32b OS only has 512MB of that, which is a fail for
this application (especially since Linux locks hard on kernel memory
fragmentation). You need a 64b install.
Do the math (which depends on the number of clients), but I think you'll
find that 1GB of RAM won't be sufficient and you'll run out of cache
before you run out of filesystem bandwidth.
Backup software:
- chain backups, so only one/two client is running at a time.
- avoid rate limiting, it's more efficient for one or two clients
racing to the finish rather than have 30 clients all talking
slowly.
- set any block sizes way big.
- work out how the indexing works. Move that off the main
backup spindles, so that index updates don't move the
disk heads on the backup spindles.
Of course, all this needs to be taken with a grain of salt. There's
a world of difference between tuning small backup server (where you
just want things to complete overnight) and a corporate backup
server (where you are more interested in how many clients each
machine can back up per night).
Finally, what is your offsite strategy? If you're ejecting diskpacks
then note that not all chassis are rated to continually do this.
Worse still your diskpacks may not fit into a borrowed chassis.
Better to use a third-party container and keep a spare container
chassis offsite with the diskpacks. Also some backup software needs
a full scan of all diskpacks if it the software is asked to do a
disaster recovery and this can take a long time.
--
Glen Turner
--
SLUG - Sydney Linux User's Group Mailing List - http://slug.org.au/
Subscription info and FAQs: http://slug.org.au/faq/mailinglists.html
