Wayne,
Thanks for the prompt response.
The issue is around performance. We have a backup area containing 4.5M
inodes in each backup, of which around 0.5M to 1M are directories.
Backing up to a new directory each time is far slower than backing up to
an almost completely filled directory. Deleting a directory with 4.5M
inodes is non-trivial too.
(We are running on an SGI Altix 4700 with 128 cores, 500 Gbyte of
memory, with XFS file systems running the Data Migration Facility
product. The target file systems for holding backups are on cheap and
slow SATA disc - we see at best 10,000 inode deletions per second, and
this can drop to fewer than 100 per second when the file systems are
busy with just a few housekeeping tasks.).
Starting with a new directory structure each time, so building 4.5M
inodes, and then destroying a similar directory and file structure which
is almost correct is very wasteful.
We do have a workaround, which keeps the hard-links numbers at nearly
their maximum level, but this involves a scan of the directory prior to
receiving each rsync and hence a performance penalty. We would rather
avoid this.
It seems to us that there is a good case for the enhanced functionality.
"When a file is found in the destination which should be replaced by one
in the source, look in the --link-dest directory first for a candidate,
and hard-link that in preference to doing a copy from source to
destination."
Thanks
Regards
Rob. Bell e-mail: [email protected]
--
Dr Robert C. Bell
CSIRO Advanced Scientific Computing, Technical Services Manager
Chief Technology Officer, Bureau of Meteorology / CSIRO
High Performance Computing and Communications Centre (HPCCC)
Street: HPCCC Level 11, 700 Collins Street, Docklands Vic 3008, Australia
Postal: HPCCC Level 11, GPO Box 1289, Melbourne Vic 3001, Australia
Phone +61 3 9669 8102, fax +61 3 9669 8112, mobile 0428 108 333, CSIRO 93 3810
http://www.hpsc.csiro.au/ http://www.hpccc.gov.au/
On Mon, 22 Dec 2008, Wayne Davison wrote:
Date: Mon, 22 Dec 2008 20:56:09 -0800
From: Wayne Davison <[email protected]>
To: Robert Bell <[email protected]>
Cc: [email protected]
Subject: Re: rsync --link-dest option with the destination directory
containing old files.
Resent-Date: Tue, 23 Dec 2008 15:56:17 +1100 (EST)
Resent-From: <[email protected]>
On Tue, Dec 23, 2008 at 10:22:42AM +1100, Robert Bell wrote:
we would like rsync to use a hard-link from ../backup.1 in
preference to a new copy from source/ to backup.0/ .
That's not something that I plan to implement. I'd suggest changing
your naming to always use a new directory, which will allow you to
remove the oldest backup directory at your leisure.
..wayne..
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
