Hello all,
At the expensive of slightly increased diff blocks, I propose we align
the librsync blocksize to be RAID-stripe-size friendly.
Currently, rdiff-backup uses a librsync blocksize of "16-byte aligned
square_root(file_length)".
For large files, square_root returns values that are not condusive to
RAID stripe sizes. For example, a 5GB file would have a block size of
73264.
The attached patch forces the librsync blocksize to multiples of
Globals.blocksize (which is currently 128k) for files > 1GB.
Please let me know what you think, I'm open to comments and suggestions.
-Eric
--
Eric Wheeler
President
eWheeler, Inc.
dba Global Linux Security
www.GlobalLinuxSecurity.pro
503-330-4277
PO Box 14707
Portland, OR 97293
Index: Rdiff.py
===================================================================
RCS file: /sources/rdiff-backup/rdiff-backup/rdiff_backup/Rdiff.py,v
retrieving revision 1.15
diff -u -r1.15 Rdiff.py
--- Rdiff.py 4 Nov 2005 22:41:13 -0000 1.15
+++ Rdiff.py 3 Jan 2011 03:27:12 -0000
@@ -26,8 +26,8 @@
def get_signature(rp, blocksize = None):
"""Take signature of rpin file and return in file object"""
if not blocksize: blocksize = find_blocksize(rp.getsize())
- log.Log("Getting signature of %s with blocksize %s" %
- (rp.get_indexpath(), blocksize), 7)
+ log.Log("Getting signature of %s with blocksize %s, filesize=%s" %
+ (rp.get_indexpath(), blocksize, rp.getsize()), 7)
return librsync.SigFile(rp.open("rb"), blocksize)
def find_blocksize(file_len):
@@ -39,8 +39,13 @@
"""
if file_len < 4096: return 64 # set minimum of 64 bytes
- else: # Use square root, rounding to nearest 16
+ elif file_len < 1024*1024*1024:
+ # Use square root, rounding to nearest 16 for files less than 1GB
return long(pow(file_len, 0.5)/16)*16
+ else:
+ # Use a blocksize-multiple for files >1 GB (this is great for RAID),
+ # and align to 16 bytes (though it probably already is)
+ return long( long((pow(file_len, 0.5)/Globals.blocksize)+1)*Globals.blocksize / 16 ) * 16
def get_delta_sigfileobj(sig_fileobj, rp_new):
"""Like get_delta but signature is in a file object"""
_______________________________________________
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
