Recently, a system on which I administer backups remotely has not been able to complete a run of rdiff-backup (the data set is 500 GB but the backup is completing less than 15% before failing). This is the case even after I remove the entire directory and retry the backup; in fact, it started after a server upgrade in which the RAID array was replaced and the backups were to be restored from the original (there are also off-site backups). In fact, we have never been able to get a full backup since the array was replaced. What appears to happen is that the backup chugs along fine for a time, with some significant memory usage mainly on the server but no obvious leaks. At some point the backup starts getting slow and then it hangs altogether for a while before dying with the memory error listed below. Running "top" shows less memory usage on the client than the server until around when the error happens, then the usage balloons on the client while the server process appears to be idle. It takes about the same amount of time each try for the backup to fail, and excluding the directory where it normally fails simply results in it failing elsewhere. A line from top output on the client:
PID UID REG RPRVT RSHRD RSIZE VPRVT VSIZE TH PRT TIME %CPU 24908 0 4765 1.25G+ 972K 1.07G+ 1.28G+ 1.28G+ 1 27 2h08m 90.4 This error occurred both with our on-site version (hacked to sort the metadata case insensitive) and with stock 1.1.5, deleting all metadata in between tests. The only differences between a couple months of error-free backups and these latest attempts are that the disk array on the server was replaced and the server OS was upgraded to a new point release of Mac OS X Tiger, although some other work may have been done that I'm not aware of. There is no indication that any hardware failure is occurring and there is plenty of disk space available on the server. The following error is when trying to do a backup run with stock rdiff-backup 1.1.5, all metadata (the rdiff-backup-data directory) removed in advance. Any help figuring out what is going on here would be appreciated, as I've worked with the software and hacked around issues we've had for a little while, but this has me stumped. Exception '' raised of class 'exceptions.MemoryError': File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error try: return function(*args) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 878, in listdir return self.conn.os.listdir(self.path) Sending back exception of type exceptions.MemoryError: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 335, in answer_request result = apply(eval(request.function_string), argument_list) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 483, in readfromid if length is None: return cls.vfiles[id].read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 302, in read if not self.addtobuffer(): break File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 325, in addtobuffer try: currentobj = self.iter.next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 342, in next next_elem = self.iter.next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py", line 132, in Iterate_fast try: rpath, val = diryield_stack[-1].next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py", line 118, in diryield for filename in self.listdir(rpath): File "/usr/local/lib/python2.3/site-packages/rdiff_backup/selection.py", line 180, in listdir dir_listing = robust.check_common_error(error_handler, dir_rp.listdir) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/robust.py", line 32, in check_common_error try: return function(*args) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 878, in listdir return self.conn.os.listdir(self.path) Sending back exception of type exceptions.MemoryError: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 335, in answer_request result = apply(eval(request.function_string), argument_list) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 483, in readfromid if length is None: return cls.vfiles[id].read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 302, in read if not self.addtobuffer(): break File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 325, in addtobuffer try: currentobj = self.iter.next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py", line 166, in get_sigs for src_rorp, dest_rorp in cls.CCPP: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py", line 306, in next source_rorp, dest_rorp = self.iter.next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 103, in Collate2Iters for relem1 in riter1: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 382, in next while not type: type, data = self._get() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 414, in _get if not self.buf: self.buf += self.file.read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 516, in read return self.connection.VirtualFile.readfromid(self.id, length) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 448, in __call__ return apply(self.connection.reval, (self.name,) + args) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 370, in reval if isinstance(result, Exception): raise result Sending back exception of type exceptions.MemoryError: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 335, in answer_request result = apply(eval(request.function_string), argument_list) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 483, in readfromid if length is None: return cls.vfiles[id].read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 302, in read if not self.addtobuffer(): break File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 325, in addtobuffer try: currentobj = self.iter.next() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py", line 103, in get_diffs for dest_sig in dest_sigiter: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 384, in next elif type == "r": return self.get_rorp(data) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 394, in get_rorp rorp.setfile(self.get_file()) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 399, in get_file type, data = self._get() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 414, in _get if not self.buf: self.buf += self.file.read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 516, in read return self.connection.VirtualFile.readfromid(self.id, length) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 448, in __call__ return apply(self.connection.reval, (self.name,) + args) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 370, in reval if isinstance(result, Exception): raise result Sending back exception of type exceptions.MemoryError: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 335, in answer_request result = apply(eval(request.function_string), argument_list) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/backup.py", line 216, in patch for diff in rorpiter.FillInIter(source_diffiter, dest_rpath): File "/usr/local/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 177, in FillInIter for rp in rpiter: File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 382, in next while not type: type, data = self._get() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 414, in _get if not self.buf: self.buf += self.file.read() File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 516, in read return self.connection.VirtualFile.readfromid(self.id, length) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 448, in __call__ return apply(self.connection.reval, (self.name,) + args) File "/usr/local/lib/python2.3/site-packages/rdiff_backup/connection.py", line 370, in reval if isinstance(result, Exception): raise result _______________________________________________ rdiff-backup-users mailing list at [email protected] http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
