[rdiff-backup-users] Error for incremental backup on Mac

2013-06-10 Thread Sai kee Wong 
Got following error, I am not familiar with troubleshooting with
rdiff-backup, can anyone suggest some way for me to troubleshoot ?

Thanks in advance.

SK

Previous backup seems to have failed, regressing destination now.
Exception 'Bad index order: ('long_filename_data', '1') = ('ctse', 'Library', 
'Mail', 'V2', 'imap-cathypy...@gmail.com@imap.gmail.com', '[Gmail].mbox', 'All 
Mail.mbox', '05D20631-00E8-46C9-A4DC-3AC8C76717F5', 'Data', '2', '6', 
'Attachments', '62150')' raised of class 'type 'exceptions.AssertionError'':
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 306, in 
error_check_Main
   try: Main(arglist)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 326, in 
Main
   take_action(rps)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 282, in 
take_action
   elif action == backup: Backup(rps[0], rps[1])
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 339, in 
Backup
   backup_final_init(rpout)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 504, in 
backup_final_init
   checkdest_if_necessary(rpout)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 938, in 
checkdest_if_necessary
   dest_rp.conn.regress.Regress(dest_rp)
 File /Library/Python/2.7/site-packages/rdiff_backup/regress.py, line 71, in 
Regress
   for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
 File /Library/Python/2.7/site-packages/rdiff_backup/rorpiter.py, line 275, 
in __call__
   assert 0, Bad index order: %s = %s % (self.index, index)

Traceback (most recent call last):
 File /usr/local/bin/rdiff-backup, line 30, in module
   rdiff_backup.Main.error_check_Main(sys.argv[1:])
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 306, in 
error_check_Main
   try: Main(arglist)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 326, in 
Main
   take_action(rps)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 282, in 
take_action
   elif action == backup: Backup(rps[0], rps[1])
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 339, in 
Backup
   backup_final_init(rpout)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 504, in 
backup_final_init
   checkdest_if_necessary(rpout)
 File /Library/Python/2.7/site-packages/rdiff_backup/Main.py, line 938, in 
checkdest_if_necessary
   dest_rp.conn.regress.Regress(dest_rp)
 File /Library/Python/2.7/site-packages/rdiff_backup/regress.py, line 71, in 
Regress
   for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
 File /Library/Python/2.7/site-packages/rdiff_backup/rorpiter.py, line 275, 
in __call__
   assert 0, Bad index order: %s = %s % (self.index, index)
AssertionError: Bad index order: ('long_filename_data', '1') = ('ctse', 
'Library', 'Mail', 'V2', 'imap-cathypy...@gmail.com@imap.gmail.com', 
'[Gmail].mbox', 'All Mail.mbox', '05D20631-00E8-46C9-A4DC-3AC8C76717F5', 
'Data', '2', '6', 'Attachments', '62150')___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

[rdiff-backup-users] Incremental backup runs extremely long time

2012-12-20 Thread Sai kee Wong 
Further to my recent report on 2012-10-17 about the fix to make rdiff-backup
works on Mac OS X.  A backup was started on an iMac for 210GB of data, so
far so good, the initial backup was 3 hrs and incr backup around 15 min
which is reasonable for normal home user.

A month later, I started another backup on another iMac for 300GB of data.

Have the following observations:

(1) Discovered that the --exclude doesn't work if the directory doesn't
exist.

(2) Discovered that the --exclude doesn't work for wildcard like
   --exclude /**.Spotlight-V100 \
   --exclude /automount** \
   --exclude /cores** \
   --exclude /Network** \
   --exclude /private/tmp** \
   --exclude /private/var/vm** \
   --exclude /tmp** \
   --exclude /Volumes/** \
nor
   --exclude /tmp* \
but only works for
--exclude /**/.Spotlight-V100 \
--exclude /**/Downloads \
--exclude /**/.Trash \
--exclude /**/Library/Caches \
--exclude /**/tmp \

(3) The initial backup was 4 hrs (reasonable) but every incr backup
takes 10 hrs which is abnormal, and it runs at 100% of a CPU 
core (the mac has quad core Intel CPU). As a normal home user,
there is only less than 1GB changes everyday.  Following is
the session statistics.

StartTime 1355880519.00 (Wed Dec 19 09:28:39 2012)
EndTime 1355917466.22 (Wed Dec 19 19:44:26 2012)
ElapsedTime 36947.22 (10 hours 15 minutes 47.22 seconds)
SourceFiles 1178810
SourceFileSize 311932438922 (291 GB)
MirrorFiles 1178805
MirrorFileSize 311931930965 (291 GB)
NewFiles 35
NewFileSize 906566 (885 KB)
DeletedFiles 30
DeletedFileSize 144514 (141 KB)
ChangedFiles 338
ChangedSourceSize 160572782 (153 MB)
ChangedMirrorSize 160826877 (153 MB)
IncrementFiles 415
IncrementFileSize 7228028 (6.89 MB)
TotalDestinationSizeChange 7735985 (7.38 MB)
Errors 0

(4) During the initial backup, it says about 200 files with error like:

UpdateError Users/skwong/gtk/inst/bin/automake-1.10 Updated mirror temp 
file /Volumes/BK2/Users/skwong/gtk/inst/bin/rdiff-backup.tmp.889172 does not 
match source

but I can open the source file without problem.  And the 
statistics as shown above also says Errors 0.

During the incremental backup, same messages appear for those
files again.

(5) I rerun the whole backup from 0 again, then notice that it runs
only at 11% of a CPU core but the inactive memory grow from 0
to 10GB (until total of the 16GB are all used) within 10 min.

For (1) and (2), may be I mis-understand the operations about
rdiff-backup, and might not be a problem.  For (5), inactive memory
doesn't really matter as the system can reclaim it as needed, but
just wonder if that's normal.

For (3), that's really out of my expectation.  Not only it takes
2.5x more than the initial new backup time, if CPU usage is also
taking into consideration, then, that is 25x; but also that make
the backup not practical.  Also, as the backup on the other iMac
runs quite normal, I really doubt there may be problem in my files
which may trigger hidden bug.  Can anyone suggest on how to debug
the problem.

Also, don't know what's wrong with (4).

Thanks in advance.

SK
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki

Re: [rdiff-backup-users] OverflowError: signed integer is greater than maximum

2012-10-21 Thread Sai kee Wong 
Thanks for the reply.

Yes I did check the related pages.  And I tried to use the Python version
2.6 which I assume should not suffer from the problem.  But it does suffer.
At that time, I was using Mac OS 10.6 which is assumed to be 64 bits.
Until recently, test was done with Mac OS 10.8 with Python 2.7, still the
same problem.  So I made the fix.

B.Rgds

SK

On 21 Oct 2012, at 6:28 AM, Joe Steele wrote:

 It looks like you were encountering the python bug identified here:
 
 http://bugs.python.org/issue1747858
 
 Before their fix, the uid was being converted to a (32 bit) signed int, 
 which, as you have noted, isn't big enough.  After the fix, the uid is 
 converted to a long.  I don't know anything about Mac OS X, but on linux, a 
 long is either 32 bits or 64 bits, depending on the architecture.  Their fix 
 should obviously work on 64 bit systems.  (And on 32 bit systems, it's 
 probably unlikely that you will encounter 32 bit uids anyway.)
 
 It also looks like their fix was implemented in python v2.5.3+, so I am a 
 little puzzled as to why you were experiencing problems with the versions of 
 python that you mention, if indeed you are using a 64 bit version.
 
 In any case, it looks like you found a fix that works for you.
 
 --Joe
 
 
 On 10/17/2012 9:43 AM, Sai kee Wong wrote:
 After the orignal post, there is no feedback.  Recently, I tried to fix
 the problem.  I didn't know any about Python, started debugging by inserting
 print statement.  Figured out in the Mac, some times it has file with uid
 and gid set to 4294967294 (nobody).  It seems Python doesn't accept unsigned
 32-bits.  And it crashed when calling the os.chown() with this high value.
 
 I don't know how to make the os.chown() to accept unsigned int instead of
 signed 32-bits int.  So I searched the web and add a function to change
 too big value to -ve, in this case is -2.  Then it works.  As in previous
 posts, somebodies asked similar questions and got no helpful response;
 I published the solution here such that if anyone is suffering from the 
 problem
 can adopt the fix.
 
 Also, please remind me if I am doing something wrong.
 
 Thanks !
 
 SK
 
 Fix:
 
 After installed the rdiff-backup 1.3.3, find the file rpath.py
 In Mac, it is in
 
 /Library/Python/2.7/site-packages/rdiff_backup/rpath.py
 
 At line 977, the line was
 
 else: os.chown(self.path, uid, gid)
 
 change it to
 
 else: os.chown(self.path, int32(uid), int32(gid))
 
 At line 46, add following
 
 def int32(x):
 if x0x:
 raise OverflowError
 if x0x7FFF:
 x=int(0x1-x)
 if x2147483648:
 return -x
 else:
 return -2147483648
 return x
 
 On 11 Apr 2011, at 9:04 AM, Sai kee Wong wrote:
 
 I'm new to the rdiff-backup
 
 I'm running Mac OS 10.6.6, installed the rdiff-backup 1.3.3 and tried
 to backup 110GB of data, at around 60GB, it stopped with following error:
 
 Exception 'signed integer is greater than maximum' raised of class 'type 
 'exceptions.OverflowError'':
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, 
 in error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, 
 in Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, 
 in take_action
elif action == backup: Backup(rps[0], rps[1])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 348, 
 in Backup
backup.Mirror(rpin, rpout)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 38, 
 in Mirror
DestS.patch(dest_rpath, source_diffiter)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 240, 
 in patch
ITR(diff.index, diff)
  File /Library/Python/2.6/site-packages/rdiff_backup/rorpiter.py, line 
 281, in __call__
last_branch.fast_process(*args)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 533, 
 in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 563, 
 in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 180, 
 in copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 977, 
 in chown
else: os.chown(self.path, uid, gid)
 
 Traceback (most recent call last):
  File /usr/local/bin/rdiff-backup, line 30, in module
rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, 
 in error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, 
 in Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, 
 in take_action
elif action == backup: Backup(rps[0], rps[1])
  File

Re: [rdiff-backup-users] OverflowError: signed integer is greater than maximum

2012-10-17 Thread Sai kee Wong 
After the orignal post, there is no feedback.  Recently, I tried to fix
the problem.  I didn't know any about Python, started debugging by inserting
print statement.  Figured out in the Mac, some times it has file with uid
and gid set to 4294967294 (nobody).  It seems Python doesn't accept unsigned
32-bits.  And it crashed when calling the os.chown() with this high value.

I don't know how to make the os.chown() to accept unsigned int instead of
signed 32-bits int.  So I searched the web and add a function to change
too big value to -ve, in this case is -2.  Then it works.  As in previous
posts, somebodies asked similar questions and got no helpful response;
I published the solution here such that if anyone is suffering from the problem
can adopt the fix.

Also, please remind me if I am doing something wrong.

Thanks !

SK

Fix:

After installed the rdiff-backup 1.3.3, find the file rpath.py
In Mac, it is in

/Library/Python/2.7/site-packages/rdiff_backup/rpath.py

At line 977, the line was

else: os.chown(self.path, uid, gid)

change it to

else: os.chown(self.path, int32(uid), int32(gid))

At line 46, add following

def int32(x):
if x0x:
raise OverflowError
if x0x7FFF:
x=int(0x1-x)
if x2147483648:
return -x
else:
return -2147483648
return x

On 11 Apr 2011, at 9:04 AM, Sai kee Wong wrote:

 I'm new to the rdiff-backup
 
 I'm running Mac OS 10.6.6, installed the rdiff-backup 1.3.3 and tried
 to backup 110GB of data, at around 60GB, it stopped with following error:
 
 Exception 'signed integer is greater than maximum' raised of class 'type 
 'exceptions.OverflowError'':
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, in 
 error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, in 
 Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, in 
 take_action
elif action == backup: Backup(rps[0], rps[1])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 348, in 
 Backup
backup.Mirror(rpin, rpout)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 38, in 
 Mirror
DestS.patch(dest_rpath, source_diffiter)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 240, 
 in patch
ITR(diff.index, diff)
  File /Library/Python/2.6/site-packages/rdiff_backup/rorpiter.py, line 281, 
 in __call__
last_branch.fast_process(*args)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 533, 
 in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 563, 
 in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 180, in 
 copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 977, in 
 chown
else: os.chown(self.path, uid, gid)
 
 Traceback (most recent call last):
  File /usr/local/bin/rdiff-backup, line 30, in module
rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, in 
 error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, in 
 Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, in 
 take_action
elif action == backup: Backup(rps[0], rps[1])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 348, in 
 Backup
backup.Mirror(rpin, rpout)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 38, in 
 Mirror
DestS.patch(dest_rpath, source_diffiter)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 240, 
 in patch
ITR(diff.index, diff)
  File /Library/Python/2.6/site-packages/rdiff_backup/rorpiter.py, line 281, 
 in __call__
last_branch.fast_process(*args)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 533, 
 in fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 563, 
 in patch_to_temp
rpath.copy_attribs(diff_rorp, new)
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 180, in 
 copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 977, in 
 chown
else: os.chown(self.path, uid, gid)
 OverflowError: signed integer is greater than maximum
 
 Believe it should be problem with the 64 bits OS. Tried python 2.6.1 
 and 2.6.6 both give above error. Tried also 2.7.1 but gives
 Segmentation fault
 
 Search through the rdiff-backup-users Archives but couldn't find
 any cue to solve the problem.
 
 Thanks in advance

[rdiff-backup-users] OverflowError: signed integer is greater than maximum

2011-04-10 Thread Sai kee Wong 
I'm new to the rdiff-backup

I'm running Mac OS 10.6.6, installed the rdiff-backup 1.3.3 and tried
to backup 110GB of data, at around 60GB, it stopped with following error:

Exception 'signed integer is greater than maximum' raised of class 'type 
'exceptions.OverflowError'':
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, in 
error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, in 
Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, in 
take_action
elif action == backup: Backup(rps[0], rps[1])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 348, in 
Backup
backup.Mirror(rpin, rpout)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 38, in 
Mirror
DestS.patch(dest_rpath, source_diffiter)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 240, in 
patch
ITR(diff.index, diff)
  File /Library/Python/2.6/site-packages/rdiff_backup/rorpiter.py, line 281, 
in __call__
last_branch.fast_process(*args)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 533, in 
fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 563, in 
patch_to_temp
rpath.copy_attribs(diff_rorp, new)
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 180, in 
copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 977, in 
chown
else: os.chown(self.path, uid, gid)

Traceback (most recent call last):
  File /usr/local/bin/rdiff-backup, line 30, in module
rdiff_backup.Main.error_check_Main(sys.argv[1:])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 306, in 
error_check_Main
try: Main(arglist)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 326, in 
Main
take_action(rps)
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 282, in 
take_action
elif action == backup: Backup(rps[0], rps[1])
  File /Library/Python/2.6/site-packages/rdiff_backup/Main.py, line 348, in 
Backup
backup.Mirror(rpin, rpout)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 38, in 
Mirror
DestS.patch(dest_rpath, source_diffiter)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 240, in 
patch
ITR(diff.index, diff)
  File /Library/Python/2.6/site-packages/rdiff_backup/rorpiter.py, line 281, 
in __call__
last_branch.fast_process(*args)
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 533, in 
fast_process
if self.patch_to_temp(mirror_rp, diff_rorp, tf):
  File /Library/Python/2.6/site-packages/rdiff_backup/backup.py, line 563, in 
patch_to_temp
rpath.copy_attribs(diff_rorp, new)
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 180, in 
copy_attribs
rpout.chown(*rpout.conn.user_group.map_rpath(rpin))
  File /Library/Python/2.6/site-packages/rdiff_backup/rpath.py, line 977, in 
chown
else: os.chown(self.path, uid, gid)
OverflowError: signed integer is greater than maximum

Believe it should be problem with the 64 bits OS. Tried python 2.6.1 
and 2.6.6 both give above error. Tried also 2.7.1 but gives
Segmentation fault

Search through the rdiff-backup-users Archives but couldn't find
any cue to solve the problem.

Thanks in advance for help.
B.rgds

SK
___
rdiff-backup-users mailing list at rdiff-backup-users@nongnu.org
http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki