[BackupPC-users] What does BackupPC_link got error -4 when calling MakeFileLink mean?

2008-10-30 Thread Jeffrey J. Kosowsky
I just got a slew of such errors:
BackupPC_link got error -4 when calling 
MakeFileLink(/var/lib/BackupPC//pc/mypc/5/fc/fcygwin/fusr/attrib, 
530bbf3350acfd3d1ce483619f9b47d0, 1)

I traced it back to the subroutine MakeFileLink, but the documentation
only details the positive return numbers and says Returns negative on
error.

So what does -4 mean and what can cause it?

Thanks


-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] What does BackupPC_link got error -4 when calling MakeFileLink mean?

2008-10-30 Thread Craig Barratt
Jeffrey writes:

 So what does -4 mean and what can cause it?

Fails to make a hardlink.  Several possible reasons: you are out
of inodes, your cpool and and pc directory are on different file
systems, your BackupPC file system doesn't support hardlinks, or
you have a permissions problem of some kind.

Craig

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] What does BackupPC_link got error -4 when calling MakeFileLink mean?

2008-10-30 Thread Jeffrey J. Kosowsky
Craig Barratt wrote at about 00:07:51 -0700 on Thursday, October 30, 2008:
  Jeffrey writes:
  
   So what does -4 mean and what can cause it?
  
  Fails to make a hardlink.  Several possible reasons: you are out
  of inodes, your cpool and and pc directory are on different file
  systems, your BackupPC file system doesn't support hardlinks, or
  you have a permissions problem of some kind.
  
  Craig

Interesting but...
1. The system supports hardlinks
   (I can create them manually and my pool and backups are full of hardlinks)
   I even was able to create hardlinks to the pool members and files
   that the errors occurred on.

2. There are plenty of inodes
   Filesystem  Inodes  Used Available Use%
   /dev/md0  60989440848948  60140492   1%

3. Permissions are unchanged and seem fine
   Topdir and everything directly in it are: backuppc.root
   All files/directories below it are: backuppc.backuppc
   Permissions are rw for files and rwx for directories

Any suggestions on how to troubleshoot this further?
(the only possible complication I can think of is that I am doing this
over nfs but not sure why I would get these errors...)

Thanks!!

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
I have found a number of files in my pool that have the same checksum
(other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
has a few links to it by the way.

Why is this happening? 
  Isn't this against the whole theory of pooling.  It also doesn't seem
  to get cleaned up by BackupPC_nightly since that has run several times
  and the pool files are now several days old.

What can I do to clean it up?
  Is there a script that goes through looking for identical checksum
  pool files that have the same content and then coalesces them all
  into one inode.

Thanks!

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] Backup aborts without apparent reason

2008-10-30 Thread Anian Wurzenberger
Hi people,
I´m getting some strange Error on Fedora 6 (Linux fileserver 2.6.20-1.2948.fc6 
#1 SMP Fri Apr 27 18:53:15 EDT 2007 i686 i686 i386 GNU/Linux) with 
BackupPC-3.1.0 . SSH is working without pw prompt. The files arrive fine (I 
have never checked if they are indeed complete, but rsync doesn´t complain, so 
it´s probably fine), I can even restore them, but I am getting Backup aborted 
(). Also there is a permission error in the global log which shouldn´t be 
there, because the folder already exists and permissions of the folder are even 
777 (of course only for testing purposes).


Part of the Backup-Log (can send more if needed):

full backup started for directory /etc/sysconfig
Running: /usr/bin/ssh -q -x -l root somepc /usr/bin/rsync --server --sender 
--numeric-ids --perms --owner --group -D --links --hard-links --times 
--block-size=2048 --recursive --ignore-times . /etc/sysconfig/ ...
Finished deltaGet phase 1
pollChild()
Parent read: stats 2286 5329 176658 0 ('errorCnt' = 0,'ExistFileSize' = 
0,'ExistFileCnt' = 1,'TotalFileCnt' = 78,'ExistFileCompSize' = 
0,'TotalFileSize' = 176658) Got stats: 2286 5329 176658 0 ('errorCnt' = 
0,'ExistFileSize' = 0,'ExistFileCnt' = 1,'TotalFileCnt' = 
78,'ExistFileCompSize' = 0,'TotalFileSize' = 176658)
pollChild()
Parent read: log Sending: 
Sending: 
pollChild()
Parent read: log attribWrite(dir=f%2fetc%2fsysconfig) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new/f%2fetc%2fsysconfig/attrib
attribWrite(dir=f%2fetc%2fsysconfig) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new/f%2fetc%2fsysconfig/attrib
pollChild()
Parent read: log attribWrite(dir=) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new//attrib
attribWrite(dir=) - /data/mirrored/mlb/!Backup/BackupPC/pc/tms/new//attrib
pollChild()
Parent read: exit
Got exit from child
Done: 78 files, 176658 bytes
Backup aborted ()
Not saving this as a partial backup since it has fewer files than the prior one 
(got 78 and 78 files versus 78)

Part of the global Log:
2008-09-30 05:00:00 Next wakeup is 2008-09-30 06:00:00 2008-09-30 05:00:01 
Started full backup on somepc (pid=19683, share=/etc/sysconfig) 2008-09-30 
05:00:08 Backup failed on somepc () 2008-09-30 05:00:08 Running BackupPC_link 
somepc (pid=19717) 2008-09-30 05:00:08 somepc: mkdir 
/data/mirrored/mlb/\!Backup/BackupPC: Permission denied at 
/usr/local/BackupPC/lib/BackupPC/Lib.pm line 899 2008-09-30 05:00:08 Finished 
somepc (BackupPC_link tms)


I hope somebody can help me...

Thanks
Anian W.
-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] What does BackupPC_link got error -4 when calling MakeFileLink mean?

2008-10-30 Thread Holger Parplies
Hi,

Jeffrey J. Kosowsky wrote on 2008-10-30 03:41:39 -0400 [Re: [BackupPC-users] 
What does BackupPC_link got error -4 when calling MakeFileLink mean?]:
 Craig Barratt wrote at about 00:07:51 -0700 on Thursday, October 30, 2008:
   Jeffrey writes:
   
So what does -4 mean and what can cause it?
   
   Fails to make a hardlink. [...]
 
 Any suggestions on how to troubleshoot this further?

well, in your case stale NFS file handle somehow springs to mind ... could
it be that?

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Backup aborts without apparent reason

2008-10-30 Thread Holger Parplies
Hi,

Anian Wurzenberger wrote on 2008-10-30 09:16:18 +0100 [[BackupPC-users] Backup 
aborts without apparent reason]:
 Hi people,
 I´m getting some strange Error on Fedora 6 (Linux fileserver
 2.6.20-1.2948.fc6 #1 SMP Fri Apr 27 18:53:15 EDT 2007 i686 i686 i386
 GNU/Linux) with BackupPC-3.1.0 . [...] I am getting Backup aborted ().
 Also there is a permission error in the global log which shouldn´t be there,
 because the folder already exists and permissions of the folder are even 777
 (of course only for testing purposes).
 [...]
 Part of the global Log:
 [...]
 2008-09-30 05:00:08 Running BackupPC_link somepc (pid=19717)
 2008-09-30 05:00:08 somepc: mkdir /data/mirrored/mlb/\!Backup/BackupPC: 
 Permission denied at /usr/local/BackupPC/lib/BackupPC/Lib.pm line 899

I don't know about the Backup aborted, but for mkpath failing I would
suspect misquoting the exclamation mark in the path to your TopDir to be
responsible. I can't really trace it down right now, but I tend to wonder why
anyone would put shell metacharacters in their TopDir - it's asking for
trouble, successfully as it appears. The same applies to long lines in emails
and missing line breaks in quoted log file excerpts ;-).

Of course any path for TopDir *should* work, as long as the OS accepts it, but
there are choices more likely to trigger bugs than others ...

I don't know if the Backup aborted is related to your choice of TopDir as
well. Are you backing up more than one host? Could you try out things like
changing the XferMethod or your TopDir?

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Tino Schwarze
Hi Jeffrey,

On Thu, Oct 30, 2008 at 03:55:16AM -0400, Jeffrey J. Kosowsky wrote:

 I have found a number of files in my pool that have the same checksum
 (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
 has a few links to it by the way.

That's intentional - what are the link counts for the files? 
If you look at BackupPC's status page, there is a line like:

* Pool hashing gives 649 repeated files with longest chain 28, 

 Why is this happening? 
   Isn't this against the whole theory of pooling.  It also doesn't seem
   to get cleaned up by BackupPC_nightly since that has run several times
   and the pool files are now several days old.

Because there is a file-system dependent limit to the number of hard
links a file may have. Look at $Conf{HardLinkMax} in config.pl.

Hm. I just took a look in my cpool and found some files which didn't
hit the hardlink count yet, but have a _0 and _1:
.../cpool/0/0 # ls -l c/00cd83be1ea3c1ffa3c6af2f4e310206* 
-rw-r- 4371 backuppc users 34 2005-01-14 17:01 
c/00cd83be1ea3c1ffa3c6af2f4e310206 
-rw-r- 3536 backuppc users 34 2005-03-02 02:22 
c/00cd83be1ea3c1ffa3c6af2f4e310206_0 
-rw-r-  439 backuppc users 34 2006-03-11 02:04 
c/00cd83be1ea3c1ffa3c6af2f4e310206_1 

MD5Sums are not equal for all files, so maybe something got corrupted
(or I updated BackupPC during the time - the files are rather old!):
.../cpool/0/0 # md5sum c/00cd83be1ea3c1ffa3c6af2f4e310206*
51ef559d1d7fa02c05fa032729c85804  c/00cd83be1ea3c1ffa3c6af2f4e310206
51ef559d1d7fa02c05fa032729c85804  c/00cd83be1ea3c1ffa3c6af2f4e310206_0
7e2276750fc478fa30142aa808df2a1f  c/00cd83be1ea3c1ffa3c6af2f4e310206_1

AFAIK, I started with $Conf{HardLinkMax} set to 32.000. As the files are
very old, a lot of links might have expired already.

I'm not sure though, how the file name is derived, I found another file
with same name but different MD5 sum:
.../cpool/0/0 # md5sum 8/0084734e7242df0fbc186ba6741d1bab*
db224998946bac7859f2448f41c58f88  8/0084734e7242df0fbc186ba6741d1bab
d1d8f3a86ae5492de0bf11f5cfb45860  8/0084734e7242df0fbc186ba6741d1bab_0

IIRC, BackupPC_nightly should perform chain cleaning.

Tino.

-- 
What we nourish flourishes. - Was wir nähren erblüht.

www.lichtkreis-chemnitz.de
www.craniosacralzentrum.de

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Holger Parplies
Hi,

Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 [[BackupPC-users] 
Duplicate files in pool with same CHECKSUM and same CONTENTS]:
 I have found a number of files in my pool that have the same checksum
 (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
 has a few links to it by the way.
 
 Why is this happening? 

presumably creating a link sometimes fails, so BackupPC copies the file,
assuming the hard link limit has been reached. I suspect problems with your
NFS server, though not a stale NFS file handle in this case, since copying
the file succeeds. Strange.

   Isn't this against the whole theory of pooling.

Well, yes :). But the action of copying the file when the method to implement
pooling (hard links) does not work for some reason (max link count reached, or
NFS file server errors if you think about it - you *do* get some level of
pooling; otherwise you'd have an independant copy or a missing file each time
linking fails) is perfectly reasonable.

   It also doesn't seem
   to get cleaned up by BackupPC_nightly since that has run several times
   and the pool files are now several days old.

BackupPC_nightly is not supposed to clean up that situation. It could be
designed to do so (the situation may arise when a link count overflow is
resolved by expired backups), but it would be horribly inefficient: for the
file to be eliminated, you would have to find() every occurrence of the inode
in all pc/* trees and replace them with links to the duplicate(s) to be kept.
You don't want that.

 What can I do to clean it up?

Fix your NFS server? :) Is there a consistent maximum number of links, or do
the copies seem to happen randomly? Honestly, I don't think the savings you
may gain from storing the pool over NFS are worth the headaches. What is
cheaper about putting a large disk into a NAS device than into your BackupPC
server? Well, yes, you can share it ... how about exporting part of the disk
from the BackupPC server (I would still recommend distinct partitions)?

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Holger Parplies
Hi,

Tino Schwarze wrote on 2008-10-30 11:13:27 +0100 [Re: [BackupPC-users] 
Duplicate files in pool with same CHECKSUM and same CONTENTS]:
 [...]
 Hm. I just took a look in my cpool and found some files which didn't
 hit the hardlink count yet, but have a _0 and _1:
 .../cpool/0/0 # ls -l c/00cd83be1ea3c1ffa3c6af2f4e310206* 
 -rw-r- 4371 backuppc users 34 2005-01-14 17:01 
 c/00cd83be1ea3c1ffa3c6af2f4e310206 
 -rw-r- 3536 backuppc users 34 2005-03-02 02:22 
 c/00cd83be1ea3c1ffa3c6af2f4e310206_0 
 -rw-r-  439 backuppc users 34 2006-03-11 02:04 
 c/00cd83be1ea3c1ffa3c6af2f4e310206_1 
 
 MD5Sums are not equal for all files,

that's intentional :-). Those files have different content but hash to the
same BackupPC hash. Quoting you:

 If you look at BackupPC's status page, there is a line like:
 
 * Pool hashing gives 649 repeated files with longest chain 28, 

That is what this line is about - you have up to 28 different files hashing to
the same BackupPC hash (some of these may coincidentally have identical
content due to link count overflows, but that would be the exception).

 AFAIK, I started with $Conf{HardLinkMax} set to 32.000. As the files are
 very old, a lot of links might have expired already.

True, but keep in mind how much 32000 really is. Unless you have many files
with identical content in your backup set (CVS/Root maybe), it will take very
many backups to reach so many links.

 I'm not sure though, how the file name is derived,

It's in the docs. Up to 256 KB of file contents (from the first 1 MB) and the
file length are taken into account, so it's quite easy to produce hash clashes
if you want to: take a file  1 MB and change the last byte. BackupPC resolves
them and they're probably infrequent enough not to be a problem (and you get
to see whether they are on the status page). Taking the length (of the
uncompressed file) into account avoids things like growing logfiles from
causing problems.

 IIRC, BackupPC_nightly should perform chain cleaning.

Unused files (i.e. link count = 1) are removed and chains renumbered. Like I
wrote, relinking identical files does not make sense.

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] Problem with CPOOL

2008-10-30 Thread Pedro M. S. Oliveira
Hi guy,
Apparently i did something really stupid, forgeting the way backuppc works and 
after a few hour struggling with another problem, in the middle of the night 
and a bit tired my brain had this great idea... why not speeding up the linking 
process by putty cpool in another fs?
i know all of you will say that can't be done because of the hard links and so 
on, while moving the data i remembered that and stop the whole process. then 
moved back the data to the original cpool dir and fs. the prob is that i didn't 
move it like hard links but as files. 
since then i can't do backups for two of my machines.
what can i do to solve this? should i erase all the backups for those machines 
and start over, can i just delete the cpool and wait it to be recreated? i 
don't mind losing those backups but if i don't have a chance no1 will die.
thanx
pedro


-- 
--
Pedro Oliveira
IT Consultant 
Email: [EMAIL PROTECTED]  
URL:   http://pedro.linux-geex.com
Telefone: +351 96 5867227
--
-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Tino Schwarze
Hi Holger,

On Thu, Oct 30, 2008 at 12:11:43PM +0100, Holger Parplies wrote:

  I'm not sure though, how the file name is derived,
 
 It's in the docs. Up to 256 KB of file contents (from the first 1 MB) and the
 file length are taken into account, so it's quite easy to produce hash clashes
 if you want to: take a file  1 MB and change the last byte. BackupPC resolves
 them and they're probably infrequent enough not to be a problem (and you get
 to see whether they are on the status page). Taking the length (of the
 uncompressed file) into account avoids things like growing logfiles from
 causing problems.

Thank you for the clarification!

Tino, hever having bothered about that before. ;-)

-- 
What we nourish flourishes. - Was wir nähren erblüht.

www.lichtkreis-chemnitz.de
www.craniosacralzentrum.de

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Tino Schwarze
Apropos link count, I just did a quick check of my pool. Here are the
top linked files:

-rw-r- 987537 backuppc users 359 2007-05-19 23:43 
./0/d/1/0d16a8f0ce1b516044a3f015b7d5ee06
-rw-r- 437446 backuppc users 98 2007-02-07 03:21 
./b/c/8/bc891581e99fb3729ea3d239a52d2b9a
-rw-r- 340062 backuppc users 98 2007-12-22 02:50 
./6/5/9/659e6651b59c8d8de4ffacdb9a27eb9f
-rw-r- 266646 backuppc users 122 2007-12-22 10:15 
./c/e/a/ceaf858b5f9ef4fdbd1b2132a9d8b14e

So almost one million links for... *drum roll* our CVS commit message template!
2nd place got *drum roll* a CVS/Tag file.
And the third is... a CVS/Root.
The fourth is another CVS/Root still featuring a quarter million links.

Bye,

Tino.

-- 
What we nourish flourishes. - Was wir nähren erblüht.

www.lichtkreis-chemnitz.de
www.craniosacralzentrum.de

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] incremental backups taking longer than full

2008-10-30 Thread Mark Maciolek
hi,

Using rsync between two linux servers, the full took 2.5 hours, the 
incremental backups are taking longer each day.

2008-10-21 23:00:01 full backup started for directory /
2008-10-22 02:32:55 full backup 0 complete, 294903 files, 203401100538 
bytes, 28 xferErrs (0 bad files, 0 bad shares, 28 other)
2008-10-22 23:00:00 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-23 04:59:40 incr backup 1 complete, 541 files, 170443233887 
bytes, 11 xferErrs (0 bad files, 0 bad shares, 11 other)
2008-10-23 23:00:00 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-24 06:41:00 incr backup 2 complete, 889 files, 169752997145 
bytes, 10 xferErrs (0 bad files, 0 bad shares, 10 other)
2008-10-24 23:00:00 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-25 06:41:42 incr backup 3 complete, 1190 files, 170136700321 
bytes, 1 xferErrs (0 bad files, 0 bad shares, 1 other)
2008-10-25 23:00:01 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-26 06:28:34 incr backup 4 complete, 1491 files, 170214964434 
bytes, 13 xferErrs (0 bad files, 0 bad shares, 13 other)
2008-10-26 23:00:01 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-27 06:54:31 incr backup 5 complete, 1826 files, 170294261665 
bytes, 2 xferErrs (0 bad files, 0 bad shares, 2 other)
2008-10-27 23:00:01 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-28 07:01:38 incr backup 6 complete, 2174 files, 170711300863 
bytes, 14 xferErrs (0 bad files, 0 bad shares, 14 other)
2008-10-28 23:12:21 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /
2008-10-29 08:27:44 incr backup 7 complete, 2481 files, 171012691775 
bytes, 2 xferErrs (0 bad files, 0 bad shares, 2 other)
2008-10-29 23:18:49 incr backup started back to 2008-10-21 23:00:01 
(backup #0) for directory /


The command that is running:

incr backup started back to 2008-10-21 23:00:01 (backup #0) for directory /
Running: /usr/bin/ssh -q -x -l root stella /usr/bin/rsync --server 
--sender --numeric-ids --perms --owner --group -D --links --hard-links 
--times --block-size=2048 --recursive . /
Xfer PIDs are now 1783
Got remote protocol 28
Negotiated protocol version 28
Sent exclude: /tmp
Sent exclude: /net
Sent exclude: /sys
Sent exclude: /proc
Sent exclude: /dev
Xfer PIDs are now 1783,1792


Any suggestions on what to look for as too why the incremental backups 
take so much longer than the full?

Mark
-- 
Mark Maciolek
Network Administrator
Morse Hall 339
862-3050
[EMAIL PROTECTED]
https://www.sr.unh.edu

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] What does BackupPC_link got error -4 when calling MakeFileLink mean?

2008-10-30 Thread Jeffrey J. Kosowsky
Holger Parplies wrote at about 10:11:33 +0100 on Thursday, October 30, 2008:
  Hi,
  
  Jeffrey J. Kosowsky wrote on 2008-10-30 03:41:39 -0400 [Re: [BackupPC-users] 
  What does BackupPC_link got error -4 when calling MakeFileLink mean?]:
   Craig Barratt wrote at about 00:07:51 -0700 on Thursday, October 30, 2008:
 Jeffrey writes:
 
  So what does -4 mean and what can cause it?
 
 Fails to make a hardlink. [...]
   
   Any suggestions on how to troubleshoot this further?
  
  well, in your case stale NFS file handle somehow springs to mind ... could
  it be that?
  
  Regards,
  Holger

I don't believe so for the following reasons:
- The NFS mount has been up the whole time
- In the latest backup, the problem affected 1737/5164 files (based
  on find -type f). 
- Specifically, 22 of the new 236 files in the backup were affected,
  however, the files affected were not consecutive -- and I couldn't
  find a common pattern with them
- Conversely, 1715 of the 4840 attrib file were affected. Again
  without any clear pattern (to me).
- My kernel has recorded about a dozen nfs timeouts over the past 12
  hours but they are not aligned with the timestamps on the error
  files. Otherwise there have been no kernel nfs errors.


Also, if the NFS server were down, I would think that I would get
other errors than just a bad link...

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Tino Schwarze wrote at about 11:13:27 +0100 on Thursday, October 30, 2008:
  Hi Jeffrey,
  
  On Thu, Oct 30, 2008 at 03:55:16AM -0400, Jeffrey J. Kosowsky wrote:
  
   I have found a number of files in my pool that have the same checksum
   (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
   has a few links to it by the way.
  
  That's intentional - what are the link counts for the files? 
  If you look at BackupPC's status page, there is a line like:
  
  * Pool hashing gives 649 repeated files with longest chain 28, 
Ah I was wondering what that line meant... (for real :)
Mine says:
 Pool hashing gives 9676 repeated files with longest chain 4
HOWEVER: my config has:
 $Conf{HardLinkMax} = 31999
And when I look at some of the repeated pool files, I see that they
only have 2-3 links each.

  
   Why is this happening? 
 Isn't this against the whole theory of pooling.  It also doesn't seem
 to get cleaned up by BackupPC_nightly since that has run several times
 and the pool files are now several days old.
  
  Because there is a file-system dependent limit to the number of hard
  links a file may have. Look at $Conf{HardLinkMax} in config.pl.
  
  Hm. I just took a look in my cpool and found some files which didn't
  hit the hardlink count yet, but have a _0 and _1:
  .../cpool/0/0 # ls -l c/00cd83be1ea3c1ffa3c6af2f4e310206* 
  -rw-r- 4371 backuppc users 34 2005-01-14 17:01 
  c/00cd83be1ea3c1ffa3c6af2f4e310206 
  -rw-r- 3536 backuppc users 34 2005-03-02 02:22 
  c/00cd83be1ea3c1ffa3c6af2f4e310206_0 
  -rw-r-  439 backuppc users 34 2006-03-11 02:04 
  c/00cd83be1ea3c1ffa3c6af2f4e310206_1 
  
  MD5Sums are not equal for all files, so maybe something got corrupted
  (or I updated BackupPC during the time - the files are rather old!):
  .../cpool/0/0 # md5sum c/00cd83be1ea3c1ffa3c6af2f4e310206*
  51ef559d1d7fa02c05fa032729c85804  c/00cd83be1ea3c1ffa3c6af2f4e310206
  51ef559d1d7fa02c05fa032729c85804  c/00cd83be1ea3c1ffa3c6af2f4e310206_0
  7e2276750fc478fa30142aa808df2a1f  c/00cd83be1ea3c1ffa3c6af2f4e310206_1
  
  AFAIK, I started with $Conf{HardLinkMax} set to 32.000. As the files are
  very old, a lot of links might have expired already.
  
  I'm not sure though, how the file name is derived, I found another file
  with same name but different MD5 sum:
  .../cpool/0/0 # md5sum 8/0084734e7242df0fbc186ba6741d1bab*
  db224998946bac7859f2448f41c58f88  8/0084734e7242df0fbc186ba6741d1bab
  d1d8f3a86ae5492de0bf11f5cfb45860  8/0084734e7242df0fbc186ba6741d1bab_0
  
  IIRC, BackupPC_nightly should perform chain cleaning.

Well, I haven't noticed any change after it runs...
I think I'm even more confused now ;)
How can I troubleshoot this further?

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Holger Parplies wrote at about 11:29:49 +0100 on Thursday, October 30, 2008:
  Hi,
  
  Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 [[BackupPC-users] 
  Duplicate files in pool with same CHECKSUM and same CONTENTS]:
   I have found a number of files in my pool that have the same checksum
   (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
   has a few links to it by the way.
   
   Why is this happening? 
  
  presumably creating a link sometimes fails, so BackupPC copies the file,
  assuming the hard link limit has been reached. I suspect problems with your
  NFS server, though not a stale NFS file handle in this case, since copying
  the file succeeds. Strange.

Yes - I am beginning to think that may be true. However as I mentioned
in the other thread, the syslog on the nfs server is clean and the one
on the client shows only about a dozen or so nfs timeouts over the
past 12 hours which is the time period I am looking at now. Otherwise,
I don't see any nfs errors.
So if it is a nfs problem, something seems to be happening somewhat
randomly and invisibly to the filesystem.

  
 Isn't this against the whole theory of pooling.
  
  Well, yes :). But the action of copying the file when the method to implement
  pooling (hard links) does not work for some reason (max link count reached, 
  or
  NFS file server errors if you think about it - you *do* get some level of
  pooling; otherwise you'd have an independant copy or a missing file each time
  linking fails) is perfectly reasonable.
  
 It also doesn't seem
 to get cleaned up by BackupPC_nightly since that has run several times
 and the pool files are now several days old.
  
  BackupPC_nightly is not supposed to clean up that situation. It could be
  designed to do so (the situation may arise when a link count overflow is
  resolved by expired backups), but it would be horribly inefficient: for the
  file to be eliminated, you would have to find() every occurrence of the inode
  in all pc/* trees and replace them with links to the duplicate(s) to be kept.
  You don't want that.

Yes but it would be nice to have a switch perhaps that allowed this
more comprehensive cleanup.
Even in a non-error case, I can imagine situations where at some point
the max file links may have been exceeded and then backups were
deleted so that the link count came back down below the max.

The logic wouldn't seem to be that horrendous. Since you would only
need to walk down the pc/* trees once -- i.e. first walk down
(c)pool/* to compile list of repeated but identical checksums. Then
walk down the pc/* tree to find the files on the list.

  
   What can I do to clean it up?
  
  Fix your NFS server? :) Is there a consistent maximum number of links, or do
  the copies seem to happen randomly? Honestly, I don't think the savings you
  may gain from storing the pool over NFS are worth the headaches. What is
  cheaper about putting a large disk into a NAS device than into your BackupPC
  server? Well, yes, you can share it ... how about exporting part of the disk
  from the BackupPC server (I would still recommend distinct partitions)?
  

You are right in theory. But I would still like to get NFS working for
various reasons and it is always a good learning experience to
troubleshoot such things ;)

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Tino Schwarze
On Thu, Oct 30, 2008 at 09:56:15AM -0400, Jeffrey J. Kosowsky wrote:

   I'm not sure though, how the file name is derived, I found another file
   with same name but different MD5 sum:
   .../cpool/0/0 # md5sum 8/0084734e7242df0fbc186ba6741d1bab*
   db224998946bac7859f2448f41c58f88  8/0084734e7242df0fbc186ba6741d1bab
   d1d8f3a86ae5492de0bf11f5cfb45860  8/0084734e7242df0fbc186ba6741d1bab_0
   
   IIRC, BackupPC_nightly should perform chain cleaning.
 
 Well, I haven't noticed any change after it runs...
 I think I'm even more confused now ;)
 How can I troubleshoot this further?

There's no trouble to shoot! ;-)

Holger explained that the pool file name is based on a checksum of the
first 256k of the file's content and the file's length, so collisions
are normal and expected.

HTH,

Tino.

-- 
What we nourish flourishes. - Was wir nähren erblüht.

www.lichtkreis-chemnitz.de
www.craniosacralzentrum.de

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Tino Schwarze wrote at about 15:08:29 +0100 on Thursday, October 30, 2008:
  On Thu, Oct 30, 2008 at 09:56:15AM -0400, Jeffrey J. Kosowsky wrote:
  
 I'm not sure though, how the file name is derived, I found another file
 with same name but different MD5 sum:
 .../cpool/0/0 # md5sum 8/0084734e7242df0fbc186ba6741d1bab*
 db224998946bac7859f2448f41c58f88  8/0084734e7242df0fbc186ba6741d1bab
 d1d8f3a86ae5492de0bf11f5cfb45860  8/0084734e7242df0fbc186ba6741d1bab_0
 
 IIRC, BackupPC_nightly should perform chain cleaning.
   
   Well, I haven't noticed any change after it runs...
   I think I'm even more confused now ;)
   How can I troubleshoot this further?
  
  There's no trouble to shoot! ;-)
  
  Holger explained that the pool file name is based on a checksum of the
  first 256k of the file's content and the file's length, so collisions
  are normal and expected.
  
Except that it my case some of the duplicated checksums truly are the
same file (probably due to the link issue I am having)...

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Adam Goryachev
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jeffrey J. Kosowsky wrote:
 Holger Parplies wrote at about 11:29:49 +0100 on Thursday, October 30, 2008:
   Hi,
   
   Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 [[BackupPC-users] 
 Duplicate files in pool with same CHECKSUM and same CONTENTS]:
I have found a number of files in my pool that have the same checksum
(other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
has a few links to it by the way.

Why is this happening? 
   
   presumably creating a link sometimes fails, so BackupPC copies the file,
   assuming the hard link limit has been reached. I suspect problems with your
   NFS server, though not a stale NFS file handle in this case, since 
 copying
   the file succeeds. Strange.
 
 Yes - I am beginning to think that may be true. However as I mentioned
 in the other thread, the syslog on the nfs server is clean and the one
 on the client shows only about a dozen or so nfs timeouts over the
 past 12 hours which is the time period I am looking at now. Otherwise,
 I don't see any nfs errors.
 So if it is a nfs problem, something seems to be happening somewhat
 randomly and invisibly to the filesystem.

See this URL which assisted me in improving the performance, and
reducing NFS errors in my environment.
http://billharlan.com/pub/papers/NFS_for_clusters.html

It was written a long time ago, but most of it is stall very relevant (I
guess NFS has not changed much).

In my case, the actual problem was faulty memory in a new server plus
some sort of strange network card driver problem corrupting the NFS
packets

It truly surprised me just how many errors I was getting even from my
existing load which I had never noticed.

Regards,
Adam
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJCc6IGyoxogrTyiURAoVnAJ9iKX9Sj8H7mDgmyrC182Uz+rIvgwCePvy5
J8OaYBtJuOvYC9a4JSNGEKI=
=gip1
-END PGP SIGNATURE-

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Backup aborts without apparent reason

2008-10-30 Thread Anian Wurzenberger
Hi,
at the moment it looks like the ! was the only problem, as my first two tries 
at backing up another machine were succesful.

Thank you for help

Anian Wurzenberger

-Original Message-
From: Anian Wurzenberger [mailto:[EMAIL PROTECTED] 
Sent: Donnerstag, 30. Oktober 2008 09:16
To: BackupPC-users@lists.sourceforge.net
Subject: [BackupPC-users] Backup aborts without apparent reason

Hi people,
I´m getting some strange Error on Fedora 6 (Linux fileserver 2.6.20-1.2948.fc6 
#1 SMP Fri Apr 27 18:53:15 EDT 2007 i686 i686 i386 GNU/Linux) with 
BackupPC-3.1.0 . SSH is working without pw prompt. The files arrive fine (I 
have never checked if they are indeed complete, but rsync doesn´t complain, so 
it´s probably fine), I can even restore them, but I am getting Backup aborted 
(). Also there is a permission error in the global log which shouldn´t be 
there, because the folder already exists and permissions of the folder are even 
777 (of course only for testing purposes).


Part of the Backup-Log (can send more if needed):

full backup started for directory /etc/sysconfig
Running: /usr/bin/ssh -q -x -l root somepc /usr/bin/rsync --server --sender 
--numeric-ids --perms --owner --group -D --links --hard-links --times 
--block-size=2048 --recursive --ignore-times . /etc/sysconfig/ ...
Finished deltaGet phase 1
pollChild()
Parent read: stats 2286 5329 176658 0 ('errorCnt' = 0,'ExistFileSize' = 
0,'ExistFileCnt' = 1,'TotalFileCnt' = 78,'ExistFileCompSize' = 
0,'TotalFileSize' = 176658) Got stats: 2286 5329 176658 0 ('errorCnt' = 
0,'ExistFileSize' = 0,'ExistFileCnt' = 1,'TotalFileCnt' = 
78,'ExistFileCompSize' = 0,'TotalFileSize' = 176658)
pollChild()
Parent read: log Sending: 
Sending: 
pollChild()
Parent read: log attribWrite(dir=f%2fetc%2fsysconfig) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new/f%2fetc%2fsysconfig/attrib
attribWrite(dir=f%2fetc%2fsysconfig) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new/f%2fetc%2fsysconfig/attrib
pollChild()
Parent read: log attribWrite(dir=) - 
/data/mirrored/mlb/!Backup/BackupPC/pc/tms/new//attrib
attribWrite(dir=) - /data/mirrored/mlb/!Backup/BackupPC/pc/tms/new//attrib
pollChild()
Parent read: exit
Got exit from child
Done: 78 files, 176658 bytes
Backup aborted ()
Not saving this as a partial backup since it has fewer files than the prior one 
(got 78 and 78 files versus 78)

Part of the global Log:
2008-09-30 05:00:00 Next wakeup is 2008-09-30 06:00:00 2008-09-30 05:00:01 
Started full backup on somepc (pid=19683, share=/etc/sysconfig) 2008-09-30 
05:00:08 Backup failed on somepc () 2008-09-30 05:00:08 Running BackupPC_link 
somepc (pid=19717) 2008-09-30 05:00:08 somepc: mkdir 
/data/mirrored/mlb/\!Backup/BackupPC: Permission denied at 
/usr/local/BackupPC/lib/BackupPC/Lib.pm line 899 2008-09-30 05:00:08 Finished 
somepc (BackupPC_link tms)


I hope somebody can help me...

Thanks
Anian W.
-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Jeffrey J. Kosowsky wrote at about 10:04:26 -0400 on Thursday, October 30, 2008:
  Holger Parplies wrote at about 11:29:49 +0100 on Thursday, October 30, 2008:
Hi,

Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 [[BackupPC-users] 
  Duplicate files in pool with same CHECKSUM and same CONTENTS]:
 I have found a number of files in my pool that have the same checksum
 (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
 has a few links to it by the way.
 
 Why is this happening? 

presumably creating a link sometimes fails, so BackupPC copies the file,
assuming the hard link limit has been reached. I suspect problems with 
  your
NFS server, though not a stale NFS file handle in this case, since 
  copying
the file succeeds. Strange.
  
  Yes - I am beginning to think that may be true. However as I mentioned
  in the other thread, the syslog on the nfs server is clean and the one
  on the client shows only about a dozen or so nfs timeouts over the
  past 12 hours which is the time period I am looking at now. Otherwise,
  I don't see any nfs errors.
Actually I traced these errors to a timout due to disks on the NAS
spinning up. They appear to be just soft timeouts (and not related to
this link problem)

  So if it is a nfs problem, something seems to be happening somewhat
  randomly and invisibly to the filesystem.
  

   Isn't this against the whole theory of pooling.

Well, yes :). But the action of copying the file when the method to 
  implement
pooling (hard links) does not work for some reason (max link count 
  reached, or
NFS file server errors if you think about it - you *do* get some level of
pooling; otherwise you'd have an independant copy or a missing file each 
  time
linking fails) is perfectly reasonable.

   It also doesn't seem
   to get cleaned up by BackupPC_nightly since that has run several times
   and the pool files are now several days old.

BackupPC_nightly is not supposed to clean up that situation. It could be
designed to do so (the situation may arise when a link count overflow is
resolved by expired backups), but it would be horribly inefficient: for 
  the
file to be eliminated, you would have to find() every occurrence of the 
  inode
in all pc/* trees and replace them with links to the duplicate(s) to be 
  kept.
You don't want that.
  
  Yes but it would be nice to have a switch perhaps that allowed this
  more comprehensive cleanup.
  Even in a non-error case, I can imagine situations where at some point
  the max file links may have been exceeded and then backups were
  deleted so that the link count came back down below the max.
  
  The logic wouldn't seem to be that horrendous. Since you would only
  need to walk down the pc/* trees once -- i.e. first walk down
  (c)pool/* to compile list of repeated but identical checksums. Then
  walk down the pc/* tree to find the files on the list.
  

 What can I do to clean it up?

Fix your NFS server? :) Is there a consistent maximum number of links, or 
  do
the copies seem to happen randomly? Honestly, I don't think the savings 
  you
may gain from storing the pool over NFS are worth the headaches. What is
cheaper about putting a large disk into a NAS device than into your 
  BackupPC
server? Well, yes, you can share it ... how about exporting part of the 
  disk
from the BackupPC server (I would still recommend distinct partitions)?

  
  You are right in theory. But I would still like to get NFS working for
  various reasons and it is always a good learning experience to
  troubleshoot such things ;)
  

Now this is interesting...
Looking through my BackupPC log files, I noticed that this problem
*FIRST* occurred on Oct 27 and has affected every backup since. The
error are only occurring when BackupPC_link runs (and I didn't have
any problems with BackupPC_link in the 10 or so previous days that I
have been using BackupPC).

So, I used both find and the incremental backups themselves to see
what happened between the last error-free backup at 18:08PM on Oct 26
and the first bad one at 1AM on Oct 27. But it doesn't seem like any
files changed on either the BackupPC server or the NFS server.

Also, interestingly, this problem occurred on the first backup attempt
*after* I rebooted my Linux server (which I hadn't rebooted in several
weeks). 

So, I'm starting to wonder whether the problem is the reboot...
I will try rebooting my server (again) to see what happens.
I will also run memtest86 for a bit just in case...

Any other suggestions?


-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world

[BackupPC-users] $Conf{BackupExcludeFiles} problem with smb

2008-10-30 Thread Madcha

Hi,

To backup Client based on Windows XP,
I would like to exlude some fodlers from my share :

I tried this:


$ConfBackupFilesExclude#125;= #123;'Home' = #91; *Nosafe*,nbsp; 'Ma 
musique', 'Mes videos', '*.mp3'#93;, #125;;


For the folder Nosafe it's ok, but it's not for me a good solution, 

I want to exclude from backup  theses folders 'Ma musique' and 'Mes videos'
For the example, the real path to these folders is (on Windows):

D:\Home\%UserName%\Mes documents\Ma musique

Someone can help me?

Regards,

+--
|This was sent by [EMAIL PROTECTED] via Backup Central.
|Forward SPAM to [EMAIL PROTECTED]
+--



-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Craig Barratt
Jeffrey writes:

 Except that it my case some of the duplicated checksums truly are the
 same file (probably due to the link issue I am having)...

Yes.  Just as Holger mentions, if the hardlink attempt fails,
a new file is created in the pool.  You appear to have some
unreliability in your NFS or network setup.

The only other time identical files will have different pool
entries, as people noted, is when $Conf{HardLinkMax} is hit.
Subsequent expiry of backups might cause the identical files
to move below $Conf{HardLinkMax}.

It's not worth the trouble to try to combine those files since
the frequency is so small and the effort to relink them is very
high.

Craig

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Craig Barratt wrote at about 11:27:41 -0700 on Thursday, October 30, 2008:
  Jeffrey writes:
  
   Except that it my case some of the duplicated checksums truly are the
   same file (probably due to the link issue I am having)...
  
  Yes.  Just as Holger mentions, if the hardlink attempt fails,
  a new file is created in the pool.  You appear to have some
  unreliability in your NFS or network setup.
  
  The only other time identical files will have different pool
  entries, as people noted, is when $Conf{HardLinkMax} is hit.
  Subsequent expiry of backups might cause the identical files
  to move below $Conf{HardLinkMax}.
  
  It's not worth the trouble to try to combine those files since
  the frequency is so small and the effort to relink them is very
  high.
  
  Craig

OK - Definitely seems to be an NFS problem -- sorry for having
troubled the BackupPC list.

When I do a shell command 'find | wc' on the cpool directory, I usually get
the right number of results but sometimes whole subdirectories are not
found.  This problem seems to come and go. As in sometimes, I get the
right results and sometimes I don't... This makes it even harder to
troubleshoot since I can't reliably reproduce the problem every time.

I am confused though why I'm not seeing any notation of this problem in my
log files (either on the nfs server or client)...

Thanks!!!

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] BackupPC quit working

2008-10-30 Thread Kenneth L. Owen
With your help, I was able to get BackupPC working.  Next I set up my
permanent storage location on a 500 GB drive mounted at /var/lib/BackupPC.
All seemed to be running well until I shut down my LAN rearrange furniture
in the room.  After placing the LAN back in service, BackupPC no longer
worked.  To the best of my knowledge, the only changes to the systems
(Archiver and Winserver, both running Fedora 8) were package downloads and
installation of a printer on Archiver.

 

The only thing that I see in the updates that potentially could affect
BackupPC operation was the new kernel.  I booted both machines to the
previous kernel and BackupPC still does not work.

 

I have gone over all of the setup files that I modified to get it going the
first time and can find no change from what worked initially.

 

(I submitted this under 'Almost Working - Working - Stopped Working' earlier
but have gotten no replies.)

 

 Latest data in yum.log on Achiver:

Oct 24 10:58:03 Installed: kernel-2.6.26.6-49.fc8.i686

Oct 24 10:58:22 Installed: kernel-devel-2.6.26.6-49.fc8.i686

Oct 24 10:58:24 Updated: kernel-headers-2.6.26.6-49.fc8.i386

Oct 24 10:58:25 Updated: kde-filesystem-4-19.fc8.noarch

 

 Last good incr backup at 10/25/2008 03:00

 

Oct 25 11:49:58 Updated: file-libs-4.21-6.fc8.i386

Oct 25 11:49:59 Updated: Xaw3d-1.5E-12.fc8.i386

Oct 25 11:50:00 Updated: Xaw3d-devel-1.5E-12.fc8.i386

Oct 25 11:50:00 Updated: file-4.21-6.fc8.i386

Oct 27 18:11:15 Installed: 1:pnm2ppa-1.04-14.fc8.i386  ( HP printer
installed)

[EMAIL PROTECTED] log]# 

 

Latest data in yum.log on Winserver:

Oct 21 21:43:34 Updated: libpurple-2.5.1-3.fc8.i386

Oct 21 21:43:37 Updated: systemtap-runtime-0.7-1.fc8.i386

Oct 24 11:17:48 Installed: kernel-2.6.26.6-49.fc8.i686

Oct 24 11:20:20 Installed: kernel-devel-2.6.26.6-49.fc8.i686

Oct 24 11:20:39 Updated: kernel-headers-2.6.26.6-49.fc8.i386

Oct 24 11:20:43 Updated: kde-filesystem-4-19.fc8.noarch

 

 Last good incr backup at 10/25/2008 03:00

 

Oct 26 15:40:54 Updated: file-libs-4.21-6.fc8.i386

Oct 26 15:40:56 Updated: Xaw3d-1.5E-12.fc8.i386

Oct 26 15:41:00 Updated: Xaw3d-devel-1.5E-12.fc8.i386

Oct 26 15:41:02 Updated: file-4.21-6.fc8.i386

[EMAIL PROTECTED] log]# 

 

 

 BackupPC Log for Winserver:

Contents of file /var/lib/BackupPC/pc/winserver/LOG.102008, modified
2008-10-28 12:41:48

 

 After creating LVM for 500 GB drive and mounting at
/var/lib/BackupPC,

 I ran a full backup?

 

2008-10-15 22:31:11 full backup started for directory /media/SHARE2

2008-10-15 23:26:45 full backup started for directory /media/SHARE1

2008-10-15 23:56:12 full backup 0 complete, 16479 files, 34508365250 bytes,
745 xferErrs (0 bad files, 0 bad shares, 745 other)

2008-10-17 03:00:00 incr backup started back to 2008-10-15 22:31:11 (backup
#0) for directory /media/SHARE2

2008-10-17 03:00:22 incr backup started back to 2008-10-15 22:31:11 (backup
#0) for directory /media/SHARE1

2008-10-17 03:00:39 incr backup 1 complete, 6 files, 23239994 bytes, 0
xferErrs (0 bad files, 0 bad shares, 0 other)

2008-10-18 03:00:00 incr backup started back to 2008-10-17 03:00:00 (backup
#1) for directory /media/SHARE2

2008-10-18 03:00:21 incr backup started back to 2008-10-17 03:00:00 (backup
#1) for directory /media/SHARE1

 Lines deleted to save space. 

2008-10-25 03:00:00 incr backup started back to 2008-10-24 03:00:00 (backup
#8) for directory /media/SHARE2

2008-10-25 03:00:21 incr backup started back to 2008-10-24 03:00:00 (backup
#8) for directory /media/SHARE1

2008-10-25 03:00:40 incr backup 9 complete, 32 files, 44569138 bytes, 0
xferErrs (0 bad files, 0 bad shares, 0 other)

 

 The LAN was down while doing some major rearrangements to the room.

 

2008-10-26 03:00:04 no ping response

until

2008-10-27 06:00:24 no ping response

 

 LAN restored when work was completed afternoon of 10/27/2008.

 New packages for Fedora 8 were downloaded on reconnect.

 

2008-10-28 01:00:06 incr backup started back to 2008-10-25 03:00:00 (backup
#9) for directory /media/SHARE2

2008-10-28 01:00:07 Got fatal error during xfer (Unable to read 4 bytes)

2008-10-28 01:00:12 Backup aborted (Unable to read 4 bytes)

until

2008-10-28 06:00:06 incr backup started back to 2008-10-25 03:00:00 (backup
#9) for directory /media/SHARE2

2008-10-28 06:00:07 Got fatal error during xfer (Unable to read 4 bytes)

2008-10-28 06:00:12 Backup aborted (Unable to read 4 bytes)

 

 Rebooted system and tried to perform a full backup.

 

2008-10-28 12:41:42 full backup started for directory /media/SHARE2
(baseline backup #9)

2008-10-28 12:41:43 Got fatal error during xfer (Unable to read 4 bytes)

2008-10-28 12:41:48 Backup aborted (Unable to read 4 bytes)

 

 

 Performed a manual dump test?  I hope someone can see something that
will help! 

[EMAIL PROTECTED] ~]$ ssh archiver

[EMAIL PROTECTED]'s password: 

Last login: Mon Oct 27 13:36:41 2008 from WinServer

[EMAIL PROTECTED] ~]$ su -

Password: 

[EMAIL 

Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
John Rouillard wrote at about 20:13:15 + on Thursday, October 30, 2008:
  On Thu, Oct 30, 2008 at 10:04:26AM -0400, Jeffrey J. Kosowsky wrote:
   Holger Parplies wrote at about 11:29:49 +0100 on Thursday, October 30, 
   2008:
 Hi,
 
 Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 
   [[BackupPC-users] Duplicate files in pool with same CHECKSUM and same 
   CONTENTS]:
  I have found a number of files in my pool that have the same checksum
  (other than a trailing _0 or _1) and also the SAME CONTENT. Each copy
  has a few links to it by the way.
  
  Why is this happening? 
 
 presumably creating a link sometimes fails, so BackupPC copies the file,
 assuming the hard link limit has been reached. I suspect problems with 
   your
 NFS server, though not a stale NFS file handle in this case,
 since the file succeeds. Strange.
   
   Yes - I am beginning to think that may be true. However as I mentioned
   in the other thread, the syslog on the nfs server is clean and the one
   on the client shows only about a dozen or so nfs timeouts over the
   past 12 hours which is the time period I am looking at now. Otherwise,
   I don't see any nfs errors.
   So if it is a nfs problem, something seems to be happening somewhat
   randomly and invisibly to the filesystem.
  
  IIRC you are using a soft nfs mount option right? If you are writing
  to an NFS share that is not recommended. Try changing it to a hard
  mount and see if the problem goes away. I only used soft mounts on
  read only filesystems.

True -- I changed it to 'hard' but am still encountering the
problem... FRUSTRATING...

It's really weird in that it seems to work the first time a directory
is read but after a directory has been read a few times, it starts
messing up. It's almost like the results are being stored in cache and
then the cache is corrupted.

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Jeffrey J. Kosowsky
Jeffrey J. Kosowsky wrote at about 20:26:35 -0400 on Thursday, October 30, 2008:
  John Rouillard wrote at about 20:13:15 + on Thursday, October 30, 2008:
On Thu, Oct 30, 2008 at 10:04:26AM -0400, Jeffrey J. Kosowsky wrote:
 Holger Parplies wrote at about 11:29:49 +0100 on Thursday, October 30, 
  2008:
   Hi,
   
   Jeffrey J. Kosowsky wrote on 2008-10-30 03:55:16 -0400 
  [[BackupPC-users] Duplicate files in pool with same CHECKSUM and same 
  CONTENTS]:
I have found a number of files in my pool that have the same 
  checksum
(other than a trailing _0 or _1) and also the SAME CONTENT. Each 
  copy
has a few links to it by the way.

Why is this happening? 
   
   presumably creating a link sometimes fails, so BackupPC copies the 
  file,
   assuming the hard link limit has been reached. I suspect problems 
  with your
   NFS server, though not a stale NFS file handle in this case,
   since the file succeeds. Strange.
 
 Yes - I am beginning to think that may be true. However as I mentioned
 in the other thread, the syslog on the nfs server is clean and the one
 on the client shows only about a dozen or so nfs timeouts over the
 past 12 hours which is the time period I am looking at now. Otherwise,
 I don't see any nfs errors.
 So if it is a nfs problem, something seems to be happening somewhat
 randomly and invisibly to the filesystem.

IIRC you are using a soft nfs mount option right? If you are writing
to an NFS share that is not recommended. Try changing it to a hard
mount and see if the problem goes away. I only used soft mounts on
read only filesystems.
  
  True -- I changed it to 'hard' but am still encountering the
  problem... FRUSTRATING...
  
  It's really weird in that it seems to work the first time a directory
  is read but after a directory has been read a few times, it starts
  messing up. It's almost like the results are being stored in cache and
  then the cache is corrupted.

In fact, I have found two ways to assuredly allow me to read the
directory again (at least for a few minutes or tries until it gets
corrupted again):
1. Remount the nfs share
2. Read the directory directly on the server (without nfs)

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Duplicate files in pool with same CHECKSUM and same CONTENTS

2008-10-30 Thread Holger Parplies
Hi,

[could we agree on a subject line without tabs? ;-]

Jeffrey J. Kosowsky wrote on 2008-10-30 20:31:15 -0400 [Re: [BackupPC-users] 
Duplicate files in pool with same CHECKSUM and same CONTENTS]:
 Jeffrey J. Kosowsky wrote at about 20:26:35 -0400 on Thursday, October 30, 
 2008:
   It's really weird in that it seems to work the first time a directory
   is read but after a directory has been read a few times, it starts
   messing up. It's almost like the results are being stored in cache and
   then the cache is corrupted.
 
 In fact, I have found two ways to assuredly allow me to read the
 directory again (at least for a few minutes or tries until it gets
 corrupted again):
 1. Remount the nfs share
 2. Read the directory directly on the server (without nfs)

bad memory on either client or server? Bug in the NFS implementation on client
or server? You said you built a kernel for the NAS device. Could anything have
gone wrong?

Have you tried the 'noac' mount option? Which NFS version are you using? Over
TCP or UDP?

Have you found out anything about ATAoE (or iSCSI, for that matter)
capabilities of the device?

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] BackupPC quit working

2008-10-30 Thread Holger Parplies
Hi,

Kenneth L. Owen wrote on 2008-10-30 19:49:24 -0400 [[BackupPC-users] BackupPC 
quit working]:
 [...]
 All seemed to be running well until I shut down my LAN rearrange furniture
 in the room.  After placing the LAN back in service, BackupPC no longer
 worked. [...]

it would make things easier if you were more precise on what you shut down and
what you didn't. From the logs, I'd guess you did not shut down Archiver?
While backups were working, were there *any* reboots to either Archiver or
Winserver?

 [lots of quotes from lots of logs]

My feeling is that it has nothing to do with updates but rather with
configuration changes or unsaved configuration information (like manual
starts of services that were not automatically restarted after reboot).

 -bash-3.2$ /usr/share/BackupPC/bin/BackupPC_dump -v -f WinServer
 
 Name server doesn't know about WinServer; trying NetBios

Name server should know about WinServer. Why are you using NetBios for
resolution of Linux host names? For two hosts you can use /etc/hosts if
you don't feel comfortable with setting up something more complicated.
Use static IP addresses, really.

 [...]
 Running: /usr/bin/ssh -q -x -l root WinServer /usr/bin/rsync --server
 --sender --numeric-ids --perms --owner --group -D --links --hard-links
 --times --block-size=2048 --recursive --ignore-times . /media/SHARE2/

That is unlikely to work, since your name server doesn't know about WinServer.
ssh will not do an nmblookup. Luckily.

This was apparently working before, though I can't find the solution in the
old thread - maybe you should have posted it?

See that you can once again 'ssh -l root WinServer' as the backuppc user on
Archiver *without password prompt or any extraneous output* - this does not
currently seem to be possible (even if Connection reset by peer is not the
error message you should be getting - unless perhaps if name resolution is
somehow working on Archiver and failing on Winserver). Testing from the
command line might give more insight into what is going wrong.

Regards,
Holger

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] 2 cpool files with same checksum, different (compressed content) but same zcatt'ed content?????

2008-10-30 Thread Jeffrey J. Kosowsky
I must be missing something on this whole compression, pooling, and
checksum matter.

I found 2 files in my cpool that have the same checksum (one is _0)
but 'cmp' to different values. However, when I zcat them, they have
the same value. I thought that (lossless) compression was a 1-1
mapping?

But here we seem to have two files that are identical (and thus have
the same checksum) but compress to 2 *different* results?
This would seem to be going against the grain of pooling where two
identical files share the same pool entry.

What am I missing?
Does backuppc add some extra header information during compression? (I
didn't see anything obvious in FileZIO.pm).

Note all files were backed up with rsync/rsyncd.

Is there anywhere (other than the code) where this is more fully documented?

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


[BackupPC-users] Pool duplicates and corrupted backups

2008-10-30 Thread Jeffrey J. Kosowsky
OK. I have been spending all day on this and am trying to understand
(and fix) the different types of pool duplication and corruption.

Types of Duplicate checksums:
1. Same checksum but contents differ -- INTENTIONAL - nothing to fix
2. Same checksum and compressed content
I have found many of these but contrary to my earlier postings
the ones that I examined were not in my error log so they
don't seem to be due to the nfs/link problems (see later)

I don't know what the source of these are though...

3. Same checksums, different compressed content, same zcatted content
As per my earlier email, I found at least one of these 
and am not sure why they even exist.

4. Files with *NO* pool entries. 
This seems to be what occurs with the
files in my error log that cite the failure of 
MakeFileLink

5. HardLinkMax exceeded -- INTENTIONAL - nothing to fix

Hopefully, there are no files with the case of same content,
different checksums).

In summary:
- 15 are by design
- 23 are unclear what is causing them
- 4 is clearly do to my filesystem problems
- 2,3,4 could all be corrected by a program that crawls through the
  cpool and pc chain.

The deeper I probe into this, the more confused I get and the more I
worry about data integrity on my system...

Thanks

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Pool duplicates and corrupted backups

2008-10-30 Thread Adam Goryachev
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Jeffrey J. Kosowsky wrote:
 Types of Duplicate checksums:
 3. Same checksums, different compressed content, same zcatted content
   As per my earlier email, I found at least one of these 
   and am not sure why they even exist.

AFAIK, it could be due to a change in the compresslevel, this should
produce different compressed content even though the original content is
identical. Also, the files are stored to the cpool based on a checksum
calculated before compression, hence they have the same name. The only
caution here is that I was of the impression that backuppc would see the
existing file with the same checksum and so not store the differently
compressed file.

 4. Files with *NO* pool entries. 
   This seems to be what occurs with the
   files in my error log that cite the failure of 
   MakeFileLink
 
 The deeper I probe into this, the more confused I get and the more I
 worry about data integrity on my system...

That is why you will need to fix your NFS problems, and/or consider
moving to a different setup. eg, ATAoE, iSCSI, or local HDD's


Did you check the details from the URL I sent through the other day? One
of the items on the list is to look at the number of time there is no
NFS thread available to service the request... I found fixing this
drastically improved my situation though didn't solve it (until I
replaced the faulty RAM in the client :)

Regards,
Adam
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFJCo8XGyoxogrTyiURAoJZAJ9ZSZckzLVwkOgwOfXJ2GpwTmcR8ACfR6iQ
3vxQzxjGxJ40oPWexKC47T4=
=MgtZ
-END PGP SIGNATURE-

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/


Re: [BackupPC-users] Pool duplicates and corrupted backups

2008-10-30 Thread Craig Barratt
Jeffrey writes:

 Types of Duplicate checksums:
 1. Same checksum but contents differ -- INTENTIONAL - nothing to fix

Right.

 2. Same checksum and compressed content
 I have found many of these but contrary to my earlier postings
 the ones that I examined were not in my error log so they
 don't seem to be due to the nfs/link problems (see later)
 
 I don't know what the source of these are though...

This could happen if there is ever a read error of a file in the
cpool.  Before linking to a cpool file (except when rsync is used
with checksum caching enabled and the checksums match exactly),
the contents are read, decompressed and compared to the incoming
file.  Any read error will cause the match to fail and a new cpool
file will be written.

 3. Same checksums, different compressed content, same zcatted
content As per my earlier email, I found at least
one of these and am not sure why they even exist.

This can happen if one file has rsync checksums and the other does not.
This would happen after case 2 above.  The checksums don't get added 
until the next time the file is read.

This can also happen since the mapping of file - file.z is one-to-many:
there are many different ways to compress a file that all uncompress
the same.  But that shouldn't happen in BackupPC unless you change
the compression level.

 4. Files with *NO* pool entries.
 This seems to be what occurs with the
 files in my error log that cite the failure of
 MakeFileLink

Yes.  There are cases when it just leaves the file there (below
pc/HOST/nnn) without linking to the pool.

 The deeper I probe into this, the more confused I get and the more I
 worry about data integrity on my system...

Look, you *do* have a data integrity problem, and yes, you should worry.

There is no further value in understanding symptoms from a relatively
filesystem-intensive application like BackupPC and working backwards.

You need to fix your hardware/software before using BackupPC again.

Craig

-
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK  win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100url=/
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/