Re: [BackupPC-users] Is there any way for BackupPC to restore hard links properly?
Craig Barratt wrote: This results in one subtle bug that can't be easily fixed: if you switch the Xfer method from tar to rsync, old backups that have hardlinks stored with tar won't be correctly restored with rsync. The workaround is generate a tar file and extract it, or switch the Xfer method back to tar before you do the restore. The opposite case should work correctly. Craig FYI, If one does ever find themselves in the situation with a bunch of files they need to re-hardlink, I found some great utilities to do that with: Name : hardlink Summary: Create a tree of hardlinks URL: http://cvs.fedora.redhat.com/viewcvs/devel/hardlink/ License: GPL+ Description: hardlink is used to create a tree of hard links. It's used by kernel installation to dramatically reduce : the amount of diskspace used by each kernel package installed. Name : fdupes Version: 1.40 Summary: Finds duplicate files in a given set of directories URL: http://netdial.caribe.net/~adrian2/fdupes.html License: MIT Description: FDUPES is a program for identifying duplicate files residing within specified directories. - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there any way for BackupPC to restore hard links properly?
Craig Barratt wrote at about 23:06:25 -0800 on Sunday, November 9, 2008: Jeffrey writes: Looking at the code and the structure of the storage and attrib files, it doesn't seem like there is any way for BackupPC to record and restore hard links. Not true. Hardlinks are stored without using hardlinks. Hardlinks are stored just like symlinks. The attribute type is BPC_FTYPE_HARDLINK and the contents of the file is the path of the file being linked to. This also adds an extra wrinkle to the 'delete' script I wrote that is supposed to allow you to delete individual files (and directories) from incremental backups. In order to delete a hardlink, it seems that you need the following information: 1. If the file is the target of other hardlinks, then you need to find all the other hard links and make another one of them the target and restructure the linking accordingly. 2. If the file is not the target, then if there are at least two remaining hard links, you can just delete the file. If there is just one remaining link (i.e., just the target), then you need to change the target's attrib back to regular file. Now is there any easy way to find all the hard-links pointing to a target or would I have to first search all the attrib files for files of type hard link and then read the corresponding files to see which targets they are linked to? Is there a better way or is trying to delete files that are hard-linked a hopeless task? (Since they are rare, I could always just add a check to make sure that you are not trying to delete a file/directory that is one or a directory tree that contains hard links). - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there any way for BackupPC to restore hard links properly?
Craig Barratt wrote at about 23:06:25 -0800 on Sunday, November 9, 2008: Jeffrey writes: Looking at the code and the structure of the storage and attrib files, it doesn't seem like there is any way for BackupPC to record and restore hard links. Not true. Hardlinks are stored without using hardlinks. Thanks good to hear! Hardlinks are stored just like symlinks. The attribute type is BPC_FTYPE_HARDLINK and the contents of the file is the path of the file being linked to. For example, if there are 4 links to a file, one instance (it doesn't matter which - depends on the Xfer Method) will be stored as a regular file. The remaining three instances will be stored as type BPC_FTYPE_HARDLINK with pointers to the first file. Dohhh... for some reason, I kept looking at the first file which of course had a standard attrib file with type=0. Note that this is independent of the hardlinks used to de-duplicate. There is one subtlety with rsync: it also needs to remember if a regular file is the target of (other) hardlinks so that the correct file list can be generated during a restore. This is done by using an extra bit in the file's mode stored in the attrib file. Based on this, I can understand why you need to add a bit for hard link target to the mode but according to RsyncFileIO.pm, it seems like otherfile types are also encoded in the mode bit via the S_IFxxx masks. For non-hard links, how does this differ from the 'type attrib? or is it redundant. Also, given the target how does rsync 'know' what other files are hard-linked to it when only doing a partial restore (or is that not necessary info)? Further, can I assume this schemme will continue to work even if you modify the exclude/include between incremental backups so that hard-links variously appear/disappear? Also, can I assume that directories are treated the same way -- and that if so then this is one case where the directory tree won't be completely replicated for incrementals (i.e. only the target tree will be replicated). This results in one subtle bug that can't be easily fixed: if you switch the Xfer method from tar to rsync, old backups that have hardlinks stored with tar won't be correctly restored with rsync. The workaround is generate a tar file and extract it, or switch the Xfer method back to tar before you do the restore. The opposite case should work correctly. Craig - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Is there any way for BackupPC to restore hard links properly?
Jeffrey writes: Looking at the code and the structure of the storage and attrib files, it doesn't seem like there is any way for BackupPC to record and restore hard links. Not true. Hardlinks are stored without using hardlinks. Hardlinks are stored just like symlinks. The attribute type is BPC_FTYPE_HARDLINK and the contents of the file is the path of the file being linked to. For example, if there are 4 links to a file, one instance (it doesn't matter which - depends on the Xfer Method) will be stored as a regular file. The remaining three instances will be stored as type BPC_FTYPE_HARDLINK with pointers to the first file. Note that this is independent of the hardlinks used to de-duplicate. There is one subtlety with rsync: it also needs to remember if a regular file is the target of (other) hardlinks so that the correct file list can be generated during a restore. This is done by using an extra bit in the file's mode stored in the attrib file. This results in one subtle bug that can't be easily fixed: if you switch the Xfer method from tar to rsync, old backups that have hardlinks stored with tar won't be correctly restored with rsync. The workaround is generate a tar file and extract it, or switch the Xfer method back to tar before you do the restore. The opposite case should work correctly. Craig - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Is there any way for BackupPC to restore hard links properly?
Looking at the code and the structure of the storage and attrib files, it doesn't seem like there is any way for BackupPC to record and restore hard links. Specifically, since Backuppc uses hard links to pool files and since the attrib database doesn't seem to record hard links, it would seem that there is no way to keep track of hard links. It would seem that this would make accurate restores impossible -- indeed not only would you end up duplicating storage but you may unwittingly lead to directories getting out of synch that should be hard-linked. On the other hand, one of the beautiful advantages of naked rsync is that it can keep track of hard links. Am I missing something obvious here? If not, is this something that can/should/will be corrected? Thanks! - This SF.Net email is sponsored by the Moblin Your Move Developer's challenge Build the coolest Linux based applications with Moblin SDK win great prizes Grand prize is a trip for two to an Open Source event anywhere in the world http://moblin-contest.org/redirect.php?banner_id=100url=/ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
