Re: [BackupPC-users] Handle Backup as inconsistent on xfer error
On 29.01.2018 12:32, Zielke, Julian, NLI wrote:
Talking about the point on not saving the rest of the archives: There is afaik
no current way to tell BackuPC to stop the backup on this share, execute
post-share using xferOK = 0 and continue
with the rest of the backups. So we'll keep it that way. Maybe this could be a
nice feature implementation to skip a share on xfer errors and return xferOK=0.
This would be a perfect way to have
a solution of partitial backups AND notification in both ways.
On the second thought, skipping certain shares is an overkill. Probably in case
if we want treat shares differently we should just create different hosts for
them.
I suggest to create an option in the configuration of the tar xfer method (bool
/ checkbox) so the user can decide whether
to abort on Xfer errors or not.
I agree with you. It would be nice to add a configuration option for that, but
for any xfer type, not only for tar xfer.
The options could be something like:
$Conf{xferErrsAreFatal} If any share had xfer errors then consider the dump bad
and mark it as partial
0 - no (default)
1 - yes
n - yes if $stat{xferErrCnt} >= n
$Conf{abortOnXferErrs} abort the dump instead of marking it partial if
$Conf{xferErrsAreFatal} condition is met.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Handle Backup as inconsistent on xfer error
Alex, yes I understand your points and they may apply in most backup scenarios. However, in my case it's a little bit different: I'm saving lucene indices, which currently rely on results of a postgres database process. The problem with lucene is that a partitial backup of an index will make the segments-file being incompatible with the rest of the files because of the segments pointing to nowhere or non-existent files, because lucene first writes data to it's data-segments and in the end updates the segments-catalogue. So having a partitial backup is the same like a corrupted backup. We have the ability to restore a consistnet index and later restore the missing part from the database tables so we're always able to restore data up to date. Also we're constantly monitoring every backup through some post-share / post-user scripts, hence we'll be notified immediately when such an xfer error occours. Normally they shouldn't because the index is configured to be in pause state during backup times so this xfer error abort will let us know if the index interfered with the backup somehow. Talking about the point on not saving the rest of the archives: There is afaik no current way to tell BackuPC to stop the backup on this share, execute post-share using xferOK = 0 and continue with the rest of the backups. So we'll keep it that way. Maybe this could be a nice feature implementation to skip a share on xfer errors and return xferOK=0. This would be a perfect way to have a solution of partitial backups AND notification in both ways. - Julian -Ursprüngliche Nachricht- Von: Alexander Moisseev via BackupPC-users [mailto:backuppc-users@lists.sourceforge.net] Gesendet: Sonntag, 28. Januar 2018 16:58 An: General list for user discussion, questions and support <backuppc-users@lists.sourceforge.net> Cc: Alexander Moisseev <mois...@mezonplus.ru> Betreff: Re: [BackupPC-users] Handle Backup as inconsistent on xfer error On 1/26/2018 5:31 PM, Zielke, Julian, NLI wrote: > It will be marked as partial but not continuing and/or doing rotation. > Also the xfer status won't be 1, because there were actual problems > saving all files within the share. This is just to prevent BackupPC from > running backups with xfer errors and doing rotation while some files in all > series always were open during backup. > Julian, If I understood you correctly, you are trying to solve the following problem: If xfer errors occurred during a backup (files were open or whatever), the backup is inconsistent as it misses some files. In current implementation such a backup is considered successful despite xfer errors. The expiration algorithm dosen't take xfer errors in account as well. Here is a problem: under certain circumstances BackupPC expiration can leave errored backups, but expire consistent ones, so we will be unable to restore some files. That is the reason why I do daily checks for xfer errors and remove backups with xfer errors. Sure, the patch you are proposing will do it automatically, but it seem not a good solution to me. What if something happened with the host's data? In many cases a recent partial backup is better than nothing. But you've aborted it! More over you haven't saved the rest of shares on this host!!! https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_dump#L1230 If some files always are open during backup, every backup will be aborted, so you will end up with no backups at all. If we mark the backup as partial instead of aborting it, the partial backup will be removed when the next successful backup completes, or if another full backup fails resulting in a newer partial backup. Advantages: files from partial backup can be restored, on the next backup we don't need to retransfer files already in the partial backup (for rsync, not for tar). The situation when some files always are open during backup should be avoided, in general. In case if there are just a few files and they are always the same, it is probably a good idea to create an exclusion list. So if all xfer errors happen when attempting to transfer files that are in the exclusion list, do not mark that backup as partial. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-use
Re: [BackupPC-users] Handle Backup as inconsistent on xfer error
On 1/26/2018 5:31 PM, Zielke, Julian, NLI wrote: It will be marked as partial but not continuing and/or doing rotation. Also the xfer status won't be 1, because there were actual problems saving all files within the share. This is just to prevent BackupPC from running backups with xfer errors and doing rotation while some files in all series always were open during backup. Julian, If I understood you correctly, you are trying to solve the following problem: If xfer errors occurred during a backup (files were open or whatever), the backup is inconsistent as it misses some files. In current implementation such a backup is considered successful despite xfer errors. The expiration algorithm dosen't take xfer errors in account as well. Here is a problem: under certain circumstances BackupPC expiration can leave errored backups, but expire consistent ones, so we will be unable to restore some files. That is the reason why I do daily checks for xfer errors and remove backups with xfer errors. Sure, the patch you are proposing will do it automatically, but it seem not a good solution to me. What if something happened with the host's data? In many cases a recent partial backup is better than nothing. But you've aborted it! More over you haven't saved the rest of shares on this host!!! https://github.com/backuppc/backuppc/blob/master/bin/BackupPC_dump#L1230 If some files always are open during backup, every backup will be aborted, so you will end up with no backups at all. If we mark the backup as partial instead of aborting it, the partial backup will be removed when the next successful backup completes, or if another full backup fails resulting in a newer partial backup. Advantages: files from partial backup can be restored, on the next backup we don't need to retransfer files already in the partial backup (for rsync, not for tar). The situation when some files always are open during backup should be avoided, in general. In case if there are just a few files and they are always the same, it is probably a good idea to create an exclusion list. So if all xfer errors happen when attempting to transfer files that are in the exclusion list, do not mark that backup as partial. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Handle Backup as inconsistent on xfer error
On 26.01.2018 12:31, Zielke, Julian, NLI wrote: I’ve decided to rewrite the code of BackupPC_dump for this case. Here’s the patch: https://pastebin.com/wv1DFVbV I suggest to create an option in the configuration of the tar xfer method (bool / checkbox) so the user can decide whether to abort on Xfer errors or not. Why do you want to abort the backup? I think we should just mark it as partial. -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Handle Backup as inconsistent on xfer error
I've decided to rewrite the code of BackupPC_dump for this case. Here's the patch: https://pastebin.com/wv1DFVbV I suggest to create an option in the configuration of the tar xfer method (bool / checkbox) so the user can decide whether to abort on Xfer errors or not. * Julian Von: Zielke, Julian, NLI Gesendet: Donnerstag, 25. Januar 2018 14:58 An: General list for user discussion, questions and support <backuppc-users@lists.sourceforge.net> Betreff: [BackupPC-users] Handle Backup as inconsistent on xfer error Hi, is there some way I can make Backup PC handle the Backup as failed if tar is stumbling over a file which was changed/deleted while reading? * Julian -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Handle Backup as inconsistent on xfer error
Hi, is there some way I can make Backup PC handle the Backup as failed if tar is stumbling over a file which was changed/deleted while reading? * Julian -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
