Re: [BackupPC-users] Xfer errors (especially bpc_sysCall_checkFileMatch(/path/to/file): file doesn't exist)
On 2019-04-03 05:02, Daniel Berteaud wrote: Am I the only one having this kind of errors, or the only one caring about the errors ? Cheers Daniel Le 12/03/2019 à 08:56, Daniel Berteaud a écrit : Hi. I've been using BackupPC since the past 13 years (starting with 2.1.2 I think), with great success. I'm now running a few installations with v4, and one of the things I don't understand is the Xfer errors. I regularily check Xfer error to be sure my backups are OK. Before v3, I could reliabily check the number of Xfer errors, but in v4, I don't understand. First, as soon as there's a single error (most common are "file has vanished"), BackupPC reports 3 errors. Often, the backup reports an error count of 2, with these kind of errors : [ 213 lignes sautées ] rsync_bpc: stat "/.config/google-chrome/CertificateRevocation/5022" failed: No such file or directory (2) rsync_bpc: stat "/.config/google-chrome/CertificateRevocation/5022/_metadata" failed: No such file or directory (2) rsync_bpc: stat "/.config/google-chrome/CertificateRevocation/5023" failed: No such file or directory (2) rsync_bpc: stat "/.config/google-chrome/CertificateRevocation/5023/_metadata" failed: No such file or directory (2) These files seem likely to have been updated or removed during the backup run. That directory contains bookmarks, cookies, extensions, and all kinds of things one might expect to change. [ 106 lignes sautées ] [...] rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1676) [generator=3.1.2.0] rsync_bpc exited with benign status 23 (5888) This essentially summarizes that files weren't found during the rsync transfer that were identified before the transfer, which matches the symptom above, which is expected when trying to back up transient files. This is quite confusing because, in this example, the directories does exist, both on the source, and in the backup accounting these errors. For files which are frequently deleted and recreated (such as cache files) this is expected behavior. So, what does this error mean ? It looks like it happens when those directories did not exist in the previous backup (the one aginst which rsync is transfering deltas), but, why would this be an error ? rsync doesn't know why files would disappear or be created during a run -- as has been pointed out elsewhere, you may want to quiesce chrome, or (more sensibly) not try to back up transient files. Last, and the most important to me, I have some hosts with hundreds of Xfer errors like these : [ 53777 lignes sautées ] R bpc_sysCall_checkFileMatch(shares/assistantedirection/files/Recycle Bin/BIBLIOTHEQUE/DOCUMENTATION/AMIANTE/Courrier FEDEREC-CRAMIF-Demande de conventionnement.pdf): file doesn't exist R bpc_sysCall_checkFileMatch(shares/assistantedirection/files/Recycle Bin/BIBLIOTHEQUE/DOCUMENTATION/BTP/Courrier FEDEREC-CRAMIF-Demande de conventionnement.pdf): file doesn't exist R bpc_sysCall_checkFileMatch(shares/assistantedirection/files/Recycle Bin/DOSSIERS COMMUNS GROUPE PENA/ACHATS/DEMANDES D'ACHAT/2019/2 FEV 19/110319CB - GESTECO Château de France.docx): file doesn't exist [ 161 lignes sautées ] Those errors are not accounted in the Xfer error count, but displayed if we open the XferErr log. My C foo is not good enough to understand exactly what's going on in https://u2182357.ct.sendgrid.net/wf/click?upn=rBK8reUlX8Sxr7Iz1fV-2F7dCyrjIWd3LVbeqZFg4ygUGvSfi41qKNj5ycIIg4GkJF3ZYo8-2Fl9JPHIyP-2FT2T5NlSfZvMVW9z8xIQ8r-2FhomH40-3D_OypFYCWzG5ApGW-2FFpGTxc4RCS9eud0Dl1htN5rYoUZ8To4zeNUFBkAGI3hzer91CBnNjwF56OFtfIV34pOjuQRpX-2B3dRh-2FjPvm7nBC7QxnAw7MCkPTmIk5XYe1BLmUKuA6iTm8N0OT4B0OgafrFblGeYsQ8kd01o5RmDhP2n6-2B8W41NzlSMQcHJX7hUnGeIeAdm5W7NTHAd2gXpp7P0Ej3e-2BOUWZ1nTVQpIsrBRs8alPGkONXI1q-2Be9bHyoBqXWW Looks like this error can be logged when a file is moved (so it exists in the pool, but not at the same path in the previous backup agsint which we delta ?) Can someone confirm this is harmless ? I so, wouldn't it make sens to remove this from the XferErr (or maybe just include it if XferLogLevel > 3 or something like that ?) The symptoms here and the directory in question matches the behavior that is expected when backing up a Samba share used by Windows systems with access to a recycle bin. Windows will rename a file into the recycle bin, so you'll see this when you back up the original file -and- the recycle bin. I can't imagine why one would normally want to back up a recycle bin, since presumably one would have already backed up the file before it was deleted, so it's probably simplest just to exclude the bin from the backups. Alternatively, one can remove the bin (or access to it) so that Windows clients delete the original files, though that also makes backup of the recycle bin somewhat pointless. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/list
Re: [BackupPC-users] Xfer errors (especially bpc_sysCall_checkFileMatch(/path/to/file): file doesn't exist)
Le 04/04/2019 à 16:20, G.W. Haywood via BackupPC-users a écrit : > Hi there, Hi, thanks for your response :-) > > Like you, after using V3 for many years I'm playing with V4, but only > seriously since about August 2018. There were a few hiccups to begin > with, but I don't see the kinds of errors that you're seeing. > > No hosts here use the 'rsync' XferMethod. I've grepped XferLOGs for > the past year for errors on hosts which use the 'rsyncd' XferMethod, > and apart from a few expected errors where machines have been shut > down mid-backup, nothing is making itself obvious. Maybe the issue only manifest itself when using the rsync transport then (I use almost exclusively rsync over ssh). Craig, if you're arround, could you explain what those errors mean ? ++ ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] 0 repeated, 0 max chain in BackupPC 4
Hello, I am sorry for "flooding" the mailing-list with so many questions at once but I am currently in the process of debugging a problem and as part of that process new questions have arisen that I could not yet formulate at the time of my previous postings. I have two backup servers. The old server uses BackupPC 3 and ext4, the new server, which is causing me problems, uses BackupPC 4 and zfs. While comparing logs I have found the two entries which I believe could point in the direction of the problem. Old: 2019-03-19 06:48:27 Cpool is 1456.03GB, 8695243 files (2303 repeated, 373 max chain, 31999 max links), 4369 directories New: 2019-04-04 01:18:05 Cpool4 is 4118.27GB, 7700145 files (0 repeated, 0 max chain, 6883996 max links), 17028 directories Despite backing up the same servers as V3 V4 finds no repeated files and no chains. This is most likely the reason why the server is running full despite being in use less than 3 month and having triple the storage capacity of the old server. In V3 about 80% percent of space was used in the cpool, with the other 20% being allocated to pc. My questions are: Is this the cause of my capacity problems? Is this a problem of ZFS? How can I fix this? Yours sincerely Stefan Stefan Schumacher Systemadministrator NetFederation GmbH Sürther Hauptstraße 180 B - Fon:+49 (0)2236/3936-701 E-Mail: stefan.schumac...@net-federation.de Internet: http://www.net-federation.de Besuchen Sie uns doch auch auf facebook, twitter, Google+, flickr, Slideshare, XING oder unserem Blog. Wir freuen uns! *** HR Benchmark 2019: Bewerber in den Mittelpunkt! *** Wie bewerberfreundlich und auf welchem digitalen Stand sind die Karriereportale großer deutscher Unternehmen? Unsere Antworten finden Sie in unserem frisch veröffentlichten HR Benchmark unter: https://www.hr-benchmark.de/human-resources-benchmark-2019/benchmark * NetFederation GmbH Geschäftsführung: Christian Berens, Thorsten Greiten Amtsgericht Köln, HRB Nr. 32660 * The information in this e-mail is confidential and may be legally privileged. It is intended solely for the addressee and access to the e-mail by anyone else is unauthorised. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. If you have received this e-mail in error please forward to: post...@net-federation.de Die in dieser E-Mail enthaltenen Informationen sind vertraulich und können von rechtlicher Bedeutung sein. Diese Mail ist ausschließlich für den Adressaten bestimmt und jeglicher Zugriff durch andere Personen ist nicht zulässig. Falls Sie nicht der beabsichtigte Empfänger sind, ist jegliche Veröffentlichung, Vervielfältigung, Verteilung oder sonstige in diesem Zusammenhang stehende Handlung untersagt und unter Umständen ungesetzlich. Falls Sie diese E-Mail irrtümlich erhalten haben, leiten Sie sie bitte weiter an: post...@net-federation.de ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] Xfer errors (especially bpc_sysCall_checkFileMatch(/path/to/file): file doesn't exist)
Hi there, On Tue, 12 Mar 2019, Daniel Berteaud wrote: I've been using BackupPC since the past 13 years (starting with 2.1.2 I think), with great success. I'm now running a few installations with v4, and one of the things I don't understand is the Xfer errors. [...snip...] On Thu, 4 Apr 2019, Daniel Berteaud wrote: Am I the only one having this kind of errors, or the only one caring about the errors ? Your patience is commendable. :) Like you, after using V3 for many years I'm playing with V4, but only seriously since about August 2018. There were a few hiccups to begin with, but I don't see the kinds of errors that you're seeing. No hosts here use the 'rsync' XferMethod. I've grepped XferLOGs for the past year for errors on hosts which use the 'rsyncd' XferMethod, and apart from a few expected errors where machines have been shut down mid-backup, nothing is making itself obvious. I've grepped BackupPC XferLOGs going all the way back to 2016 for the strings 'No such file or directory' and 'bpc_sysCall_checkFileMatch'. There was no instance of 'No such file or directory', and just a few 'bpc_sysCall_checkFileMatch ... file doesn't exist'. The majority of the latter were in the hidden sub-directories of home directories on machines which run X sessions/desktops. The errors are reasonable and since moving to BPC V4 I've routinely excluding those directories from backups anyway. The error has never appeared for any of our server machines which offer only shell access. HTH -- 73, Ged. ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Another seeming BUG in BackupPC_migrateV3toV4
When I run 'BackupPC_migrateV3toV4' (v4.3.0), I get occasional errors
of form:
setVarInt botch: got negative argument -1015087104; setting to 0
NOTE: that the error is rare -- I migrated about 2 million pool files
repeated across 100 backups... yet I only encountered a couple of
dozen such errors.
Note the error seems to be generated in bpc_attrib.c in the following
subroutine:
/*
* Write variable-length unsigned integer in 7 bit chunks, LSB first
*/
static void setVarInt(uchar **bufPP, uchar *bufEnd, int64 value)
{
uchar *bufP = *bufPP;
if ( value < 0 ) {
bpc_logErrf("setVarInt botch: got negative argument %ld;
setting to 0\n", (long int)value);
value = 0;
}
do {
uchar c = value & 0x7f;
value >>= 7;
if ( value ) c |= 0x80;
if ( bufP < bufEnd ) {
*bufP++ = c;
} else {
bufP++;
}
} while ( value );
*bufPP = bufP;
}
--
Here is some context from the (verbose) output of
BackupPC_migrateV3toV4:
Wrote file
cache_http%3A%2F%2Fi.ebayimg.com%2F00%2Fs%2FMTc2WDE3Ng%3D%3D%2F%24%28KGrHqNHJE%21FCdYT6%21EDBQt9otljTQ%7E%7E60_57.PNG%3Fset_id%3D4002:
type = 0, mode = 0100444, uid/gid = 490/490, size = 49237, inode = 231882,
nlinks = 0, digest = 0x0cb6af..., bufUsed = 636
Wrote file
cache_http%3A%2F%2Fi.ebayimg.com%2F00%2Fs%2FMjExWDMwNg%3D%3D%2Fz%2Fo5UAAOxyMQtSaXHu%2F%24%28KGrHqZHJBoFJfGTBIQCBS%2C%28HuG2FQ%7E%7E60_57.PNG%3Fset_id%3D4002:
type = 0, mode = 0100444, uid/gid = 490/490, size = 10548, inode = 231885,
nlinks = 0, digest = 0x79a2c2..., bufUsed = 810
setVarInt botch: got negative argument -1015087104; setting to 0
Wrote file
cache_http%3A%2F%2Fi.ebayimg.com%2F00%2Fs%2FMTE3WDMxNA%3D%3D%2Fz%2FH7sAAOxyRNJScAaq%2F%24%28KGrHqF%2C%21k8FJlR29NWMBSc%21%2Cp7SSw%7E%7E60_57.PNG%3Fset_id%3D4002:
type = 0, mode = 00, uid/gid = 0/0, size = 0, inode = 231883, nlinks = 0,
digest = 0x7b428e..., bufUsed = 1006
Wrote file
cache_http%3A%2F%2Fi.ebayimg.com%2F00%2Fs%2FMzMxWDQwMA%3D%3D%2Fz%2FFMgAAOxydgZTI2GK%2F%24_1.JPG%3Fset_id%3D2:
type = 0, mode = 0100444, uid/gid = 490/490, size = 17859, inode = 231884,
nlinks = 0, digest = 0x99c51e..., bufUsed = 1197
Wrote file
cache_http%3A%2F%2Fi.ebayimg.com%2F00%2Fs%2FMTc2WDE3Ng%3D%3D%2F%24T2eC16ZHJHIE9nyseyoZBQt9oyl1o%21%7E%7E60_57.PNG%3Fset_id%3D4002:
type = 0, mode = 0100444, uid/gid = 490/490, size = 15206, inode = 231881,
nlinks = 0, digest = 0x97c615..., bufUsed = 1345
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
