Dear Craig,
On Fri, Feb 02, 2018 at 09:19:02AM -0800, Craig Barratt via BackupPC-users
wrote:
> Here are a few things to check:
>
> - can you manually create files and directories below /vol/bpc4/BackupPC
> as the backuppc user (eg, "su backuppc; cd /vol/bpc4/BackupPC; mkdir foo;
> rmdir foo; touch foobar; rm foobar")
yes, works as expected.
> - is the file system mounted readonly?
no, mount says:
... /vol/bpc4 type ext4 (rw,noatime,data=ordered)
> - has the backuppc user lost permissions?
in what sense? All BackupPC processes run as user backuppc; all files
and directories /vol/bpc4/BackupPC and below are owned by backuppc:backuppc.
df says
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/data-bpc4 1.6T 955G 535G 65% /vol/bpc4
The file system lives on a mirror raid, which seems to be ok:
md127 : active raid1 sdb1[1] sdc1[0]
2930266432 blocks [2/2] [UU]
There are no SMART errors logged on the disks (even after SMART test).
I have copied several GB to the file system and removed them again -
no errors.
I have launched a forced fsck. It succeeded without any errors:
Durchgang 1: Inodes, Blöcke und Größen werden geprüft
Durchgang 2: Verzeichnisstruktur wird geprüft
Durchgang 3: Verzeichnisverknüpfungen werden geprüft
Durchgang 4: Referenzzähler werden überprüft
Durchgang 5: Zusammengefasste Gruppeninformation wird geprüft
/dev/data/bpc4: 104312919/104505344 Dateien (0.0% nicht
zusammenhängend), 256909
738/417993728 Blöcke
I went through the logs. There are more error messages from a backup
which was presented as "running" in the summary. Rather strange sound
the messages "IO error encountered" and "No space left on device "
(please see below). I cannot figure out the origin of these messages.
...
G bpc_fileReadAll: can't open
/vol/bpc4/BackupPC/cpool/20/9a/219b13a5a32e6cd66ab3cf139bd8d35c (from
alternatives/pg_config)
file has vanished: "alternatives/pg_config"
...
...
IO error encountered -- skipping file deletion
G bpc_attrib_dirRead: can't open
/vol/bpc4/BackupPC/cpool/26/24/27243e59e0d4404f14ddc7ebb0202ce6
G bpc_attribCache_loadPath:
bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, fetc/fapache2/attrib)
returned -1
...
...
R bpc_attrib_dirRead: can't open
/vol/bpc4/BackupPC/cpool/1e/aa/1eab8cca5e477b6fe2530e955cca3978
R bpc_attribCache_loadPath:
bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, fetc/fcups/fssl/attrib)
returned -1
[ skipped 6 lines ]
...
...
R
R
bpc_sysCall_checkFileMatch(.cpan/build/DateTime-Locale-1.05-GcMGii/pm_to_blib):
file doesn't exist
...
...
R bpc_poolWrite_write: can't open/create
/vol/bpc4/BackupPC/cpool/18801.736.0 for writingIOdone: pool
.cpan/build/DateTime-Locale-1.05-GcMGii/blib/lib/DateTime/Locale/.en_NR.pod.000000
...
...
R
bpc_poolWrite_unmarkPendingDelete(/vol/bpc4/BackupPC/cpool/50/02/50032f7b52be451f6a6c396067253906)
failed; errno = 2
R Couldn't unmark candidate matching file
/vol/bpc4/BackupPC/cpool/50/02/50032f7b52be451f6a6c396067253906
(skipped; errno = 2)
...
The following is very strange:
...
rsync_bpc: recv_generator: mkdir
"/.cpan/build/Dist-CheckConflicts-0.11-RNgNPa/t/lib" failed: No space left on
device (28)
...
dumpe2fs says:
...
Inode count: 104505344
Block count: 417993728
Reserved block count: 20899686
Free blocks: 243883988
Free inodes: 97119479
...
Best regards
Torsten Finke
> It does look like the failure to update the filesystem exposes a bug which
> I'll fix, but this is secondary to your main issue:
>
> Can't use an undefined value as an ARRAY reference at
> /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380
>
>
> Craig
>
> On Fri, Feb 2, 2018 at 5:56 AM, <f...@igh.de> wrote:
>
> > Dear BackupPC Users,
> >
> >
> > I am running BackupPC-4.1.3 for several months without problems
> > (really a great piece of software).
> >
> > About a week ago a strange type of error occured and keeps staying,
> > which I cannot understand. The error messages are like this ("fs" is
> > the name of one of my systems to be backed up; I did some linebreaks
> > to increase readability):
> >
> >
> > 2018-01-28 23:43:34 Can't create /vol/bpc4/BackupPC/pc/fs/149/refCnt
> > 2018-01-29 02:09:34 Serious error: last backup
> > /vol/bpc4/BackupPC/pc/fs/149 directory doesn't exist!!! Need to
> > remove back to last filled backup
> > 2018-01-29 02:09:34 Deleting backup 147
> > 2018-01-29 02:09:35 BackupPC_backupDelete: removing #147
> > 2018-01-29 02:09:35 BackupPC_backupDelete: No prior backup for merge
> > 2018-01-29 02:09:52 BackupPC_refCountUpdate: doing fsck on fs #149
> > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist
> > 2018-01-29 02:09:52 Can't use an undefined value as an ARRAY reference
> > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380.
> > 2018-01-29 02:09:52 BackupPC_backupDelete: got 1 errors
> > 2018-01-29 02:09:52 Finished BackupPC_backupDelete, status = 256
> > (running time: 18 sec)
> > 2018-01-29 02:09:52 Deleting backup 148
> > 2018-01-29 02:09:52 BackupPC_backupDelete: removing #148
> > 2018-01-29 02:09:52 BackupPC_backupDelete: No prior backup for merge
> > 2018-01-29 02:09:55 BackupPC_refCountUpdate: doing fsck on fs #149
> > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist
> > 2018-01-29 02:09:55 Can't use an undefined value as an ARRAY reference
> > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380.
> > 2018-01-29 02:09:55 BackupPC_backupDelete: got 1 errors
> > 2018-01-29 02:09:55 Finished BackupPC_backupDelete, status = 256
> > (running time: 3 sec)
> > 2018-01-29 02:09:55 Deleting backup 149
> > 2018-01-29 02:09:56 BackupPC_backupDelete: removing #149
> > 2018-01-29 02:09:56 BackupPC_backupDelete: No prior backup for merge
> > 2018-01-29 02:38:20 BackupPC_refCountUpdate: host fs got 0 errors
> > (took 1704 secs)
> > 2018-01-29 02:38:20 Finished BackupPC_backupDelete, status = 0
> > (running time: 1705 sec)
> > 2018-01-29 03:16:29 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-29 03:55:15 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-29 04:15:15 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-29 23:43:54 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-30 01:24:02 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-30 02:36:01 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-30 03:42:01 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-30 04:12:01 dump failed: unable to open/create
> > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > 2018-01-30 23:57:53 Copying backup #146 to #147
> >
> > 2018-01-31 01:05:38 Can't mkpath
> > /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00
> > 2018-01-31 01:05:38 Can't copy
> > ./fsysbak/f.../fLocked/f00/attrib_dc793dc4650baa1a8cc3b69d211573c1
> > to /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00/attrib_
> > dc793dc4650baa1a8cc3b69d211573c1
> > ...
> >
> > The last pair of messages is then repeated for thousands of files.
> >
> >
> > It seems, that there is an error concerning the sequence.
> >
> > I had checked for several default reasons:
> > - No files or directories have been manipulated manually
> > - BackupPC is almost the only task running on my backup server (there
> > should be no side effects from other services; no user is normally
> > working on that machine)
> > - the file system has about 1.6 TB at a usage of 65% (this rate was
> > never exceeded)
> > - the system had no crash (uptime is 217 days, since setup of
> > BackupPC).
> > - /vol/bpc4 is a mount point of an ext4 file system (normally very
> > reliable)
> > - Kernel 4.1.39, OpenSuSE 42.1 - not really outdated
> >
> >
> >
> >
> > Any advice to repair this error is highly appretiated.
> >
> >
> > Best regards
> >
> >
> > Torsten Finke
> >
> >
> >
--
------------------------------------------------------------------------
Torsten Finke
f...@igh.de
Ingenieurgemeinschaft IgH
Gesellschaft für Ingenieurleistungen mbH
Heinz-Bäcker-Str. 34
D-45356 Essen
------------------------------------------------------------------------
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
