Torsten,
Thanks for the additional info, but I can't explain the behavior your are
seeing. Sending to the user-list to see if others have suggestions.
BackupPC seems to be unable to create directories and files below
/vol/bpc4/BackupPC. The "No space left on device" errno seems inconsistent
with the output of df.
Although it won't address any of your problems, I pushed a fix
<https://github.com/backuppc/backuppc/commit/265431fff889bca2624541b05cdfbbaeae7a0039>
for the error caused by the failure to open a directory.
Craig
On Sat, Feb 3, 2018 at 3:56 AM, <f...@igh.de> wrote:
> Dear Craig,
>
> On Fri, Feb 02, 2018 at 09:19:02AM -0800, Craig Barratt via BackupPC-users
> wrote:
> > Here are a few things to check:
> >
> > - can you manually create files and directories below
> /vol/bpc4/BackupPC
> > as the backuppc user (eg, "su backuppc; cd /vol/bpc4/BackupPC; mkdir
> foo;
> > rmdir foo; touch foobar; rm foobar")
>
> yes, works as expected.
>
> > - is the file system mounted readonly?
>
> no, mount says:
> ... /vol/bpc4 type ext4 (rw,noatime,data=ordered)
>
> > - has the backuppc user lost permissions?
>
> in what sense? All BackupPC processes run as user backuppc; all files
> and directories /vol/bpc4/BackupPC and below are owned by
> backuppc:backuppc.
>
>
> df says
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/data-bpc4 1.6T 955G 535G 65% /vol/bpc4
>
> The file system lives on a mirror raid, which seems to be ok:
>
> md127 : active raid1 sdb1[1] sdc1[0]
> 2930266432 blocks [2/2] [UU]
>
> There are no SMART errors logged on the disks (even after SMART test).
>
> I have copied several GB to the file system and removed them again -
> no errors.
>
> I have launched a forced fsck. It succeeded without any errors:
> Durchgang 1: Inodes, Blöcke und Größen werden geprüft
> Durchgang 2: Verzeichnisstruktur wird geprüft
> Durchgang 3: Verzeichnisverknüpfungen werden geprüft
> Durchgang 4: Referenzzähler werden überprüft
> Durchgang 5: Zusammengefasste Gruppeninformation wird geprüft
> /dev/data/bpc4: 104312919/104505344 Dateien (0.0% nicht
> zusammenhängend), 256909
> 738/417993728 Blöcke
>
>
> I went through the logs. There are more error messages from a backup
> which was presented as "running" in the summary. Rather strange sound
> the messages "IO error encountered" and "No space left on device "
> (please see below). I cannot figure out the origin of these messages.
>
> ...
> G bpc_fileReadAll: can't open
> /vol/bpc4/BackupPC/cpool/20/9a/219b13a5a32e6cd66ab3cf139bd8d35c (from
> alternatives/pg_config)
> file has vanished: "alternatives/pg_config"
> ...
>
> ...
> IO error encountered -- skipping file deletion
> G bpc_attrib_dirRead: can't open
> /vol/bpc4/BackupPC/cpool/26/24/27243e59e0d4404f14ddc7ebb0202ce6
> G bpc_attribCache_loadPath:
> bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, fetc/fapache2/attrib)
> returned -1
> ...
>
> ...
> R bpc_attrib_dirRead: can't open
> /vol/bpc4/BackupPC/cpool/1e/aa/1eab8cca5e477b6fe2530e955cca3978
> R bpc_attribCache_loadPath:
> bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169,
> fetc/fcups/fssl/attrib) returned -1
> [ skipped 6 lines ]
> ...
>
>
> ...
> R
> R
> bpc_sysCall_checkFileMatch(.cpan/build/DateTime-Locale-1.
> 05-GcMGii/pm_to_blib):
> file doesn't exist
> ...
>
> ...
> R bpc_poolWrite_write: can't open/create
> /vol/bpc4/BackupPC/cpool/18801.736.0 for writingIOdone: pool
> .cpan/build/DateTime-Locale-1.05-GcMGii/blib/lib/DateTime/
> Locale/.en_NR.pod.000000
> ...
>
> ...
> R
> bpc_poolWrite_unmarkPendingDelete(/vol/bpc4/BackupPC/cpool/50/02/
> 50032f7b52be451f6a6c396067253906)
> failed; errno = 2
> R Couldn't unmark candidate matching file
> /vol/bpc4/BackupPC/cpool/50/02/50032f7b52be451f6a6c396067253906
> (skipped; errno = 2)
> ...
>
>
> The following is very strange:
> ...
> rsync_bpc: recv_generator: mkdir
> "/.cpan/build/Dist-CheckConflicts-0.11-RNgNPa/t/lib" failed: No space
> left on device (28)
> ...
>
> dumpe2fs says:
>
> ...
> Inode count: 104505344
> Block count: 417993728
> Reserved block count: 20899686
> Free blocks: 243883988
> Free inodes: 97119479
> ...
>
>
>
> Best regards
>
>
> Torsten Finke
>
>
>
>
> > It does look like the failure to update the filesystem exposes a bug
> which
> > I'll fix, but this is secondary to your main issue:
> >
> > Can't use an undefined value as an ARRAY reference at
> > /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380
> >
> >
> > Craig
> >
> > On Fri, Feb 2, 2018 at 5:56 AM, <f...@igh.de> wrote:
> >
> > > Dear BackupPC Users,
> > >
> > >
> > > I am running BackupPC-4.1.3 for several months without problems
> > > (really a great piece of software).
> > >
> > > About a week ago a strange type of error occured and keeps staying,
> > > which I cannot understand. The error messages are like this ("fs" is
> > > the name of one of my systems to be backed up; I did some linebreaks
> > > to increase readability):
> > >
> > >
> > > 2018-01-28 23:43:34 Can't create /vol/bpc4/BackupPC/pc/fs/149/refCnt
> > > 2018-01-29 02:09:34 Serious error: last backup
> > > /vol/bpc4/BackupPC/pc/fs/149 directory doesn't exist!!! Need to
> > > remove back to last filled backup
> > > 2018-01-29 02:09:34 Deleting backup 147
> > > 2018-01-29 02:09:35 BackupPC_backupDelete: removing #147
> > > 2018-01-29 02:09:35 BackupPC_backupDelete: No prior backup for merge
> > > 2018-01-29 02:09:52 BackupPC_refCountUpdate: doing fsck on fs #149
> > > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist
> > > 2018-01-29 02:09:52 Can't use an undefined value as an ARRAY reference
> > > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380.
> > > 2018-01-29 02:09:52 BackupPC_backupDelete: got 1 errors
> > > 2018-01-29 02:09:52 Finished BackupPC_backupDelete, status = 256
> > > (running time: 18 sec)
> > > 2018-01-29 02:09:52 Deleting backup 148
> > > 2018-01-29 02:09:52 BackupPC_backupDelete: removing #148
> > > 2018-01-29 02:09:52 BackupPC_backupDelete: No prior backup for merge
> > > 2018-01-29 02:09:55 BackupPC_refCountUpdate: doing fsck on fs #149
> > > since /vol/bpc4/BackupPC/pc/fs/149/refCnt doesn't exist
> > > 2018-01-29 02:09:55 Can't use an undefined value as an ARRAY reference
> > > at /vol/opt/BackupPC-4.1.3/bin/BackupPC_refCountUpdate line 380.
> > > 2018-01-29 02:09:55 BackupPC_backupDelete: got 1 errors
> > > 2018-01-29 02:09:55 Finished BackupPC_backupDelete, status = 256
> > > (running time: 3 sec)
> > > 2018-01-29 02:09:55 Deleting backup 149
> > > 2018-01-29 02:09:56 BackupPC_backupDelete: removing #149
> > > 2018-01-29 02:09:56 BackupPC_backupDelete: No prior backup for merge
> > > 2018-01-29 02:38:20 BackupPC_refCountUpdate: host fs got 0 errors
> > > (took 1704 secs)
> > > 2018-01-29 02:38:20 Finished BackupPC_backupDelete, status = 0
> > > (running time: 1705 sec)
> > > 2018-01-29 03:16:29 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-29 03:55:15 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-29 04:15:15 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-29 23:43:54 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-30 01:24:02 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-30 02:36:01 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-30 03:42:01 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-30 04:12:01 dump failed: unable to open/create
> > > /vol/bpc4/BackupPC/pc/fs/XferLOG.147.z
> > > 2018-01-30 23:57:53 Copying backup #146 to #147
> > >
> > > 2018-01-31 01:05:38 Can't mkpath
> > > /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00
> > > 2018-01-31 01:05:38 Can't copy
> > > ./fsysbak/f.../fLocked/f00/attrib_dc793dc4650baa1a8cc3b69d211573c1
> > > to /vol/bpc4/BackupPC/pc/fs/147/./fsysbak/f.../fLocked/f00/attrib_
> > > dc793dc4650baa1a8cc3b69d211573c1
> > > ...
> > >
> > > The last pair of messages is then repeated for thousands of files.
> > >
> > >
> > > It seems, that there is an error concerning the sequence.
> > >
> > > I had checked for several default reasons:
> > > - No files or directories have been manipulated manually
> > > - BackupPC is almost the only task running on my backup server (there
> > > should be no side effects from other services; no user is normally
> > > working on that machine)
> > > - the file system has about 1.6 TB at a usage of 65% (this rate was
> > > never exceeded)
> > > - the system had no crash (uptime is 217 days, since setup of
> > > BackupPC).
> > > - /vol/bpc4 is a mount point of an ext4 file system (normally very
> > > reliable)
> > > - Kernel 4.1.39, OpenSuSE 42.1 - not really outdated
> > >
> > >
> > >
> > >
> > > Any advice to repair this error is highly appretiated.
> > >
> > >
> > > Best regards
> > >
> > >
> > > Torsten Finke
> > >
> > >
> > >
> --
> ------------------------------------------------------------------------
> Torsten Finke
> f...@igh.de
>
> Ingenieurgemeinschaft IgH
> Gesellschaft für Ingenieurleistungen mbH
> Heinz-Bäcker-Str. 34
> D-45356 Essen
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki: http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List: https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki: http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/
