Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[fi]

Dear Holger, 

> f...@igh.de wrote on 2018-03-08 16:59:37 +0100 [Re: [BackupPC-users] Serious 
> error: last backup ... directory doesn't exist!!! - reason found]:
> > [...]
> > Meanwhile I found the reason: the partition ran out of inodes. As you
> > wrote under "How much disk space do I need?" one has to have "plenty
> > of inodes". But what does that mean? 
> 
> as has been said, that depends directly on what you are backing up.

yes of course, but there are other signicant influences. I have
counted the number of inodes in the cpool and in the pc
directories. The latter was about 100 times higher. Obviously the
number of kept backups is a factor to consider. It seems, the number
n_i of inodes required is about 

   n_i = n_f (2 n_b + 1), 

where 
  n_b: number of backups kept 
  n_f: number of files to backup
  
could anyone confirm or disconfirm this? 



> > May I ask the following:
> > 
> > - in the "General Server Information" you give some statistical
> >   information about disk usage; would it be a good idea also to give
> >   information about inode consumption? 
> 
> You mean 'df -i'?

yes exactly! 

I have created a post backup notification e-mail which now contains
also the output of "df -i". 

BTW.: such a post backup notification I would recommend as
default. It is not a good idea to conclude a successful backup from a
missing message...


> > - is it possible and would it make sense to separate the "pc" and the
> >   "pool/cpool" directories into different partitions? I just did an
> >   rsync of a BackupPC-directory and found that the files on "pc" are
> >   mostly empty or small. The file sizes in "pool/cpool" are remarkable
> >   bigger - I assume these are the "real" files. So one could create
> >   one partition for "pool/cpool" having about e.g. 64kB per inode and
> >   another partition having a block size of 1 kB and also 1 kB per
> >   inode. Maybe this would reduce disk space consumption and also allow 
> >   rsyncing somewhat faster. 
> 
> My first thought is to avoid the issue altogether by using a file system
> that doesn't statically allocate inodes (e.g. XFS or reiserfs, the latter
> I wouldn't recommend for other reasons, though; I don't know about ext4,
> btrfs and ZFS, but my guess would be that ext4 has static allocation and
> the others dynamic). Why worry about a problem modern file systems simply
> don't have?

I have made several tests with different file systems (xfs, reiserfs,
jfs). After some time of operation I had different kinds of trouble with
all of them; especially xfs seems to be sensitive against power
blackout (I know UPS could help, but this increases the effort of battery
life cycle management; sometimes I found UPS caused more trouble than solved). 

I consider ext[3/4] the far most robust file system. So if I know how
to calculate the required number of inodes it will stay my favorite. 


Best regards


Torsten


-- 

Torsten Finke
f...@igh.de

Ingenieurgemeinschaft IgH
Gesellschaft für Ingenieurleistungen mbH
Heinz-Bäcker-Str. 34
D-45356 Essen



--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[Michael Stowe]

On 2018-03-13 16:33, Holger Parplies wrote:
My first thought is to avoid the issue altogether by using a file 
system
that doesn't statically allocate inodes (e.g. XFS or reiserfs, the 
latter
I wouldn't recommend for other reasons, though; I don't know about 
ext4,
btrfs and ZFS, but my guess would be that ext4 has static allocation 
and
the others dynamic). Why worry about a problem modern file systems 
simply

don't have?

Regards,
Holger


I happen to have used BackupPC with ext4, xfs, and btrfs.  ext4 (unlike 
ext2 and ext3) does indeed have a way to extend the number of inodes 
beyond the static allocation.  XFS, elegant and well designed, gave me 
no end of trouble, frankly.  Your mileage may vary, but on more than one 
occasion, I found subtle corruption for unknown reasons had caused weird 
ripple effects throughout the file system.  btrfs has, perhaps 
ironically, proven the most reliable of the three.  Again, your mileage 
may vary, and while I can go into considerable detail about my own 
experiences and testing, they are limited in scope and nature.
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[Holger Parplies]

Hi,

f...@igh.de wrote on 2018-03-08 16:59:37 +0100 [Re: [BackupPC-users] Serious 
error: last backup ... directory doesn't exist!!! - reason found]:
> [...]
> Meanwhile I found the reason: the partition ran out of inodes. As you
> wrote under "How much disk space do I need?" one has to have "plenty
> of inodes". But what does that mean? 

as has been said, that depends directly on what you are backing up.

> May I ask the following:
> 
> - in the "General Server Information" you give some statistical
>   information about disk usage; would it be a good idea also to give
>   information about inode consumption? 

You mean 'df -i'?

> - is it possible and would it make sense to separate the "pc" and the
>   "pool/cpool" directories into different partitions? I just did an
>   rsync of a BackupPC-directory and found that the files on "pc" are
>   mostly empty or small. The file sizes in "pool/cpool" are remarkable
>   bigger - I assume these are the "real" files. So one could create
>   one partition for "pool/cpool" having about e.g. 64kB per inode and
>   another partition having a block size of 1 kB and also 1 kB per
>   inode. Maybe this would reduce disk space consumption and also allow 
>   rsyncing somewhat faster. 

My first thought is to avoid the issue altogether by using a file system
that doesn't statically allocate inodes (e.g. XFS or reiserfs, the latter
I wouldn't recommend for other reasons, though; I don't know about ext4,
btrfs and ZFS, but my guess would be that ext4 has static allocation and
the others dynamic). Why worry about a problem modern file systems simply
don't have?

Regards,
Holger

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[frush]

One of the more challenging aspects of this is the number of inodes that 
BackupPC consumes
is directly related to how many inodes you're backing up.   If I recall 
correctly, in
BackupPC 4.x, each 'filled' backup will use 2 inodes per object backed up in 
addition to
the inode consumed by the unique hashed file.
One has to understand well the environment they are backing up in order to make 
a informed
choice about the number of inodes required by BackupPC.   I know I didn't on my 
first
pass!  Our current set of backups (~30 days) uses 93 Million inodes and 4TB of 
space to
backup 126 hosts.   Metadata alone takes up 357MB on our NAS.
--Ray FrushColorado State University.
On Thu, 2018-03-08 at 19:54 +0300, Alexander Moisseev via BackupPC-users wrote:
> On 3/8/2018 6:59 PM, f...@igh.de wrote:
> > Craig,
> > 
> > again I return to my issue "No space left on device".
> > 
> > Meanwhile I found the reason: the partition ran out of inodes. As you
> > wrote under "How much disk space do I need?" one has to have "plenty
> > of inodes". But what does that mean?
> > 
> > May I ask the following:
> > 
> > - in the "General Server Information" you give some statistical
> >information about disk usage; would it be a good idea also to give
> >information about inode consumption?
> > 
> 
> It is a really good idea, but obtaining inode consumption with du command is 
> complicated
> since it returns different sets of columns on different OSes.
> I think the simplest way is to replace CheckFileSystemUsage subroutine with
> Filesys::DiskSpace module.
> 
> Craig, is it ok to introduce another dependency?
> 
> --
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> ___
> BackupPC-users mailing list
> BackupPC-users@lists.sourceforge.net
> List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
> Wiki:http://backuppc.wiki.sourceforge.net
> Project: http://backuppc.sourceforge.net/--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[Alexander Moisseev via BackupPC-users]

On 3/8/2018 6:59 PM, f...@igh.de wrote:

Craig,

again I return to my issue "No space left on device".

Meanwhile I found the reason: the partition ran out of inodes. As you
wrote under "How much disk space do I need?" one has to have "plenty
of inodes". But what does that mean?

May I ask the following:

- in the "General Server Information" you give some statistical
   information about disk usage; would it be a good idea also to give
   information about inode consumption?



It is a really good idea, but obtaining inode consumption with du command is 
complicated since it returns different sets of columns on different OSes.
I think the simplest way is to replace CheckFileSystemUsage subroutine with 
Filesys::DiskSpace module.

Craig, is it ok to introduce another dependency?

--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Serious error: last backup ... directory doesn't exist!!! - reason found

[fi]

Craig,

again I return to my issue "No space left on device". 

Meanwhile I found the reason: the partition ran out of inodes. As you
wrote under "How much disk space do I need?" one has to have "plenty
of inodes". But what does that mean? 

May I ask the following:

- in the "General Server Information" you give some statistical
  information about disk usage; would it be a good idea also to give
  information about inode consumption? 

- is it possible and would it make sense to separate the "pc" and the
  "pool/cpool" directories into different partitions? I just did an
  rsync of a BackupPC-directory and found that the files on "pc" are
  mostly empty or small. The file sizes in "pool/cpool" are remarkable
  bigger - I assume these are the "real" files. So one could create
  one partition for "pool/cpool" having about e.g. 64kB per inode and
  another partition having a block size of 1 kB and also 1 kB per
  inode. Maybe this would reduce disk space consumption and also allow 
  rsyncing somewhat faster. 


Best regards


Torsten



> Thanks for the additional info, but I can't explain the behavior your are
> seeing.  Sending to the user-list to see if others have suggestions.
> 
> BackupPC seems to be unable to create directories and files below
> /vol/bpc4/BackupPC.  The "No space left on device" errno seems inconsistent
> with the output of df.
> 
> Although it won't address any of your problems, I pushed a fix
> 
> for the error caused by the failure to open a directory.
> 
> Craig
> 
> On Sat, Feb 3, 2018 at 3:56 AM,  wrote:
> 
> > Dear Craig,
> >
> > On Fri, Feb 02, 2018 at 09:19:02AM -0800, Craig Barratt via BackupPC-users
> > wrote:
> > > Here are a few things to check:
> > >
> > >- can you manually create files and directories below
> > /vol/bpc4/BackupPC
> > >as the backuppc user (eg, "su backuppc; cd /vol/bpc4/BackupPC; mkdir
> > foo;
> > >rmdir foo; touch foobar; rm foobar")
> >
> > yes, works as expected.
> >
> > >- is the file system mounted readonly?
> >
> > no, mount says:
> > ... /vol/bpc4 type ext4 (rw,noatime,data=ordered)
> >
> > >- has the backuppc user lost permissions?
> >
> > in what sense? All BackupPC processes run as user backuppc; all files
> > and directories /vol/bpc4/BackupPC and below are owned by
> > backuppc:backuppc.
> >
> >
> > df says
> > Filesystem Size  Used Avail Use% Mounted on
> > /dev/mapper/data-bpc4  1.6T  955G  535G  65% /vol/bpc4
> >
> > The file system lives on a mirror raid, which seems to be ok:
> >
> > md127 : active raid1 sdb1[1] sdc1[0]
> >   2930266432 blocks [2/2] [UU]
> >
> > There are no SMART errors logged on the disks (even after SMART test).
> >
> > I have copied several GB to the file system and removed them again -
> > no errors.
> >
> > I have launched a forced fsck. It succeeded without any errors:
> > Durchgang 1: Inodes, Blöcke und Größen werden geprüft
> > Durchgang 2: Verzeichnisstruktur wird geprüft
> > Durchgang 3: Verzeichnisverknüpfungen werden geprüft
> > Durchgang 4: Referenzzähler werden überprüft
> > Durchgang 5: Zusammengefasste Gruppeninformation wird geprüft
> > /dev/data/bpc4: 104312919/104505344 Dateien (0.0% nicht
> > zusammenhängend), 256909
> > 738/417993728 Blöcke
> >
> >
> > I went through the logs. There are more error messages from a backup
> > which was presented as "running" in the summary. Rather strange sound
> > the messages "IO error encountered" and "No space left on device "
> > (please see below). I cannot figure out the origin of these messages.
> >
> > ...
> > G bpc_fileReadAll: can't open
> > /vol/bpc4/BackupPC/cpool/20/9a/219b13a5a32e6cd66ab3cf139bd8d35c (from
> > alternatives/pg_config)
> > file has vanished: "alternatives/pg_config"
> > ...
> >
> > ...
> > IO error encountered -- skipping file deletion
> > G bpc_attrib_dirRead: can't open
> > /vol/bpc4/BackupPC/cpool/26/24/27243e59e0d4404f14ddc7ebb0202ce6
> > G bpc_attribCache_loadPath:
> > bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169, fetc/fapache2/attrib)
> > returned -1
> > ...
> >
> > ...
> > R bpc_attrib_dirRead: can't open
> > /vol/bpc4/BackupPC/cpool/1e/aa/1eab8cca5e477b6fe2530e955cca3978
> > R bpc_attribCache_loadPath:
> > bpc_attrib_dirRead(/vol/bpc4/BackupPC/pc/zuse/169,
> > fetc/fcups/fssl/attrib) returned -1
> > [ skipped 6 lines ]
> > ...
> >
> >
> > ...
> > R
> > R
> > bpc_sysCall_checkFileMatch(.cpan/build/DateTime-Locale-1.
> > 05-GcMGii/pm_to_blib):
> > file doesn't exist
> > ...
> >
> > ...
> > R bpc_poolWrite_write: can't open/create
> > /vol/bpc4/BackupPC/cpool/18801.736.0 for writingIOdone: pool
> > .cpan/build/DateTime-Locale-1.05-GcMGii/blib/lib/DateTime/
> > Locale/.en_NR.pod.00
> > ...
> >
> > ...
> > R
> > bpc_poolWrite_unmarkPendingDelete(/vol/bpc4/BackupPC/cpool/50/02/
> > 50032f7b52be451f6a6c396067253906)
> > failed; errno = 2
> > R Couldn't unmark candidate matching file
> >