Re: [BackupPC-users] errors on network-mounted pool filesystem (backu...@kosowsky.org)

[Sandro Bordacchini]

So, it actually happened that was exactly the problem.

It seems that a mounted sshfs "simulates" support to hardlinks creating a
copy of the file when you actually issue the creation of a hardlink.
I did some tests and when i created a hardlink to a file (both within the
sshfs) an identical file is created with DIFFERENT inode.

I did some research and i found out that is the default behaviour ... but
it can be disabled adding:
disable_hardlink
to the mount options.
With this option the filesystem throws an error if you try to create a
hardlink in it.

So, in the end i was able to mount the sshfs and make it work (aside from
performance issues) with backuppc with these options:

sshfs#
 
disable_hardlink,idmap=user,allow_other,IdentityFile=,port=,ServerAliveInterval=15,reconnect,_netdev,uid=108,gid=114,cache=yes,kernel_cache,compression=no,Ciphers=
aes128-...@openssh.com 0 0

where uid and gid are the backupc user id and group id

the ssh options ServerAliveInterval=15 and ServerAliveCountMax (not present
here) can be added/edited to improve reconnect time in case of a networking
problem
the options cache=yes,kernel_cache,compression=no were used to improve
performance and also the cipher-forcing option was used to force the
least-cpu-intense common cipher between client and server (cfr.
nmap ssh2-enum-algos script).

A word about performance:
Anyway after a promising 10MB/s for the first two hosts, average backup
speed dropped to an average 0.5MB/s.
I don't know if the access speed to the sshfs has been capped by my cloud
provider due to the intense use. Anyway, I think that could still be a
feasible solution under some limitations.
Also CPU usage was pretty high (probably due to the
lack-of-hardlink-support fallback strategy), but I was using a pretty
entry-level vm template.

Also, but this is peculiar with my setup, it seems that sshfs doesn't
handle well dhcp renewals.

I hope this can be useful to someone.

SB




Il giorno dom 4 set 2022 alle ore 21:24 Sandro Bordacchini <
sandro.bordacch...@gmail.com> ha scritto:

> Thank you for your answer.
> I am aware that's not an optimal solution, K.
> For what i can understand here:
> https://backuppc.github.io/backuppc/BackupPC.html#What-type-of-storage-space-do-I-need
>
> Starting with 4.0.0, BackupPC no longer uses hardlinks for storage of
> deduplicated files. However, hardlinks are still used temporarily in a few
> places for doing atomic renames, *with a fallback doing a file copy if
> the hardlink fails*, and files are moved (renamed) across various paths
> that turn into expensive file copies if they span multiple file systems.
>
> So there is a "fallback" if the filesystem does not support hardlinks.
> Or maybe I got that wrong (I am not an English mother tongue speaker)...
>
>
>
>> -- Forwarded message --
>> From: backu...@kosowsky.org
>> To: "General list for user discussion, questions and support" <
>> backuppc-users@lists.sourceforge.net>
>> Cc:
>> Bcc:
>> Date: Fri, 2 Sep 2022 14:31:18 -0400
>> Subject: Re: [BackupPC-users] errors on network-mounted pool filesystem
>> Are you sure that BackupPC is compatible with sshfs?
>> While v4 drastically reduces number of hard links by eliminating the
>> hard-link pool scheme, it still uses hard links for *temporary* atomic
>> renames.
>> See: http://backuppc.sourceforge.net/BackupPC-4.2.1.html
>>
>> Sshfs may have other limitations as it is not a full-fledged
>> filesystem...
>>
>>
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem (backu...@kosowsky.org)

[Sandro Bordacchini]

Thank you for your answer.
I am aware that's not an optimal solution, K.
For what i can understand here:
https://backuppc.github.io/backuppc/BackupPC.html#What-type-of-storage-space-do-I-need

Starting with 4.0.0, BackupPC no longer uses hardlinks for storage of
deduplicated files. However, hardlinks are still used temporarily in a few
places for doing atomic renames, *with a fallback doing a file copy if the
hardlink fails*, and files are moved (renamed) across various paths that
turn into expensive file copies if they span multiple file systems.

So there is a "fallback" if the filesystem does not support hardlinks.
Or maybe I got that wrong (I am not an English mother tongue speaker)...



> -- Forwarded message --
> From: backu...@kosowsky.org
> To: "General list for user discussion, questions and support" <
> backuppc-users@lists.sourceforge.net>
> Cc:
> Bcc:
> Date: Fri, 2 Sep 2022 14:31:18 -0400
> Subject: Re: [BackupPC-users] errors on network-mounted pool filesystem
> Are you sure that BackupPC is compatible with sshfs?
> While v4 drastically reduces number of hard links by eliminating the
> hard-link pool scheme, it still uses hard links for *temporary* atomic
> renames.
> See: http://backuppc.sourceforge.net/BackupPC-4.2.1.html
>
> Sshfs may have other limitations as it is not a full-fledged
> filesystem...
>
>
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem (Greg Harris)

[Sandro Bordacchini]

Hello Greg,

Thank you for answering.
Something about permissions/features of the underlying fs was my first idea
too.

I don't understand how (or why) can a process running by user backuppc
write a filesystem as a different user..

backuppc@:~$ id
uid=108(backuppc) gid=114(backuppc) groups=114(backuppc)

That's the default for my debian installation.

I think i tracked down the first error line
G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
here: 
*https://github.com/backuppc/rsync-bpc/blob/dcb672942e4094849ff035506729a998b340e7e3/backuppc/bpc_dirOps.c#L62
<https://github.com/backuppc/rsync-bpc/blob/dcb672942e4094849ff035506729a998b340e7e3/backuppc/bpc_dirOps.c#L62>*

and that's a mkdir with permission 0777.

Issuing a:
mkdir -m 0777 attrib
in the *~/pc//0/f%2fetc%2f* directory -> the attrib
directory is correctly created with the right permissions:

drwxr-x--- 1 backuppc backuppc  20 Sep  4 18:21 .
drwxr-x--- 1 backuppc backuppc 139 Sep  4 03:23 ..
drwxrwxrwx 1 backuppc backuppc   2 Sep  4 18:21 attrib
-rw-r- 1 backuppc backuppc   0 Sep  4 03:20
attrib_00a0ed124f6656e346bf0098285fa240
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 falternatives
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fapparmor.d
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fapt
drwxr-x--- 1 backuppc backuppc   2 Sep  4 03:20 fbackup.d
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fcloud
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fcron.d
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fcron.daily
drwxr-x--- 1 backuppc backuppc   2 Sep  4 03:20 fcron.monthly
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fcryptsetup-initramfs
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fdefault
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fdehydrated
drwxr-x--- 1 backuppc backuppc   2 Sep  4 03:20 fdpkg
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fgrub.d
drwxr-x--- 1 backuppc backuppc   2 Sep  4 03:20 fmdadm
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fnginx
drwxr-x--- 1 backuppc backuppc   3 Sep  4 03:20 fssh

So I really don't understand why this fails.
There is something about this attrib directory, because I get this very
same error for every debian host I add.




> -- Forwarded message --
> From: Greg Harris 
> To: "General list for user discussion, questions and support" <
backuppc-users@lists.sourceforge.net>
> Cc:
> Bcc:
> Date: Fri, 2 Sep 2022 13:53:10 +
> Subject: Re: [BackupPC-users] errors on network-mounted pool filesystem
> I’d suspect that the rsync_bpc call is using a user that isn’t what you
expect.  However, I can’t find the exact setting I was expecting.  In my
setup, the group that owns the backup location is the web group, (I think
www for Ubuntu), not backuppc, and backuppc is part of the web group.
>
> Thanks,
>
> Greg Harris
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem

[G.W. Haywood via BackupPC-users]

Hi there,

On Sat, 3 Sep 2022, Sandro Bordacchini wrote:


...
I needed to switch to a cloud solution, though.
...


I think that's a very bad idea.

--

73,
Ged.


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem

[Les Mikesell]

On Fri, Sep 2, 2022 at 1:34 PM  wrote:
>
> Sshfs may have other limitations as it is not a full-fledged
> filesystem...
>

And conceptually, think about what has to happen when using rysnc with
a remote filesystem.  Computing the block checksums that is supposed
to be a quick local disk operation now has to run the whole contents
of each file over the network.

--
   Les Mikesell
 lesmikes...@gmail.com


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem

[backuppc]

Are you sure that BackupPC is compatible with sshfs?
While v4 drastically reduces number of hard links by eliminating the
hard-link pool scheme, it still uses hard links for *temporary* atomic
renames.
See: http://backuppc.sourceforge.net/BackupPC-4.2.1.html

Sshfs may have other limitations as it is not a full-fledged
filesystem...


Sandro Bordacchini wrote at about 15:01:35 +0200 on Friday, September 2, 2022:
 > Hello all.
 > 
 > Just wanted to share with you what happened to me and check if you have any
 > idea on what is going on.
 > I have been a satisfied user of backuppc for a long time. So far I am using
 > a 3.3.0 version on a debian 7 on a local vm (a local vmware esxi i have
 > full control on): no problem with this approach.
 > 
 > I needed to switch to a cloud solution, though.
 > 
 > No need to migrate the pool. I thought about installing backuppc v4 and
 > just starting from that, adding my old hosts (to avoid v3/v4 pool version
 > issues)
 > My pool is 0.7 TB so to keep the budget low I planned to buy a low-middle
 > sized vm plus some "cloud storage space".
 > I read that hardlinks support is no longer strictly mandatory (but it can
 > affect performance) on v4 so I was about to give it a try. I hope I
 > understood this crucial point correctly.
 > 
 > So i installed backuppc 4.4.0-3 (standard repo version on debian 11)
 > I mounted the cloud space with sshfs as user backuppc, this are the options
 > in the fstab line:
 > 
 > idmap=user,allow_other,default_permissions,IdentityFile=/root/.ssh/,port=23,ServerAliveInterval=15,reconnect,_netdev,uid=108,gid=114
 > 
 > uid and gid are backuppc user and group
 > I got a working installation and I tried a successful backup (full and
 > incr) of localhost (tar method), with no errors.
 > 
 > Then I tried to add some remote linux servers (rsync over ssh method) and I
 > got several transfer errors... very similar across all the servers... for
 > example a sample from xferLOG:
 > 
 > [...]
 > This is the rsync child about to exec /usr/libexec/backuppc-rsync/rsync_bpc
 > Xfer PIDs are now 39411,39414
 > xferPids 39411,39414
 > G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
 > G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
 > => [lots of repeated lines]
 > rsync_bpc: recv_generator: mkdir "/NetworkManager" failed: No such file or
 > directory (2)
 > *** Skipping any contents from this failed directory ***
 > G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
 > G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
 > rsync_bpc: recv_generator: mkdir "/X11/xkb" failed: No such file or
 > directory (2)
 > *** Skipping any contents from this failed directory ***
 > G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
 > G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
 > rsync_bpc: recv_generator: mkdir "/acpi/events" failed: No such file or
 > directory (2)
 > *** Skipping any contents from this failed directory ***
 > G bpc_fileReadAll: can't open
 > /var/lib/backuppc/cpool/36/16/37170a6409f1fa9cf67d20660b1d0f2d (from
 > alternatives/ALTER_OPERATOR_CLASS.7.gz)
 > rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_CLASS.7.gz" ->
 > "/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_CLASS.7.gz" failed: File
 > exists (17)
 > rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_FAMILY.7.gz" ->
 > "/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_FAMILY.7.gz" failed: File
 > exists (17)
 > G bpc_fileReadAll: can't open
 > /var/lib/backuppc/cpool/e2/48/e34810c68ce29a133fbc30d73c6ad6a0 (from
 > alternatives/ALTER_TABLESPACE.7.gz)
 > [...]
 > 
 > All the remote servers i added so far (all debian servers) got several
 > errors and if i try to browse their backups, the web interface tells me
 > they are empty.
 > 
 > My main concern is: why the backup with tar method is working and the other
 > with rsync is not?
 > If it was something about the pool filesystem ... should not affect all the
 > backups?
 > Is something about the options i mount the sshfs filesystem? Should I set
 > write syncs?
 > Seeing the log someone could argue user backuppc has no rw access but i can
 > tell you that backuppc created a lot of files and folders in that fs.
 > 
 > Thank you in advance.
 > S
 > ___
 > BackupPC-users mailing list
 > BackupPC-users@lists.sourceforge.net
 > List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
 > Wiki:https://github.com/backuppc/backuppc/wiki
 > Project: https://backuppc.github.io/backuppc/


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

Re: [BackupPC-users] errors on network-mounted pool filesystem

[Greg Harris]

I’d suspect that the rsync_bpc call is using a user that isn’t what you expect. 
 However, I can’t find the exact setting I was expecting.  In my setup, the 
group that owns the backup location is the web group, (I think www for Ubuntu), 
not backuppc, and backuppc is part of the web group.

Thanks,

Greg Harris

On Sep 2, 2022, at 9:01 AM, Sandro Bordacchini 
mailto:sandro.bordacch...@gmail.com>> wrote:

Hello all.

Just wanted to share with you what happened to me and check if you have any 
idea on what is going on.
I have been a satisfied user of backuppc for a long time. So far I am using a 
3.3.0 version on a debian 7 on a local vm (a local vmware esxi i have full 
control on): no problem with this approach.

I needed to switch to a cloud solution, though.

No need to migrate the pool. I thought about installing backuppc v4 and just 
starting from that, adding my old hosts (to avoid v3/v4 pool version issues)
My pool is 0.7 TB so to keep the budget low I planned to buy a low-middle sized 
vm plus some "cloud storage space".
I read that hardlinks support is no longer strictly mandatory (but it can 
affect performance) on v4 so I was about to give it a try. I hope I understood 
this crucial point correctly.

So i installed backuppc 4.4.0-3 (standard repo version on debian 11)
I mounted the cloud space with sshfs as user backuppc, this are the options in 
the fstab line:

idmap=user,allow_other,default_permissions,IdentityFile=/root/.ssh/,port=23,ServerAliveInterval=15,reconnect,_netdev,uid=108,gid=114

uid and gid are backuppc user and group
I got a working installation and I tried a successful backup (full and incr) of 
localhost (tar method), with no errors.

Then I tried to add some remote linux servers (rsync over ssh method) and I got 
several transfer errors... very similar across all the servers... for example a 
sample from xferLOG:

[...]
This is the rsync child about to exec /usr/libexec/backuppc-rsync/rsync_bpc
Xfer PIDs are now 39411,39414
xferPids 39411,39414
G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
=> [lots of repeated lines]
rsync_bpc: recv_generator: mkdir "/NetworkManager" failed: No such file or 
directory (2)
*** Skipping any contents from this failed directory ***
G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
rsync_bpc: recv_generator: mkdir "/X11/xkb" failed: No such file or directory 
(2)
*** Skipping any contents from this failed directory ***
G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
rsync_bpc: recv_generator: mkdir "/acpi/events" failed: No such file or 
directory (2)
*** Skipping any contents from this failed directory ***
G bpc_fileReadAll: can't open 
/var/lib/backuppc/cpool/36/16/37170a6409f1fa9cf67d20660b1d0f2d (from 
alternatives/ALTER_OPERATOR_CLASS.7.gz)
rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_CLASS.7.gz" -> 
"/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_CLASS.7.gz" failed: File 
exists (17)
rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_FAMILY.7.gz" -> 
"/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_FAMILY.7.gz" failed: File 
exists (17)
G bpc_fileReadAll: can't open 
/var/lib/backuppc/cpool/e2/48/e34810c68ce29a133fbc30d73c6ad6a0 (from 
alternatives/ALTER_TABLESPACE.7.gz)
[...]

All the remote servers i added so far (all debian servers) got several errors 
and if i try to browse their backups, the web interface tells me they are empty.

My main concern is: why the backup with tar method is working and the other 
with rsync is not?
If it was something about the pool filesystem ... should not affect all the 
backups?
Is something about the options i mount the sshfs filesystem? Should I set write 
syncs?
Seeing the log someone could argue user backuppc has no rw access but i can 
tell you that backuppc created a lot of files and folders in that fs.

Thank you in advance.
S

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/

[BackupPC-users] errors on network-mounted pool filesystem

[Sandro Bordacchini]

Hello all.

Just wanted to share with you what happened to me and check if you have any
idea on what is going on.
I have been a satisfied user of backuppc for a long time. So far I am using
a 3.3.0 version on a debian 7 on a local vm (a local vmware esxi i have
full control on): no problem with this approach.

I needed to switch to a cloud solution, though.

No need to migrate the pool. I thought about installing backuppc v4 and
just starting from that, adding my old hosts (to avoid v3/v4 pool version
issues)
My pool is 0.7 TB so to keep the budget low I planned to buy a low-middle
sized vm plus some "cloud storage space".
I read that hardlinks support is no longer strictly mandatory (but it can
affect performance) on v4 so I was about to give it a try. I hope I
understood this crucial point correctly.

So i installed backuppc 4.4.0-3 (standard repo version on debian 11)
I mounted the cloud space with sshfs as user backuppc, this are the options
in the fstab line:

idmap=user,allow_other,default_permissions,IdentityFile=/root/.ssh/,port=23,ServerAliveInterval=15,reconnect,_netdev,uid=108,gid=114

uid and gid are backuppc user and group
I got a working installation and I tried a successful backup (full and
incr) of localhost (tar method), with no errors.

Then I tried to add some remote linux servers (rsync over ssh method) and I
got several transfer errors... very similar across all the servers... for
example a sample from xferLOG:

[...]
This is the rsync child about to exec /usr/libexec/backuppc-rsync/rsync_bpc
Xfer PIDs are now 39411,39414
xferPids 39411,39414
G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
G bpc_path_create: can't create f%2fetc%2f/attrib (errno 2)
=> [lots of repeated lines]
rsync_bpc: recv_generator: mkdir "/NetworkManager" failed: No such file or
directory (2)
*** Skipping any contents from this failed directory ***
G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
G bpc_path_create: can't create f%2fetc%2f/fX11 (errno 2)
rsync_bpc: recv_generator: mkdir "/X11/xkb" failed: No such file or
directory (2)
*** Skipping any contents from this failed directory ***
G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
G bpc_path_create: can't create f%2fetc%2f/facpi (errno 2)
rsync_bpc: recv_generator: mkdir "/acpi/events" failed: No such file or
directory (2)
*** Skipping any contents from this failed directory ***
G bpc_fileReadAll: can't open
/var/lib/backuppc/cpool/36/16/37170a6409f1fa9cf67d20660b1d0f2d (from
alternatives/ALTER_OPERATOR_CLASS.7.gz)
rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_CLASS.7.gz" ->
"/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_CLASS.7.gz" failed: File
exists (17)
rsync_bpc: symlink "/alternatives/ALTER_OPERATOR_FAMILY.7.gz" ->
"/usr/share/postgresql/13/man/man7/ALTER_OPERATOR_FAMILY.7.gz" failed: File
exists (17)
G bpc_fileReadAll: can't open
/var/lib/backuppc/cpool/e2/48/e34810c68ce29a133fbc30d73c6ad6a0 (from
alternatives/ALTER_TABLESPACE.7.gz)
[...]

All the remote servers i added so far (all debian servers) got several
errors and if i try to browse their backups, the web interface tells me
they are empty.

My main concern is: why the backup with tar method is working and the other
with rsync is not?
If it was something about the pool filesystem ... should not affect all the
backups?
Is something about the options i mount the sshfs filesystem? Should I set
write syncs?
Seeing the log someone could argue user backuppc has no rw access but i can
tell you that backuppc created a lot of files and folders in that fs.

Thank you in advance.
S
___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:https://github.com/backuppc/backuppc/wiki
Project: https://backuppc.github.io/backuppc/