Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Pim Rupert]

Hi Falko,

>> Apparently a shell is required to use a jump host from the ssh command in 
>> this situation?
> 
> There's an analysis about apparently same problem:
> 
> https://unix.stackexchange.com/questions/457692/does-ssh-proxyjump-require-local-shell-access
> 
> They mention, that setting the "SHELL" variable is sufficient

Good catch!

As often with StackExchange, the first answer is not the correct answer. :-)

The answer at https://unix.stackexchange.com/a/496092 seems to hit the nail on 
the head:

> Basically I think SSH is trying to be clever and looking up the user's shell 
> and then running the proxy command using the shell. But when the shell 
> doesn't exist it fails.
> 
> If you set the environment variable SHELL before you run ssh then it will fix 
> the problem.


Unfortunately, this work-around (inserting an environment variable) seems 
incompatible with the current way BackupPC executes the commands.

The /sbin/nologin shell assignment doesn't really add that much security, but 
it would be cool if it still was possible to initiate SSH commands while using 
jump hosts without having to assign a shell to the BackupPC user.

Cheers!

Pim



___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Falko Trojahn via BackupPC-users]

Hi Pim,


The reason for this had nothing to do with any "remote host", it was that the 
"backuppc" user had no shell configured!

# grep backuppc /etc/passwd
backuppc:x:994:990::/var/lib/BackupPC:/sbin/nologin

After changing the shell to /bin/bash for backuppc, all errors disappeared. 
Running backups automatically and from the web application succeeded with the 
jump host.



Glad you got it to work.


Apparently a shell is required to use a jump host from the ssh command in this 
situation?


There's an analysis about apparently same problem:

https://unix.stackexchange.com/questions/457692/does-ssh-proxyjump-require-local-shell-access

They mention, that setting the "SHELL" variable is sufficient ..



I will further investigate if this is really required and will report back in 
this list.

@Falko: since you got it working without changes, is a shell set for the 
"backuppc" user?


Yes, it's /bin/sh or /bin/bash.

Greetings,
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Pim Rupert]

Hello Falko,

> On 22 Apr 2020, at 09:59, Falko Trojahn via BackupPC-users 
>  wrote:
> 
> Usually, there are some more informations in the Xferlog. What shows the 
> Xferlog of a failed attempt?

The XferLog and verbose output for three attempts are attached below. 

The same user is used for all attempts: "backuppc". 

Basically, it boils down to this:

1. Manually started from command-line in bash shell for "backuppc" user: ALL OK
2. Scheduled via crontab for "backuppc" user: ALL OK
3. Initiated from hourly schedule by BackupPC daemon running as "backuppc": 
FAILS
4. Manually started from web interface running as "backuppc": FAILS 

When it fails, it fails on the rsync_bpc command. With the following error 
(example):

Xfer PIDs are now 31322
This is the rsync child about to exec /usr/bin/rsync_bpc
ssh_exchange_identification: Connection closed by remote host
rsync_bpc: connection unexpectedly closed (0 bytes received so far) 
[Receiver]

This implies something goes wrong with the SSH connection. However, no outgoing 
TCP packets are visible in tcpdump. The error is not reproducible when 
initiated manually from shell or cron, in that case we do see outgoing TCP 
packets and all is well. This is severely odd. 

I still think I am doing something stupid, or I am hitting some weird bug in 
this specific combination of rsync_bpc version and my system. 

Your help is greatly appreciated.

XferLOG from full backup initiated through web interface (backup fails):

Running: /usr/bin/rsync_bpc --bpc-top-dir /var/lib/BackupPC/ 
--bpc-host-name myhostname --bpc-share-name / --bpc-bkup-num 124 
--bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 
--bpc-bkup-inode0 175546 --bpc-attrib-new --bpc-log-level 0 -e /usr/bin/ssh\ 
-l\ mycompany_backup --rsync-path=/usr/bin/sudo\ /usr/bin/rsync --super 
--recursive --protect-args --numeric-ids --perms --owner --group -D --times 
--links --hard-links --delete --delete-excluded --one-file-system --partial 
--log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum 
--timeout=72000 --exclude=/mnt --exclude=/proc --exclude=/dev --exclude=/sys 
--exclude=/var/lib/pgsql/\*/data/base --exclude=/var/lib/pgsql/\*/data/pg_wal 
--exclude=/var/lib/mysql/\*/\* --exclude=/var/lib/mysql/ibdata\* 
--exclude=/var/lib/mysql/ib_logfile\* --exclude=/var/lib/BackupPC/\* 
--exclude=/var/mycompany/mysql/mariabackup/tmp_export 
--exclude=/mariabackup/tmp_export myhostname:/ /
full backup started for directory /
Xfer PIDs are now 31322
This is the rsync child about to exec /usr/bin/rsync_bpc
ssh_exchange_identification: Connection closed by remote host
rsync_bpc: connection unexpectedly closed (0 bytes received so far) 
[Receiver]
Done: 0 errors, 0 filesExist, 0 sizeExist, 0 sizeExistComp, 0 filesTotal, 0 
sizeTotal, 0 filesNew, 0 sizeNew, 0 sizeNewComp, 175546 inode
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2.0]
rsync_bpc exited with fatal status 255 (65280) (rsync error: unexplained 
error (code 255) at io.c(226) [Receiver=3.1.2.0])
Xfer PIDs are now 
Got fatal error during xfer (No files dumped for share /)
Backup aborted (No files dumped for share /)
(...)

XferLOG from full backup initiated through bash shell (backup succeeds):

(...)
Backup prep: type = full, case = 6, inPlace = 1, doDuplicate = 0, 
newBkupNum = 124, newBkupIdx = 29, lastBkupNum = , lastBkupIdx =  (FillCycle = 
0, noFillCnt = 0)
Running: /usr/bin/rsync_bpc --bpc-top-dir /var/lib/BackupPC/ 
--bpc-host-name myhostname --bpc-share-name / --bpc-bkup-num 124 
--bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 
--bpc-bkup-inode0 175546 --bpc-attrib-new --bpc-log-level 0 -e /usr/bin/ssh\ 
-l\ mycompany_backup --rsync-path=/usr/bin/sudo\ /usr/bin/rsync --super 
--recursive --protect-args --numeric-ids --perms --owner --group -D --times 
--links --hard-links --delete --delete-excluded --one-file-system --partial 
--log-format=log:\ %o\ %i\ %B\ %8U,%8G\ %9l\ %f%L --stats --checksum 
--timeout=72000 --exclude=/mnt --exclude=/proc --exclude=/dev --exclude=/sys 
--exclude=/var/lib/pgsql/\*/data/base --exclude=/var/lib/pgsql/\*/data/pg_wal 
--exclude=/var/lib/mysql/\*/\* --exclude=/var/lib/mysql/ibdata\* 
--exclude=/var/lib/mysql/ib_logfile\* --exclude=/var/lib/BackupPC/\* 
--exclude=/var/mycompany/mysql/mariabackup/tmp_export 
--exclude=/mariabackup/tmp_export myhostname:/ /
full backup started for directory /
Xfer PIDs are now 341
This is the rsync child about to exec /usr/bin/rsync_bpc
cmdExecOrEval: about to exec /usr/bin/rsync_bpc --bpc-top-dir 
/var/lib/BackupPC/ --bpc-host-name myhostname --bpc-share-name / --bpc-bkup-num 
124 --bpc-bkup-comp 3 --bpc-bkup-prevnum -1 --bpc-bkup-prevcomp -1 
--bpc-bkup-inode0 175546 --bpc-attrib-new --bpc-log-level 0 -e /usr/bin/ssh\ 
-l\ mycompany_backup --rsync-path=/usr/bin/sudo\ /usr/bin/rsync --super 
--recursive --protect-args --numeric-ids 

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Falko Trojahn via BackupPC-users]

Hi Pim,


Without DumpPreUserCmd I get the following error when initiating an incremental 
backup through the web interface:

Got fatal error during xfer (rsync error: unexplained error (code 255) 
at io.c(226) [Receiver=3.1.2.0])

that reminds me, that I'm using rsync-bpc version 3.0.9.14, yet.
Dunno why any more, sorry. May be there were not so much differences 
between the versions.




And the following error when initiating a full backup through the web interface:

Got fatal error during xfer (No files dumped for share /)

Usually, there are some more informations in the Xferlog. What shows the 
Xferlog of a failed attempt?



Also, both backup attempts do not yield any visible outgoing traffic via 
tcpdump. So apparently it is not even initiating the outbound SSH connections.


That's weird, indeed.


Thanks. Already tried a manual backup through the CLI, and this works.


You mean by starting BackupPC_dump? Then, that sounds great, at least 
this is working. You are using the same backuppc user to start the 
backup manually, right?




So, I am only seeing this issue with backups automatically initiated from the 
hourly schedule or manually initiated through the web interface.

That's weird, right? I guess since you guys are able to get it working without 
any issues, I must be doing something awfully stupid or I am hitting a weird 
bug in my CentOS environment.



 Can I increase logging of the scheduling processes or of the web 
interface so I can see more details about which exact commands are being 
executed?


Do you see any difference between XferLOG



I am still thinking the SSH command is somehow getting mixed up when started 
from the scheduler or web application.


ok, could you please try and put a
  /usr/share/BackupPC/bin/BackupPC_dump -f -vvv (hostname)
in the crontab of your backuppc user?

Perhaps set it to some minutes apart, one time only. Maybe something 
like this (adjust the time), if you get no cron mails - redirect to file:


0 1 * * * /usr/share/BackupPC/bin/BackupPC_dump -f -vvv hostname 2>&1 > 
/tmp/my-backup-log.txt


If this fails, too: please check, what in PATH is missing compared to 
your normal shell use.


Greetings,
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Pim Rupert]

Hi Greg and Falko

> On 21 Apr 2020, at 20:09, Greg Harris  wrote:
> 
> You seem to have a pretty good handle on stuff, but you aren’t suffering from 
> CentOS’ silent killer are you?  SELinux?

Thank for the suggestion. Unfortunately it's not SELinux, I have it fully 
disabled to be certain.

> On 21 Apr 2020, at 20:05, Falko Trojahn via BackupPC-users 
>  wrote:
> 
> So, the differences seem to be:
> 
> * you're using sudo and a special backup user
> 
> * as your DumpPreUserCmd errors out, did you try without?
>  *  does the backup work without DumpPreUserCmd?

Yes, I have tried without DumpPreUserCmd.

Without DumpPreUserCmd I get the following error when initiating an incremental 
backup through the web interface:

Got fatal error during xfer (rsync error: unexplained error (code 255) 
at io.c(226) [Receiver=3.1.2.0])

And the following error when initiating a full backup through the web interface:

Got fatal error during xfer (No files dumped for share /)

Also, both backup attempts do not yield any visible outgoing traffic via 
tcpdump. So apparently it is not even initiating the outbound SSH connections.

> > DumpPreUserCmd returned error status 65280... exiting
>  *  what is this errorstatus from?

I think it is a wrapper script multiplying an exit code 255 with 256. The 255 
may be coming from the SSh command. But that is just speculation from my end. 

> May be this helps, too:
> (...)
> (on Debian, this is in another location:
>  /usr/share/backuppc/bin/BackupPC_dump
> don't know about CentOS)

Thanks. Already tried a manual backup through the CLI, and this works.
The outbound SSH connection to the jump host is visible in "tcpdump" and the 
backups succeed. There is nothing interesting in the verbose output, it's just 
showing a successful backup process. :-)

For later reference: the command on CentOS is:

/usr/share/BackupPC/bin/BackupPC_dump

So, I am only seeing this issue with backups automatically initiated from the 
hourly schedule or manually initiated through the web interface.

That's weird, right? I guess since you guys are able to get it working without 
any issues, I must be doing something awfully stupid or I am hitting a weird 
bug in my CentOS environment. Can I increase logging of the scheduling 
processes or of the web interface so I can see more details about which exact 
commands are being executed? 

I am still thinking the SSH command is somehow getting mixed up when started 
from the scheduler or web application.

> Yes, you can use another user. Is it a normal sshd what you use for jump 
> host, or something like sshportal or ssh-bastion?

All hosts use normal sshd from CentOS 7.

Best regards,

Pim Rupert

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Greg Harris]

You seem to have a pretty good handle on stuff, but you aren’t suffering from 
CentOS’ silent killer are you?  SELinux?

Thanks,

Greg Harris

On Apr 21, 2020, at 8:10 AM, Falko Trojahn via BackupPC-users 
mailto:backuppc-users@lists.sourceforge.net>>
 wrote:

Hi Pim,

using jumphost here for backing up a remote host and it's VMs without any 
problems. What BackupPC version do you use?
I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to 
hear that you got it working on your installation.

ok, so I'll try it on an 4.3.2 installation, too, and give you some information 
if it works there.

Greetings,
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Falko Trojahn via BackupPC-users]

Hello Pim,

using jumphost here for backing up a remote host and it's VMs without 
any problems. What BackupPC version do you use?
I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very 
good to hear that you got it working on your installation.


ok, so I'll try it on an 4.3.2 installation, too, and give you some 
information if it works there.


working here without any clue as described. No changes in BackupPC 
configuration needed.


So, the differences seem to be:

* you're using sudo and a special backup user

* as your DumpPreUserCmd errors out, did you try without?

> DumpPreUserCmd returned error status 65280... exiting
  *  what is this errorstatus from?
  *  does the backup work without DumpPreUserCmd?

May be this helps, too:

On Tue, 11 Jun 19, Adam Goryachev wrote:
> Finally, the one single command I've found to be the *most* helpful in
> debugging any such issues is this:
>
> /usr/lib/backuppc/bin/BackupPC_dump -f -vvv hostname
>
> Which will just try to do a full backup, but show you on the console
> what it is doing through each step. You should make sure there is no
> scheduled backup for this host, and no in-progress backup for this host
> when you run this command. Under normal operation, you shouldn't use
> this command.

(on Debian, this is in another location:
  /usr/share/backuppc/bin/BackupPC_dump
 don't know about CentOS)


> Also, not trying to be cheeky here: for added security you don't
> actually need AgentForwarding
Yes, you're right - this was an copy leftover.

> nor root logins when using a jump host.
Yes, you can use another user. Is it a normal sshd what you use for jump 
host, or something like sshportal or ssh-bastion?


There are different ways for ssh proxy/jump, too:
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Jump_Hosts_--_Passing_Through_a_Gateway_or_Two

Best regards,
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Falko Trojahn via BackupPC-users]

Hi Pim,


using jumphost here for backing up a remote host and it's VMs without any 
problems. What BackupPC version do you use?

I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to 
hear that you got it working on your installation.


ok, so I'll try it on an 4.3.2 installation, too, and give you some 
information if it works there.


Greetings,
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Pim Rupert]

On 21 Apr 2020, at 12:09, Pim Rupert  wrote:
> 
> I am just speculating here, but could it be that when the BackupPC software 
> is handling the SSH process, it is somehow unable to deal with the implictly 
> added ProxyCommand arguments?

Following this hunch I tried starting the BackupPC_dump process manually from 
the shell.

# su - backuppc -s /bin/bash

-bash-4.2$ /usr/bin/perl /usr/share/BackupPC/bin/BackupPC_dump -i 
mysqlimport.paragin.nl
__bpc_progress_state__ pre-cmd
__bpc_progress_state__ backup share "/"
started incr dump, share=/
xferPids 19255
xferPids 19255,19259
__bpc_progress_fileCnt__ 1
__bpc_progress_fileCnt__ 21
__bpc_progress_fileCnt__ 41
(...)
__bpc_progress_state__ rename total
xferPids
xferPids
incr backup complete

-bash-4.2$ echo $?
0

I saw backup traffic via tcpdump and successful authentication on jump host and 
client host.
This succeeds just fine! 

However, when started from the BackupPC web interface, the backup fails when a 
jump host is configured. So, again, just speculating, but could it be that I am 
hitting a bug in the web interface? Is anyone else seeing the same behaviour in 
their setups when using the BackupPC web interface? Unfortunately my Perl is 
too rusty for proper debugging.

Best regards,

Pim Rupert

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Pim Rupert]

Hi Falko,

> On 21 Apr 2020, at 00:03, Falko Trojahn via BackupPC-users 
>  wrote:
> 
> using jumphost here for backing up a remote host and it's VMs without any 
> problems. What BackupPC version do you use?

I am using BackupPC 4.3.1-3 from the yum repository for CentOS 7. Very good to 
hear that you got it working on your installation.

I am positive that SSH authentication is set up correctly. The problem doesn't 
seem to be authentication. There isn't even an authentication attempt being 
logged on either server. It's even worse, tcpdump does not even report outgoing 
packets to port 22. 

Weirdly, when the SSH commands are run manually, the outgoing packets are 
visible, and the whole SSH connection is being set up successfully. 

I am just speculating here, but could it be that when the BackupPC software is 
handling the SSH process, it is somehow unable to deal with the implictly added 
ProxyCommand arguments?

Also, not trying to be cheeky here: for added security you don't actually need 
AgentForwarding nor root logins when using a jump host.

Best regards,

Pim Rupert

___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/

Re: [BackupPC-users] Using a jump host to backup via rsync over SSH

[Falko Trojahn via BackupPC-users]

Hi,


Option B: using an SSH client config file

Alternatively I have tried using an implicit jump host through SSH client 
config with a slightly different way of setting up the jump host (through 
netcat). This results in exactly the same errors.

Host client-machine
   ProxyCommand ssh jumphost nc %h %p 2> /dev/null

Host jumphost
   Hostname jumphostname
   User jumphostuser



using jumphost here for backing up a remote host and it's VMs without 
any problems. What BackupPC version do you use?


Doing it like this on Debian 10 buster with BackupPC 3.3.2-2:

:~# su - backuppc
$ ssh target  # confirm fingerprint
$ cat ~/.ssh/config

#
Host your-real-host
  HostName your-real-ip-here
  Port 22   # or whatever you use
  ForwardAgent yes
  ForwardX11 no
  User root
#
Host first-vm-on-real-host
  HostName first-vm-ip-here
  ForwardAgent no
  ForwardX11 no
  User root
  Port 22
  ProxyCommand ssh root@your-real-host nc %h %p
#
Host 2nd-vm-on-real-host
  HostName 2nd-vm-ip-here
  ForwardAgent no
  ForwardX11 no
  User root
  Port 22
  ProxyCommand ssh root@your-real-host nc %h %p

$ ssh-keygen -t rsa -b 4096
$ ssh-copy-id your-real-host
$ ssh-copy-id first-vm-on-real-host
$ ssh-copy-id 2nd-vm-on-real-host

Perhaps you have to adjust /etc/ssh/sshd_config to allow ssh-key only 
access.


If you use any backuppc-wrapper-script on the real host, maybe adapt it 
to the ssh forwarding.


When trying manually, make sure you do not use your own loaded ssh key
thru ssh-agent, but really use the ssh key of the backuppc user. Prove 
that by:

ssh-add -l

HTH
Falko


___
BackupPC-users mailing list
BackupPC-users@lists.sourceforge.net
List:https://lists.sourceforge.net/lists/listinfo/backuppc-users
Wiki:http://backuppc.wiki.sourceforge.net
Project: http://backuppc.sourceforge.net/