In the message dated: Wed, 13 Feb 2019 11:42:44 +0000,
The pithy ruminations from Martin Simmons on
[[External] Re: [Bacula-users] multi-homed bacula-dir + asymetrical routing =
authentication failure?] were:
=> >>>>> On Tue, 12 Feb 2019 14:12:08 -0500, mark bergman said:
=> >
=> > We run the bacula-dir and bacula-sd (v9.4.1) on a CentOS6 server
=> > with multiple network interfaces. I'm seeing a problem with a new client
=> > that I believe is related to asymetric routing and maybe the use of the
=> > server's IP to generate the MD5 CRAM digest.
=>
=> FWIW, the MD5 CRAM algorithm is not affected by the IP addresses.
Ok. Good to know.
=>
=> According to these debug messages, the server is connecting to client IP
=> 10.20.20.1, not 10.20.0.10.
That's my typo in creating this mail. In reality, I've obfuscated the
actual addresses. Yes, that makes reporting the issue and getting any
feedback more difficult, but the discrepency you picked up was from my
editing and the bad ascii-art, not the actual config.
=> >
=> > In particular, note Line 8, where the bacula server seems to identify
=> > itself as 172.16.1.159 (the primary interface), even though the traffic
=> > was sent via the 10.20.0.0 network.
=>
=> The host IP address in line 8 is the address of the server as returned by
=> accept(2), i.e. at the TCP/IP level, not within the application data.
That's what I find very odd. The bacula server couldn't initiate a
connection directly to the client via 172.16.0.0, since the bacula server
has an /etc/hosts entry for the client within 10.20.0.0 and since the
client has no interface on the 172.16.0.0 network.
>From the client, the bacula_server resolves to only the 10.20.0.0 address,
so any client-initiated communication would be directed to that address,
not via the client's default gateway.
As I see it, the only way that the bacula server would use 172.16.0.0
in any traffic with the client would be if:
bacula server contacts client via 10.20.0.0 and supplies
172.16.0.0 within application data
client answers at application level (ISO 7) and the client's
default gateway does NAT from 10.20.0.0 to 172.16.0.0
the bacula server then replies to the traffic at layer 4, using
the 170.16.0.0 address given by the gatway
I think I've got to get packet dumps from both sides to analyze this further.
=>
=> I suggest you temporarily remove the DNAT rules from the client, because I
=> don't see why they are needed for the estimate command.
I didn't think they were needed either -- they were an attempt to deal
with what seems to be traffic to new_client that's being sent out through
the client's default gateway (and NAT'ed) onto 172.16.0.0.
There is no successful communication without the client DNAT rules either
-- I see the same client-side messages.
Here's something interesting, with DNAT removed:
==================== bacula_server =================
1 [root@bacula_server]# printf "estimate job=new_client\n" |
/opt/bacula/bin/bconsole
2 Connecting to Director bacula_server:9101
3 1000 OK: 103 bacula_server Version: 9.4.1 (20 December 2018)
4 Enter a period to cancel a command.
5 estimate job=new_client
6 Using Catalog "MyCatalog"
7 Connecting to Client new_client at 10.20.20.1:9102
8 Failed to connect to Client.
9 You have messages.
10 [root@bacula_server]# getent hosts new_client
11 10.20.20.1 new_client.local new_client
12 [root@bacula_server]# telnet new_client 9102
13 Trying 10.20.20.1...
14 Connected to new_client.local (10.20.20.1).
15 Escape character is '^]'.
16
17
18 Connection closed by foreign host.
19 [root@bacula_server]# more /opt/bacula/etc/clients/new_client.conf
20 Client {
21 Name = new_client
22 Address = 10.20.20.1
23 FDPort = 9102
24 Catalog = MyCatalog
25 Password = "01234"
26 AutoPrune = yes # Prune expired Jobs/Files
27 Maximum Concurrent Jobs = 4
28 File Retention = 12 months # not these retention values are
because the lesser of the values in the [default|pool|client] are used.
29 Job Retention = 12 months # this specification is to ensure that
the default (6 mo for job records) is not applied
30 }
31 [root@bacula_server]# grep 159 /etc/hosts
32 172.16.0.159 bacula_server.fqdn
32 10.20.110.159 bacula_server.fqdn bacula_server bacula_server.local
=====================================================
Line 7 -- bacula_server claims it is connecting to the client on 10.20.20.1
Line 14 -- actual connection to the client at 10.20.20.1
================== new_client =======================
1 [root@new_client ~] # cat /tmp/bacula-fd.debug.log
2 new_client: bsock.c:847-0 socket=4 who=client host=172.16.0.159
port=7306
3 new_client: job.c:317-0 <dird: Hello Director bacula_server calling 103
4 new_client: job.c:340-0 Executing Hello Dir Hello Director
bacula_server calling 103 command.
5 new_client: cram-md5.c:69-0 send: auth cram-md5 challenge
<23169800.1550097552@new_client> ssl=0
6 new_client: cram-md5.c:71-0 Send challenge comm error. ERR=Connection
reset by peer
7 new_client: authenticate.c:101-0 cram_auth challenge failed for
Director client
8 new_client: Fatal Error at authenticate.c:105 because:
9 Incorrect password given by Director at client.
10 new_client: job.c:343-0 Quit command loop. Canceled=0
11 new_client: job.c:467-0 Calling term_find_files
12 new_client: job.c:470-0 Done with term_find_files
13 new_client: job.c:473-0 Done with free_jcr
14 new_client: bsock.c:847-0 socket=4 who=client host=10.20.200.5
port=41072
15 new_client: job.c:505-0 Bad command from client. Len=-4.
16 [root@new_client ~]# getent hosts 10.20.110.159
17 10.20.110.159 bacula_server bacula_server.fqdn
18 [root@new_client ~]# getent hosts 172.16.98.159
19 [root@new_client ~]# getent hosts bacula_server
20 10.20.110.159 bacula_server bacula_server.fqdn
=====================================================
Line 2 -- the client claims it receives the connection from a host that
identifies itself as 172.16.0.159 (that is the IP address of the hostname for
the bacula server VIP).
Line 14 -- the telnet connection from the bacula_server is correctly shown as
coming from 10.20.200.5 (another VIP for the bacula_server, putting it on the
10.20.0.0 subnet)
Thanks,
Mark
=> __Martin
=>
--
Mark Bergman voice: 215-746-4061
mark.berg...@uphs.upenn.edu fax: 215-614-0266
http://www.med.upenn.edu/cbica/
IT Technical Director, Center for Biomedical Image Computing and Analytics
Department of Radiology University of Pennsylvania
_______________________________________________
Bacula-users mailing list
Bacula-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bacula-users
