Public bug reported:

We have observed the following behaviour on our LVS systems, which is
causing issues with our monitor scripts.  The systems are running Ubuntu
14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and
-116) and the 4.4.0-72 xenial kernel.

Our systems are set up in direct routing mode for the services they
handle.  Our monitor scripts for DNS send test queries to our DNS
servers using their real IPs.  Sporadically, we have seen these checks
fail as the DNS answers are coming back from the wrong IP address.

We debugged this using tcpdump, and found that the response packets
coming into the LVS systems were using the correct IPs (i.e. the real
IPs on the DNS servers).  However, applications see the responses as
coming from a VIP instead.

All of this has been established using UDP traffic.

I have tracked this behaviour down to a specific case, which I can only assume 
is associated with how the kernel handles LVS NAT connections (i.e. masquerade 
mode):
- If a DNS query is made on the LVS server to a DNS VIP, that creates an entry 
in the connection table, and is keyed on (srcip:sport -> dstip:dport) and 
associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated 
with (dnsvip:53)
- If a subsequent DNS query is made from the same UDP port, the response is 
correct as seen by tcpdump
- When the response is seen by the application, the source IP address for the 
response is wrong

I have inferred that as the response comes back from dnsip:53 and there
is a connection table entry, IPVS seems to assume NAT is in use, and
translates it using the entry (lvsip:50000 -> dnsip:53).  The
application layer then sees the response from (dnsvip:53), which is
incorrect.

/proc/version_signature (from both nodes):
  Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39
  Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49

IPVS Configuration:

root@lvs5:~# ipvsadm -Sn
-A -t 144.32.128.183:25 -s wlc
-a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10
-a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10
-A -t 144.32.128.183:465 -s wlc
-a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10
-a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10
-A -t 144.32.128.183:587 -s wlc
-a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10
-a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10
-A -t 144.32.128.240:53 -s rr
-a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -t 144.32.128.242:53 -s rr
-a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
-A -t 144.32.129.39:25 -s wlc
-a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10
-a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10
-A -t 144.32.129.39:465 -s wlc
-a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10
-a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10
-A -t 144.32.129.39:587 -s wlc
-a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10
-a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10
-A -u 144.32.128.240:53 -s rr
-a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -u 144.32.128.242:53 -s rr
-a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10

** Affects: linux (Ubuntu)
     Importance: Undecided
         Status: Incomplete

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1681847

Title:
  IPVS incorrectly reverse-NATs traffic to LVS host

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  We have observed the following behaviour on our LVS systems, which is
  causing issues with our monitor scripts.  The systems are running
  Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel
  (-100 and -116) and the 4.4.0-72 xenial kernel.

  Our systems are set up in direct routing mode for the services they
  handle.  Our monitor scripts for DNS send test queries to our DNS
  servers using their real IPs.  Sporadically, we have seen these checks
  fail as the DNS answers are coming back from the wrong IP address.

  We debugged this using tcpdump, and found that the response packets
  coming into the LVS systems were using the correct IPs (i.e. the real
  IPs on the DNS servers).  However, applications see the responses as
  coming from a VIP instead.

  All of this has been established using UDP traffic.

  I have tracked this behaviour down to a specific case, which I can only 
assume is associated with how the kernel handles LVS NAT connections (i.e. 
masquerade mode):
  - If a DNS query is made on the LVS server to a DNS VIP, that creates an 
entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and 
associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated 
with (dnsvip:53)
  - If a subsequent DNS query is made from the same UDP port, the response is 
correct as seen by tcpdump
  - When the response is seen by the application, the source IP address for the 
response is wrong

  I have inferred that as the response comes back from dnsip:53 and
  there is a connection table entry, IPVS seems to assume NAT is in use,
  and translates it using the entry (lvsip:50000 -> dnsip:53).  The
  application layer then sees the response from (dnsvip:53), which is
  incorrect.

  /proc/version_signature (from both nodes):
    Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39
    Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49

  IPVS Configuration:

  root@lvs5:~# ipvsadm -Sn
  -A -t 144.32.128.183:25 -s wlc
  -a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10
  -a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10
  -A -t 144.32.128.183:465 -s wlc
  -a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10
  -a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10
  -A -t 144.32.128.183:587 -s wlc
  -a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10
  -a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10
  -A -t 144.32.128.240:53 -s rr
  -a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
  -A -t 144.32.128.242:53 -s rr
  -a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
  -A -t 144.32.129.39:25 -s wlc
  -a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10
  -a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10
  -A -t 144.32.129.39:465 -s wlc
  -a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10
  -a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10
  -A -t 144.32.129.39:587 -s wlc
  -a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10
  -a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10
  -A -u 144.32.128.240:53 -s rr
  -a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
  -A -u 144.32.128.242:53 -s rr
  -a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1681847/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to     : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

Reply via email to