Public bug reported:
We have observed the following behaviour on our LVS systems, which is
causing issues with our monitor scripts. The systems are running Ubuntu
14.04.5 LTS and I've tested both with the stock 3.13.0 kernel (-100 and
-116) and the 4.4.0-72 xenial kernel.
Our systems are set up in direct routing mode for the services they
handle. Our monitor scripts for DNS send test queries to our DNS
servers using their real IPs. Sporadically, we have seen these checks
fail as the DNS answers are coming back from the wrong IP address.
We debugged this using tcpdump, and found that the response packets
coming into the LVS systems were using the correct IPs (i.e. the real
IPs on the DNS servers). However, applications see the responses as
coming from a VIP instead.
All of this has been established using UDP traffic.
I have tracked this behaviour down to a specific case, which I can only assume
is associated with how the kernel handles LVS NAT connections (i.e. masquerade
mode):
- If a DNS query is made on the LVS server to a DNS VIP, that creates an entry
in the connection table, and is keyed on (srcip:sport -> dstip:dport) and
associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated
with (dnsvip:53)
- If a subsequent DNS query is made from the same UDP port, the response is
correct as seen by tcpdump
- When the response is seen by the application, the source IP address for the
response is wrong
I have inferred that as the response comes back from dnsip:53 and there
is a connection table entry, IPVS seems to assume NAT is in use, and
translates it using the entry (lvsip:50000 -> dnsip:53). The
application layer then sees the response from (dnsvip:53), which is
incorrect.
/proc/version_signature (from both nodes):
Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39
Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49
IPVS Configuration:
root@lvs5:~# ipvsadm -Sn
-A -t 144.32.128.183:25 -s wlc
-a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10
-a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10
-A -t 144.32.128.183:465 -s wlc
-a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10
-a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10
-A -t 144.32.128.183:587 -s wlc
-a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10
-a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10
-A -t 144.32.128.240:53 -s rr
-a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -t 144.32.128.242:53 -s rr
-a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
-A -t 144.32.129.39:25 -s wlc
-a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10
-a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10
-A -t 144.32.129.39:465 -s wlc
-a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10
-a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10
-A -t 144.32.129.39:587 -s wlc
-a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10
-a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10
-A -u 144.32.128.240:53 -s rr
-a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -u 144.32.128.242:53 -s rr
-a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
** Affects: linux (Ubuntu)
Importance: Undecided
Status: Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1681847
Title:
IPVS incorrectly reverse-NATs traffic to LVS host
Status in linux package in Ubuntu:
Incomplete
Bug description:
We have observed the following behaviour on our LVS systems, which is
causing issues with our monitor scripts. The systems are running
Ubuntu 14.04.5 LTS and I've tested both with the stock 3.13.0 kernel
(-100 and -116) and the 4.4.0-72 xenial kernel.
Our systems are set up in direct routing mode for the services they
handle. Our monitor scripts for DNS send test queries to our DNS
servers using their real IPs. Sporadically, we have seen these checks
fail as the DNS answers are coming back from the wrong IP address.
We debugged this using tcpdump, and found that the response packets
coming into the LVS systems were using the correct IPs (i.e. the real
IPs on the DNS servers). However, applications see the responses as
coming from a VIP instead.
All of this has been established using UDP traffic.
I have tracked this behaviour down to a specific case, which I can only
assume is associated with how the kernel handles LVS NAT connections (i.e.
masquerade mode):
- If a DNS query is made on the LVS server to a DNS VIP, that creates an
entry in the connection table, and is keyed on (srcip:sport -> dstip:dport) and
associated with (vip:vport) - for example, (lvsip:50000 -> dnsip:53) associated
with (dnsvip:53)
- If a subsequent DNS query is made from the same UDP port, the response is
correct as seen by tcpdump
- When the response is seen by the application, the source IP address for the
response is wrong
I have inferred that as the response comes back from dnsip:53 and
there is a connection table entry, IPVS seems to assume NAT is in use,
and translates it using the entry (lvsip:50000 -> dnsip:53). The
application layer then sees the response from (dnsvip:53), which is
incorrect.
/proc/version_signature (from both nodes):
Ubuntu 3.13.0-100.147-generic 3.13.11-ckt39
Ubuntu 4.4.0-72.93~14.04.1-generic 4.4.49
IPVS Configuration:
root@lvs5:~# ipvsadm -Sn
-A -t 144.32.128.183:25 -s wlc
-a -t 144.32.128.183:25 -r 144.32.129.29:25 -g -w 10
-a -t 144.32.128.183:25 -r 144.32.129.64:25 -g -w 10
-A -t 144.32.128.183:465 -s wlc
-a -t 144.32.128.183:465 -r 144.32.129.29:465 -g -w 10
-a -t 144.32.128.183:465 -r 144.32.129.64:465 -g -w 10
-A -t 144.32.128.183:587 -s wlc
-a -t 144.32.128.183:587 -r 144.32.129.29:587 -g -w 10
-a -t 144.32.128.183:587 -r 144.32.129.64:587 -g -w 10
-A -t 144.32.128.240:53 -s rr
-a -t 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -t 144.32.128.242:53 -s rr
-a -t 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
-A -t 144.32.129.39:25 -s wlc
-a -t 144.32.129.39:25 -r 144.32.129.163:25 -g -w 10
-a -t 144.32.129.39:25 -r 144.32.129.164:25 -g -w 10
-A -t 144.32.129.39:465 -s wlc
-a -t 144.32.129.39:465 -r 144.32.129.163:465 -g -w 10
-a -t 144.32.129.39:465 -r 144.32.129.164:465 -g -w 10
-A -t 144.32.129.39:587 -s wlc
-a -t 144.32.129.39:587 -r 144.32.129.163:587 -g -w 10
-a -t 144.32.129.39:587 -r 144.32.129.164:587 -g -w 10
-A -u 144.32.128.240:53 -s rr
-a -u 144.32.128.240:53 -r 144.32.128.227:53 -g -w 10
-A -u 144.32.128.242:53 -s rr
-a -u 144.32.128.242:53 -r 144.32.128.143:53 -g -w 10
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1681847/+subscriptions
--
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help : https://help.launchpad.net/ListHelp
