Hey all,
I’ve found what I think could be a possible issue with the redundant VPC router
pairs in Clousdstack. The issue was first noticed when routers were failing
over from master to backup. When the backup router became master, everything
continued to work properly and traffic flowed as normal. However, when it
failed from the new master back to the original master the virtual router
stopped allowing traffic through any network interfaces and any failover after
that resulted in virtual routers that were not passing traffic.
I can reproduce this behavior by doing a manual failover (logging in and
issuing a reboot command on the router) from master to backup and then back to
the original master. From what I can tell, the iptables rules on the router
are somehow modified during the failover (or a manual reboot) in such a way as
to make them completely nonfunctional. I did a side-by-side comparison of the
iptables rules before and after a failover (or a manual reboot) and there are
definite differences. Sometimes rules are changed, sometimes they are
duplicated, and I’ve even found that some rules are missing completely out of
iptables.
We are running in a CentOS 7 environment and using KVM as our hypervisor. Our
CS version is 4.8 with standard images for the VRs. As mentioned previously,
our VRs are in redundant pairs for VPCs.
I’ve attached two iptables outputs, one from a working router and one from a
broken router after failover.
Any help or direction you could provide to help me further identify why this is
happening would be appreciated.
Thanks!
Tim Gipson
<https://www.ena.com/>
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:08:17 2017
*mangle
:PREROUTING ACCEPT [445:57066]
:INPUT ACCEPT [547:62882]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [537:50055]
:POSTROUTING ACCEPT [537:50055]
:ACL_OUTBOUND_eth2 - [0:0]
:VPN_STATS_eth1 - [0:0]
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark
--nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark
--nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i eth2 -m state --state NEW -j CONNMARK --set-xmark
0x2/0xffffffff
-A PREROUTING -s 172.16.64.0/24 ! -d 172.16.64.1/32 -i eth2 -m state --state
NEW -j ACL_OUTBOUND_eth2
-A PREROUTING -i eth1 -m state --state NEW -j CONNMARK --set-xmark
0x1/0xffffffff
-A FORWARD -j VPN_STATS_eth1
-A ACL_OUTBOUND_eth2 -d 224.0.0.18/32 -j ACCEPT
-A ACL_OUTBOUND_eth2 -j ACCEPT
-A ACL_OUTBOUND_eth2 -d 225.0.0.50/32 -j ACCEPT
-A VPN_STATS_eth1 -o eth1 -m mark --mark 0x525
-A VPN_STATS_eth1 -i eth1 -m mark --mark 0x524
COMMIT
# Completed on Tue Aug 29 21:08:17 2017
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:08:17 2017
*filter
:INPUT DROP [36:4240]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [537:50055]
:ACL_INBOUND_eth2 - [0:0]
:NETWORK_STATS - [0:0]
:NETWORK_STATS_eth1 - [0:0]
-A INPUT -i eth0 -p tcp -m tcp --dport 10086 -j ACCEPT
-A INPUT -j NETWORK_STATS
-A INPUT -d 172.16.64.3/32 -i eth2 -p tcp -m tcp --dport 80 -m state --state
NEW -j ACCEPT
-A INPUT -d 172.16.64.3/32 -i eth2 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -d 172.16.64.3/32 -i eth2 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i eth2 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -j NETWORK_STATS
-A INPUT -i eth2 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i eth2 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT
-A INPUT -d 224.0.0.18/32 -j ACCEPT
-A INPUT -d 225.0.0.50/32 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j
ACCEPT
-A INPUT -d 224.0.0.18/32 -j ACCEPT
-A INPUT -d 225.0.0.50/32 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j
ACCEPT
-A FORWARD -j NETWORK_STATS
-A FORWARD -j NETWORK_STATS_eth1
-A FORWARD -j NETWORK_STATS
-A FORWARD -d 172.16.64.0/24 -o eth2 -j ACL_INBOUND_eth2
-A FORWARD -s 172.16.64.0/22 ! -d 172.16.64.0/22 -j ACCEPT
-A OUTPUT -j NETWORK_STATS
-A OUTPUT -j NETWORK_STATS
-A ACL_INBOUND_eth2 -d 225.0.0.50/32 -j ACCEPT
-A ACL_INBOUND_eth2 -d 224.0.0.18/32 -j ACCEPT
-A NETWORK_STATS -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 -o eth0 -p tcp
-A NETWORK_STATS ! -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 ! -o eth0 -p tcp
-A NETWORK_STATS -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 -o eth0 -p tcp
-A NETWORK_STATS ! -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 ! -o eth0 -p tcp
-A NETWORK_STATS_eth1 -d 172.16.64.0/24 -o eth1
-A NETWORK_STATS_eth1 -s 172.16.64.0/24 -o eth1
COMMIT
# Completed on Tue Aug 29 21:08:17 2017
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:08:17 2017
*nat
:PREROUTING ACCEPT [70:3660]
:INPUT ACCEPT [16:1104]
:OUTPUT ACCEPT [10:641]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 172.16.64.0/24 -o eth2 -j SNAT --to-source 172.16.64.3
-A POSTROUTING -o eth1 -j SNAT --to-source 207.191.181.154
COMMIT
# Completed on Tue Aug 29 21:08:17 2017
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:02:26 2017
*mangle
:PREROUTING ACCEPT [4793:505137]
:INPUT ACCEPT [5180:515989]
:FORWARD ACCEPT [4:304]
:OUTPUT ACCEPT [6985:654900]
:POSTROUTING ACCEPT [6989:655204]
:ACL_OUTBOUND_eth2 - [0:0]
:VPN_STATS_eth1 - [0:0]
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark
--nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -m state --state RELATED,ESTABLISHED -j CONNMARK --restore-mark
--nfmask 0xffffffff --ctmask 0xffffffff
-A PREROUTING -i eth2 -m state --state NEW -j CONNMARK --set-xmark
0x2/0xffffffff
-A PREROUTING -s 172.16.64.0/24 ! -d 172.16.64.1/32 -i eth2 -m state --state
NEW -j ACL_OUTBOUND_eth2
-A PREROUTING -i eth1 -m state --state NEW -j CONNMARK --set-xmark
0x1/0xffffffff
-A FORWARD -j VPN_STATS_eth1
-A OUTPUT -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
-A ACL_OUTBOUND_eth2 -d 224.0.0.18/32 -j ACCEPT
-A ACL_OUTBOUND_eth2 -j ACCEPT
-A ACL_OUTBOUND_eth2 -d 225.0.0.50/32 -j ACCEPT
-A VPN_STATS_eth1 -o eth1 -m mark --mark 0x525
-A VPN_STATS_eth1 -i eth1 -m mark --mark 0x524
COMMIT
# Completed on Tue Aug 29 21:02:26 2017
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:02:26 2017
*filter
:INPUT DROP [46:2884]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [6985:654900]
:ACL_INBOUND_eth2 - [0:0]
:NETWORK_STATS - [0:0]
:NETWORK_STATS_eth1 - [0:0]
-A INPUT -d 172.16.64.3/32 -i eth2 -p tcp -m tcp --dport 80 -m state --state
NEW -j ACCEPT
-A INPUT -d 172.16.64.3/32 -i eth2 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -d 172.16.64.3/32 -i eth2 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -j NETWORK_STATS
-A INPUT -d 224.0.0.18/32 -j ACCEPT
-A INPUT -d 225.0.0.50/32 -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 3922 -m state --state NEW,ESTABLISHED -j
ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i eth2 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i eth2 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A INPUT -i eth2 -p tcp -m tcp --dport 8080 -m state --state NEW -j ACCEPT
-A FORWARD -j NETWORK_STATS_eth1
-A FORWARD -j NETWORK_STATS
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -d 172.16.64.0/24 -o eth2 -j ACL_INBOUND_eth2
-A FORWARD -s 172.16.64.0/22 ! -d 172.16.64.0/22 -j ACCEPT
-A OUTPUT -j NETWORK_STATS
-A ACL_INBOUND_eth2 -d 225.0.0.50/32 -j ACCEPT
-A ACL_INBOUND_eth2 -d 224.0.0.18/32 -j ACCEPT
-A ACL_INBOUND_eth2 -j DROP
-A NETWORK_STATS -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 -o eth0 -p tcp
-A NETWORK_STATS ! -i eth0 -o eth2 -p tcp
-A NETWORK_STATS -i eth2 ! -o eth0 -p tcp
-A NETWORK_STATS_eth1 -d 172.16.64.0/24 -o eth1
-A NETWORK_STATS_eth1 -s 172.16.64.0/24 -o eth1
COMMIT
# Completed on Tue Aug 29 21:02:26 2017
# Generated by iptables-save v1.4.14 on Tue Aug 29 21:02:26 2017
*nat
:PREROUTING ACCEPT [1736:87596]
:INPUT ACCEPT [164:11672]
:OUTPUT ACCEPT [42:2722]
:POSTROUTING ACCEPT [2:472]
-A POSTROUTING -s 172.16.64.0/24 -o eth2 -j SNAT --to-source 172.16.64.3
-A POSTROUTING -o eth1 -j SNAT --to-source 207.191.181.154
COMMIT
# Completed on Tue Aug 29 21:02:26 2017
