MASQ/SNAT and ARP are interacting in a way that is causing outbound
connectivity issues in periods of low traffic (when ARP entries timeout).
Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I
need them to have the Firewall's Interface IP address as their source.
I have modified MASQ to SNAT to the Firewall's Interface IP address for the
Peering network (via 198.32.212.73), but outbound traffic is normally to
more distant networks and my default route is to the Paid Internet (via
121.200.226.210).
I have seen some have scripted ARP watchers that could assist but I believe
this is something Shorewall can cope with, but I am lacking in the
knowledge.
r...@per-r1:/etc/shorewall# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:121.200.226.210 Bcast:121.200.226.211
Mask:255.255.255.252
eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0
eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0
eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91
inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0
r...@per-r1:/etc/shorewall# ip route show table main | grep -v zebra
121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210
198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73
180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7
10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1
default via 121.200.226.209 dev eth0 metric 100
#
# Shorewall version 4 - Masq file
#
eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7
eth0:198.32.212.0/24 10.240.1.0/24 198.32.212.73
r...@per-r1:/etc/shorewall# cat shorewall.conf | grep -v \# | grep -v ^$
STARTUP_ENABLED=Yes
VERBOSITY=1
SHOREWALL_COMPILER=
LOGFILE=/var/log/messages
LOGFORMAT="Shorewall:%s:%s:"
LOGTAGONLY=No
LOGRATE=12/minute
LOGBURST=3
LOGALLNEW=
BLACKLIST_LOGLEVEL=
MACLIST_LOG_LEVEL=info
TCP_FLAGS_LOG_LEVEL=info
RFC1918_LOG_LEVEL=info
SMURF_LOG_LEVEL=info
LOG_MARTIANS=No
IPTABLES=
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
SHOREWALL_SHELL=/bin/sh
SUBSYSLOCK=""
MODULESDIR=
CONFIG_PATH=/etc/shorewall:/usr/share/shorewall
RESTOREFILE=
IPSECFILE=zones
LOCKFILE=
DROP_DEFAULT="Drop"
REJECT_DEFAULT="Reject"
ACCEPT_DEFAULT="none"
QUEUE_DEFAULT="none"
NFQUEUE_DEFAULT="none"
RSH_COMMAND='ssh ${ro...@${system} ${command}'
RCP_COMMAND='scp ${files} ${ro...@${system}:${destination}'
IP_FORWARDING=On
ADD_IP_ALIASES=Yes
ADD_SNAT_ALIASES=No
RETAIN_ALIASES=No
TC_ENABLED=No
TC_EXPERT=No
CLEAR_TC=No
MARK_IN_FORWARD_CHAIN=No
CLAMPMSS=No
ROUTE_FILTER=Yes
DETECT_DNAT_IPADDRS=No
MUTEX_TIMEOUT=60
ADMINISABSENTMINDED=Yes
BLACKLISTNEWONLY=Yes
DELAYBLACKLISTLOAD=No
MODULE_SUFFIX=
DISABLE_IPV6=Yes
BRIDGING=No
DYNAMIC_ZONES=No
PKTTYPE=Yes
RFC1918_STRICT=No
MACLIST_TABLE=filter
MACLIST_TTL=
SAVE_IPSETS=No
MAPOLDACTIONS=No
FASTACCEPT=Yes
IMPLICIT_CONTINUE=Yes
HIGH_ROUTE_MARKS=No
USE_ACTIONS=Yes
OPTIMIZE=0
EXPORTPARAMS=Yes
EXPAND_POLICIES=Yes
KEEP_RT_TABLES=No
DELETE_THEN_ADD=Yes
MULTICAST=No
DONT_LOAD=
BLACKLIST_DISPOSITION=DROP
MACLIST_DISPOSITION=REJECT
TCP_FLAGS_DISPOSITION=DROP
Only masq, hosts, rules, zones, policy, interfaces, params, and
shorewall.conf have non default settings.
r...@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212
198.32.212.12 dev eth0 lladdr 00:1f:12:8b:28:29 DELAY
198.32.212.2 dev eth0 lladdr 00:24:14:2b:98:00 REACHABLE
198.32.212.37 dev eth0 lladdr 00:1b:0d:ef:7e:80 REACHABLE
198.32.212.17 dev eth0 lladdr 00:02:17:ee:ee:01 REACHABLE
198.32.212.64 dev eth0 lladdr 00:90:69:df:60:7e REACHABLE
198.32.212.15 dev eth0 lladdr 00:1a:2f:c6:9e:a0 STALE
198.32.212.22 dev eth0 lladdr 00:0d:65:ec:20:01 DELAY
198.32.212.240 dev eth0 lladdr 00:24:dc:12:ad:80 REACHABLE
198.32.212.253 dev eth0 lladdr 00:24:dc:0b:eb:80 REACHABLE
198.32.212.46 dev eth0 lladdr 00:1b:2b:4f:44:0a DELAY
198.32.212.5 dev eth0 lladdr 00:0b:60:a5:78:1b REACHABLE
198.32.212.44 dev eth0 lladdr 00:02:fc:fa:9c:19 DELAY
198.32.212.33 dev eth0 lladdr 00:13:5f:1c:8a:80 DELAY
198.32.212.52 dev eth0 lladdr 00:d0:ff:e2:58:38 STALE
198.32.212.57 dev eth0 lladdr 00:13:1a:27:a2:1b STALE
198.32.212.16 dev eth0 lladdr 00:01:63:39:ec:1b STALE
198.32.212.40 dev eth0 lladdr 00:19:e7:71:d7:a8 STALE
198.32.212.7 dev eth0 lladdr 00:0f:8f:e2:a1:01 STALE
198.32.212.72 dev eth0 lladdr 00:0f:35:a2:00:19 STALE
TCPDUMP of ARP traffic:
r...@per-r1:/etc/shorewall# tcpdump -nei eth0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
12:06:38.102489 00:15:17:cc:dd:90 > 00:1b:2b:4f:44:0a, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.46 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:38.103212 00:1b:2b:4f:44:0a > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.46 is-at 00:1b:2b:4f:44:0a
12:06:38.132488 00:15:17:cc:dd:90 > 00:21:05:46:63:db, ethertype ARP
(0x0806), length 42: arp who-has 121.200.226.209 tell 121.200.226.210
12:06:38.134412 00:21:05:46:63:db > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 121.200.226.209 is-at 00:21:05:46:63:db
12:06:39.123738 00:15:17:cc:dd:90 > 00:1f:12:8b:28:29, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.12 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:39.125163 00:1f:12:8b:28:29 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.12 is-at 00:1f:12:8b:28:29
12:06:40.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:40.984990 00:15:17:cc:dd:90 > 00:0d:65:ec:20:01, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.22 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:40.985862 00:0d:65:ec:20:01 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.22 is-at 00:0d:65:ec:20:01
12:06:41.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:42.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:43.701238 00:15:17:cc:dd:90 > 00:02:fc:fa:9c:19, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.44 tell 198.32.212.73
12:06:43.702478 00:02:fc:fa:9c:19 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.44 is-at 00:02:fc:fa:9c:19
12:06:44.018817 00:17:0e:87:6a:c1 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp who-has 198.32.212.73 (00:15:17:cc:dd:90) tell
198.32.212.206
12:06:44.018832 00:15:17:cc:dd:90 > 00:17:0e:87:6a:c1, ethertype ARP
(0x0806), length 42: arp reply 198.32.212.73 is-at 00:15:17:cc:dd:90
^C
15 packets captured
18 packets received by filter
0 packets dropped by kernel
Since capturing this tcpdump I have improved performance with:
printf "net.ipv4.neigh.eth0.gc_stale_time = 3600" >> /etc/sysctl.conf
sudo /sbin/sysctl -p /etc/sysctl.conf
now seeing:
r...@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212 | wc
65 390 3578
Was 22 132 1247
While performance has improved the gc_stale_time change could have
undesirable side effects so I would still like to find out if Shorewall can
improve the SNAT / ARP issue at the root of this?
Kind regards,
Trent O'Callaghan
www.nearmap.com
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
_______________________________________________
Shorewall-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/shorewall-users
