Re: [Shorewall-users] Packet loss when changing mangle rule
Hi, I've tried fiddling around with the following parameters, but it seems to make no difference: net.core.default_qdisc = pfifo_fast net.ipv4.tcp_congestion_control = cubic net.ipv4.tcp_available_congestion_control = cubic reno bbr cdg The above values are default in my system. I tried the following: # sysctl -w net.core.default_qdisc=fq_codel # sysctl -w net.ipv4.tcp_congestion_control=cdg I restarted shorewall, but not the OS. I'm still seeing the same packet loss behavior after a while. Can anyone with more experience please tell me if I should revert to defaults (qdisc, congestion control) for "shorewall router" purposes. I mean that maybe fq_codel and cdg "won't hurt" as they seem to be unrelated to my packet loss issue. Thanks, Vieri -- Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot ___ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users
Re: [Shorewall-users] Packet loss when changing mangle rule
On Wednesday, August 15, 2018, 12:40:29 AM GMT+2, Tom Eastep
wrote:
> The only thing that I see is heavy upload traffic to port 80 at IP
> address 31.216.144.11.
>
> There are no ARP packets in the entire pcap file, so it isn't a problem
> of the upstream router not being able to determine the L2 address of
> your gateway. Given that the echo-request packets are being sent, that
> is about the only thing that your gateway could be involved in that
> would result in dropped incoming echo-reply packets.
>
> There are also some retransmissions of TCP packets going on, but that
> isn't all that unusual.
>
> I think that the next step I would take would be to pick a destination
> "closer" to your gateway; use 'traceroute' to determine the IP address
> of the router that is a couple of hops from your gateway and ping that.
> If you don't see packet loss in those pings, then the problem is likely
> beyond that router.
Thanks for helping out.
I noticed that whichever interface I use to run "traceroute" (enp9s4 for ISP1,
enp9s5 for ISP2, enp9s6 for ISP3) I always get an unknown and unpingable
private IP address at hop #2 (192.168.144.1):
# traceroute -n -i enp9s6 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
1 192.168.101.1 0.378 ms 0.521 ms 0.657 ms
2 192.168.144.1 5.366 ms 5.435 ms 5.437 ms
[...]
The modem/router's LAN IPv4 address is 192.168.101.1, and is pingable from the
Shorewall gateway.
I checked the modem/router's configuration for anything related to
192.168.144.*, but found nothing.
I ran this on the gateway:
# nmap -vv -Pn -e enp9s6 192.168.144.1
Starting Nmap 7.40 ( https://nmap.org ) at 2018-08-16 09:02 CEST
Initiating Parallel DNS resolution of 1 host. at 09:02
Completed Parallel DNS resolution of 1 host. at 09:02, 13.04s elapsed
Initiating SYN Stealth Scan at 09:02
Scanning 192.168.144.1 [1000 ports]
SYN Stealth Scan Timing: About 15.45% done; ETC: 09:05 (0:02:50 remaining)
SYN Stealth Scan Timing: About 30.00% done; ETC: 09:05 (0:02:22 remaining)
SYN Stealth Scan Timing: About 44.55% done; ETC: 09:05 (0:01:53 remaining)
SYN Stealth Scan Timing: About 59.55% done; ETC: 09:05 (0:01:22 remaining)
SYN Stealth Scan Timing: About 74.65% done; ETC: 09:05 (0:00:51 remaining)
Completed SYN Stealth Scan at 09:05, 202.83s elapsed (1000 total ports)
Nmap scan report for 192.168.144.1
Host is up, received user-set.
All 1000 scanned ports on 192.168.144.1 are filtered because of 1000
no-responses
Read data files from: /usr/bin/../share/nmap
Nmap done: 1 IP address (1 host up) scanned in 216.03 seconds
Raw packets sent: 2000 (88.000KB) | Rcvd: 2 (484B)
Anyway, the hops after the second one are usually different when I run
"traceroute" several times.
For instance, if I run "traceroute -n -i enp9s6 www.shorewall.net" several
times I get different IP addresses from hop #3 onwards until I reach
63.135.54.24.
Same thing for www.fltk.org or www.gentoo.org. Somehow, whenever I run
"traceroute -n -i enp9s6 8.8.8.8" I always get the same IP addresses at least
for hops #3 and #4 (out of a total of 10). In any case, none of these hosts
with these IP addresses reply to ping requests ("ping -n -I enp9s6
HOP_IP_ADDR"). In other words, when trying to access Google's primary DNS
server at 8.8.8.8, the only pingable IP addresses in the path are 8.8.8.8
itself and the modem/router's LAN IP addr. at 192.168.101.1.
I guess I need to ask my ISP to allow ICMP traffic for these intermediate
routers.
Looking back at the traceroute results for shorewall.net, I notice that hop #6
is the first pingable IP address even though it's not always the same. I can
also confirm by geoIP db lookup that it belongs to my ISP.
So I wrote it down and waited until I detected another packet loss event.
When it happened, I pinged that host to find out that it behaved just like in
my previous trace to 8.8.8.8 (ie. echo requests leave the shorewall gateway but
there are missing replies).
Also, this is what happens when I run a traceroute to shorewall:
# traceroute -n -i enp9s6 www.shorewall.net (63.135.54.24), 30 hops max, 60
byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
Another try shortly after shows this:
# traceroute -n -i enp9s6 www.shorewall.net (63.135.54.24), 30 hops max, 60
byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *
10 * * *
11 * * 4.35.175.6 198.850 ms
12 206.130.130.241 197.610 ms * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
# ping -n -I enp9s6 192.168.101.1
shows no packet loss (modem/router's
Re: [Shorewall-users] Packet loss when changing mangle rule
On 08/13/2018 12:55 AM, Vieri Di Paola via Shorewall-users wrote: > > On Saturday, August 11, 2018, 8:07:36 PM GMT+2, Tom Eastep > wrote: > >> My suggestion for debugging this further is to use a packet sniffer to >> see what is happening on the wire during the period of loss: >> >> a) Are the echo-request packets being sent? >> b) If not, is there unsuccessful ARPing occurring? > > The echo requests are being sent as seen below. > > # tcpdump -i enp9s6 host 8.8.8.8 > dropped privs to tcpdump > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on enp9s6, link-type EN10MB (Ethernet), capture size 262144 bytes > 08:12:18.603636 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 1, length 64 > 08:12:19.642887 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 2, length 64 > 08:12:20.682810 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 3, length 64 > 08:12:21.721237 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 4, length 64 > 08:12:22.762528 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 5, length 64 > 08:12:23.802597 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 6, length 64 > 08:12:24.841126 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 7, length 64 > 08:12:25.881195 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 8, length 64 > 08:12:26.922607 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 9, length 64 > 08:12:27.962655 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 10, length 64 > 08:12:29.001091 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 11, length 64 > 08:12:30.042569 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 12, length 64 > 08:12:31.082581 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 13, length 64 > 08:12:32.122591 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 14, length 64 > 08:12:33.162544 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 15, length 64 > 08:12:33.177647 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 15, length 64 > 08:12:34.171027 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 16, length 64 > 08:12:34.186632 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 16, length 64 > 08:12:35.180972 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 17, length 64 > 08:12:35.196782 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 17, length 64 > 08:12:36.191086 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 18, length 64 > 08:12:36.206656 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 18, length 64 > 08:12:37.201017 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 19, length 64 > 08:12:37.216632 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 19, length 64 > 08:12:38.210990 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 20, length 64 > 08:12:38.226712 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 20, length 64 > 08:12:39.219069 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 21, length 64 > 08:12:39.234713 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 21, length 64 > 08:12:40.229018 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 22, length 64 > 08:12:40.244660 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 22, length 64 > 08:12:41.230962 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo > request, id 4825, seq 23, length 64 > 08:12:41.246677 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo > reply, id 4825, seq 23, length 64 > > The pattern of echo replies failing to arrive is variable. At times, there's > no packet loss, but I frequently get a 10-20-40% packet loss (sometimes even > as high as 65%). > I also noticed (but it's just an impression) that right after rebooting the > modem/router I get no packet loss whatsoever for several hours. > > The shorewall gateway's enp9s6 interface is 100Mbps full duplex, and is > connected to the modem/router which has a built-in 4-port Gbps switch with an > ethernet cable. If I connect a test laptop to that mini switch while I'm > experiencing the packet
Re: [Shorewall-users] Packet loss when changing mangle rule
On Saturday, August 11, 2018, 8:07:36 PM GMT+2, Tom Eastep wrote: > My suggestion for debugging this further is to use a packet sniffer to > see what is happening on the wire during the period of loss: > > a) Are the echo-request packets being sent? > b) If not, is there unsuccessful ARPing occurring? The echo requests are being sent as seen below. # tcpdump -i enp9s6 host 8.8.8.8 dropped privs to tcpdump tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp9s6, link-type EN10MB (Ethernet), capture size 262144 bytes 08:12:18.603636 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 1, length 64 08:12:19.642887 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 2, length 64 08:12:20.682810 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 3, length 64 08:12:21.721237 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 4, length 64 08:12:22.762528 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 5, length 64 08:12:23.802597 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 6, length 64 08:12:24.841126 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 7, length 64 08:12:25.881195 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 8, length 64 08:12:26.922607 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 9, length 64 08:12:27.962655 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 10, length 64 08:12:29.001091 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 11, length 64 08:12:30.042569 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 12, length 64 08:12:31.082581 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 13, length 64 08:12:32.122591 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 14, length 64 08:12:33.162544 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 15, length 64 08:12:33.177647 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 15, length 64 08:12:34.171027 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 16, length 64 08:12:34.186632 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 16, length 64 08:12:35.180972 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 17, length 64 08:12:35.196782 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 17, length 64 08:12:36.191086 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 18, length 64 08:12:36.206656 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 18, length 64 08:12:37.201017 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 19, length 64 08:12:37.216632 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 19, length 64 08:12:38.210990 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 20, length 64 08:12:38.226712 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 20, length 64 08:12:39.219069 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 21, length 64 08:12:39.234713 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 21, length 64 08:12:40.229018 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 22, length 64 08:12:40.244660 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 22, length 64 08:12:41.230962 IP 192.168.101.2 > google-public-dns-a.google.com: ICMP echo request, id 4825, seq 23, length 64 08:12:41.246677 IP google-public-dns-a.google.com > 192.168.101.2: ICMP echo reply, id 4825, seq 23, length 64 The pattern of echo replies failing to arrive is variable. At times, there's no packet loss, but I frequently get a 10-20-40% packet loss (sometimes even as high as 65%). I also noticed (but it's just an impression) that right after rebooting the modem/router I get no packet loss whatsoever for several hours. The shorewall gateway's enp9s6 interface is 100Mbps full duplex, and is connected to the modem/router which has a built-in 4-port Gbps switch with an ethernet cable. If I connect a test laptop to that mini switch while I'm experiencing the packet loss issues on the shorewall box, I can see the same problems there too. Disconnecting the shorewall gateway ethernet cable from that 4-port switch, hence making my test laptop the only client device in the network makes the packet loss is
Re: [Shorewall-users] Packet loss when changing mangle rule
On 08/09/2018 01:38 AM, Vieri Di Paola via Shorewall-users wrote:
> Hi,
>
> I've encountered a weird issue.
>
> I have 3 ISP links (WAN) connected to a shorewall gateway, each on their own
> NIC.
>
> After about 24 hours working with apparently no issues, I start to get
> network issues on only one of the three.
>
> A simple test from the shorewall gateway shows the following packet loss when
> pinging from the NIC that's connected to the failing ISP:
>
> # shorewall reset ; ping -n -I enp9s6 8.8.8.8 ; shorewall dump >
> /home/vieri/swdump
> Shorewall Counters Reset
> PING 8.8.8.8 (8.8.8.8) from 192.168.101.2 enp9s6: 56(84) bytes of data.
> 64 bytes from 8.8.8.8: icmp_seq=12 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=13 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=14 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=15 ttl=120 time=10.9 ms
> 64 bytes from 8.8.8.8: icmp_seq=16 ttl=120 time=11.0 ms
> 64 bytes from 8.8.8.8: icmp_seq=17 ttl=120 time=11.0 ms
> 64 bytes from 8.8.8.8: icmp_seq=18 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=19 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=20 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=21 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=22 ttl=120 time=11.1 ms
> 64 bytes from 8.8.8.8: icmp_seq=23 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=24 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=25 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=26 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=27 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=28 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=29 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=30 ttl=120 time=11.4 ms
> 64 bytes from 8.8.8.8: icmp_seq=31 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=32 ttl=120 time=11.2 ms
> 64 bytes from 8.8.8.8: icmp_seq=33 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=34 ttl=120 time=11.5 ms
> 64 bytes from 8.8.8.8: icmp_seq=35 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=36 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=37 ttl=120 time=11.4 ms
> 64 bytes from 8.8.8.8: icmp_seq=38 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=39 ttl=120 time=11.3 ms
> 64 bytes from 8.8.8.8: icmp_seq=40 ttl=120 time=11.8 ms
> 64 bytes from 8.8.8.8: icmp_seq=41 ttl=120 time=11.6 ms
> 64 bytes from 8.8.8.8: icmp_seq=42 ttl=120 time=11.4 ms
> ^C
> --- 8.8.8.8 ping statistics ---
> 42 packets transmitted, 31 received, 26% packet loss, time 41698ms
> rtt min/avg/max/mdev = 10.981/11.303/11.890/0.212 ms
>
> The same test on the other 2 ISP links are OK.
>
> Hence, if ISP3 is the failing link and ISP1, ISP2 are OK, I try to move some
> traffic from ISP3 to ISP2 like so in the mangle file:
>
> MARK(2):P ${HMAN_EXTRA_CORP_NETWORKS}
> (2: ISP2, 3: ISP3,
> HMAN_EXTRA_CORP_NETWORKS="192.168.210.0/23,192.168.212.0/24")
>
> Now, the same ping test from the NIC that's connected to ISP2 starts showing
> the same packet loss stats while the test on the NIC connected to ISP3 has 0%
> packet loss.
>
> Wherever I move the traffic with this line in the mangle file, I get ICMP
> packet loss, ie., moving it back to MARK(3) (ISP3) shows packet loss again
> only on that line.
>
> The shorewall dump taken during the test above is here:
>
> https://drive.google.com/open?id=1a6RlQhi2w_JJF9ZuFt6aI9G-JAQbFC9n
>
> Finally, to top it all off, if I reboot the modem/router on the ISP3 link,
> all's well again (no packet loss whatsoever, no matter which rule I use in
> the mangle file). Until the next day...
>
> So, how can I go about this to determine what's causing this issue? My
> Internet Provider has already passed the buck and thinks that it's an issue
> with my shorewall gateway...
>
> Help appreciated.
>
I don't see anything in the dump that explains this behavior. I do,
however, notice this conntrack table entry:
icmp 1 29 src=192.168.101.2 dst=8.8.8.8 type=8 code=0 id=3380
packets=42 bytes=3528 src=8.8.8.8 dst=192.168.101.2 type=0 code=0
id=3380 packets=31 bytes=2604 mark=3 use=1
'mark=3' indicates that the flow is using the correct interface (enp9s6).
My suggestion for debugging this further is to use a packet sniffer to
see what is happening on the wire during the period of loss:
a) Are the echo-request packets being sent?
b) If not, is there unsuccessful ARPing occurring?
-Tom
--
Tom Eastep\ Q: What do you get when you cross a mobster with
Shoreline, \ an international standard?
Washington, USA \ A: Someone who makes you an offer you can't
http://shorewall.org \ understand
\___
signature.asc
Description: OpenPGP digital signature
--
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/
