Re: Gateway server experiencing degraded performance
On 05/16, Andrew K Adams wrote: > I've looked at traffic with tcpdump, but there is so much in both > scenarios that I can't really say that one service, or a > packet-of-death is triggering the degraded state. But thinking in > aggregate, if there was a way to monitor the average packets/sec the > gateway is forwarding, in both scenarios, I'd at least know for > certain that it is or is not my gateway (I guess it's conceivable that > by rebooting I'm tweaking state in Comcast's next-hop during DHCP > negotiation or something ...) Hi, Andrew! I doubt it's the same problem, but there's another thread where rebooting helps: https://mail-index.netbsd.org/tech-net/2022/04/12/msg008179.html Re monitoring in aggregate, I know there are people here way more knowledgeable than me, but if you haven't been able to make any progress on tracking this down, I wonder if periodically capturing the output of vmstat -CmW and netstat -m might help on the outside chance that they might show some statistic change over time that could lead you to the source of the problem? Regards, Lewis
Re: Gateway server experiencing degraded performance
On Mon, 16 May 2022, Andrew K Adams wrote: I've looked at traffic with tcpdump, but there is so much in both scenarios that I can't really say that one service, or a packet-of-death is triggering the degraded state. But thinking in aggregate, if there was a way to monitor the average packets/sec the gateway is forwarding, in both scenarios, I'd at least know for certain that it is or is not my gateway (I guess it's conceivable that by rebooting I'm tweaking state in Comcast's next-hop during DHCP negotiation or something ...) net/vnstat in pkgsrc might be useful for this. Hi, I’ve noticed a problem with my NetBSD server that I really could use some help with. The server is acting as my home router (gateway) in ‘pass-through’ mode with Comcast Xfinity as the service provider. The only functions/services enabled on the server are: ip-forwarding, dhcp-client, and NTP. And on the internal-facing lan: NAT, DHCP, and SSH. The internal network supports 5+ wired hosts & 10+ wireless devices. I currently have the 1Gig option with Comcast and when things are operating as expected, I see from an internal (wired) host 750mbs of throughput according to Comcast’s Xfinity speed test website. The problem is that performance doesn’t last. In anywhere from 3 weeks to 24 hours, my network performance (again, using the same wired host measured by Xfinity’s speed test) will drop to ~20mbs. And the change is clearly observable in all networking applications, e.g., Netflix. I’m querying here, though, because the fix I’ve found is to reboot the server. Upon reboot, I immediately go back to 750mbs. Hence, I’d like to figure out how to troubleshoot what on my server is (eventually) degrading my network performance. The server is relatively new; I built it using an ASUS Mini ITX with two onboard GigE ethernet ports. Here’s some relevant dmesg output: [ 1.00] NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020 [ 1.00] mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC [ 1.00] total memory = 3457 MB [ 1.00] avail memory = 3380 MB [ 1.025286] wm0 at pci0 dev 25 function 0: I217 LM Ethernet Connection (rev. 0x05) [ 1.025286] wm0: interrupting at msi2 vec 0 [ 1.025286] wm0: PCI-Express bus [ 1.025286] wm0: 2048 words FLASH, version 0.13.4 [ 1.025286] wm0: 0x6a4480 [ 1.025286] re0 at pci3 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 0x0c) [ 1.025286] re0: interrupting at msix4 vec 0 [ 1.025286] re0: using 256 tx descriptors And from ifconfig (note, I’ve removed my IP & MAC address from the output): iquitos# ifconfig -a wm0: flags=0x8843 mtu 1500 capabilities=7ff80 capabilities=7ff80 capabilities=7ff80 enabled=0 ec_capabilities=17 ec_enabled=2 media: Ethernet autoselect (1000baseT full-duplex,flowcontrol,rxpause,txpause) status: active inet X.X.X.X/20 broadcast 255.255.255.255 flags 0x0 Inet6 X%wm0/64 flags 0x0 scopeid 0x1 re0: flags=0x8843 mtu 1500 capabilities=3f80 capabilities=3f80 enabled=0 ec_capabilities=3 ec_enabled=0 media: Ethernet autoselect (1000baseT full-duplex) status: active inet 10.19.34.1/24 broadcast 10.19.34.255 flags 0x0 inet6 fe80::728b:cdff:febc:831b%re0/64 flags 0x0 scopeid 0x2 lo0: flags=0x8049 mtu 33176 inet 127.0.0.1/8 flags 0x0 inet6 ::1/128 flags 0x20 inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x3 I’m not sure what other information would be useful, just let me know, and thank you in advance for help you can give me! Just a question, not a requirement, but have you looked at Wireshark traces comparing good performance vs. bad performance? It's just another way to find possibly find a problem. I'm sure someone else will have better ideas. Andy
Re: Gateway server experiencing degraded performance
I've looked at traffic with tcpdump, but there is so much in both scenarios that I can't really say that one service, or a packet-of-death is triggering the degraded state. But thinking in aggregate, if there was a way to monitor the average packets/sec the gateway is forwarding, in both scenarios, I'd at least know for certain that it is or is not my gateway (I guess it's conceivable that by rebooting I'm tweaking state in Comcast's next-hop during DHCP negotiation or something ...) Andrew On Sun, May 15, 2022 at 12:19 PM Andy Ruhl wrote: > > On Sun, May 15, 2022 at 7:52 AM Andrew K Adams wrote: > > > > Hi, I’ve noticed a problem with my NetBSD server that I really could > > use some help with. The server is acting as my home router (gateway) > > in ‘pass-through’ mode with Comcast Xfinity as the service provider. > > The only functions/services enabled on the server are: ip-forwarding, > > dhcp-client, and NTP. And on the internal-facing lan: NAT, DHCP, and > > SSH. The internal network supports 5+ wired hosts & 10+ wireless > > devices. I currently have the 1Gig option with Comcast and when > > things are operating as expected, I see from an internal (wired) host > > 750mbs of throughput according to Comcast’s Xfinity speed test > > website. The problem is that performance doesn’t last. In anywhere > > from 3 weeks to 24 hours, my network performance (again, using the > > same wired host measured by Xfinity’s speed test) will drop to ~20mbs. > > And the change is clearly observable in all networking applications, > > e.g., Netflix. I’m querying here, though, because the fix I’ve found > > is to reboot the server. Upon reboot, I immediately go back to > > 750mbs. Hence, I’d like to figure out how to troubleshoot what on my > > server is (eventually) degrading my network performance. > > > > The server is relatively new; I built it using an ASUS Mini ITX with > > two onboard GigE ethernet ports. Here’s some relevant dmesg output: > > > > [ 1.00] NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020 > > [ 1.00] > > mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC > > [ 1.00] total memory = 3457 MB > > [ 1.00] avail memory = 3380 MB > > > > [ 1.025286] wm0 at pci0 dev 25 function 0: I217 LM Ethernet > > Connection (rev. 0x05) > > [ 1.025286] wm0: interrupting at msi2 vec 0 > > [ 1.025286] wm0: PCI-Express bus > > [ 1.025286] wm0: 2048 words FLASH, version 0.13.4 > > [ 1.025286] wm0: 0x6a4480 > > > > [ 1.025286] re0 at pci3 dev 0 function 0: RealTek 8168/8111 PCIe > > Gigabit Ethernet (rev. 0x0c) > > [ 1.025286] re0: interrupting at msix4 vec 0 > > [ 1.025286] re0: using 256 tx descriptors > > > > > > And from ifconfig (note, I’ve removed my IP & MAC address from the output): > > > > iquitos# ifconfig -a > > wm0: flags=0x8843 mtu 1500 > > capabilities=7ff80 > > capabilities=7ff80 > > capabilities=7ff80 > > enabled=0 > > ec_capabilities=17 > > ec_enabled=2 > > media: Ethernet autoselect (1000baseT > > full-duplex,flowcontrol,rxpause,txpause) > > status: active > > inet X.X.X.X/20 broadcast 255.255.255.255 flags 0x0 > > Inet6 X%wm0/64 flags 0x0 scopeid 0x1 > > re0: flags=0x8843 mtu 1500 > > > > capabilities=3f80 > > capabilities=3f80 > > enabled=0 > > ec_capabilities=3 > > ec_enabled=0 > > media: Ethernet autoselect (1000baseT full-duplex) > > status: active > > inet 10.19.34.1/24 broadcast 10.19.34.255 flags 0x0 > > inet6 fe80::728b:cdff:febc:831b%re0/64 flags 0x0 scopeid 0x2 > > lo0: flags=0x8049 mtu 33176 > > inet 127.0.0.1/8 flags 0x0 > > inet6 ::1/128 flags 0x20 > > inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x3 > > > > > > I’m not sure what other information would be useful, just let me know, > > and thank you in advance for help you can give me! > > Just a question, not a requirement, but have you looked at Wireshark > traces comparing good performance vs. bad performance? > > It's just another way to find possibly find a problem. I'm sure > someone else will have better ideas. > > Andy
Re: Gateway server experiencing degraded performance
On Sun, May 15, 2022 at 7:52 AM Andrew K Adams wrote: > > Hi, I’ve noticed a problem with my NetBSD server that I really could > use some help with. The server is acting as my home router (gateway) > in ‘pass-through’ mode with Comcast Xfinity as the service provider. > The only functions/services enabled on the server are: ip-forwarding, > dhcp-client, and NTP. And on the internal-facing lan: NAT, DHCP, and > SSH. The internal network supports 5+ wired hosts & 10+ wireless > devices. I currently have the 1Gig option with Comcast and when > things are operating as expected, I see from an internal (wired) host > 750mbs of throughput according to Comcast’s Xfinity speed test > website. The problem is that performance doesn’t last. In anywhere > from 3 weeks to 24 hours, my network performance (again, using the > same wired host measured by Xfinity’s speed test) will drop to ~20mbs. > And the change is clearly observable in all networking applications, > e.g., Netflix. I’m querying here, though, because the fix I’ve found > is to reboot the server. Upon reboot, I immediately go back to > 750mbs. Hence, I’d like to figure out how to troubleshoot what on my > server is (eventually) degrading my network performance. > > The server is relatively new; I built it using an ASUS Mini ITX with > two onboard GigE ethernet ports. Here’s some relevant dmesg output: > > [ 1.00] NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020 > [ 1.00] > mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC > [ 1.00] total memory = 3457 MB > [ 1.00] avail memory = 3380 MB > > [ 1.025286] wm0 at pci0 dev 25 function 0: I217 LM Ethernet > Connection (rev. 0x05) > [ 1.025286] wm0: interrupting at msi2 vec 0 > [ 1.025286] wm0: PCI-Express bus > [ 1.025286] wm0: 2048 words FLASH, version 0.13.4 > [ 1.025286] wm0: 0x6a4480 > > [ 1.025286] re0 at pci3 dev 0 function 0: RealTek 8168/8111 PCIe > Gigabit Ethernet (rev. 0x0c) > [ 1.025286] re0: interrupting at msix4 vec 0 > [ 1.025286] re0: using 256 tx descriptors > > > And from ifconfig (note, I’ve removed my IP & MAC address from the output): > > iquitos# ifconfig -a > wm0: flags=0x8843 mtu 1500 > capabilities=7ff80 > capabilities=7ff80 > capabilities=7ff80 > enabled=0 > ec_capabilities=17 > ec_enabled=2 > media: Ethernet autoselect (1000baseT > full-duplex,flowcontrol,rxpause,txpause) > status: active > inet X.X.X.X/20 broadcast 255.255.255.255 flags 0x0 > Inet6 X%wm0/64 flags 0x0 scopeid 0x1 > re0: flags=0x8843 mtu 1500 > capabilities=3f80 > capabilities=3f80 > enabled=0 > ec_capabilities=3 > ec_enabled=0 > media: Ethernet autoselect (1000baseT full-duplex) > status: active > inet 10.19.34.1/24 broadcast 10.19.34.255 flags 0x0 > inet6 fe80::728b:cdff:febc:831b%re0/64 flags 0x0 scopeid 0x2 > lo0: flags=0x8049 mtu 33176 > inet 127.0.0.1/8 flags 0x0 > inet6 ::1/128 flags 0x20 > inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x3 > > > I’m not sure what other information would be useful, just let me know, > and thank you in advance for help you can give me! Just a question, not a requirement, but have you looked at Wireshark traces comparing good performance vs. bad performance? It's just another way to find possibly find a problem. I'm sure someone else will have better ideas. Andy
Gateway server experiencing degraded performance
Hi, I’ve noticed a problem with my NetBSD server that I really could use some help with. The server is acting as my home router (gateway) in ‘pass-through’ mode with Comcast Xfinity as the service provider. The only functions/services enabled on the server are: ip-forwarding, dhcp-client, and NTP. And on the internal-facing lan: NAT, DHCP, and SSH. The internal network supports 5+ wired hosts & 10+ wireless devices. I currently have the 1Gig option with Comcast and when things are operating as expected, I see from an internal (wired) host 750mbs of throughput according to Comcast’s Xfinity speed test website. The problem is that performance doesn’t last. In anywhere from 3 weeks to 24 hours, my network performance (again, using the same wired host measured by Xfinity’s speed test) will drop to ~20mbs. And the change is clearly observable in all networking applications, e.g., Netflix. I’m querying here, though, because the fix I’ve found is to reboot the server. Upon reboot, I immediately go back to 750mbs. Hence, I’d like to figure out how to troubleshoot what on my server is (eventually) degrading my network performance. The server is relatively new; I built it using an ASUS Mini ITX with two onboard GigE ethernet ports. Here’s some relevant dmesg output: [ 1.00] NetBSD 9.1 (GENERIC) #0: Sun Oct 18 19:24:30 UTC 2020 [ 1.00] mkre...@mkrepro.netbsd.org:/usr/src/sys/arch/i386/compile/GENERIC [ 1.00] total memory = 3457 MB [ 1.00] avail memory = 3380 MB [ 1.025286] wm0 at pci0 dev 25 function 0: I217 LM Ethernet Connection (rev. 0x05) [ 1.025286] wm0: interrupting at msi2 vec 0 [ 1.025286] wm0: PCI-Express bus [ 1.025286] wm0: 2048 words FLASH, version 0.13.4 [ 1.025286] wm0: 0x6a4480 [ 1.025286] re0 at pci3 dev 0 function 0: RealTek 8168/8111 PCIe Gigabit Ethernet (rev. 0x0c) [ 1.025286] re0: interrupting at msix4 vec 0 [ 1.025286] re0: using 256 tx descriptors And from ifconfig (note, I’ve removed my IP & MAC address from the output): iquitos# ifconfig -a wm0: flags=0x8843 mtu 1500 capabilities=7ff80 capabilities=7ff80 capabilities=7ff80 enabled=0 ec_capabilities=17 ec_enabled=2 media: Ethernet autoselect (1000baseT full-duplex,flowcontrol,rxpause,txpause) status: active inet X.X.X.X/20 broadcast 255.255.255.255 flags 0x0 Inet6 X%wm0/64 flags 0x0 scopeid 0x1 re0: flags=0x8843 mtu 1500 capabilities=3f80 capabilities=3f80 enabled=0 ec_capabilities=3 ec_enabled=0 media: Ethernet autoselect (1000baseT full-duplex) status: active inet 10.19.34.1/24 broadcast 10.19.34.255 flags 0x0 inet6 fe80::728b:cdff:febc:831b%re0/64 flags 0x0 scopeid 0x2 lo0: flags=0x8049 mtu 33176 inet 127.0.0.1/8 flags 0x0 inet6 ::1/128 flags 0x20 inet6 fe80::1%lo0/64 flags 0x0 scopeid 0x3 I’m not sure what other information would be useful, just let me know, and thank you in advance for help you can give me! Andrew -- Andrew K. Adams, MCS, MIS CISO, NSF Cybersecurity Center of Excellence / trustedci.org PISO, Security Manager, Pittsburgh Supercomputing Center / psc.edu Carnegie Mellon University akad...@psc.edu / +1 412-268-5142
