[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
** Changed in: linux (Ubuntu) Status: Expired => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Triaged Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 50210, offset 0, flags [DF], proto TCP (6), length 64) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc29a (incorrect -> 0x356e), ack 1, win 487,
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Expired Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 50210, offset 0, flags [DF], proto TCP (6), length 64) 10.42.1.8.43060 >
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
The patch highlighted in #5 does not fix this issue for me without other patches from the refactor, so it may be we need to wait for the 5.19 hwe kernel to arrive to fix this issue. ** Changed in: linux (Ubuntu) Status: Expired => Incomplete -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
[Expired for linux (Ubuntu) because there has been no activity for 60 days.] ** Changed in: linux (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Expired Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 78: (tos 0x0, ttl 64, id 50210, offset 0, flags [DF], proto TCP (6), length 64) 10.42.1.8.43060 >
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
Frode, After investigating the issue in our lab, it looks like the used kernel is missing with the following patch: https://lore.kernel.org/netdev/20220504070256.694458-9-sae...@nvidia.com/ The problem should be solved with this patch, please pull it into your 22.04 5.15 kernel. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad,
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
Investigating this issue with Frode, we saw that its is related to the Security Group applied to the HW Offload (switchdev) capable ports. If an "allow all traffic" security group is attached to the port, the issue is not seen. We see the issue when applying a security group with some explicit rules to allow specific traffic and an implicit DROP rule. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
The fix for the issue boils down to these commits: 314e1105831b net/mlx5e: Add post act offload/unoffload API 8300f225268b net/mlx5e: Create new flow attr for multi table actions a81283263bb0 net/mlx5e: Use multi table support for CT and sample actions 2a829fe25d28 net/mlx5e: TC, Clean redundant counter flag from tc action parsers 7843bd604081 net/mlx5e: TC, Make post_act parse CT and sample actions However, they rely on both previous commits in the same v5.18 merge commit as well as a refactoring which has been going on over the v5.16 and v5.17 cycles. This may be complicated to backport to v5.15. So I guess the question becomes: - Would there be something we could do to adapt the fix to what is currently in v5.15? - Would an alternative be to release note this problem and point forward to a fix coming in the 22.04 HWE kernel based on v5.19 which is going to appear in not a too distant future? -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66:
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
fwiw; the fix is somewhere to be found in these mainline commits: b070e70381ee343dc10d2b1f0f3a6b1f940f0ae5 7843bd604081b5f005b31343cde1bb680bcc1776 2a829fe25d2809a33a6842359323bb139556c03d a81283263bb0c38b52fce6d599c2434a8d232dd6 8300f225268be9ee2c0daf5a3f23929fcdcbf213 314e1105831b45a0fe3045411d7fec4997936e86 0610f8dc0309cd0ea017d129efb95a942325344d d1a3138f7913014e0714cb1d3d44793d76fc38a1 bfbdd77ac52fecc198b049bff9574e84519ed9a5 7c5f940d264e4c3d4f4fdbb4dc84b9eca6d3ea43 4b5fba4a3ac790ad79bb4d36e7a4398702429b4d 1d5024f88dadb5949c9fd10876b3a9d2a202fc1e b98d2d722f81b2043a4458adebfbe45e239d1f79 c1e80bf4ad3e91dd21a8ccdee129052e25ace7e6 9625bf39bd4140cebd60cc658846505f4ab6c8ca 71753b8ec103fd71d6ee90e522d797ccf978e4ed 3a9e5fff2ab0d6f4af701757d35b9453dc563b78 3c87aedd48997f9ab3afdeb2084f6b6163a21db8 ed5f9cf06b20f74c1098d6d62313e3e9af217fcb 62f7991feab69ed398ad6a03b911fa0ee0549516 3ab45777a27c5f53a55a2c353b70648449ca7a33 6b23f6ab86a4f3a608b9bfa98616e99a34bb650c 8bf30be75069d6080659de9a28565c048f6cef9b 17c84cb46e33fcd682d5d5d6180162374cb24a34 6ce204eac38723d3a24c7bf148221de933bba54d d08c6e2a4d0308a7922d7ef3b1b3af45d4096aad befa41771f9e07f0a95b4a4a0d6b77fd334f4e44 76c31e5f758596509fbab120b8d055bf927ed165 c27bd1718c06eb356a4ea0ab8588ad442d953947 9536923d3f353873e54d96769fbd2e7f867a73e1 60dc0ef674ecbc944e2c236c8a71ce26440cb1a9 8348b71ccd921d0bff21b6f480ecc1e3a5033359 a572c0a748e6f7e04fb56d65139edbece4cb113d 73a3f1bcab1e66206c2de4bf11083fb8ddd5f49c 84ba8062e383c4a4f8f6388d1cfc87328f2a6115 e5d4e1da65564bb8631ecf5f08f0b2960fb0bc78 efe6f961cd2e54aca4385c7fee1c997f7c286c73 eeed226ed110ed40598e60e29b66643012277be7 3b49a7edec1df4f7019b3b872124231197267205 ff99316700799b84e842f819a44db608557bae3e 8be9686d2479a12f986dfce2f792e33ab747ac0f 918ed7bf76267e43863205e0d61a949c301f367b 09bf97923224fea7f3633648b64867a934d627f2 df67ad625b9ef1c7400f3573df8ac36ab9f56260 c118ebc98233eb1383f1adfeea4fa8e59f635088 39542e234b5215dc267b425f8315747849c5e520 There appears to have been a major refactor of drivers/net/ethernet/mellanox/mlx5/core/en_tc.c going on for the past several kernel releases. It is still not known if the fix is at the tip of that refactor or if it will be isolated and easily backportable. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4
