[Kernel-packages] [Bug 1999229] Re: mlx5 VF LAG flapping
I dont see the issue after updating Ubuntu 22.04 with kernel 5.15 to 5.19.0-28-generic. HW offload seems to work as expected. Itai -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1999229 Title: mlx5 VF LAG flapping Status in linux package in Ubuntu: Confirmed Bug description: # sudo lsb_release -a No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy # mlxfwmanager Querying Mellanox devices firmware ... Device #1: -- Device Type: ConnectX6DX Part Number: MCX623106AN-CDA_Ax Description: ConnectX-6 Dx EN adapter card; 100GbE; Dual-port QSFP56; PCIe 4.0/3.0 x16; PSID: MT_000359 PCI Device Name: :41:00.0 Base GUID:08c0eb03006fb26e Base MAC: 08c0eb6fb26e Versions: CurrentAvailable FW 22.34.4000 N/A PXE3.6.0700 N/A UEFI 14.27.0015 N/A # uname -a Linux ps6-ra1-n2 5.19.0-24-generic #25~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Fri Nov 18 14:28:08 UTC 2 x86_64 x86_64 x86_64 GNU/Linux Kernel from linux-generic-hwe-22.04-edge package in jammy-proposed, see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Problem: Severe packet loss to high speed NIC due to what appears as VF LAG flapping: [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.0: mlx5_cmd_out_err:778:(pid 3383): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad resource(0x5), syndrome (0xf2ff71), err(-22) [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.0: E-Switch: Failed to create termination table rule, err -EINVAL [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.0: E-Switch: Failed to get termination table, err -EINVAL [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.1: mlx5_cmd_out_err:778:(pid 3383): SET_FLOW_TABLE_ENTRY(0x936) op_mod(0x0) failed, status bad resource(0x5), syndrome (0xf2ff71), err(-22) [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.1: E-Switch: Failed to create termination table rule, err -EINVAL [Fri Dec 9 07:27:19 2022] mlx5_core :41:00.1: E-Switch: Failed to get termination table, err -EINVAL [Fri Dec 9 07:27:20 2022] mlx5_core :41:00.0: lag map active ports: 1, 2 [Fri Dec 9 07:27:20 2022] mlx5_core :41:00.0: lag map active ports: 2 [Fri Dec 9 07:27:20 2022] mlx5_core :41:00.0: lag map active ports: 1, 2 [Fri Dec 9 07:27:21 2022] mlx5_core :41:00.0: lag map active ports: 2 [Fri Dec 9 07:27:21 2022] mlx5_core :41:00.0: lag map active ports: 1, 2 [Fri Dec 9 07:27:22 2022] mlx5_core :41:00.0: lag map active ports: 2 [Fri Dec 9 07:27:23 2022] mlx5_core :41:00.0: lag map active ports: 1, 2 [Fri Dec 9 07:27:23 2022] mlx5_core :41:00.0: lag map active ports: 2 This does not happen when using the Jammy 5.15 kernel, everything else in the environment being equal. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1999229/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
Frode, After investigating the issue in our lab, it looks like the used kernel is missing with the following patch: https://lore.kernel.org/netdev/20220504070256.694458-9-sae...@nvidia.com/ The problem should be solved with this patch, please pull it into your 22.04 5.15 kernel. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr 4081686770], length 0 07:16:59.171956 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad,
[Kernel-packages] [Bug 1987618] Re: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways
Investigating this issue with Frode, we saw that its is related to the Security Group applied to the HW Offload (switchdev) capable ports. If an "allow all traffic" security group is attached to the port, the issue is not seen. We see the issue when applying a security group with some explicit rules to allow specific traffic and an implicit DROP rule. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1987618 Title: [jammy mlx5] Slow outbound N/S TCP connections for instances on gateways Status in linux package in Ubuntu: Incomplete Bug description: Charmed OpenStack on Jammy. Hardware offloaded instances that reside on the hypervisor which is also the active OVN gateway for their network have degraded connectivity. $ time wget --debug -O /dev/null http://nova.clouds.archive.ubuntu.com/ubuntu/ Setting --output-document (outputdocument) to /dev/null DEBUG output created by Wget 1.21.2 on linux-gnu. Reading HSTS entries from /home/ubuntu/.wget-hsts URI encoding = ‘UTF-8’ --2022-08-25 07:16:54-- http://nova.clouds.archive.ubuntu.com/ubuntu/ Resolving nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)... 91.189.91.121, 185.125.190.38, 185.125.190.41, ... Caching nova.clouds.archive.ubuntu.com => 91.189.91.121 185.125.190.38 185.125.190.41 91.189.91.122 2001:67c:1562::29 2620:2d:4000:1::1b 2620:2d:4000:1::18 2001:67c:1562::26 Connecting to nova.clouds.archive.ubuntu.com (nova.clouds.archive.ubuntu.com)|91.189.91.121|:80... connected. Created socket 4. Releasing 0x55ef05654490 (new refcount 1). ---request begin--- GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Date: Thu, 25 Aug 2022 07:16:54 GMT Server: Apache/2.4.29 (Ubuntu) Vary: Accept-Encoding Content-Length: 1779 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html;charset=UTF-8 ---response end--- 200 OK Registered socket 4 for persistent reuse. URI content encoding = ‘UTF-8’ Length: 1779 (1.7K) [text/html] Saving to: ‘/dev/null’ /dev/null 100%[>] 1.74K --.-KB/sin 0.07s 2022-08-25 07:16:59 (24.3 KB/s) - ‘/dev/null’ saved [1779/1779] real 0m5.591s user 0m0.000s sys 0m0.005s $ sudo tcpdump -nevi ens3 not port 22 07:16:54.017265 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 64, id 50207, offset 0, flags [DF], proto TCP (6), length 60) 10.42.1.8.43060 > 91.189.91.121.80: Flags [S], cksum 0xc296 (incorrect -> 0x4190), seq 2905559525, win 62314, options [mss 8902,sackOK,TS val 4081686695 ecr 0,nop,wscale 7], length 0 07:16:54.091378 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 58, id 0, offset 0, flags [DF], proto TCP (6), length 60) 91.189.91.121.80 > 10.42.1.8.43060: Flags [S.], cksum 0x809d (correct), seq 3404088064, ack 2905559526, win 65160, options [mss 1460,sackOK,TS val 932285789 ecr 4081686695,nop,wscale 7], length 0 07:16:54.091402 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 64, id 50208, offset 0, flags [DF], proto TCP (6), length 52) 10.42.1.8.43060 > 91.189.91.121.80: Flags [.], cksum 0xc28e (incorrect -> 0xabc0), ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 0 07:16:54.091512 fa:16:3e:6d:94:75 > fa:16:3e:47:3a:ad, ethertype IPv4 (0x0800), length 218: (tos 0x0, ttl 64, id 50209, offset 0, flags [DF], proto TCP (6), length 204) 10.42.1.8.43060 > 91.189.91.121.80: Flags [P.], cksum 0xc326 (incorrect -> 0x7d0c), seq 1:153, ack 1, win 487, options [nop,nop,TS val 4081686770 ecr 932285789], length 152: HTTP, length: 152 GET /ubuntu/ HTTP/1.1 Host: nova.clouds.archive.ubuntu.com User-Agent: Wget/1.21.2 Accept: */* Accept-Encoding: identity Connection: Keep-Alive 07:16:54.166170 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15660, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [.], cksum 0xaac7 (correct), ack 153, win 508, options [nop,nop,TS val 932285865 ecr 4081686770], length 0 07:16:59.171924 fa:16:3e:47:3a:ad > fa:16:3e:6d:94:75, ethertype IPv4 (0x0800), length 66: (tos 0x0, ttl 58, id 15669, offset 0, flags [DF], proto TCP (6), length 52) 91.189.91.121.80 > 10.42.1.8.43060: Flags [F.], cksum 0x8f61 (correct), seq 2008, ack 153, win 508, options [nop,nop,TS val 932290871 ecr
[Kernel-packages] [Bug 1978820] Re: Jammy Charmed OpenStack deployment fails over connectivity issues when using converged OVS bridge for control and data planes
I can verify the fix is working with the released 5.15.0-43.46 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1978820 Title: Jammy Charmed OpenStack deployment fails over connectivity issues when using converged OVS bridge for control and data planes Status in linux package in Ubuntu: Confirmed Status in linux source package in Jammy: Fix Released Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic Charmed Openstack deployment with HW Offload over Jammy series will look ok until Vault initialization phase, then after initializing Vault all DB-related apps will end up in block/error state over "Failed to connect to MYSQL". Connectivity testing between DB containers located on different nodes will show there is unexplained sporadic packet loss preventing proper communication between the DB related apps. This will happen when the following conditions are met: 1. Control plane (oam, internal spaces) is configured as vlan interfaces on the same OVS bridge used for data plane (over high speed NIC with HW Offload capabilities). 2. OVS was set with HW offload=true (will happen by OVN chrams after Vault initialization) 3. NIC was not yet set to "switchdev" mode (netplan file will be created by OVN chrams after Vault initialization, however will take affect only after node is rebooted) The root cause is the following missing kernel patch: https://patchwork.kernel.org/project/netdevbpf/patch/20220531205447.99236-3-sa...@kernel.org/ To reproduce: Deploy charmed openstack with HW offload while using control plane on the high speed NIC OVS bridge. Before initializing Vault login to one of the innoDB instances and ping the other 2 instances - all ok. Manually enable OVS HW Offload, ping will become inconsistent. Workaround: After the deployment bring-up phase, BEFORE enabling Vault, login to the nodes and manually create 150-charm-ovn.yaml (example below). Then reboot one node after another. When nodes recover proceed with Vault initialization to complete the deployment. #root@node3:/home/ubuntu# cat /etc/netplan/150-charm-ovn.yaml ### # [ WARNING ] # Configuration file maintained by Juju. Local changes may be overwritten. # Config managed by ovn-chassis charm ### network: version: 2 ethernets: ens1f0: virtual-function-count: 8 embedded-switch-mode: switchdev delay-virtual-functions-rebind: true ens1f1: virtual-function-count: 8 embedded-switch-mode: switchdev delay-virtual-functions-rebind: true --- ProblemType: Bug AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 4 10:46 seq crw-rw 1 root audio 116, 33 Jul 4 10:46 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 22.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: HP ProLiant DL360 Gen9 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-40-generic root=UUID=db1801a9-daa1-4386-b2ec-c65a40bc5dd3 ro intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 ProcVersionSignature: Ubuntu 5.15.0-40.43-generic 5.15.35 RelatedPackageVersions: linux-restricted-modules-5.15.0-40-generic N/A linux-backports-modules-5.15.0-40-generic N/A linux-firmware 20220329.git681281e4-0ubuntu3.2 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: jammy uec-images Uname: Linux 5.15.0-40-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 03/05/2015 dmi.bios.release: 1.32 dmi.bios.vendor: HP dmi.bios.version: P89 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.ec.firmware.release: 2.53 dmi.modalias: dmi:bvnHP:bvrP89:bd03/05/2015:br1.32:efr2.53:svnHP:pnProLiantDL360Gen9:pvr:cvnHP:ct23:cvr:sku755258-B21: dmi.product.family: ProLiant dmi.product.name: ProLiant DL360 Gen9 dmi.product.sku: 755258-B21 dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978820/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe :
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
Hi Zack, This cannot be considered as regression testing, I did specific testing for specific (important) use cases. Itai -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
Verified with the kernel 5.15.0-37.39+lp1980730-1-generic you provided. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
Hi Zack, I can confirm the HW offload is working properly for the "ovn internal port" use cases! Thanks Itai -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
it got the raid config. the problem is that the raid kernel module is part of the modules-extra pkg and not the modules pkg. once I installed the modules-extra pkg the initramfs image was regenerated with the relevant module and the server was able to boot up. thanks again. Itai -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
it was a missing raid controller module. it boots up ok after I installed the modules-extra pkgs. Thanks. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1980730] Re: OVS internal port Hardware Offload implementation is missing in Jammy kernel
Hi Zack, I tried to install this kernel pkgs (headers, image, modules), however when trying to boot with this kernel it fails over a failure to detect root file system (see screenshot). When I revert back to the default kernel it boots. What did I miss? ** Attachment added: "kernel_rootfs.PNG" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+attachment/5605217/+files/kernel_rootfs.PNG -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1980730 Title: OVS internal port Hardware Offload implementation is missing in Jammy kernel Status in linux package in Ubuntu: Incomplete Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic When deploying Charmed Openstack deployment with HW Offload over Jammy series, some of the traffic workloads (mainly those related to Floating IP and NAT) will not be properly offloaded. In order to solve it, "OVS internal port offload" feature was introduced and pushed to upstream kernel. These are the main base patches for the feature we call "OVS internal port offload": b16eb3c81fe27978afdb2c111908d4d627a88d99 net/mlx5: Support internal port as decap route device 5e9942721749fc96b9df4b0545474153316c0571 net/mlx5e: Term table handling of internal port rules 166f431ec6beaf472bc2e116a202a127b64779e4 net/mlx5e: Add indirect tc offload of ovs internal port 100ad4e2d75837c9b42f49b3814b4b42ec9ebe46 net/mlx5e: Offload internal port as encap route device 27484f7170edabbda7b53650cd24d38295cffe60 net/mlx5e: Offload tc rules that redirect to ovs internal port dbac71f22954276633e525f958994f84a7bd303f net/mlx5e: Accept action skbedit in the tc actions list 4f4edcc2b84fecec66748ecbb90a84b981ecdaae net/mlx5: E-Switch, Add ovs internal port mapping to metadata support 189ce08ebf876df2b51f625877731055475352df net/mlx5e: Use generic name for the forwarding dev pointer 28e7606fa8f106cdc0355e0548396c037443e063 net/mlx5e: Refactor rx handler of represetor device Please make sure to take as well all patches the came later on top of those base patches To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1980730/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1978820] Re: Jammy Charmed OpenStack deployment fails over connectivity issues when using converged OVS bridge for control and data planes
** Tags removed: apport-collected jammy uec-images verification-needed-jammy ** Tags added: verification-done-jammy -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1978820 Title: Jammy Charmed OpenStack deployment fails over connectivity issues when using converged OVS bridge for control and data planes Status in linux package in Ubuntu: Confirmed Status in linux source package in Jammy: Fix Committed Bug description: Platform: OpenStack Yoga, Ubuntu 22.04 Jammy, Kernel 5.15.0-37-generic Charmed Openstack deployment with HW Offload over Jammy series will look ok until Vault initialization phase, then after initializing Vault all DB-related apps will end up in block/error state over "Failed to connect to MYSQL". Connectivity testing between DB containers located on different nodes will show there is unexplained sporadic packet loss preventing proper communication between the DB related apps. This will happen when the following conditions are met: 1. Control plane (oam, internal spaces) is configured as vlan interfaces on the same OVS bridge used for data plane (over high speed NIC with HW Offload capabilities). 2. OVS was set with HW offload=true (will happen by OVN chrams after Vault initialization) 3. NIC was not yet set to "switchdev" mode (netplan file will be created by OVN chrams after Vault initialization, however will take affect only after node is rebooted) The root cause is the following missing kernel patch: https://patchwork.kernel.org/project/netdevbpf/patch/20220531205447.99236-3-sa...@kernel.org/ To reproduce: Deploy charmed openstack with HW offload while using control plane on the high speed NIC OVS bridge. Before initializing Vault login to one of the innoDB instances and ping the other 2 instances - all ok. Manually enable OVS HW Offload, ping will become inconsistent. Workaround: After the deployment bring-up phase, BEFORE enabling Vault, login to the nodes and manually create 150-charm-ovn.yaml (example below). Then reboot one node after another. When nodes recover proceed with Vault initialization to complete the deployment. #root@node3:/home/ubuntu# cat /etc/netplan/150-charm-ovn.yaml ### # [ WARNING ] # Configuration file maintained by Juju. Local changes may be overwritten. # Config managed by ovn-chassis charm ### network: version: 2 ethernets: ens1f0: virtual-function-count: 8 embedded-switch-mode: switchdev delay-virtual-functions-rebind: true ens1f1: virtual-function-count: 8 embedded-switch-mode: switchdev delay-virtual-functions-rebind: true --- ProblemType: Bug AlsaDevices: total 0 crw-rw 1 root audio 116, 1 Jul 4 10:46 seq crw-rw 1 root audio 116, 33 Jul 4 10:46 timer AplayDevices: Error: [Errno 2] No such file or directory: 'aplay' ApportVersion: 2.20.11-0ubuntu82.1 Architecture: amd64 ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord' CRDA: N/A CasperMD5CheckResult: unknown DistroRelease: Ubuntu 22.04 IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig' MachineType: HP ProLiant DL360 Gen9 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl icp Package: linux (not installed) PciMultimedia: ProcEnviron: TERM=xterm-256color PATH=(custom, no user) LANG=C.UTF-8 SHELL=/bin/bash ProcFB: 0 mgag200drmfb ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-40-generic root=UUID=db1801a9-daa1-4386-b2ec-c65a40bc5dd3 ro intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 ProcVersionSignature: Ubuntu 5.15.0-40.43-generic 5.15.35 RelatedPackageVersions: linux-restricted-modules-5.15.0-40-generic N/A linux-backports-modules-5.15.0-40-generic N/A linux-firmware 20220329.git681281e4-0ubuntu3.2 RfKill: Error: [Errno 2] No such file or directory: 'rfkill' Tags: jammy uec-images Uname: Linux 5.15.0-40-generic x86_64 UpgradeStatus: No upgrade log present (probably fresh install) UserGroups: N/A _MarkForUpload: True dmi.bios.date: 03/05/2015 dmi.bios.release: 1.32 dmi.bios.vendor: HP dmi.bios.version: P89 dmi.chassis.type: 23 dmi.chassis.vendor: HP dmi.ec.firmware.release: 2.53 dmi.modalias: dmi:bvnHP:bvrP89:bd03/05/2015:br1.32:efr2.53:svnHP:pnProLiantDL360Gen9:pvr:cvnHP:ct23:cvr:sku755258-B21: dmi.product.family: ProLiant dmi.product.name: ProLiant DL360 Gen9 dmi.product.sku: 755258-B21 dmi.sys.vendor: HP To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1978820/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to :
