[Kernel-packages] [Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
** Changed in: linux (Ubuntu Focal) Status: New => In Progress ** Changed in: linux (Ubuntu Groovy) Status: New => In Progress ** Changed in: linux (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Focal) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: linux (Ubuntu Groovy) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Summary changed: - Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure + qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP tx csum offload -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: qede: Kubernetes Internal DNS Failure due to QL41xxx NIC not supporting IPIP tx csum offload Status in linux package in Ubuntu: Confirmed Status in linux source package in Focal: In Progress Status in linux source package in Groovy: In Progress Bug description: With QL41xxx and Ubuntu DNS server DNS failures are seen when updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Inbox driver qede v8.37.0.20 Complete Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Similar issue is described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
** Also affects: linux (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Focal) Importance: Undecided Status: New -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure Status in linux package in Ubuntu: Confirmed Status in linux source package in Focal: New Status in linux source package in Groovy: New Bug description: With QL41xxx and Ubuntu DNS server DNS failures are seen when updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Inbox driver qede v8.37.0.20 Complete Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Similar issue is described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
** Description changed: - Customer is reporting a problem with QL41xxx and Ubuntu internal DNS - server. The issue appeared when the customer updated to the latest - Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not - observed with 4.5 ubuntu-linux. + With QL41xxx and Ubuntu DNS server DNS failures are seen when updated to + the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was + not observed with 4.5 ubuntu-linux. Problem Definition: - - Product: PowerEdge R740xd - Serial: C7J90W2 - Hostname: xkubmin1r12 OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) - Firmware: 15.15.11 Inbox driver qede v8.37.0.20 - - Completed Detailed Problem Description: + Complete Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. - Customer is experiencing the same issue described in this article. + Similar issue is described in this article. https://github.com/kubernetes/kubernetes/issues/95365 - - Customer Impact: Production site - The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment. Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 - qede: fix offload for IPIP tunnel packets + qede: fix offload for IPIP tunnel packets - IPIP tunnels packets are unknown to device, - hence these packets are incorrectly parsed and - caused the packet corruption, so disable offlods - for such packets at run time. + IPIP tunnels packets are unknown to device, + hence these packets are incorrectly parsed and + caused the packet corruption, so disable offlods + for such packets at run time. - Signed-off-by: Manish Chopra - Signed-off-by: Sudarsana Kalluru - Signed-off-by: Igor Russkikh - Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com - Signed-off-by: Jakub Kicinski - + Signed-off-by: Manish Chopra + Signed-off-by: Sudarsana Kalluru + Signed-off-by: Igor Russkikh + Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com + Signed-off-by: Jakub Kicinski Thanks, Manish -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure Status in linux package in Ubuntu: Confirmed Bug description: With QL41xxx and Ubuntu DNS server DNS failures are seen when updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Inbox driver qede v8.37.0.20 Complete Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Similar issue is described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
We don't have any logs to post, attached wire traces which we got from customer as we were only interested in that. Having said that, this bug is resolved with the above fix posted upstream, I guess we don't need any further logs. ** Attachment added: "wire_traces.zip" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+attachment/5446099/+files/wire_traces.zip ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure Status in linux package in Ubuntu: Confirmed Bug description: Customer is reporting a problem with QL41xxx and Ubuntu internal DNS server. The issue appeared when the customer updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: Product: PowerEdge R740xd Serial: C7J90W2 Hostname: xkubmin1r12 OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Firmware: 15.15.11 Inbox driver qede v8.37.0.20 Completed Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Customer is experiencing the same issue described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Customer Impact: Production site The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment. Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1909062] Re: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure
** Package changed: ubuntu => linux (Ubuntu) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1909062 Title: Ubuntu kernel 5.x QL41xxx NIC (qede driver) Kubernetes internal DNS failure Status in linux package in Ubuntu: Incomplete Bug description: Customer is reporting a problem with QL41xxx and Ubuntu internal DNS server. The issue appeared when the customer updated to the latest Ubuntu kernel 20.04.1 LTS version 5.4.0-52-generic. Issue was not observed with 4.5 ubuntu-linux. Problem Definition: Product: PowerEdge R740xd Serial: C7J90W2 Hostname: xkubmin1r12 OS Version: /etc/os-release shows Ubuntu 18.04.4 LTS, but Booted kernel is the latest Ubuntu 20.04.1 LTS version 5.4.0-52-generic NIC: 2 dual-port (4) QLogic Corp. FastLinQ QL41000 Series 10/25/40/50GbE Controller [1077:8070] (rev 02) Firmware: 15.15.11 Inbox driver qede v8.37.0.20 Completed Detailed Problem Description: Anything that uses the internal Kubernetes DNS server fails. If an external DNS server is used resolution works for non-Kubernetes IPs. Customer is experiencing the same issue described in this article. https://github.com/kubernetes/kubernetes/issues/95365 Customer Impact: Production site The QLogic Nic 41262 is their main nic for all of their 14G environment thousands of servers. Unclear how many of those hosts are Kubernetes, but the point is they want the QL41000 to work since it is very prevalent in the entire environment. Below patch recently on upstream fixes this - [Note that issue was introduced by driver's tunnel offload support which was added in after 4.5 kernel] commit 5d5647dad259bb416fd5d3d87012760386d97530 Author: Manish Chopra Date: Mon Dec 21 06:55:30 2020 -0800 qede: fix offload for IPIP tunnel packets IPIP tunnels packets are unknown to device, hence these packets are incorrectly parsed and caused the packet corruption, so disable offlods for such packets at run time. Signed-off-by: Manish Chopra Signed-off-by: Sudarsana Kalluru Signed-off-by: Igor Russkikh Link: https://lore.kernel.org/r/20201221145530.7771-1-mani...@marvell.com Signed-off-by: Jakub Kicinski Thanks, Manish To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1909062/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
