[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi Jens, I highly recommend you go through the pain to upgrade the kernel on your GPU cluster to something modern, like 4.15.0-91-generic. There was quite a few regressions around the 4.15.0-56 to 4.15.0-58 mark, as we merged a lot of upstream stable patches in at that time. 4.15.0-91 is pretty stable these days, and you can probably leave it long term on that kernel. In this bug, the fix landed in the mlx5_core driver, which is a kernel module. Kernel modules are only compatible with the kernel that they were compiled for, since Linux does not have a stable ABI / binary interface. So, this isn't as easy as just copying over a fixed kernel module. The kmod package doesn't actually have any kernel modules in it, just the blacklists and things defined in /etc/modules-load.d and /etc/modprobe.d Nvidia drivers should be built with dkms, and *should* work without too much hassle. I know that theory doesn't always align with reality though. Anyway, I recommend you upgrade to a newer kernel on your GPU cluster. Thanks, Matthew -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi Jeff, hmm, didn't get notified by launchpad about your answer :(((. Anyway, tried another machine with 4.15.0-91-generic and indeed, it seems to be fixed. Now the problem is, that our GPU machines are running 4.15.0-58-generic and cannot be upgraded because all the nvidia stuff is very picky and we do not have the time to upgrade the cluster to a new kernel version. Therefore the question: is this just a driver problem? Copying over the kmod from a 4.15.0-91 isn't probably a problem, I guess. Or rebuilding the kmod for 4.15.0-58 may work out of the box as well... -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase] Simply try and bring an interface up on a machine with Mellanox series 5 NICs. When a packet comes through which is smaller
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi Jens, As the fix was landed in 4.15.0-59, I would expect that you would likely still see issues in 4.15.0-58. The current Bionic GA kernel is 4.15.0.91.83 in -updates. You should try an updated kernel and see if that resolves the issue. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase] Simply try and bring an interface up on a machine with Mellanox series 5 NICs. When a packet comes through which is smaller than required and padding is added, the problem will be triggered. The 4.15.0-59 from -proposed has been tested by the customer, and resolves the issue. [Regression Potential] This patch has a low chance of regression since it fixes a regression introduced by 88078d98d1bb085d72af8437707279e203524fa5 in 4.15.0-56.
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
We use 'Linux kino6 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux' (Ubuntu 18.04.3 LTS) and see all the time 'hw csum failure's: [ +28.297139] kino6_0: hw csum failure [ +0.003607] CPU: 12 PID: 0 Comm: swapper/12 Tainted: P O 4.15.0-58-generic #64-Ubuntu [ +0.03] Hardware name: GIGABYTE G291-281-00/MG51-G21-00, BIOS R06 11/19/2019 [ +0.01] Call Trace: [ +0.02] [ +0.11] dump_stack+0x63/0x8b [ +0.08] netdev_rx_csum_fault+0x38/0x40 [ +0.03] __skb_checksum_complete+0xbc/0xd0 [ +0.05] nf_ip_checksum+0xc3/0xf0 [ +0.18] tcp_error+0x162/0x1c0 [nf_conntrack] [ +0.06] ? ttwu_do_wakeup+0x1e/0x140 [ +0.11] nf_conntrack_in+0x14f/0x500 [nf_conntrack] [ +0.07] ? csum_partial_ext+0x9/0x10 [ +0.07] ? __skb_checksum+0x6b/0x300 [ +0.06] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] [ +0.05] nf_hook_slow+0x48/0xc0 [ +0.04] ? skb_send_sock+0x50/0x50 [ +0.05] ip_rcv+0x2fa/0x360 [ +0.03] ? inet_del_offload+0x40/0x40 [ +0.04] __netif_receive_skb_core+0x432/0xb40 [ +0.03] ? update_curr+0xf2/0x1d0 [ +0.04] ? tcp4_gro_receive+0x137/0x1a0 [ +0.03] __netif_receive_skb+0x18/0x60 [ +0.03] ? __netif_receive_skb+0x18/0x60 [ +0.03] netif_receive_skb_internal+0x45/0xe0 [ +0.04] napi_gro_receive+0xc5/0xf0 [ +0.36] mlx5e_handle_rx_cqe_mpwrq+0x465/0x860 [mlx5_core] [ +0.28] mlx5e_poll_rx_cq+0xd1/0x8b0 [mlx5_core] [ +0.25] mlx5e_napi_poll+0x9d/0x290 [mlx5_core] [ +0.04] net_rx_action+0x140/0x3a0 [ +0.05] __do_softirq+0xe4/0x2d4 [ +0.06] irq_exit+0xc5/0xd0 [ +0.03] do_IRQ+0x8a/0xe0 [ +0.03] common_interrupt+0x8c/0x8c [ +0.02] [ +0.05] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0 [ +0.02] RSP: 0018:ad7a00283e68 EFLAGS: 0246 ORIG_RAX: ffdd [ +0.04] RAX: 89f13fd22840 RBX: 02273da74e91 RCX: 001f [ +0.02] RDX: 02273da74e91 RSI: feba65558937 RDI: [ +0.02] RBP: ad7a00283ea8 R08: 0004 R09: 00022080 [ +0.01] R10: ad7a00283e38 R11: 07e0dcda6658 R12: cd5a00503298 [ +0.02] R13: 0003 R14: b3f72e78 R15: [ +0.04] ? cpuidle_enter_state+0x97/0x2f0 [ +0.03] cpuidle_enter+0x17/0x20 [ +0.05] call_cpuidle+0x23/0x40 [ +0.04] do_idle+0x18c/0x1f0 [ +0.04] cpu_startup_entry+0x73/0x80 [ +0.05] start_secondary+0x1ab/0x200 [ +0.05] secondary_startup_64+0xa5/0xb0 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi @mohamadh, Are you using the in-tree drivers that Ubuntu supplies? I have seen some customers using out of tree MOFED drivers from the Mellanox website that have been out of date, and being able to reproduce this on newer kernels like 4.15.0-66. Upon updating to newer MOFED drivers or reverting to the normal Ubuntu supplied driver seemed to fix things for them. If you are using the normal Ubuntu kernel with inbuilt drivers, can you open a new bug report and include your /var/log/kern.log with the Call trace? We can't simply ship a set of upstream patches untested, and every patch needs to fix a specific problem, so we will work with you to find the patch that fixes things. Thanks, Matthew -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase]
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi, I see that the issue still reproduced on newer kernels > 4.15.0-69, to fix the issue should get all the following upstream patches: net/mlx5e: Rx, Fix checksum calculation for new hardware --> db849faa9bef993a1379dc510623f750a72fa7ce net/mlx5e: Rx, Check ip headers sanity - > 0318a7b7fcad9765931146efa7ca3a034194737c net/mlx5e: Rx, Fixup skb checksum for packets with tail padding --> 0aa1d18615c163f92935b806dcaff9157645233a net/mlx5e: XDP, Avoid checksum complete when XDP prog is loaded --> 5d0bb3bac4b9f6c22280b04545626fdfd99edc6b mlx5: fix get_ip_proto() --> ef6fcd455278c2be3032a346cc66d9dd9866b787 net/mlx5e: Allow reporting of checksum unnecessary --> b856df28f9230a47669efbdd57896084caadb2b3 net/mlx5e: don't set CHECKSUM_COMPLETE on SCTP packets --> fe1dc069990c1f290ef6b99adb46332c03258f38 net/mlx5e: Set ECN for received packets using CQE indication --> f007c13d4ad62f494c83897eda96437005df4a91 net/mlx5e: Add likely to the common RX checksum flow --> 63a612f984a1fae040ab6f1c6a0f1fdcdf1954b8 net/mlx5e: CHECKSUM_COMPLETE offload for VLAN/QinQ packets --> f938daeee95eb36ef6b431bf054a5cc6cdada112 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is:
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Patches could be found in Eoan as well, close it with Fix-released. ** Changed in: linux (Ubuntu) Status: Incomplete => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Fix Released Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase] Simply try and bring an interface up on a machine with Mellanox series 5 NICs. When a packet comes through which is smaller than required and padding is added, the problem will be triggered. The 4.15.0-59 from -proposed has been tested by the customer, and resolves the issue. [Regression Potential] This patch has a low chance of regression since it fixes a regression introduced by 88078d98d1bb085d72af8437707279e203524fa5 in 4.15.0-56. The changes are limited to mlx5_core driver, and have been well tested and accepted by the community
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
Hi Matthew, I was going through some kernel bugs and it looks this one has already been released, but just didn't get updated automatically as its LP number is not directly mentioned in the changelog. Marking it as Fix Released. Hope this helps! cheers, Mauricio ** Changed in: linux (Ubuntu Bionic) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: Incomplete Status in linux source package in Bionic: Fix Released Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase] Simply try and bring an interface up on a machine with Mellanox series 5 NICs. When a packet comes through which is smaller than required and padding is added, the problem will be triggered. The 4.15.0-59 from -proposed has been tested by the customer, and resolves the issue. [Regression Potential] This patch has a low chance of
[Kernel-packages] [Bug 1840854] Re: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs
The customer installed 4.15.0-59 from -proposed to a machine with Mellanox Ethernet CX4LX cards, using the mlx5_core kernel module. Checksums are now calculated correctly and the kernel spat does not occur when the devices are brought up. Marking this as verified. ** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1840854 Title: mlx5_core reports hardware checksum error for padded packets on Mellanox NICs Status in linux package in Ubuntu: New Status in linux source package in Bionic: Fix Committed Bug description: BugLink: https://bugs.launchpad.net/bugs/1840854 [Impact] On machines equipped with Mellanox NIC's, in this particular case, Mellanox 5 series NICs using the mlx5_core driver, after installing 4.15.0-56 or later there is the following kernel splat: bond0: hw csum failure CPU: 63 PID: 2473 Comm: in:imklog Tainted: P OE 4.15.0-58-generic #64~16.04.1-Ubuntu Call Trace: dump_stack+0x63/0x8b netdev_rx_csum_fault+0x38/0x40 __skb_checksum_complete+0xc0/0xd0 nf_ip_checksum+0xca/0xf0 tcp_error+0xe0/0x1a0 [nf_conntrack] ? tcp_v4_rcv+0x7c6/0xa70 nf_conntrack_in+0xde/0x520 [nf_conntrack] ipv4_conntrack_in+0x1c/0x20 [nf_conntrack_ipv4] nf_hook_slow+0x48/0xd0 ? skb_send_sock+0x50/0x50 ip_rcv+0x30f/0x370 ? inet_del_offload+0x40/0x40 __netif_receive_skb_core+0x879/0xba0 ? tcp4_gro_receive+0x117/0x1b0 __netif_receive_skb+0x18/0x60 ? __netif_receive_skb+0x18/0x60 netif_receive_skb_internal+0x45/0xf0 napi_gro_receive+0xd0/0xf0 mlx5e_handle_rx_cqe_mpwrq+0x4a1/0x8a0 [mlx5_core] mlx5e_poll_rx_cq+0xc3/0x880 [mlx5_core] mlx5e_napi_poll+0x9b/0x280 [mlx5_core] net_rx_action+0x265/0x3b0 __do_softirq+0xf5/0x2a8 irq_exit+0xca/0xd0 do_IRQ+0x57/0xe0 common_interrupt+0x8c/0x8c In 4.15.0-56, a commit was added from upstream -stable that introduced an optimisation for checksumming packets which have had zero bytes padded to the end of the packet. commit 88078d98d1bb085d72af8437707279e203524fa5 Author: Eric Dumazet Date: Wed Apr 18 11:43:15 2018 -0700 subject: net: pskb_trim_rcsum() and CHECKSUM_COMPLETE are friends You can read it here: https://github.com/torvalds/linux/commit/88078d98d1bb085d72af8437707279e203524fa5 It was discussed in this bugzilla link: https://bugzilla.kernel.org/show_bug.cgi?id=201849 This commit causes problems with a number of NIC devices, including Mellanox. This is best described by the maintainer, Dimitris Michailidis: > > > MLNX devices have an issue with packets that are padded past the end of > > > the L3 payload with bytes that aren't all 0s. They use a mode of checksum > > > reporting which should be including the padding bytes but MLNX devices > > > leave those out. When the padding bytes aren't all 0 this omission causes > > > a checksum error. This device behavior has existed for a long time but it > > > has begun causing errors only this year. Before a padded packet had its > HW > > > checksum ignored so it wasn't material what HW had reported. More > recently > > > padded packet checksums started using the HW value and now it is > > > noticeable when that value isn't right. Now, some routers stick additional information in the zero padding section on occasion, which will change the hardware checksum. Since the hardware checksum was ignored until 4.15.0-56 with 88078d98d1bb085d72af8437707279e203524fa5, this wasn't an issue. But with the optimisation, we start running into trouble since the hardware checksums no longer match what the kernel is expecting. [Fix] This was fixed for Mellanox 4 and 5 series drivers recently. Mellanox 4: 74abc07dee613086f9c0ded9e263ddc959a6de04 https://github.com/torvalds/linux/commit/74abc07dee613086f9c0ded9e263ddc959a6de04 Mellanox 5: e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a https://github.com/torvalds/linux/commit/e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a This customer hit the issue with mlx5_core driver, so the fix is: commit e8c8b53ccaff568fef4c13a6ccaf08bf241aa01a Author: Cong Wang Date: Mon Dec 3 22:14:04 2018 -0800 subject: net/mlx5e: Force CHECKSUM_UNNECESSARY for short ethernet frames This is actually present in 4.15.0-59, which is currently sitting in -proposed. The commits are a part of 4.9.156, 4.14.99, 4.19.21 upstream -stable releases, and have been pulled into bionic as a part of LP #1837664 [Testcase] Simply try and bring an interface up on a machine with Mellanox series 5 NICs. When a packet comes through which is smaller than required and padding is added, the problem will be triggered. The 4.15.0-59 from -proposed has been tested by the customer, and resolves the issue. [Regression Potential] This patch has a low chance of regression since it fixes a regression introduced by
