[Kernel-packages] [Bug 1784684] Re: Waste padding after PPPoE payload
** Description changed: After a new Kernel e.g. 4.4.0-131 is installed on our Ubuntu 16.04 LTS clients, unneeded (waste) 14 bytes paddings are unexpectedly present after PPPoE payload in client's Ethernet frames which are sent towards a NAS access router. The content of such a waste padding is not all zero filled bytes, but rather parts of memory, sometimes with readable text. Thus it is also a security issue. Sometimes the waste padding is more than 14 bytes. For example, a PPPoED PADI frame (single tagged VLAN) with text "amf/application/12" (18 bytes) at the end of the frame shown as padding in Wireshark: ff ff ff ff ff ff 00 0a cd 2a ea 9f 81 00 00 07 ÿÿ..Í*ê. 0010 88 63 11 09 00 00 00 0c 01 01 00 00 01 03 00 04 .c.. 0020 d9 1f 00 00 61 6d 66 2f 61 70 70 6c 69 63 61 74 Ù...amf/applicat 0030 69 6f 6e 2f 31 32 ion/12 It is a critical bug of Kernel 4.4.0-131. Some affected client Ethernet frames are large enough (>64 bytes), and thus they do not need to be padded for Ethernet. Padding should be included and is required in Ethernet frames only to achieve the minimum 64 byte size of an Ethernet frame sent on wire. For a packet larger than e.g. 100 bytes no padding is needed in Ethernet frame. However we can see such 14 Byte paddings also in large client Ethernet frames (in PPPoE/PPP/IP session packets), with payload size of several hundred bytes (200, 300, 400 bytes and more). Our Ubuntu 16.04 LTS Clients are always updated via apt update / apt upgrade early enough, after a new Ubuntu update is released in Internet. Our statistics after analyzing the collected archived pcap traces of our clients with Ubuntu 16.04 LTS and the latest Kernel installed can help to identify the date and the version when the kernel bug first appeared: From 15.06.2018 to 29.06.2018 - still no waste padding on all clients (OK) 02.07.2018 - OK, still no waste padding on all clients (OK) 03.07.2018 - OK, still no waste padding on all clients (OK) 04.07.2018 - OK, still no waste padding on all clients (OK) 05.07.2018 - OK, still no waste padding on all clients (OK) 06.07.2018 - OK, still no waste padding on all clients (OK) !!! Kernel Update to 4.4.0-131 on 09-Jul-2018 !!! 09.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! 10.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! 12.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! 13.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! 19.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! ... - So the padding issue occurred on our clients for the first time on - 09-Jul-2018, after the Linux Kernel was updated to 4.4.0-131 on our - Ubuntu 16.04 LTS clients. Still then the issue is present on all updated - clients. The kernel bug disappeared and the padding behavior is OK again - (without bug), after clients were downgraded to and booted with an older - kernel version like e.g. 4.4.0-116 (older than 4.4.0-131). + So the padding issue occurred on our Ubuntu 16.04 LTS clients for the + first time on 09-Jul-2018, after the Linux Kernel was updated to + 4.4.0-131. Since then, the issue is present on all the updated clients. + The kernel bug disappeared and the padding behavior is OK again (without + bug), after clients were downgraded to and booted with an older kernel + version like e.g. 4.4.0-116 (older than 4.4.0-131). - The kernel version affected with this BUG: + Information about the Ubuntu clients and the kernel version affected + with this BUG: + @client:~$ uname -a Linux client 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux @client:~$ cat /proc/version_signature Ubuntu 4.4.0-131.157-generic 4.4.134 @client:~$ lsb_release -rd Description:Ubuntu 16.04.5 LTS Release:16.04 @client:~$ lsb_release -ci Distributor ID: Ubuntu Codename: xenial - The kernel bug does not occur on Ubuntu 16.04 LTS clients and waste - paddings (14 byte on single-stacked VLAN or 18 byte on double-stacked - VLAN) are not present in client frames, if an old Kernel e.g. 4.4.0-116 - (of 12-Feb-2018) is installed on the client and activated at boot time. + The kernel bug does not occur and waste paddings (14 byte on single- + stacked VLAN, 18 byte on double-stacked VLAN) are not present in client + frames, if an old Kernel e.g. 4.4.0-116 (of 12-Feb-2018) is installed on + the client and activated at boot time. This old kernel 4.4.0-116 is still without the padding bug: @client:~$ uname -a Linux client 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux A PADI packet with unneeded padding is included as attachment. It
[Kernel-packages] [Bug 1784684] Re: Waste padding after PPPoE payload
** Description changed: After a new Kernel e.g. 4.4.0-131 is installed on our Ubuntu 16.04 LTS clients, unneeded (waste) 14 bytes paddings are unexpectedly present after PPPoE payload in client's Ethernet frames which are sent towards a - NAS access router. The content of the waste padding is not all zero + NAS access router. The content of such a waste padding is not all zero filled bytes, but rather parts of memory, sometimes with readable text. - Sometimes the waste padding is more than 14 bytes. For example, a PPPoED - PADI Frame with "amf/application/12" (18 bytes) shown as padding in - Wireshark: + Thus it is also a security issue. Sometimes the waste padding is more + than 14 bytes. For example, a PPPoED PADI frame (single tagged VLAN) + with text "amf/application/12" (18 bytes) at the end of the frame shown + as padding in Wireshark: ff ff ff ff ff ff 00 0a cd 2a ea 9f 81 00 00 07 ÿÿ..Í*ê. 0010 88 63 11 09 00 00 00 0c 01 01 00 00 01 03 00 04 .c.. 0020 d9 1f 00 00 61 6d 66 2f 61 70 70 6c 69 63 61 74 Ù...amf/applicat 0030 69 6f 6e 2f 31 32 ion/12 - It is a critical bug of Kernel 4.4.0-131 and some previous releases - (since July 2018). + It is a critical bug of Kernel 4.4.0-131. - The affected client Ethernet frames are large enough (>64 bytes), and - thus they do not need to be padded. Padding should be included and is - required in Ethernet frames only to achieve the minimum 64 byte size of - an Ethernet frame sent on wire. For a packet larger than e.g. 100 bytes - no padding is needed in Ethernet frame. However we see such 14 Byte - paddings also in large client Ethernet Frames (in PPPoE/PPP/IP session - packets), with payload size of several hundred bytes (200, 300, 400 - bytes and more). + Some affected client Ethernet frames are large enough (>64 bytes), and + thus they do not need to be padded for Ethernet. Padding should be + included and is required in Ethernet frames only to achieve the minimum + 64 byte size of an Ethernet frame sent on wire. For a packet larger than + e.g. 100 bytes no padding is needed in Ethernet frame. However we can + see such 14 Byte paddings also in large client Ethernet frames (in + PPPoE/PPP/IP session packets), with payload size of several hundred + bytes (200, 300, 400 bytes and more). Our Ubuntu 16.04 LTS Clients are always updated via apt update / apt upgrade early enough, after a new Ubuntu update is released in Internet. - Our statistics after analyzing the collected archived pcap traces - (Clients with Ubuntu 16.04 LTS and the latest Kernel updates installed) - can help to identify the date when the kernel bug first appeared: + Our statistics after analyzing the collected archived pcap traces of our + clients with Ubuntu 16.04 LTS and the latest Kernel installed can help + to identify the date and the version when the kernel bug first appeared: - From 15.Jul.2018 till 29.Jul.2018 - OK, still no waste padding - 10.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! - 12.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! - 13.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! - 19.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! + From 15.06.2018 to 29.06.2018 - still no waste padding on all clients (OK) + 02.07.2018 - OK, still no waste padding on all clients (OK) + 03.07.2018 - OK, still no waste padding on all clients (OK) + 04.07.2018 - OK, still no waste padding on all clients (OK) + 05.07.2018 - OK, still no waste padding on all clients (OK) + 06.07.2018 - OK, still no waste padding on all clients (OK) - A kernel version affected with this BUG: + !!! Kernel Update to 4.4.0-131 on 09-Jul-2018 !!! + + 09.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! + 10.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! + 12.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! + 13.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! + 19.07.2018 - 14 byte padding (PADI, PADR, etc) on all updated clients - a kernel BUG ! + ... + + So the padding issue occurred on our clients for the first time on + 09-Jul-2018, after the Linux Kernel was updated to 4.4.0-131 on our + Ubuntu 16.04 LTS clients. Still then the issue is present on all updated + clients. The kernel bug disappeared and the padding behavior is OK again + (without bug), after clients were downgraded to and booted with an older + kernel version like e.g. 4.4.0-116 (older than 4.4.0-131). + + The kernel version affected with this BUG: @client:~$ uname -a Linux client 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux + + @client:~$ cat /proc/version_signature + Ubuntu 4.4.0-131.157-generic 4.4.134 @client:~$ lsb_release -rd Description:Ubuntu 16.04.5 LTS Release:16.04 - The
[Kernel-packages] [Bug 1784684] Re: Waste padding after PPPoE payload
** Attachment added: "two_packets.pcap" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784684/+attachment/5170301/+files/two_packets.pcap -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1784684 Title: Waste padding after PPPoE payload Status in linux package in Ubuntu: Confirmed Bug description: After a new Kernel e.g. 4.4.0-131 is installed on our Ubuntu 16.04 LTS clients, unneeded (waste) 14 bytes paddings are unexpectedly present after PPPoE payload in client's Ethernet frames which are sent towards a NAS access router. The content of the waste padding is not all zero filled bytes, but rather parts of memory, sometimes with readable text. Sometimes the waste padding is more than 14 bytes. For example, a PPPoED PADI Frame with "amf/application/12" (18 bytes) shown as padding in Wireshark: ff ff ff ff ff ff 00 0a cd 2a ea 9f 81 00 00 07 ÿÿ..Í*ê. 0010 88 63 11 09 00 00 00 0c 01 01 00 00 01 03 00 04 .c.. 0020 d9 1f 00 00 61 6d 66 2f 61 70 70 6c 69 63 61 74 Ù...amf/applicat 0030 69 6f 6e 2f 31 32 ion/12 It is a critical bug of Kernel 4.4.0-131 and some previous releases (since July 2018). The affected client Ethernet frames are large enough (>64 bytes), and thus they do not need to be padded. Padding should be included and is required in Ethernet frames only to achieve the minimum 64 byte size of an Ethernet frame sent on wire. For a packet larger than e.g. 100 bytes no padding is needed in Ethernet frame. However we see such 14 Byte paddings also in large client Ethernet Frames (in PPPoE/PPP/IP session packets), with payload size of several hundred bytes (200, 300, 400 bytes and more). Our Ubuntu 16.04 LTS Clients are always updated via apt update / apt upgrade early enough, after a new Ubuntu update is released in Internet. Our statistics after analyzing the collected archived pcap traces (Clients with Ubuntu 16.04 LTS and the latest Kernel updates installed) can help to identify the date when the kernel bug first appeared: From 15.Jul.2018 till 29.Jul.2018 - OK, still no waste padding 10.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 12.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 13.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 19.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! A kernel version affected with this BUG: @client:~$ uname -a Linux client 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux @client:~$ lsb_release -rd Description:Ubuntu 16.04.5 LTS Release:16.04 The kernel bug does not occur on Ubuntu 16.04 LTS Client and waste paddings (14 byte or 18 byte) are not present in Client frames, if an old Kernel e.g. 4.4.0-116 (of 12-Feb-2018) is installed on the client and activated at boot time. This old kernel 4.4.0-116 is still without the padding bug: @client:~$ uname -a Linux client 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux A PADI packet with unneeded padding is included as attachment. It was captured via tcpdump directly on the Ubuntu Client. It is a padding inserted by Ubuntu, not by ethernet HW or driver. If it were padding included by Ethernet adapter or driver, we could not capture it on the same host and we could not see such real padding for this client packet in Wireshark. The issue affects both PPPoED client packets (PADI, PADR) and session PPPoE/PPP/IP client packets. We expect no padding is included after PPPoE payload in Ethernet Frames if the packets are not small (>64 byte) and thus are not required to be padded for Ethernet. Such correct behavior (without waste paddings) can be observed on Ubuntu 16.04 LTS with old Kernel versions released before June 2018, like e.g. 4.4.0-116. Please implement the fix in the next Kernel update. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784684/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1784684] Re: Waste padding after PPPoE payload
A second pcap trace (two_packets.pcap) with two Client packets is attached to this ticket. The following Client packets in the pcap trace demonstrate the issue exemplarily: - PADI frame (PPPoED; Client Ethernet Frame of 87 byte size); - a session Client packet (Ethernet frame of 473 Byte size). The both packets are large enough (>64 Bytes) and do not need to be padded. However a waste 14 byte padding is present after PPPoE payload in the both packets (Linux kernel bug). One can see the Problem while displaying and analyzing these packets in Wireshark. Evaluation of the Ubuntu (kernel) failure: 1. Such 14 Byte paddings included by Ubuntu kernel are waste and not required by the Ethernet standard. 2. Such incorrect Ethernet Frames from Client can be evaluated as "malformed" on NAS and thus can be silently dropped by NAS, resulting a critical connectivity issue in client PPPoE/PPP sessions. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1784684 Title: Waste padding after PPPoE payload Status in linux package in Ubuntu: Confirmed Bug description: After a new Kernel e.g. 4.4.0-131 is installed on our Ubuntu 16.04 LTS clients, unneeded (waste) 14 bytes paddings are unexpectedly present after PPPoE payload in client's Ethernet frames which are sent towards a NAS access router. The content of the waste padding is not all zero filled bytes, but rather parts of memory, sometimes with readable text. Sometimes the waste padding is more than 14 bytes. For example, a PPPoED PADI Frame with "amf/application/12" (18 bytes) shown as padding in Wireshark: ff ff ff ff ff ff 00 0a cd 2a ea 9f 81 00 00 07 ÿÿ..Í*ê. 0010 88 63 11 09 00 00 00 0c 01 01 00 00 01 03 00 04 .c.. 0020 d9 1f 00 00 61 6d 66 2f 61 70 70 6c 69 63 61 74 Ù...amf/applicat 0030 69 6f 6e 2f 31 32 ion/12 It is a critical bug of Kernel 4.4.0-131 and some previous releases (since July 2018). The affected client Ethernet frames are large enough (>64 bytes), and thus they do not need to be padded. Padding should be included and is required in Ethernet frames only to achieve the minimum 64 byte size of an Ethernet frame sent on wire. For a packet larger than e.g. 100 bytes no padding is needed in Ethernet frame. However we see such 14 Byte paddings also in large client Ethernet Frames (in PPPoE/PPP/IP session packets), with payload size of several hundred bytes (200, 300, 400 bytes and more). Our Ubuntu 16.04 LTS Clients are always updated via apt update / apt upgrade early enough, after a new Ubuntu update is released in Internet. Our statistics after analyzing the collected archived pcap traces (Clients with Ubuntu 16.04 LTS and the latest Kernel updates installed) can help to identify the date when the kernel bug first appeared: From 15.Jul.2018 till 29.Jul.2018 - OK, still no waste padding 10.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 12.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 13.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 19.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! A kernel version affected with this BUG: @client:~$ uname -a Linux client 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux @client:~$ lsb_release -rd Description:Ubuntu 16.04.5 LTS Release:16.04 The kernel bug does not occur on Ubuntu 16.04 LTS Client and waste paddings (14 byte or 18 byte) are not present in Client frames, if an old Kernel e.g. 4.4.0-116 (of 12-Feb-2018) is installed on the client and activated at boot time. This old kernel 4.4.0-116 is still without the padding bug: @client:~$ uname -a Linux client 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux A PADI packet with unneeded padding is included as attachment. It was captured via tcpdump directly on the Ubuntu Client. It is a padding inserted by Ubuntu, not by ethernet HW or driver. If it were padding included by Ethernet adapter or driver, we could not capture it on the same host and we could not see such real padding for this client packet in Wireshark. The issue affects both PPPoED client packets (PADI, PADR) and session PPPoE/PPP/IP client packets. We expect no padding is included after PPPoE payload in Ethernet Frames if the packets are not small (>64 byte) and thus are not required to be padded for Ethernet. Such correct behavior (without waste paddings) can be observed on Ubuntu 16.04 LTS with old Kernel versions released before June 2018, like e.g. 4.4.0-116. Please implement the fix in the next Kernel update. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784684/+subscriptions -- Mailing list:
[Kernel-packages] [Bug 1784684] Re: Waste padding after PPPoE payload
The command you requested on a affected Ubuntu 16.04 LTS host cannot be successfully executed (no connection to the Server): @client:~$ apport-collect 1784684 ERROR: connecting to Launchpad failed: Unable to find the server at launchpad.net As we are unable to run this command due to missing direct Internet access on the affected Ubuntu clients, the bug Status in the ticket will be manually changed into 'Confirmed'. ** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1784684 Title: Waste padding after PPPoE payload Status in linux package in Ubuntu: Confirmed Bug description: After a new Kernel e.g. 4.4.0-131 is installed on our Ubuntu 16.04 LTS clients, unneeded (waste) 14 bytes paddings are unexpectedly present after PPPoE payload in client's Ethernet frames which are sent towards a NAS access router. The content of the waste padding is not all zero filled bytes, but rather parts of memory, sometimes with readable text. Sometimes the waste padding is more than 14 bytes. For example, a PPPoED PADI Frame with "amf/application/12" (18 bytes) shown as padding in Wireshark: ff ff ff ff ff ff 00 0a cd 2a ea 9f 81 00 00 07 ÿÿ..Í*ê. 0010 88 63 11 09 00 00 00 0c 01 01 00 00 01 03 00 04 .c.. 0020 d9 1f 00 00 61 6d 66 2f 61 70 70 6c 69 63 61 74 Ù...amf/applicat 0030 69 6f 6e 2f 31 32 ion/12 It is a critical bug of Kernel 4.4.0-131 and some previous releases (since July 2018). The affected client Ethernet frames are large enough (>64 bytes), and thus they do not need to be padded. Padding should be included and is required in Ethernet frames only to achieve the minimum 64 byte size of an Ethernet frame sent on wire. For a packet larger than e.g. 100 bytes no padding is needed in Ethernet frame. However we see such 14 Byte paddings also in large client Ethernet Frames (in PPPoE/PPP/IP session packets), with payload size of several hundred bytes (200, 300, 400 bytes and more). Our Ubuntu 16.04 LTS Clients are always updated via apt update / apt upgrade early enough, after a new Ubuntu update is released in Internet. Our statistics after analyzing the collected archived pcap traces (Clients with Ubuntu 16.04 LTS and the latest Kernel updates installed) can help to identify the date when the kernel bug first appeared: From 15.Jul.2018 till 29.Jul.2018 - OK, still no waste padding 10.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 12.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 13.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! 19.07.2018 - 14 byte padding (PADI etc) - a kernel BUG ! A kernel version affected with this BUG: @client:~$ uname -a Linux client 4.4.0-131-generic #157-Ubuntu SMP Thu Jul 12 15:51:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux @client:~$ lsb_release -rd Description:Ubuntu 16.04.5 LTS Release:16.04 The kernel bug does not occur on Ubuntu 16.04 LTS Client and waste paddings (14 byte or 18 byte) are not present in Client frames, if an old Kernel e.g. 4.4.0-116 (of 12-Feb-2018) is installed on the client and activated at boot time. This old kernel 4.4.0-116 is still without the padding bug: @client:~$ uname -a Linux client 4.4.0-116-generic #140-Ubuntu SMP Mon Feb 12 21:23:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux A PADI packet with unneeded padding is included as attachment. It was captured via tcpdump directly on the Ubuntu Client. It is a padding inserted by Ubuntu, not by ethernet HW or driver. If it were padding included by Ethernet adapter or driver, we could not capture it on the same host and we could not see such real padding for this client packet in Wireshark. The issue affects both PPPoED client packets (PADI, PADR) and session PPPoE/PPP/IP client packets. We expect no padding is included after PPPoE payload in Ethernet Frames if the packets are not small (>64 byte) and thus are not required to be padded for Ethernet. Such correct behavior (without waste paddings) can be observed on Ubuntu 16.04 LTS with old Kernel versions released before June 2018, like e.g. 4.4.0-116. Please implement the fix in the next Kernel update. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1784684/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp