[Kernel-packages] [Bug 2016881] Re: Building for Focal fails on riscv64

[Heitor Alves de Siqueira]

Bionic is EoL, if RISC-V support is needed there we'll need to target
Pro 18.04.

** Tags removed: patch se-sponsor-halves

** Description changed:

- Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a single
- patch missing:
- https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80
+ [Impact]
+  * riscv64 builds fail completely, so users on that architecture can't use 
ZFS.
+ 
+ [Test Plan]
+  * Build zfs-linux on target architecture. No test failure is expected.
+ 
+ [Where problems could occur]
+  * Test failures and bugs that are exclusive to RISCV64 might show up
+ 
+ [Other Info]
+  * Upstream introduced support riscv64 with 4254e407294b
+  * Builds succeed on Jammy and newer, Focal is missing the enablement patch
+ 
+ --
+ [Original Description]
+ Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a single patch 
missing: 
https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: Invalid => Won't Fix

** Summary changed:

- Building for Focal fails on riscv64
+ zfs-linux FTBFS for riscv64 on Focal

** Changed in: zfs-linux (Ubuntu Bionic)
   Importance: Medium => Undecided

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016881

Title:
  zfs-linux FTBFS for riscv64 on Focal

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Won't Fix
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  [Impact]
   * riscv64 builds fail completely, so users on that architecture can't use 
ZFS.

  [Test Plan]
   * Build zfs-linux on target architecture. No test failure is expected.

  [Where problems could occur]
   * Test failures and bugs that are exclusive to RISCV64 might show up

  [Other Info]
   * Upstream introduced support riscv64 with 4254e407294b
   * Builds succeed on Jammy and newer, Focal is missing the enablement patch

  --
  [Original Description]
  Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a single patch 
missing: 
https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2016881/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1987052] Re: Running MySQL8 in Docker on ZFS only works if you have ZFS 2.1.5. Ubuntu 22.04 only has 2.1.4 right now.

[Heitor Alves de Siqueira]

** Changed in: zfs-linux (Ubuntu)
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1987052

Title:
  Running MySQL8 in Docker on ZFS only works if you have ZFS 2.1.5.
  Ubuntu 22.04 only has 2.1.4 right now.

Status in zfs-linux package in Ubuntu:
  Fix Released

Bug description:
  Running MySQL 8 using Docker on a ZFS system currently is not possible due to 
a bug in ZFS.
  It has been fixed in ZFS 2.1.5, but jammy-updates is still on 2.1.4

  Can someone please update from 2.1.4 to 2.1.5?

  MySQL 8 is a popular database and running using Docker (and ZFS) is
  more and more common.

  More details on this and confirmed by the MySQL Docker team:
  https://github.com/docker-library/mysql/issues/868

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1987052/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1980848] Re: arc_summary doesn't work with HWE kernel 5.15

[Heitor Alves de Siqueira]

** Also affects: zfs-linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Changed in: zfs-linux (Ubuntu Focal)
   Status: New => In Progress

** Changed in: zfs-linux (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: zfs-linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu)
   Status: Confirmed => Fix Released

** Changed in: zfs-linux (Ubuntu)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1980848

Title:
  arc_summary doesn't work with HWE kernel 5.15

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  # Issue description

  `arc_summary` no longer works with kernel 5.15. It used to work with
  previous kernel like 5.13.

  # Steps to reproduce

  1) setup 20.04 with HWE kernel 5.15
  2) install `zfsutils-linux`
  3) run `arc_summary`
  $ arc_summary 
  Traceback (most recent call last):
File "/usr/sbin/arc_summary", line 875, in 
  main()
File "/usr/sbin/arc_summary", line 826, in main
  kstats = get_kstats()
File "/usr/sbin/arc_summary", line 259, in get_kstats
  with open(PROC_PATH+section, 'r') as proc_location:
  FileNotFoundError: [Errno 2] No such file or directory: 
'/proc/spl/kstat/zfs/xuio_stats'


  Indeed, the xuio_stats file isn't there anymore:
  $ ll /proc/spl/kstat/zfs/
  total 0
  dr-xr-xr-x 21 root root 0 Jul  6 10:49 ./
  dr-xr-xr-x  4 root root 0 Jul  6 10:49 ../
  -rw-r--r--  1 root root 0 Jul  6 10:49 abdstats
  -rw-r--r--  1 root root 0 Jul  6 10:49 arcstats
  dr-xr-xr-x 20 root root 0 Jul  6 10:49 data/
  -rw---  1 root root 0 Jul  6 10:49 dbgmsg
  -rw---  1 root root 0 Jul  6 10:49 dbufs
  -rw-r--r--  1 root root 0 Jul  6 10:49 dbufstats
  dr-xr-xr-x 70 root root 0 Jul  6 10:49 default/
  -rw-r--r--  1 root root 0 Jul  6 10:49 dmu_tx
  -rw-r--r--  1 root root 0 Jul  6 10:49 dnodestats
  -rw-r--r--  1 root root 0 Jul  6 10:49 fletcher_4_bench
  -rw-r--r--  1 root root 0 Jul  6 10:49 fm
  -rw-r--r--  1 root root 0 Jul  6 10:49 import_progress
  -rw-r--r--  1 root root 0 Jul  6 10:49 metaslab_stats
  -rw-r--r--  1 root root 0 Jul  6 10:49 vdev_cache_stats
  -rw-r--r--  1 root root 0 Jul  6 10:49 vdev_mirror_stats
  -rw-r--r--  1 root root 0 Jul  6 10:49 vdev_raidz_bench
  -rw-r--r--  1 root root 0 Jul  6 10:49 zfetchstats
  -rw-r--r--  1 root root 0 Jul  6 10:49 zil
  -rw-r--r--  1 root root 0 Jul  6 10:49 zstd

  
  # Workaround

  This (naive) patch sidesteps the problem:

  $ diff -Naur /usr/sbin/arc_summary.old /usr/sbin/arc_summary
  --- /usr/sbin/arc_summary.old 2022-07-06 10:59:50.752833101 -0400
  +++ /usr/sbin/arc_summary 2022-07-06 10:59:22.449113169 -0400
  @@ -255,6 +255,8 @@
   secs = SECTION_PATHS.values()
   
   for section in secs:
  +if not os.path.exists(PROC_PATH+section):
  +continue
   
   with open(PROC_PATH+section, 'r') as proc_location:
   lines = [line for line in proc_location]

  
  # Additional information
  $ lsb_release -rd
  Description:  Ubuntu 20.04.4 LTS
  Release:  20.04
  $ uname -r
  5.15.0-41-generic
  $ apt-cache policy zfsutils-linux linux-image-generic-hwe-20.04
  zfsutils-linux:
Installed: 0.8.3-1ubuntu12.14
Candidate: 0.8.3-1ubuntu12.14
Version table:
   *** 0.8.3-1ubuntu12.14 500
  500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 
Packages
  100 /var/lib/dpkg/status
   0.8.3-1ubuntu12.9 500
  500 http://security.ubuntu.com/ubuntu focal-security/main amd64 
Packages
   0.8.3-1ubuntu12 500
  500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages
  linux-image-generic-hwe-20.04:
Installed: 5.15.0.41.44~20.04.13
Candidate: 5.15.0.41.44~20.04.13
Version table:
   *** 5.15.0.41.44~20.04.13 400
  400 http://us.archive.ubuntu.com/ubuntu focal-proposed/main amd64 
Packages
  100 /var/lib/dpkg/status
   5.13.0.52.59~20.04.31 500
  500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 
Packages
  500 http://security.ubuntu.com/ubuntu focal-security/main amd64 
Packages
   5.4.0.26.32 500
  500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1980848/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2061986] Re: Mount CIFS fails with Permission denied

[Heitor Alves de Siqueira]

Marking Ubuntu as "Fix Released" according to parent description, as fix
was introduced upstream in v5.16.

** Changed in: linux (Ubuntu Focal)
   Status: Confirmed => In Progress

** Changed in: linux (Ubuntu Jammy)
   Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2061986

Title:
  Mount CIFS fails with Permission denied

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Focal:
  In Progress
Status in linux source package in Jammy:
  In Progress

Bug description:
  [ Impact ]

   * Mounting SMB share from server without Key Exchange capability is
  failing with Access Denied error

   * Even though SMB server during Session Setup Response in NTLMSSP_CHALLANGE 
message does not advertise
 Key Exchange capabilities SMB client < 5.16 will forcefully use it leading 
to error response during
 TCON requests.

   * Issue can be reproduced on 5.15 or older Kernels, there is no reproduction 
on 6.5 Kernel
   
   * This scenario was fixed in upstream commit 
9de0737d5ba0425c3154d5d83da12a8fa8595c0f
   
   * An example of server without Key Exchange capability is Oracle Solaris 
11.4 SMB zfs, meaning
 mounting share from that server will result in ACCESS_DENIED error.
   
  [ Test Plan ]

   * So far issue was reported only with Oracle Solaris 11.04 smb server
  and Ubuntu with Kernel <= 5.15

   * To reproduce, setup Oracle Solaris SMB server and try to mount share on 
22.04/20.04 (5.15/5.04)
 Steps to configure SMB server:
  1. Download the ISO for Oracle Solaris Common Build Edition [1]
  2. Create a VM with at least 16 GB of memory - I have experienced 
installation issues with less memory
  3. Install Oracle Solaris using the downloaded ISO
  a. Make sure to create a test user
  4. Log into the VM as the root user
  5. Create a test directory for the share:
  a. mkdir /smbshare && chmod 777 /smbshare 
  6. Disable the normal Samba daemon: [2]
  a. svcadm disable svc:/network/samba
  b. svcadm disable svc:/network/wins
  7 Configure the server to serve Samba shares using ZFS in Workgroup mode [3]
  a. svcadm enable -r smb/server
  b. smbadm join -w workgroup
  8 Update the /etc/pam.d/other file to require authentication by adding the 
following line:
  a. password requiredpam_smb_passwd.so.1nowarn
  9. Reset the password for the test user so that it is updated in the SMB 
password database
  10. Create the pool and share it using Samba: [4]
  a. zfs create -o mountpoint=/smbshare/ rpool/smbshare
  b. zfs share -o share.smb=on rpool/smbshare%share

  [1] 

  [2] 

  [3] 

  [4] 


   * With server configured, mount share using ubuntu SMB client
 Expected result: mount operation should succeed
 Actual result: mount returns following error:
  root@ubuntu20:/mnt# mount -t cifs -o username=rmalz //192.168.50.217/smbshare 
test
  Password for rmalz@//192.168.50.217/smbshare:  
  mount error(13): Permission denied
  Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log 
messages (dmesg) 

  [ Where problems could occur ]

   * Upstream patch is changing smb client behavior based on server 
NTLMSSP_CHALLENGE Negotiate Flags,
 if server does not advertise Key Exchange Capability but requires it from 
client communication might
 be broken. It is unknown if such servers are used, such instance should be 
treated as a server bug.

   * Patch is available in upstream kernel since 5.16, any issues associated 
with it should be already
 detected.

   * Patch adds additional requirement checks on server NTLM flags, although it 
is possible to hit
 these checks, I was not able to find any instances of that occurring.

   * To lower regression potential, upstream patch backported to Ubuntu 5.15 
and 5.04 Kernels have been
 tested in following environments:
 smb server: Oracle Solaris 11.04, Ubuntu 22.04 HWE
 smb client: Ubuntu 22.04, Ubuntu 20.04
 During testing no issues have been detected.

  [ Other Info ]
   
   * Error message coming from SMB client is the same as providing incorrect 
credentials, which might
 confuse users. 
   * Attaching tcpdump pcaps with SMB operations from 5.15 Kernel with and 
without patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2061986/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : 

[Kernel-packages] [Bug 2061986] Re: Mount CIFS fails with Permission denied

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu)
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2061986

Title:
  Mount CIFS fails with Permission denied

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Focal:
  In Progress
Status in linux source package in Jammy:
  In Progress

Bug description:
  [ Impact ]

   * Mounting SMB share from server without Key Exchange capability is
  failing with Access Denied error

   * Even though SMB server during Session Setup Response in NTLMSSP_CHALLANGE 
message does not advertise
 Key Exchange capabilities SMB client < 5.16 will forcefully use it leading 
to error response during
 TCON requests.

   * Issue can be reproduced on 5.15 or older Kernels, there is no reproduction 
on 6.5 Kernel
   
   * This scenario was fixed in upstream commit 
9de0737d5ba0425c3154d5d83da12a8fa8595c0f
   
   * An example of server without Key Exchange capability is Oracle Solaris 
11.4 SMB zfs, meaning
 mounting share from that server will result in ACCESS_DENIED error.
   
  [ Test Plan ]

   * So far issue was reported only with Oracle Solaris 11.04 smb server
  and Ubuntu with Kernel <= 5.15

   * To reproduce, setup Oracle Solaris SMB server and try to mount share on 
22.04/20.04 (5.15/5.04)
 Steps to configure SMB server:
  1. Download the ISO for Oracle Solaris Common Build Edition [1]
  2. Create a VM with at least 16 GB of memory - I have experienced 
installation issues with less memory
  3. Install Oracle Solaris using the downloaded ISO
  a. Make sure to create a test user
  4. Log into the VM as the root user
  5. Create a test directory for the share:
  a. mkdir /smbshare && chmod 777 /smbshare 
  6. Disable the normal Samba daemon: [2]
  a. svcadm disable svc:/network/samba
  b. svcadm disable svc:/network/wins
  7 Configure the server to serve Samba shares using ZFS in Workgroup mode [3]
  a. svcadm enable -r smb/server
  b. smbadm join -w workgroup
  8 Update the /etc/pam.d/other file to require authentication by adding the 
following line:
  a. password requiredpam_smb_passwd.so.1nowarn
  9. Reset the password for the test user so that it is updated in the SMB 
password database
  10. Create the pool and share it using Samba: [4]
  a. zfs create -o mountpoint=/smbshare/ rpool/smbshare
  b. zfs share -o share.smb=on rpool/smbshare%share

  [1] 

  [2] 

  [3] 

  [4] 


   * With server configured, mount share using ubuntu SMB client
 Expected result: mount operation should succeed
 Actual result: mount returns following error:
  root@ubuntu20:/mnt# mount -t cifs -o username=rmalz //192.168.50.217/smbshare 
test
  Password for rmalz@//192.168.50.217/smbshare:  
  mount error(13): Permission denied
  Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log 
messages (dmesg) 

  [ Where problems could occur ]

   * Upstream patch is changing smb client behavior based on server 
NTLMSSP_CHALLENGE Negotiate Flags,
 if server does not advertise Key Exchange Capability but requires it from 
client communication might
 be broken. It is unknown if such servers are used, such instance should be 
treated as a server bug.

   * Patch is available in upstream kernel since 5.16, any issues associated 
with it should be already
 detected.

   * Patch adds additional requirement checks on server NTLM flags, although it 
is possible to hit
 these checks, I was not able to find any instances of that occurring.

   * To lower regression potential, upstream patch backported to Ubuntu 5.15 
and 5.04 Kernels have been
 tested in following environments:
 smb server: Oracle Solaris 11.04, Ubuntu 22.04 HWE
 smb client: Ubuntu 22.04, Ubuntu 20.04
 During testing no issues have been detected.

  [ Other Info ]
   
   * Error message coming from SMB client is the same as providing incorrect 
credentials, which might
 confuse users. 
   * Attaching tcpdump pcaps with SMB operations from 5.15 Kernel with and 
without patch.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2061986/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2016881] Re: Building for Focal fails on riscv64

[Heitor Alves de Siqueira]

** Changed in: zfs-linux (Ubuntu Focal)
   Importance: Medium => Low

** Changed in: zfs-linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu Focal)
   Status: Incomplete => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016881

Title:
  Building for Focal fails on riscv64

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Invalid
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a
  single patch missing:
  https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2016881/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2036239] Re: Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

[Heitor Alves de Siqueira]

Yes, HWE kernels based on the Jammy/Mantic/Noble kernels should get this
fix automatically when the GA versions get released.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2036239

Title:
  Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Jammy:
  Fix Committed
Status in linux source package in Mantic:
  Fix Committed
Status in linux source package in Noble:
  In Progress

Bug description:
  [Impact]
   * Issue is causing transmit hang on E810 ports with bonding enabled.
   * Based on the provided logs, TX hang can last for even a couple of 
minutes, but in most scenarios, the network will be recovered after the ice 
driver performs a PF reset (TX hang handler routine).
   * Originally, the issue was observed during Tempest tests on a newly 
created OpenStack cluster, resulting in a lack of certification.
  
  [Fix]
  * Initially, a workaround has been proposed by Intel engineers to disable 
LAG initialization [1].
This change has been tested in an environment where reproduction is 
easily achieved.
After multiple iterations, no reproduction has been observed.
  * Shortly after, Intel proposed a patch [2] to disable LAG initialization 
if NVM does not expose proper capabilities.
  
  [Test Plan]
  * To reproduce the issue, over a 20-node cluster was used with Ceph-based 
storage. The problem could sometimes manifest while deploying a cluster or 
after the cluster was already deployed during the Tempest test run.
  * The issue could appear on a random node, making reproduction hard to 
achieve.
  * Multiple stress tests on single host with similar configuration did not 
trigger a reproduction.
  
  [Where problems could occur]
  * All ice drivers with ice_lag_event_handler registered can expose the 
issue. This handler is not implemented in 20.04
  * CVL4.2 and older NVM images for E810 does not expose SRIOV LAG 
capabilities (CVL4.3 wasn't checked) meaning at some point NVM with this 
capability will be released.
Although potentialy issue is caused by using features without proper FW 
support [2], we want to take a closer look once NVMs with proper support are 
introduced.

  [1] - 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/40
  [2] - 
https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html
 4d50fcdc2476eef94c14c6761073af5667bb43b6

  [Other Info]
  * Issue could be reproduced on custom 6.2 jammy-hwe kernel with ice 
driver backported from mainline kernel from before patch [2] was added.
  * Original description of the case below:
  
  

  I'm having issues with an Intel E810-XXV card on a Dell server under
  Ubuntu Jammy.

  Details:

  - hardware --> a1:00.0 Ethernet controller: Intel Corporation Ethernet
  Controller E810-XXV for SFP (rev 02)

  - tested with both GA and HWE kernels (`5.15.0-83-generic #92` and
  `6.2.0-32-generic #32~22.04.1-Ubuntu`) with the same results.

  - using a bond over the two ports of the same card, at 25Gbps to two
  different switches, bond is using LACP with hash layer3+4 and fast
  timeout. But I believe the bug is not directly related to bonding as
  the problem seems to be in the interface.

  - machine installed by maas. No issues during installation, but at
  that time bond is not formed yet, later when linux is booted, the bond
  is formed and works without issues for a while

  - it works for about 2 to 3 hours fine, then the issue starts (may or
  may not be related to network load, but it seems that it is triggered
  by some tests that I run after openstack finishes installing)

  - one of the legs of the bond freezes and everything that would go to
  that lag is discarded, in and out, ping to random external hosts start
  losing every second packet

  - after some time you can see on the kernel log messages about "NETDEV
  WATCHDOG: enp161s0f0 (ice): transmit queue 166 timed out" and a stack
  trace

  - the switch does log that the bond is flapping
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Sep 12 20:05 seq
   crw-rw 1 root audio 116, 33 Sep 12 20:05 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  CasperMD5CheckResult: pass
  CloudArchitecture: x86_64
  CloudID: none
  CloudName: none
  CloudPlatform: none
  CloudSubPlatform: config
  DistroRelease: Ubuntu 22.04
  InstallationDate: Installed on 2023-08-22 (24 days ago)
  InstallationMedia: 

[Kernel-packages] [Bug 2036239] Re: Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

[Heitor Alves de Siqueira]

@christian-rhomann "Fix committed" here means that the patches have been merged 
into Ubuntu's kernel tree for that specific release. The patch Robert submitted 
is the one from upstream, not the test patch from the comments here.
E.g. for Jammy:
https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/commit/?h=master-next=fc26d7737e3a

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2036239

Title:
  Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Jammy:
  Fix Committed
Status in linux source package in Mantic:
  Fix Committed
Status in linux source package in Noble:
  In Progress

Bug description:
  [Impact]
   * Issue is causing transmit hang on E810 ports with bonding enabled.
   * Based on the provided logs, TX hang can last for even a couple of 
minutes, but in most scenarios, the network will be recovered after the ice 
driver performs a PF reset (TX hang handler routine).
   * Originally, the issue was observed during Tempest tests on a newly 
created OpenStack cluster, resulting in a lack of certification.
  
  [Fix]
  * Initially, a workaround has been proposed by Intel engineers to disable 
LAG initialization [1].
This change has been tested in an environment where reproduction is 
easily achieved.
After multiple iterations, no reproduction has been observed.
  * Shortly after, Intel proposed a patch [2] to disable LAG initialization 
if NVM does not expose proper capabilities.
  
  [Test Plan]
  * To reproduce the issue, over a 20-node cluster was used with Ceph-based 
storage. The problem could sometimes manifest while deploying a cluster or 
after the cluster was already deployed during the Tempest test run.
  * The issue could appear on a random node, making reproduction hard to 
achieve.
  * Multiple stress tests on single host with similar configuration did not 
trigger a reproduction.
  
  [Where problems could occur]
  * All ice drivers with ice_lag_event_handler registered can expose the 
issue. This handler is not implemented in 20.04
  * CVL4.2 and older NVM images for E810 does not expose SRIOV LAG 
capabilities (CVL4.3 wasn't checked) meaning at some point NVM with this 
capability will be released.
Although potentialy issue is caused by using features without proper FW 
support [2], we want to take a closer look once NVMs with proper support are 
introduced.

  [1] - 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/40
  [2] - 
https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html
 4d50fcdc2476eef94c14c6761073af5667bb43b6

  [Other Info]
  * Issue could be reproduced on custom 6.2 jammy-hwe kernel with ice 
driver backported from mainline kernel from before patch [2] was added.
  * Original description of the case below:
  
  

  I'm having issues with an Intel E810-XXV card on a Dell server under
  Ubuntu Jammy.

  Details:

  - hardware --> a1:00.0 Ethernet controller: Intel Corporation Ethernet
  Controller E810-XXV for SFP (rev 02)

  - tested with both GA and HWE kernels (`5.15.0-83-generic #92` and
  `6.2.0-32-generic #32~22.04.1-Ubuntu`) with the same results.

  - using a bond over the two ports of the same card, at 25Gbps to two
  different switches, bond is using LACP with hash layer3+4 and fast
  timeout. But I believe the bug is not directly related to bonding as
  the problem seems to be in the interface.

  - machine installed by maas. No issues during installation, but at
  that time bond is not formed yet, later when linux is booted, the bond
  is formed and works without issues for a while

  - it works for about 2 to 3 hours fine, then the issue starts (may or
  may not be related to network load, but it seems that it is triggered
  by some tests that I run after openstack finishes installing)

  - one of the legs of the bond freezes and everything that would go to
  that lag is discarded, in and out, ping to random external hosts start
  losing every second packet

  - after some time you can see on the kernel log messages about "NETDEV
  WATCHDOG: enp161s0f0 (ice): transmit queue 166 timed out" and a stack
  trace

  - the switch does log that the bond is flapping
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Sep 12 20:05 seq
   crw-rw 1 root audio 116, 33 Sep 12 20:05 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  CasperMD5CheckResult: pass

[Kernel-packages] [Bug 2036239] Re: Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

[Heitor Alves de Siqueira]

** Also affects: linux (Ubuntu Noble)
   Importance: Medium
 Assignee: Robert Malz (rmalz)
   Status: In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2036239

Title:
  Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Jammy:
  In Progress
Status in linux source package in Mantic:
  In Progress
Status in linux source package in Noble:
  In Progress

Bug description:
  [Impact]
   * Issue is causing transmit hang on E810 ports with bonding enabled.
   * Based on the provided logs, TX hang can last for even a couple of 
minutes, but in most scenarios, the network will be recovered after the ice 
driver performs a PF reset (TX hang handler routine).
   * Originally, the issue was observed during Tempest tests on a newly 
created OpenStack cluster, resulting in a lack of certification.
  
  [Fix]
  * Initially, a workaround has been proposed by Intel engineers to disable 
LAG initialization [1].
This change has been tested in an environment where reproduction is 
easily achieved.
After multiple iterations, no reproduction has been observed.
  * Shortly after, Intel proposed a patch [2] to disable LAG initialization 
if NVM does not expose proper capabilities.
  
  [Test Plan]
  * To reproduce the issue, over a 20-node cluster was used with Ceph-based 
storage. The problem could sometimes manifest while deploying a cluster or 
after the cluster was already deployed during the Tempest test run.
  * The issue could appear on a random node, making reproduction hard to 
achieve.
  * Multiple stress tests on single host with similar configuration did not 
trigger a reproduction.
  
  [Where problems could occur]
  * All ice drivers with ice_lag_event_handler registered can expose the 
issue. This handler is not implemented in 20.04
  * CVL4.2 and older NVM images for E810 does not expose SRIOV LAG 
capabilities (CVL4.3 wasn't checked) meaning at some point NVM with this 
capability will be released.
Although potentialy issue is caused by using features without proper FW 
support [2], we want to take a closer look once NVMs with proper support are 
introduced.

  [1] - 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/40
  [2] - 
https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html
 4d50fcdc2476eef94c14c6761073af5667bb43b6

  [Other Info]
  * Issue could be reproduced on custom 6.2 jammy-hwe kernel with ice 
driver backported from mainline kernel from before patch [2] was added.
  * Original description of the case below:
  
  

  I'm having issues with an Intel E810-XXV card on a Dell server under
  Ubuntu Jammy.

  Details:

  - hardware --> a1:00.0 Ethernet controller: Intel Corporation Ethernet
  Controller E810-XXV for SFP (rev 02)

  - tested with both GA and HWE kernels (`5.15.0-83-generic #92` and
  `6.2.0-32-generic #32~22.04.1-Ubuntu`) with the same results.

  - using a bond over the two ports of the same card, at 25Gbps to two
  different switches, bond is using LACP with hash layer3+4 and fast
  timeout. But I believe the bug is not directly related to bonding as
  the problem seems to be in the interface.

  - machine installed by maas. No issues during installation, but at
  that time bond is not formed yet, later when linux is booted, the bond
  is formed and works without issues for a while

  - it works for about 2 to 3 hours fine, then the issue starts (may or
  may not be related to network load, but it seems that it is triggered
  by some tests that I run after openstack finishes installing)

  - one of the legs of the bond freezes and everything that would go to
  that lag is discarded, in and out, ping to random external hosts start
  losing every second packet

  - after some time you can see on the kernel log messages about "NETDEV
  WATCHDOG: enp161s0f0 (ice): transmit queue 166 timed out" and a stack
  trace

  - the switch does log that the bond is flapping
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Sep 12 20:05 seq
   crw-rw 1 root audio 116, 33 Sep 12 20:05 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  CasperMD5CheckResult: pass
  CloudArchitecture: x86_64
  CloudID: none
  CloudName: none
  CloudPlatform: none
  CloudSubPlatform: config
  DistroRelease: Ubuntu 22.04
  InstallationDate: Installed on 2023-08-22 (24 days ago)
  InstallationMedia: Ubuntu-Server 

[Kernel-packages] [Bug 2036239] Re: Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu)
   Status: Invalid => Confirmed

** Changed in: linux (Ubuntu)
   Status: Confirmed => In Progress

** Changed in: linux (Ubuntu)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Robert Malz (rmalz)

** Changed in: linux (Ubuntu Jammy)
 Assignee: (unassigned) => Robert Malz (rmalz)

** Changed in: linux (Ubuntu Mantic)
 Assignee: (unassigned) => Robert Malz (rmalz)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2036239

Title:
  Intel E810-XXV - NETDEV WATCHDOG: (ice): transmit queue timed out

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Jammy:
  In Progress
Status in linux source package in Mantic:
  In Progress

Bug description:
  [Impact]
   * Issue is causing transmit hang on E810 ports with bonding enabled.
   * Based on the provided logs, TX hang can last for even a couple of 
minutes, but in most scenarios, the network will be recovered after the ice 
driver performs a PF reset (TX hang handler routine).
   * Originally, the issue was observed during Tempest tests on a newly 
created OpenStack cluster, resulting in a lack of certification.
  
  [Fix]
  * Initially, a workaround has been proposed by Intel engineers to disable 
LAG initialization [1].
This change has been tested in an environment where reproduction is 
easily achieved.
After multiple iterations, no reproduction has been observed.
  * Shortly after, Intel proposed a patch [2] to disable LAG initialization 
if NVM does not expose proper capabilities.
  
  [Test Plan]
  * To reproduce the issue, over a 20-node cluster was used with Ceph-based 
storage. The problem could sometimes manifest while deploying a cluster or 
after the cluster was already deployed during the Tempest test run.
  * The issue could appear on a random node, making reproduction hard to 
achieve.
  * Multiple stress tests on single host with similar configuration did not 
trigger a reproduction.
  
  [Where problems could occur]
  * All ice drivers with ice_lag_event_handler registered can expose the 
issue. This handler is not implemented in 20.04
  * CVL4.2 and older NVM images for E810 does not expose SRIOV LAG 
capabilities (CVL4.3 wasn't checked) meaning at some point NVM with this 
capability will be released.
Although potentialy issue is caused by using features without proper FW 
support [2], we want to take a closer look once NVMs with proper support are 
introduced.

  [1] - 
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2036239/comments/40
  [2] - 
https://lists.osuosl.org/pipermail/intel-wired-lan/Week-of-Mon-20231211/038588.html
 4d50fcdc2476eef94c14c6761073af5667bb43b6

  [Other Info]
  * Issue could be reproduced on custom 6.2 jammy-hwe kernel with ice 
driver backported from mainline kernel from before patch [2] was added.
  * Original description of the case below:
  
  

  I'm having issues with an Intel E810-XXV card on a Dell server under
  Ubuntu Jammy.

  Details:

  - hardware --> a1:00.0 Ethernet controller: Intel Corporation Ethernet
  Controller E810-XXV for SFP (rev 02)

  - tested with both GA and HWE kernels (`5.15.0-83-generic #92` and
  `6.2.0-32-generic #32~22.04.1-Ubuntu`) with the same results.

  - using a bond over the two ports of the same card, at 25Gbps to two
  different switches, bond is using LACP with hash layer3+4 and fast
  timeout. But I believe the bug is not directly related to bonding as
  the problem seems to be in the interface.

  - machine installed by maas. No issues during installation, but at
  that time bond is not formed yet, later when linux is booted, the bond
  is formed and works without issues for a while

  - it works for about 2 to 3 hours fine, then the issue starts (may or
  may not be related to network load, but it seems that it is triggered
  by some tests that I run after openstack finishes installing)

  - one of the legs of the bond freezes and everything that would go to
  that lag is discarded, in and out, ping to random external hosts start
  losing every second packet

  - after some time you can see on the kernel log messages about "NETDEV
  WATCHDOG: enp161s0f0 (ice): transmit queue 166 timed out" and a stack
  trace

  - the switch does log that the bond is flapping
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Sep 12 20:05 seq
   crw-rw 1 root audio 116, 33 Sep 12 20:05 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.5
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit 

[Kernel-packages] [Bug 1837227] Re: systemd mount units fail during boot, while file system is correctly mounted

[Heitor Alves de Siqueira]

I've successfully confirmed that the mount issues seem to be fixed using
systemd from focal-proposed. Using the 'rep-tmpfs.sh' script, all
variants ran without any issues for multiple rounds. Basic testing on a
VM also looks good. Below are the versions used for this test:

$ dpkg -l systemd
...
||/ Name   Version   Architecture Description
+++-==-=--=
ii  systemd245.4-4ubuntu3.23 amd64system and service manager

$ uname -rv
5.4.0-169-generic #187-Ubuntu SMP Thu Nov 23 14:52:28 UTC 2023

I'll look into the reported regressions next, to confirm none of them
are caused by this LP.

** Tags removed: se-sponsor-halves verification-needed verification-needed-focal
** Tags added: verification-done verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1837227

Title:
  systemd mount units fail during boot, while file system is correctly
  mounted

Status in systemd:
  New
Status in Ubuntu Pro:
  In Progress
Status in Ubuntu Pro 18.04 series:
  In Progress
Status in linux package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Won't Fix
Status in systemd source package in Bionic:
  Won't Fix
Status in linux source package in Focal:
  Fix Released
Status in systemd source package in Focal:
  Fix Committed
Status in linux source package in Jammy:
  Fix Released
Status in systemd source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  systemd mount units fail during boot, and the system boots into emergency mode

  [Test Plan]
  This issue seems to happen randomly, and doesn't seem related to a specific 
mount unit.

  We've used a test script with good results during investigation to
  reproduce similar mount failures in a running system, and have seen a
  strong correlation between the script failures and the boot time mount
  failures.

  The attached 'rep-tmpfs.sh' script should be used to validate that
  mount points are working correctly under stress. One can run through
  the different variants as below:

  # ./rep-tmpfs.sh --variant-0
  # ./rep-tmpfs.sh --variant-1
  # ./rep-tmpfs.sh --variant-2
  # ./rep-tmpfs.sh --variant-3
  # ./rep-tmpfs.sh --variant-4

  All of these should run successfully without any reported errors.

  [Where problems could occur]
  The patches change the way systemd tracks and handles mount points in 
general, so potential regressions could affect other mount units. We should 
keep an eye out for any issues with mounting file systems, as well as rapid 
mount/unmount operations. Successful test runs with the reproducer script 
should increase reliability in having no new regressions.

  [Other Info]
  This has been tackled upstream with several attempts, which have resulted in 
the final patch from 2022:
    01400460ae16 core/mount: adjust deserialized state based on 
/proc/self/mountinfo

  For Bionic, systemd requires several dependency patches as below:
    6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a 
new unit cycle
    7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the 
other way round
    350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() 
results
    1d086a6e5972 mount: mark an existing "mounting" unit from 
/proc/self/mountinfo as "just_mounted"

  Additionally, the kernel also requires the following patches:
    28ca0d6d39ab list: introduce list_for_each_continue()
    9f6c61f96f2d proc/mounts: add cursor

  [Original Description]
  In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode 
with a failed mount unit (ext4 file system), while the corresponding file 
system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  
(https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1837227/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2038249] Re: The dump file parsing issue arises from structural changes in Linux kernel 6.2

[Heitor Alves de Siqueira]

As this will need fixing in the development release, I'll re-subscribe
the ubuntu-sponsors team. After this has been fixed there, we can take a
look at the SRU for the other stable releases.

Thanks, Chengen!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to crash in Ubuntu.
https://bugs.launchpad.net/bugs/2038249

Title:
  The dump file parsing issue arises from structural changes in Linux
  kernel 6.2

Status in crash package in Ubuntu:
  In Progress
Status in crash source package in Jammy:
  In Progress
Status in crash source package in Lunar:
  In Progress
Status in crash source package in Mantic:
  In Progress

Bug description:
  [Impact]
  Linux kernel 6.2 includes patches with structural changes that may render the 
crash utility unable to parse the dump file.
  ==
  d122019bf061 mm: Split slab into its own type
  401fb12c68c2 mm: Differentiate struct slab fields by sl*b implementations
  07f910f9b729 mm: Remove slab from struct page
  0d9b1ffefabe arm64: mm: make vabits_actual a build time constant if possible
  e36ce448a08d mm/slab: use kmalloc_node() for off slab freelist_idx_t array 
allocation
  130d4df57390 mm/sl[au]b: rearrange struct slab fields to allow larger rcu_head
  ac3b43283923 module: replace module_layout with module_memory
  b69f0aeb0689 pid: Replace struct pid 1-element array with flex-array
  ==

  [Fix]
  It is advisable to adopt commits that address the structural changes issue.
  ==
  [v8.0.0]
  14f8c460473c memory: Handle struct slab changes on Linux 5.17-rc1 and later
  5f390ed811b0 Fix for "kmem -s|-S" and "bt -F[F]" on Linux 5.17-rc1
  b89f9ccf511a Fix for "kmem -s|-S" on Linux 5.17+ with CONFIG_SLAB
  [v8.0.1]
  f02c8e87fccb arm64: use TCR_EL1_T1SZ to get the correct info if vabits_actual 
is missing
  [v8.0.2]
  d83df2fb66cd SLUB: Fix for offset change of struct slab members on Linux 
6.2-rc1
  df1f0cba729f x86_64: Fix for move of per-cpu variables into struct pcpu_hot
  120d6e89fc14 SLAB: Fix for "kmem -s|-S" options on Linux 6.1 and later
  ac96e17d1de5 SLAB: Fix for "kmem -s|-S" options on Linux 6.2-rc1 and later
  7750e61fdb2a Support module memory layout change on Linux 6.4
  88580068b7dd Fix failure of gathering task table on Linux 6.5-rc1 and later
  4ee56105881d Fix compilation error due to new strlcpy function that glibc 
added
  ==

  [Test Plan]
  1. Install the required packages and then proceed to reboot the machine.
  # sudo apt install crash linux-crashdump -y
  # reboot
  2. To check the status of kdump, use the `kdump-config show` command.
  # kdump-config show
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x6400
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-6.2.0-33-generic
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-6.2.0-33-generic
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-6.2.0-33-generic 
root=UUID=3e72f5d5-870b-4b8e-9a0d-8ba920391379 ro console=tty1 console=ttyS0 
reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 irqpoll 
usbcore.nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz
  3. To trigger a crash dump forcefully, execute the `echo c | sudo tee 
/proc/sysrq-trigger` command.
  4. Download the kernel .ddeb file, which will be used for analyzing the dump 
file.
  # sudo -i
  # cd /var/crash
  # pull-lp-ddebs linux-image-unsigned-$(uname -r)
  # dpkg-deb -x linux-image-unsigned-$(uname -r)-*.ddeb dbgsym-$(uname -r)
  5. Utilize the "crash" utility to parse and analyze the dump file.
  crash 8.0.0
  Copyright (C) 2002-2021  Red Hat, Inc.
  Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
  Copyright (C) 1999-2006  Hewlett-Packard Co
  Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
  Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
  Copyright (C) 2005, 2011, 2020-2021  NEC Corporation
  Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
  Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
  Copyright (C) 2015, 2021  VMware, Inc.
  This program is free software, covered by the GNU General Public License,
  and you are welcome to change it and/or distribute copies of it under
  certain conditions.  Enter "help copying" to see the conditions.
  This program has absolutely no warranty.  Enter "help warranty" for details.
   
  WARNING: VA_BITS: calculated: 46  vmcoreinfo: 48
  GNU gdb (GDB) 10.2
  Copyright (C) 2021 Free Software Foundation, Inc.
  License GPLv3+: GNU GPL version 3 or later 
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.
  Type "show copying" and "show warranty" for details.
  This GDB was configured as "aarch64-unknown-linux-gnu".
  Type "show configuration" for 

[Kernel-packages] [Bug 1837227] Re: systemd mount units fail during boot, while file system is correctly mounted

[Heitor Alves de Siqueira]

I've tested the kernel from focal-proposed, with the systemd packages
from my personal PPA (as the systemd patches aren't yet available in
focal-proposed).

All test variants from the rep-tmpfs.sh script ran succesfully, and
general smoke testing revealed no further issues.

ubuntu@z-rotomvm35:~$ uname -rv
5.4.0-164-generic #181-Ubuntu SMP Fri Sep 1 13:41:22 UTC 2023
ubuntu@z-rotomvm35:~$ dpkg -l systemd
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name   VersionArchitecture Description
+++-==-==--=
ii  systemd245.4-4ubuntu3.22+20230626dbg2 amd64system and 
service manager


** Tags removed: verification-needed-focal-linux
** Tags added: verification-done-focal-linux

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1837227

Title:
  systemd mount units fail during boot, while file system is correctly
  mounted

Status in systemd:
  New
Status in Ubuntu Pro:
  In Progress
Status in Ubuntu Pro 18.04 series:
  In Progress
Status in linux package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Won't Fix
Status in systemd source package in Bionic:
  Won't Fix
Status in linux source package in Focal:
  Fix Committed
Status in systemd source package in Focal:
  In Progress
Status in linux source package in Jammy:
  Fix Released
Status in systemd source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  systemd mount units fail during boot, and the system boots into emergency mode

  [Test Plan]
  This issue seems to happen randomly, and doesn't seem related to a specific 
mount unit.

  We've used a test script with good results during investigation to
  reproduce similar mount failures in a running system, and have seen a
  strong correlation between the script failures and the boot time mount
  failures.

  The attached 'rep-tmpfs.sh' script should be used to validate that
  mount points are working correctly under stress. One can run through
  the different variants as below:

  # ./rep-tmpfs.sh --variant-0
  # ./rep-tmpfs.sh --variant-1
  # ./rep-tmpfs.sh --variant-2
  # ./rep-tmpfs.sh --variant-3
  # ./rep-tmpfs.sh --variant-4

  All of these should run successfully without any reported errors.

  [Where problems could occur]
  The patches change the way systemd tracks and handles mount points in 
general, so potential regressions could affect other mount units. We should 
keep an eye out for any issues with mounting file systems, as well as rapid 
mount/unmount operations. Successful test runs with the reproducer script 
should increase reliability in having no new regressions.

  [Other Info]
  This has been tackled upstream with several attempts, which have resulted in 
the final patch from 2022:
    01400460ae16 core/mount: adjust deserialized state based on 
/proc/self/mountinfo

  For Bionic, systemd requires several dependency patches as below:
    6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a 
new unit cycle
    7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the 
other way round
    350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() 
results
    1d086a6e5972 mount: mark an existing "mounting" unit from 
/proc/self/mountinfo as "just_mounted"

  Additionally, the kernel also requires the following patches:
    28ca0d6d39ab list: introduce list_for_each_continue()
    9f6c61f96f2d proc/mounts: add cursor

  [Original Description]
  In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode 
with a failed mount unit (ext4 file system), while the corresponding file 
system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  
(https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1837227/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1837227] Re: systemd mount units fail during boot, while file system is correctly mounted

[Heitor Alves de Siqueira]

** Also affects: linux (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: linux (Ubuntu Bionic)
   Status: New => Won't Fix

** Changed in: linux (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Focal)
   Status: New => Fix Committed

** Changed in: linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Focal)
   Importance: Medium => High

** Changed in: systemd (Ubuntu Focal)
   Status: Fix Committed => In Progress

** Changed in: linux (Ubuntu)
   Status: New => Fix Released

** Changed in: linux (Ubuntu Jammy)
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1837227

Title:
  systemd mount units fail during boot, while file system is correctly
  mounted

Status in systemd:
  New
Status in Ubuntu Pro:
  In Progress
Status in Ubuntu Pro 18.04 series:
  In Progress
Status in linux package in Ubuntu:
  Fix Released
Status in systemd package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Won't Fix
Status in systemd source package in Bionic:
  Won't Fix
Status in linux source package in Focal:
  Fix Committed
Status in systemd source package in Focal:
  In Progress
Status in linux source package in Jammy:
  Fix Released
Status in systemd source package in Jammy:
  Fix Released

Bug description:
  [Impact]
  systemd mount units fail during boot, and the system boots into emergency mode

  [Test Plan]
  This issue seems to happen randomly, and doesn't seem related to a specific 
mount unit.

  We've used a test script with good results during investigation to
  reproduce similar mount failures in a running system, and have seen a
  strong correlation between the script failures and the boot time mount
  failures.

  The attached 'rep-tmpfs.sh' script should be used to validate that
  mount points are working correctly under stress. One can run through
  the different variants as below:

  # ./rep-tmpfs.sh --variant-0
  # ./rep-tmpfs.sh --variant-1
  # ./rep-tmpfs.sh --variant-2
  # ./rep-tmpfs.sh --variant-3
  # ./rep-tmpfs.sh --variant-4

  All of these should run successfully without any reported errors.

  [Where problems could occur]
  The patches change the way systemd tracks and handles mount points in 
general, so potential regressions could affect other mount units. We should 
keep an eye out for any issues with mounting file systems, as well as rapid 
mount/unmount operations. Successful test runs with the reproducer script 
should increase reliability in having no new regressions.

  [Other Info]
  This has been tackled upstream with several attempts, which have resulted in 
the final patch from 2022:
    01400460ae16 core/mount: adjust deserialized state based on 
/proc/self/mountinfo

  For Bionic, systemd requires several dependency patches as below:
    6a1d4d9fa6b9 core: properly reset all ExecStatus structures when entering a 
new unit cycle
    7eba1463dedc mount: flush out cycle state on DEAD→MOUNTED only, not the 
other way round
    350804867dbc mount: rescan /proc/self/mountinfo before processing waitid() 
results
    1d086a6e5972 mount: mark an existing "mounting" unit from 
/proc/self/mountinfo as "just_mounted"

  Additionally, the kernel also requires the following patches:
    28ca0d6d39ab list: introduce list_for_each_continue()
    9f6c61f96f2d proc/mounts: add cursor

  [Original Description]
  In Ubuntu 18.04 at least, we sometimes get a random server in emergency mode 
with a failed mount unit (ext4 file system), while the corresponding file 
system is in fact correctly mounted. It happens roughly once every 1000 reboots.

  It seems to be related with this bug :
  https://github.com/systemd/systemd/issues/10872

  Is it possible to apply the fix
  
(https://github.com/systemd/systemd/commit/350804867dbcc9b7ccabae1187d730d37e2d8a21)
  in Ubuntu 18.04 ?

  Thanks in advance.

To manage notifications about this bug go to:
https://bugs.launchpad.net/systemd/+bug/1837227/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2016881] Re: Building for Focal fails on riscv64

[Heitor Alves de Siqueira]

Thank you for the patch, xypron! A few comments on your debdiff:

I don't see the Bug-Ubuntu: tag in the patch, could you please add it so it's 
DEP-3 compliant?
Looking at the Origin: tag, it's also not clear whether the patch is a clean 
cherry-pick or needed to be backported with some changes. Could you please add 
the "upstream" or "backport" tag before the URL there? Thanks!

** Changed in: zfs-linux (Ubuntu Focal)
   Status: Confirmed => Incomplete

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016881

Title:
  Building for Focal fails on riscv64

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Invalid
Status in zfs-linux source package in Focal:
  Incomplete

Bug description:
  Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a
  single patch missing:
  https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2016881/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2016881] Re: Building for Focal fails on riscv64

[Heitor Alves de Siqueira]

** Tags added: se-sponsor-halves

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016881

Title:
  Building for Focal fails on riscv64

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Invalid
Status in zfs-linux source package in Focal:
  Confirmed

Bug description:
  Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a
  single patch missing:
  https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2016881/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2016881] Re: Building for Focal fails on riscv64

[Heitor Alves de Siqueira]

Note that the riscv64 patch is also missing from Focal (20.04), and the
package also fails to build there. Upstream included initial support in
zfs-2.0.0, so newer Ubuntu releases should have it by default:

$ git describe --contains 4254e407294b211f3399da2ee131b45fe9f4ac80
zfs-2.0.0-rc1~665

** Also affects: zfs-linux (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Also affects: zfs-linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Changed in: zfs-linux (Ubuntu)
   Status: New => Fix Released

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Focal)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Bionic)
   Importance: Undecided => Medium

** Changed in: zfs-linux (Ubuntu Focal)
   Importance: Undecided => Medium

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/2016881

Title:
  Building for Focal fails on riscv64

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed

Bug description:
  Building zfs-linux for riscv64 fails on Ubuntu 18.04. There is a
  single patch missing:
  https://github.com/openzfs/zfs/commit/4254e407294b211f3399da2ee131b45fe9f4ac80

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2016881/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

zfs-linux FTBS on riscv64, but that's been the case for the versions
from -updates for some time now. It's not related to this upload, but
rather to the ZFS version in focal not supporting riscv64:

In file included from ../../lib/libspl/include/sys/types.h:36,
 from ../../module/avl/avl.c:101:
../../lib/libspl/include/sys/isa_defs.h:200:2: error: #error "Unsupported ISA 
type"
  200 | #error "Unsupported ISA type"
  |  ^
../../lib/libspl/include/sys/isa_defs.h:216:2: error: #error "Neither 
_LITTLE_ENDIAN nor _BIG_ENDIAN are defined"
  216 | #error "Neither _LITTLE_ENDIAN nor _BIG_ENDIAN are defined"
  |  ^
make[5]: *** [Makefile:709: avl.lo] Error 1

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  Fix Committed

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"
   - e945e8d7f4fc "Restore FreeBSD sysctl processing for arc.min and arc.max"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

** Tags added: verification-needed-focal

** Tags added: se-sponsor-halves

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  Fix Committed

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"
   - e945e8d7f4fc "Restore FreeBSD sysctl processing for arc.min and arc.max"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

Thank you for the v2, Ghadi! I made some minor modifications to your
changelog entry (added "d/p/" before the patch files), and included a
"backport-notes:" tag in your second patch mentioning the FreeBSD-
specific sections were dropped from upstream.

With the v2, I ran the regression tests again and did some basic
validation. Both min and max ARC parameters were set as expected, and
invalid values are being ignored. I was able to set arc_min below RAM/32
as well as above RAM/2, and the same for arc_max.

Uploaded to focal, thank you for the help!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"
   - e945e8d7f4fc "Restore FreeBSD sysctl processing for arc.min and arc.max"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 2004262] Re: Intel E810 NICs driver in causing hangs when booting and bonds configured

[Heitor Alves de Siqueira]

tatus: New => Confirmed

** Changed in: linux (Ubuntu Kinetic)
   Status: New => Confirmed

** Changed in: linux (Ubuntu Jammy)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Kinetic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Lunar)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Jammy)
   Importance: Undecided => High

** Changed in: linux (Ubuntu Kinetic)
   Importance: Undecided => High

** Changed in: linux (Ubuntu Lunar)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/2004262

Title:
  Intel E810 NICs driver in causing hangs when booting and bonds
  configured

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Jammy:
  Confirmed
Status in linux source package in Kinetic:
  Confirmed
Status in linux source package in Lunar:
  Confirmed

Bug description:
  [Impact]
* Intel E810-family NICs cause system hangs when booting with bonding 
enabled
* This happens due to the driver unplugging auxiliary devices
* The unplug event happens under RTNL lock context, which causes a deadlock 
where the RDMA driver waits for the RNL lock to complete removal

  [Test Plan]
* Users have reported that after setting up bonding on switch and server 
side, the system will hang when starting network services

  [Fix]
* The upstream patch defers unplugging/re-plugging of the auxiliary device, 
so that it's not performed under the RTNL lock context.
* Fix was introduced by commit:
248401cb2c46 ice: avoid bonding causing auxiliary plug/unplug under 
RTNL lock

  [Regression Potential]
* Regressions would manifest in devices that support RDMA functionality and
  have been added to a bond
* We should look out for auxiliary devices that haven't been properly
  unplugged, or that cause further issues with
  ice_plug_aux_dev()/ice_unplug_aux_dev()

  
  [Original Description]
  jammy 22.04.1
  linux-image-generic 5.15.0-58-generic
  Intel E810-XXV Dual Port NICs in Dell PowerEdge 650

  - 5.15 in jammy -> reproducible
  - 5.19 in hwe-edge -> reproducible
  - 6.2.rc6 in the mainline build -> works
  - Intel's ice driver 1.10.1.2.2 -> works

  After beonding is enabled on switch and server side, the system will
  hang at initialing ubuntu.  The kernel loads but around starting the
  Network Services the system can hang for sometimes 5 minutes, and in
  other cases, indefinitely.

  The message of:

  echo 0 > /proc/sys/kernel/hung_task_timeout_sec”  systemd-resolve
  blocked for more than 120 seconds

  appears, and eventually the Network services just attempts to start
  and never does.  This is with or without DHCP enabled.

  Tried this same setup with the hwe-22.04, hwe-20.04, hwe-22.04-ege and
  linux-oem kernels and all exhibit the same failure.

  To work around this. installing the Intel 'ice' driver of version
  1.10.1.2.2 works.  The system doesn't even remotely hang at startup
  and all networking functions remain working (ping, DNS, general
  accessibility).

  The driver can be found at 
https://downloadmirror.intel.com/763930/ice-1.10.1.2.2.tar.gz
  ---
  ProblemType: Bug
  AlsaDevices:
   total 0
   crw-rw 1 root audio 116,  1 Jan 31 13:08 seq
   crw-rw 1 root audio 116, 33 Jan 31 13:08 timer
  AplayDevices: Error: [Errno 2] No such file or directory: 'aplay'
  ApportVersion: 2.20.11-0ubuntu82.3
  Architecture: amd64
  ArecordDevices: Error: [Errno 2] No such file or directory: 'arecord'
  AudioDevicesInUse: Error: command ['fuser', '-v', '/dev/snd/seq', 
'/dev/snd/timer'] failed with exit code 1:
  CRDA: N/A
  CasperMD5json:
   {
     "result": "skip"
   }DistroRelease: Ubuntu 22.04
  InstallationDate: Installed on 2023-01-27 (3 days ago)InstallationMedia: 
Ubuntu-Server 22.04.1 LTS "Jammy Jellyfish" - Release amd64 (20220809)
  IwConfig: Error: [Errno 2] No such file or directory: 'iwconfig'
  MachineType: Dell Inc. PowerEdge R650
  Package: linux (not installed)
  PciMultimedia:

  ProcFB: 0 mgag200drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.15.0-58-generic 
root=UUID=668aab7c-abe9-434b-a810-acc6eab76cbc ro fsck.mode=skip
  ProcVersionSignature: Ubuntu 5.15.0-58.64-generic 5.15.74
  RelatedPackageVersions:
   linux-restricted-modules-5.15.0-58-generic N/A
   linux-backports-modules-5.15.0-58-generic  N/A
   linux-firmware 20220329.git681281e4-0ubuntu3.9
  RfKill: Error: [Errno 2] No such file or directory: 'rfkill'Tags:  jammy 
uec-images
  Uname: Linux 5.15.0-58-generic x86_64
  UpgradeStatus: No upgrade log present (probably fresh install)
  UserGroups: N/A
  _MarkForUpload: True
  dmi.bios.date: 09/14/2022
  dmi.bios.release

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

Thanks, Dimitri! I've tested Ghadi's patches and confirmed they work and
don't trigger any regressions in the ZFS test suite.

I've asked Ghadi to rework some portions of his backport in order to be closer 
to the upstream patches though, so we should have a v2 soon. We'll make sure 
that previous changelog entries aren't modified, and I'll re-run the ZFS 
regression tests against the new version as well.
Thank you for the work on this one, Ghadi!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"
   - e945e8d7f4fc "Restore FreeBSD sysctl processing for arc.min and arc.max"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

** Patch removed: "lp1964992-focal.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+attachment/5571326/+files/lp1964992-focal.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

No worries, thank you for the patience, Robie!

With further testing, we've found out that there's an additional upstream
patch required for Focal, which fixes the "inversion" when setting the ARC
limits.

I'll amend the description and re-spin this SRU to include the proper
fixes. I'll then re-upload, after running the regression tests on the new 
version.

** Changed in: zfs-linux (Ubuntu Focal)
   Status: Triaged => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  In Progress

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1978333] Re: Remove "ata_piix.prefer_ms_hyperv=0" parameter

[Heitor Alves de Siqueira]

** Tags removed: verification-needed
** Tags added: verification-done

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to makedumpfile in Ubuntu.
https://bugs.launchpad.net/bugs/1978333

Title:
  Remove "ata_piix.prefer_ms_hyperv=0" parameter

Status in kdump-tools package in Ubuntu:
  Fix Committed
Status in makedumpfile package in Ubuntu:
  Invalid
Status in kdump-tools source package in Bionic:
  Invalid
Status in makedumpfile source package in Bionic:
  Fix Committed
Status in kdump-tools source package in Focal:
  Invalid
Status in makedumpfile source package in Focal:
  Fix Committed
Status in kdump-tools source package in Impish:
  Invalid
Status in makedumpfile source package in Impish:
  Invalid
Status in kdump-tools source package in Jammy:
  Fix Committed
Status in makedumpfile source package in Jammy:
  Invalid
Status in kdump-tools source package in Kinetic:
  Fix Committed
Status in makedumpfile source package in Kinetic:
  Invalid

Bug description:
  [Impact]

  Azure VM instances hit I/O error on boot causing kernel crash

  [Test Plan]

  Create Ubuntu Marketplace VM on Azure

  ```
  ssh -i .ssh/id_rsa ubuntu@ipaddr 
  ```

  Install crash dump utilities (from guide:
  https://ubuntu.com/server/docs/kernel-crash-dump)

  apt-get install kdump-tools

  Say (y) to all questions during install

  kdump-config show 
  *shows the vm is not yet ready to kdump 

  root@bionic3: kdump-config show 
   * no crashkernel= parameter in the kernel cmdline
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 
 /var/lib/kdump/vmlinuz
  kdump initrd: 
 /var/lib/kdump/initrd.img
  current state:Not ready to kdump

  kexec command:
no kexec command recorded

  
  Reboot the VM

  sudo su

  As root on the VM after reboot:

  kdump-config show

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img 
/var/lib/kdump/vmlinuz

  
  #verify kdump is on

  cat /proc/cmdline
  ... crashkernel=512M-:192M

  dmesg | grep -i crash
  [0.071660] kexec: Reserving the low 1M of memory for crashkernel
  [0.269823] Reserving 192MB of memory at 640MB for crashkernel (System 
RAM: 4095MB)

  cat /proc/sys/kernel/sysrq
  # make sure this value is greater than 0
  #set it to 1
  sudo sysctl -w kernel.sysrq=1

  the directory of /var/crash should have no crashes yet as well.

  Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command:
  # perform crash
  sudo su
  echo c > /proc/sysrq-trigger

  After a couple of minuties 
  Open new terminal and try to ssh to azure  VM, 
  It does not succeed 

  Force a reboot of the VM through the portal or serial console 
  #kdump doesnt work and hangs indefinitely
  #force reboot VM from Azure console

  # verify package version of kdump-tool
  # verify parameter is not listed in the kdump-config show output 
  kdump-config unload
  kdump-config load 
  kdump-config show 

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

  
  #trigger another crash
  echo c > /proc/sysrq-trigger

  # open new terminal and ssh back into vm

  cd /var/crash

  #verify a dump was created 
  linux-image-5.4.0-1086-azure-202208041658.crash

  [Where Problems Could Occur]

  This change modifies the debian/rules.
  The package could fail to build properly if mistyped.

  [Other]
  Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" 
parameter.

  target series - Bionic, Focal, Jammy

  upstream patch

  https://salsa.debian.org/debian/kdump-
  tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

Hi Robie,

You're right, the patch does essentially invert the problem. This is
still the behavior upstream, and it currently works like you mentioned:
if the user tries to set a min above the default max (ramsize/2), it
fails.

I'm working on a patch to propose upstream that should fix this. We
should be setting min/max values as a pairs else we'll run into a
similar issue as the one reported here. I'm also going to double check
other tunables to see if they exhibit similar issues, so we can avoid
further problems on those too.

For this particular LP bug, do you think we should wait until a "proper"
fix upstream? I do understand the point about breaking setups relying on
the current min/max behavior, but that will also happen when upgrading
to newer releases. My (subjective) opinion is that users trying to
reduce ZFS memory footprint are much more common than the alternative,
and for high memory systems this is currently not possible due to this
bug. What do you think?

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  Incomplete

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1978333] Re: Remove "ata_piix.prefer_ms_hyperv=0" parameter

[Heitor Alves de Siqueira]

Failed autopkgtest result for i386 seemed to be due to a flaky test, not an
actual regression. Test passed after another run, and no further autopkgtest
regressions have been reported.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to makedumpfile in Ubuntu.
https://bugs.launchpad.net/bugs/1978333

Title:
  Remove "ata_piix.prefer_ms_hyperv=0" parameter

Status in kdump-tools package in Ubuntu:
  Fix Committed
Status in makedumpfile package in Ubuntu:
  Invalid
Status in kdump-tools source package in Bionic:
  Invalid
Status in makedumpfile source package in Bionic:
  Fix Committed
Status in kdump-tools source package in Focal:
  Invalid
Status in makedumpfile source package in Focal:
  Fix Committed
Status in kdump-tools source package in Impish:
  Invalid
Status in makedumpfile source package in Impish:
  Invalid
Status in kdump-tools source package in Jammy:
  In Progress
Status in makedumpfile source package in Jammy:
  Invalid
Status in kdump-tools source package in Kinetic:
  Fix Committed
Status in makedumpfile source package in Kinetic:
  Invalid

Bug description:
  [Impact]

  Azure VM instances hit I/O error on boot causing kernel crash

  [Test Plan]

  Create Ubuntu Marketplace VM on Azure

  ```
  ssh -i .ssh/id_rsa ubuntu@ipaddr 
  ```

  Install crash dump utilities (from guide:
  https://ubuntu.com/server/docs/kernel-crash-dump)

  apt-get install kdump-tools

  Say (y) to all questions during install

  kdump-config show 
  *shows the vm is not yet ready to kdump 

  root@bionic3: kdump-config show 
   * no crashkernel= parameter in the kernel cmdline
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 
 /var/lib/kdump/vmlinuz
  kdump initrd: 
 /var/lib/kdump/initrd.img
  current state:Not ready to kdump

  kexec command:
no kexec command recorded

  
  Reboot the VM

  sudo su

  As root on the VM after reboot:

  kdump-config show

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img 
/var/lib/kdump/vmlinuz

  
  #verify kdump is on

  cat /proc/cmdline
  ... crashkernel=512M-:192M

  dmesg | grep -i crash
  [0.071660] kexec: Reserving the low 1M of memory for crashkernel
  [0.269823] Reserving 192MB of memory at 640MB for crashkernel (System 
RAM: 4095MB)

  cat /proc/sys/kernel/sysrq
  # make sure this value is greater than 0
  #set it to 1
  sudo sysctl -w kernel.sysrq=1

  the directory of /var/crash should have no crashes yet as well.

  Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command:
  # perform crash
  sudo su
  echo c > /proc/sysrq-trigger

  After a couple of minuties 
  Open new terminal and try to ssh to azure  VM, 
  It does not succeed 

  Force a reboot of the VM through the portal or serial console 
  #kdump doesnt work and hangs indefinitely
  #force reboot VM from Azure console

  # verify package version of kdump-tool
  # verify parameter is not listed in the kdump-config show output 
  kdump-config unload
  kdump-config load 
  kdump-config show 

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

  
  #trigger another crash
  echo c > /proc/sysrq-trigger

  # open new terminal and ssh back into vm

  cd /var/crash

  #verify a dump was created 
  linux-image-5.4.0-1086-azure-202208041658.crash

  [Where Problems Could Occur]

  This change modifies the debian/rules.
  The package could fail to build properly if mistyped.

  [Other]
  Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" 
parameter.

  target series - Bionic, Focal, Jammy

  

[Kernel-packages] [Bug 1978333] Re: Remove "ata_piix.prefer_ms_hyperv=0" parameter

[Heitor Alves de Siqueira]

** Tags removed: sts-sponsor-halves
** Tags added: se-sponsor-halves

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to makedumpfile in Ubuntu.
https://bugs.launchpad.net/bugs/1978333

Title:
  Remove "ata_piix.prefer_ms_hyperv=0" parameter

Status in kdump-tools package in Ubuntu:
  Fix Committed
Status in makedumpfile package in Ubuntu:
  Invalid
Status in kdump-tools source package in Bionic:
  Invalid
Status in makedumpfile source package in Bionic:
  Fix Committed
Status in kdump-tools source package in Focal:
  Invalid
Status in makedumpfile source package in Focal:
  Fix Committed
Status in kdump-tools source package in Impish:
  Invalid
Status in makedumpfile source package in Impish:
  Invalid
Status in kdump-tools source package in Jammy:
  In Progress
Status in makedumpfile source package in Jammy:
  Invalid
Status in kdump-tools source package in Kinetic:
  Fix Committed
Status in makedumpfile source package in Kinetic:
  Invalid

Bug description:
  [Impact]

  Azure VM instances hit I/O error on boot causing kernel crash

  [Test Plan]

  Create Ubuntu Marketplace VM on Azure

  ```
  ssh -i .ssh/id_rsa ubuntu@ipaddr 
  ```

  Install crash dump utilities (from guide:
  https://ubuntu.com/server/docs/kernel-crash-dump)

  apt-get install kdump-tools

  Say (y) to all questions during install

  kdump-config show 
  *shows the vm is not yet ready to kdump 

  root@bionic3: kdump-config show 
   * no crashkernel= parameter in the kernel cmdline
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 
 /var/lib/kdump/vmlinuz
  kdump initrd: 
 /var/lib/kdump/initrd.img
  current state:Not ready to kdump

  kexec command:
no kexec command recorded

  
  Reboot the VM

  sudo su

  As root on the VM after reboot:

  kdump-config show

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb ata_piix.prefer_ms_hyperv=0" --initrd=/var/lib/kdump/initrd.img 
/var/lib/kdump/vmlinuz

  
  #verify kdump is on

  cat /proc/cmdline
  ... crashkernel=512M-:192M

  dmesg | grep -i crash
  [0.071660] kexec: Reserving the low 1M of memory for crashkernel
  [0.269823] Reserving 192MB of memory at 640MB for crashkernel (System 
RAM: 4095MB)

  cat /proc/sys/kernel/sysrq
  # make sure this value is greater than 0
  #set it to 1
  sudo sysctl -w kernel.sysrq=1

  the directory of /var/crash should have no crashes yet as well.

  Outcome with "ata_piix.prefer_ms_hyperv=0" in kexec command:
  # perform crash
  sudo su
  echo c > /proc/sysrq-trigger

  After a couple of minuties 
  Open new terminal and try to ssh to azure  VM, 
  It does not succeed 

  Force a reboot of the VM through the portal or serial console 
  #kdump doesnt work and hangs indefinitely
  #force reboot VM from Azure console

  # verify package version of kdump-tool
  # verify parameter is not listed in the kdump-config show output 
  kdump-config unload
  kdump-config load 
  kdump-config show 

  kdump-config show 
  DUMP_MODE:kdump
  USE_KDUMP:1
  KDUMP_SYSCTL: kernel.panic_on_oops=1
  KDUMP_COREDIR:/var/crash
  crashkernel addr: 0x3200
 /var/lib/kdump/vmlinuz: symbolic link to /boot/vmlinuz-5.4.0-1086-azure
  kdump initrd: 
 /var/lib/kdump/initrd.img: symbolic link to 
/var/lib/kdump/initrd.img-5.4.0-1086-azure
  current state:ready to kdump

  kexec command:
/sbin/kexec -p --command-line="BOOT_IMAGE=/boot/vmlinuz-5.4.0-1086-azure 
root=UUID=143c811b-9b9c-48f3-b0c8-040f6e65f50a ro console=tty1 console=ttyS0 
earlyprintk=ttyS0 reset_devices systemd.unit=kdump-tools-dump.service nr_cpus=1 
irqpoll nousb" --initrd=/var/lib/kdump/initrd.img /var/lib/kdump/vmlinuz

  
  #trigger another crash
  echo c > /proc/sysrq-trigger

  # open new terminal and ssh back into vm

  cd /var/crash

  #verify a dump was created 
  linux-image-5.4.0-1086-azure-202208041658.crash

  [Where Problems Could Occur]

  This change modifies the debian/rules.
  The package could fail to build properly if mistyped.

  [Other]
  Back-porting a fix from upstream to remove "ata_piix.prefer_ms_hyperv=0" 
parameter.

  target series - Bionic, Focal, Jammy

  upstream patch

  https://salsa.debian.org/debian/kdump-
  tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503

  

[Kernel-packages] [Bug 1978333] Re: Remove "ata_piix.prefer_ms_hyperv=0" parameter

[Heitor Alves de Siqueira]

** Tags added: sts sts-sponsor-halves

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to makedumpfile in Ubuntu.
https://bugs.launchpad.net/bugs/1978333

Title:
  Remove "ata_piix.prefer_ms_hyperv=0" parameter

Status in kdump-tools package in Ubuntu:
  Fix Committed
Status in makedumpfile package in Ubuntu:
  New
Status in kdump-tools source package in Bionic:
  Invalid
Status in makedumpfile source package in Bionic:
  New
Status in kdump-tools source package in Focal:
  Invalid
Status in makedumpfile source package in Focal:
  New
Status in kdump-tools source package in Impish:
  New
Status in makedumpfile source package in Impish:
  New
Status in kdump-tools source package in Jammy:
  New
Status in makedumpfile source package in Jammy:
  New
Status in kdump-tools source package in Kinetic:
  Fix Committed
Status in makedumpfile source package in Kinetic:
  New

Bug description:
  Back-porting a fix from upstream to remove
  "ata_piix.prefer_ms_hyperv=0" parameter.

  target series - Kinetic-> Bionic

  upstream patch

  https://salsa.debian.org/debian/kdump-
  tools/-/commit/b1bac9396ddbbce3817c34be3161630698e4a503

  *Note: There are two source packages needed changes, kdump-tools for Impish 
-> Kinetic 
  and makedumpfile for series Focal -> Bionic 


  Thanks!

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/kdump-tools/+bug/1978333/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1942624] Re: NVMe devices fail to probe due to ACPI power state change

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  * Specific NVMe devices fail to probe and become unusable after boot
  * Caused by an ACPI regression that doesn't correctly handle power states
  * Upstream regression commit:
    7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally
  * Regression window for Ubuntu kernels includes 5.13 and 5.14
  
  [Test Plan]
  * Boot affected kernel and validate whether NVMe device is usuble
  * Check kernel logs for failed probe message:
    "can't change power state from D3Cold to D0 (config space inaccessible"
  
  [Fix]
  * Fixed by not turning off power resources in unknown state
  * Fix was introduced by commit:
    bc2836859643 ACPI: PM: Do not turn off power resources in unknown state
- * Ubuntu kernels starting with 5.15 (Jammy) not affected, as they contain the 
fix above
+ * Kernels starting with 5.15 (e.g. Jammy) not affected, as they already 
contain the fix above
  
  [Regression Potential]
  * NVMe devices continue failing to probe
  * Other devices become unusable after power state changes
  * Further regressions would affect power state of devices, possibly after boot
  
  --
  [Original Description]
  NVME "can't change power state from D3Cold to D0 (config space inaccessible)"
  
  Bug with kernels after version 5.11.0-18 on Lenovo Ideapad 330-15ICH.
  The NVME drive with my root partition cannot be mounted at boot with an
  error "can't change power state from D3Cold to D0 (config space
  inaccessible)". I'm willing to help find a root cause if I don't need to
  spent too many hours. All Ubuntu kernels after 5.11.0-18 exhibit this
  bug, but I could boot properly with the official linux kernel 5.13.0.
  Thanks a lot for your help
  
  ProblemType: Bug
  DistroRelease: Ubuntu 21.04
  Package: linux-image-5.11.0-18-generic 5.11.0-18.19
  ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
  Uname: Linux 5.11.0-18-generic x86_64
  ApportVersion: 2.20.11-0ubuntu65.1
  Architecture: amd64
  AudioDevicesInUse:
   USERPID ACCESS COMMAND
   /dev/snd/controlC0:  chris  7503 F pulseaudio
   /dev/snd/pcmC0D0p:   chris  7503 F...m pulseaudio
  CasperMD5CheckResult: unknown
  CurrentDesktop: ubuntu:GNOME
  Date: Fri Sep  3 18:46:35 2021
  InstallationDate: Installed on 2019-07-17 (779 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 5986:210e Acer, Inc EasyCamera
   Bus 001 Device 004: ID 8087:0a2a Intel Corp. Bluetooth wireless interface
   Bus 001 Device 007: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: LENOVO 81FK
  ProcFB: 0 i915drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-18-generic 
root=UUID=9d963312-3a16-428d-8efd-f1323c6528f1 ro quiet nosplash 
crashkernel=512M-:192M
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-18-generic N/A
   linux-backports-modules-5.11.0-18-generic  N/A
   linux-firmware 1.197.3
  SourcePackage: linux
  UpgradeStatus: Upgraded to hirsute on 2021-06-03 (92 days ago)
  dmi.bios.date: 10/24/2018
  dmi.bios.release: 1.29
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7ZCN29WW
  dmi.board.asset.tag: NO Asset Tag
  dmi.board.name: LNVNB161216
  dmi.board.vendor: LENOVO
  dmi.board.version: SDK0J40709 WIN
  dmi.chassis.asset.tag: NO Asset Tag
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Lenovo ideapad 330-15ICH
  dmi.ec.firmware.release: 1.29
  dmi.modalias: 
dmi:bvnLENOVO:bvr7ZCN29WW:bd10/24/2018:br1.29:efr1.29:svnLENOVO:pn81FK:pvrLenovoideapad330-15ICH:rvnLENOVO:rnLNVNB161216:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad330-15ICH:
  dmi.product.family: ideapad 330-15ICH
  dmi.product.name: 81FK
  dmi.product.sku: LENOVO_MT_81FK_BU_idea_FM_ideapad 330-15ICH
  dmi.product.version: Lenovo ideapad 330-15ICH
  dmi.sys.vendor: LENOVO

** Description changed:

  [Impact]
  * Specific NVMe devices fail to probe and become unusable after boot
  * Caused by an ACPI regression that doesn't correctly handle power states
  * Upstream regression commit:
    7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally
  * Regression window for Ubuntu kernels includes 5.13 and 5.14
  
  [Test Plan]
- * Boot affected kernel and validate whether NVMe device is usuble
+ * Boot affected kernel and validate whether NVMe device is usable
  * Check kernel logs for failed probe message:
-   "can't change power state from D3Cold to D0 (config space inaccessible"
+   "can't change power state from D3Cold to D0 (config space inaccessible)"
  
  [Fix]
  * Fixed by not turning off power resources in unknown state
  * Fix was introduced by commit:
    bc2836859643 ACPI: PM: Do not turn off power resources in unknown state
  * Kernels starting with 5.15 (e.g. Jammy) not affected, as they already 
contain the fix 

[Kernel-packages] [Bug 1942624] Re: NVMe devices fail to probe due to ACPI power state change

[Heitor Alves de Siqueira]

** Summary changed:

- NVME "can't change power state from D3Cold to D0 (config space inaccessible)"
+ NVMe devices fail to probe due to ACPI power state change

** Description changed:

+ [Impact]
+ * Specific NVMe devices fail to probe and become unusable after boot
+ * Caused by an ACPI regression that doesn't correctly handle power states
+ * Upstream regression commit:
+   7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally
+ 
+ [Test Plan]
+ * Boot affected kernel and validate whether NVMe device is usuble
+ * Check kernel logs for failed probe message:
+   "can't change power state from D3Cold to D0 (config space inaccessible"
+ 
+ [Fix]
+ * Fixed by not turning off power resources in unknown state
+ * Fix was introduced by commit:
+   bc2836859643 ACPI: PM: Do not turn off power resources in unknown state
+ 
+ [Regression Potential]
+ * NVMe devices continue failing to probe
+ * Other devices become unusable after power state changes
+ * Further regressions would affect power state of devices, possibly after boot
+ 
+ --
+ [Original Description]
+ NVME "can't change power state from D3Cold to D0 (config space inaccessible)"
+ 
  Bug with kernels after version 5.11.0-18 on Lenovo Ideapad 330-15ICH.
  The NVME drive with my root partition cannot be mounted at boot with an
  error "can't change power state from D3Cold to D0 (config space
  inaccessible)". I'm willing to help find a root cause if I don't need to
  spent too many hours. All Ubuntu kernels after 5.11.0-18 exhibit this
  bug, but I could boot properly with the official linux kernel 5.13.0.
  Thanks a lot for your help
  
  ProblemType: Bug
  DistroRelease: Ubuntu 21.04
  Package: linux-image-5.11.0-18-generic 5.11.0-18.19
  ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
  Uname: Linux 5.11.0-18-generic x86_64
  ApportVersion: 2.20.11-0ubuntu65.1
  Architecture: amd64
  AudioDevicesInUse:
   USERPID ACCESS COMMAND
   /dev/snd/controlC0:  chris  7503 F pulseaudio
   /dev/snd/pcmC0D0p:   chris  7503 F...m pulseaudio
  CasperMD5CheckResult: unknown
  CurrentDesktop: ubuntu:GNOME
  Date: Fri Sep  3 18:46:35 2021
  InstallationDate: Installed on 2019-07-17 (779 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 5986:210e Acer, Inc EasyCamera
   Bus 001 Device 004: ID 8087:0a2a Intel Corp. Bluetooth wireless interface
   Bus 001 Device 007: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: LENOVO 81FK
  ProcFB: 0 i915drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-18-generic 
root=UUID=9d963312-3a16-428d-8efd-f1323c6528f1 ro quiet nosplash 
crashkernel=512M-:192M
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-18-generic N/A
   linux-backports-modules-5.11.0-18-generic  N/A
   linux-firmware 1.197.3
  SourcePackage: linux
  UpgradeStatus: Upgraded to hirsute on 2021-06-03 (92 days ago)
  dmi.bios.date: 10/24/2018
  dmi.bios.release: 1.29
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7ZCN29WW
  dmi.board.asset.tag: NO Asset Tag
  dmi.board.name: LNVNB161216
  dmi.board.vendor: LENOVO
  dmi.board.version: SDK0J40709 WIN
  dmi.chassis.asset.tag: NO Asset Tag
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Lenovo ideapad 330-15ICH
  dmi.ec.firmware.release: 1.29
  dmi.modalias: 
dmi:bvnLENOVO:bvr7ZCN29WW:bd10/24/2018:br1.29:efr1.29:svnLENOVO:pn81FK:pvrLenovoideapad330-15ICH:rvnLENOVO:rnLNVNB161216:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad330-15ICH:
  dmi.product.family: ideapad 330-15ICH
  dmi.product.name: 81FK
  dmi.product.sku: LENOVO_MT_81FK_BU_idea_FM_ideapad 330-15ICH
  dmi.product.version: Lenovo ideapad 330-15ICH
  dmi.sys.vendor: LENOVO

** Description changed:

  [Impact]
  * Specific NVMe devices fail to probe and become unusable after boot
  * Caused by an ACPI regression that doesn't correctly handle power states
  * Upstream regression commit:
-   7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally
+   7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally
+ * Regression window for Ubuntu kernels includes 5.13 and 5.14
  
  [Test Plan]
  * Boot affected kernel and validate whether NVMe device is usuble
  * Check kernel logs for failed probe message:
-   "can't change power state from D3Cold to D0 (config space inaccessible"
+   "can't change power state from D3Cold to D0 (config space inaccessible"
  
  [Fix]
  * Fixed by not turning off power resources in unknown state
  * Fix was introduced by commit:
-   bc2836859643 ACPI: PM: Do not turn off power resources in unknown state
+   bc2836859643 ACPI: PM: Do not turn off power resources in unknown state
+ * Ubuntu kernels starting with 5.15 (Jammy) not affected, as they contain the 
fix above
  
  

[Kernel-packages] [Bug 1964512] Re: Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

Validated for Ubuntu Impish with the current kernel from impish-proposed.
Tested on a 10G NIC that's affected by this patch (device ID 0x0903). iperf3 
shows that we're able to almost saturate the NIC successfully in both standard 
and reverse sender mode:

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.1 GBytes  8.69 Gbits/sec  990 sender
[  5]   0.00-9.99   sec  10.1 GBytes  8.69 Gbits/sec  receiver

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec  155 sender
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec  receiver

ubuntu@duduo:~$ uname -rv
5.13.0-40-generic #45-Ubuntu SMP Tue Mar 29 14:48:14 UTC 2022

Other basic smoke testing confirms no major regressions.


** Tags removed: verification-needed-impish
** Tags added: verification-done-impish

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  Fix Committed
Status in linux source package in Focal:
  Fix Committed
Status in linux source package in Impish:
  Fix Committed
Status in linux source package in Jammy:
  Fix Committed

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964512] Re: Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

In addition to my own validations above, impacted users that have 40G
NICs have also confirmed the patch behaves as expected without major
regressions.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  Fix Committed
Status in linux source package in Focal:
  Fix Committed
Status in linux source package in Impish:
  Fix Committed
Status in linux source package in Jammy:
  Fix Committed

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964512] Re: Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

Validated for Ubuntu Bionic with the current HWE kernel from bionic-proposed.
Tested on a 10G NIC that's affected by this patch (device ID 0x0903). iperf3 
shows that we're able to almost saturate the NIC successfully in both standard 
and reverse sender mode:

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.2 GBytes  8.80 Gbits/sec  376 sender
[  5]   0.00-10.00  sec  10.2 GBytes  8.80 Gbits/sec  receiver

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec0 sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec  receiver

ubuntu@duduo:~$ uname -rv
5.4.0-108-generic #122~18.04.1-Ubuntu SMP Wed Apr 6 16:57:12 UTC 2022

Other basic smoke testing confirms no major regressions.


** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  Fix Committed
Status in linux source package in Focal:
  Fix Committed
Status in linux source package in Impish:
  Fix Committed
Status in linux source package in Jammy:
  Fix Committed

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964512] Re: Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

Validated for Ubuntu Focal with the current kernel from focal-proposed.
Tested on a 10G NIC that's affected by this patch (device ID 0x0903). iperf3 
shows that we're able to almost saturate the NIC successfully in both standard 
and reverse sender mode:

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.3 GBytes  8.83 Gbits/sec  266 sender
[  5]   0.00-10.00  sec  10.3 GBytes  8.82 Gbits/sec  receiver

[ ID] Interval   Transfer Bitrate Retr
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec0 sender
[  5]   0.00-10.00  sec  10.8 GBytes  9.30 Gbits/sec  receiver

ubuntu@duduo:~$ uname -rv
5.4.0-109-generic #123-Ubuntu SMP Fri Apr 8 09:10:54 UTC 2022


** Tags removed: verification-needed-focal
** Tags added: verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  Fix Committed
Status in linux source package in Focal:
  Fix Committed
Status in linux source package in Impish:
  Fix Committed
Status in linux source package in Jammy:
  Fix Committed

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1942624] Re: NVME "can't change power state from D3Cold to D0 (config space inaccessible)"

[Heitor Alves de Siqueira]

Hi all,

This seems to be related to a regression introduced by the following commit:
* 7e4fdeafa61f ACPI: power: Turn off unused power resources unconditionally

The fix seems to indeed be the commit @sclarkson mentioned:
* bc2836859643 ACPI: PM: Do not turn off power resources in unknown state

Looking at our kernels, it seems the affected version window is between v5.12 
and v5.15. I've tagged the relevant packages in this bug, and have built a set 
of test kernels to validate the fixes. For the 5.13 kernels, we additionally 
require the commit below:
* 9b7ff25d129d ACPI: power: Refine turning off unused power resources

I'd greatly appreciate if anyone affected by this could give these test
kernels a try, as I don't have the appropriate hardware to test it
myself. I've uploaded packages for Focal-HWE/Impish (5.13) and Focal-OEM
(5.14) to a public PPA, but please consider these packages for testing
purposes only. If you can reproduce this bug on a different kernel,
please add a comment and I'll look into backporting the required patches
there as well.

Cheers,
Heitor

[0] https://launchpad.net/~halves/+archive/ubuntu/test-1942624

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-oem-5.14 in Ubuntu.
https://bugs.launchpad.net/bugs/1942624

Title:
  NVME "can't change power state from D3Cold to D0 (config space
  inaccessible)"

Status in linux package in Ubuntu:
  Fix Released
Status in linux-oem-5.14 package in Ubuntu:
  In Progress
Status in linux source package in Focal:
  Invalid
Status in linux-oem-5.14 source package in Focal:
  In Progress
Status in linux source package in Impish:
  In Progress
Status in linux-oem-5.14 source package in Impish:
  Invalid

Bug description:
  Bug with kernels after version 5.11.0-18 on Lenovo Ideapad 330-15ICH.
  The NVME drive with my root partition cannot be mounted at boot with
  an error "can't change power state from D3Cold to D0 (config space
  inaccessible)". I'm willing to help find a root cause if I don't need
  to spent too many hours. All Ubuntu kernels after 5.11.0-18 exhibit
  this bug, but I could boot properly with the official linux kernel
  5.13.0. Thanks a lot for your help

  ProblemType: Bug
  DistroRelease: Ubuntu 21.04
  Package: linux-image-5.11.0-18-generic 5.11.0-18.19
  ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
  Uname: Linux 5.11.0-18-generic x86_64
  ApportVersion: 2.20.11-0ubuntu65.1
  Architecture: amd64
  AudioDevicesInUse:
   USERPID ACCESS COMMAND
   /dev/snd/controlC0:  chris  7503 F pulseaudio
   /dev/snd/pcmC0D0p:   chris  7503 F...m pulseaudio
  CasperMD5CheckResult: unknown
  CurrentDesktop: ubuntu:GNOME
  Date: Fri Sep  3 18:46:35 2021
  InstallationDate: Installed on 2019-07-17 (779 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 5986:210e Acer, Inc EasyCamera
   Bus 001 Device 004: ID 8087:0a2a Intel Corp. Bluetooth wireless interface
   Bus 001 Device 007: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: LENOVO 81FK
  ProcFB: 0 i915drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-18-generic 
root=UUID=9d963312-3a16-428d-8efd-f1323c6528f1 ro quiet nosplash 
crashkernel=512M-:192M
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-18-generic N/A
   linux-backports-modules-5.11.0-18-generic  N/A
   linux-firmware 1.197.3
  SourcePackage: linux
  UpgradeStatus: Upgraded to hirsute on 2021-06-03 (92 days ago)
  dmi.bios.date: 10/24/2018
  dmi.bios.release: 1.29
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7ZCN29WW
  dmi.board.asset.tag: NO Asset Tag
  dmi.board.name: LNVNB161216
  dmi.board.vendor: LENOVO
  dmi.board.version: SDK0J40709 WIN
  dmi.chassis.asset.tag: NO Asset Tag
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Lenovo ideapad 330-15ICH
  dmi.ec.firmware.release: 1.29
  dmi.modalias: 
dmi:bvnLENOVO:bvr7ZCN29WW:bd10/24/2018:br1.29:efr1.29:svnLENOVO:pn81FK:pvrLenovoideapad330-15ICH:rvnLENOVO:rnLNVNB161216:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad330-15ICH:
  dmi.product.family: ideapad 330-15ICH
  dmi.product.name: 81FK
  dmi.product.sku: LENOVO_MT_81FK_BU_idea_FM_ideapad 330-15ICH
  dmi.product.version: Lenovo ideapad 330-15ICH
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942624/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1942624] Re: NVME "can't change power state from D3Cold to D0 (config space inaccessible)"

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu Impish)
   Status: Confirmed => In Progress

** Changed in: linux-oem-5.14 (Ubuntu Focal)
   Status: Confirmed => In Progress

** Changed in: linux-oem-5.14 (Ubuntu)
   Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-oem-5.14 in Ubuntu.
https://bugs.launchpad.net/bugs/1942624

Title:
  NVME "can't change power state from D3Cold to D0 (config space
  inaccessible)"

Status in linux package in Ubuntu:
  Fix Released
Status in linux-oem-5.14 package in Ubuntu:
  In Progress
Status in linux source package in Focal:
  Invalid
Status in linux-oem-5.14 source package in Focal:
  In Progress
Status in linux source package in Impish:
  In Progress
Status in linux-oem-5.14 source package in Impish:
  Invalid

Bug description:
  Bug with kernels after version 5.11.0-18 on Lenovo Ideapad 330-15ICH.
  The NVME drive with my root partition cannot be mounted at boot with
  an error "can't change power state from D3Cold to D0 (config space
  inaccessible)". I'm willing to help find a root cause if I don't need
  to spent too many hours. All Ubuntu kernels after 5.11.0-18 exhibit
  this bug, but I could boot properly with the official linux kernel
  5.13.0. Thanks a lot for your help

  ProblemType: Bug
  DistroRelease: Ubuntu 21.04
  Package: linux-image-5.11.0-18-generic 5.11.0-18.19
  ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
  Uname: Linux 5.11.0-18-generic x86_64
  ApportVersion: 2.20.11-0ubuntu65.1
  Architecture: amd64
  AudioDevicesInUse:
   USERPID ACCESS COMMAND
   /dev/snd/controlC0:  chris  7503 F pulseaudio
   /dev/snd/pcmC0D0p:   chris  7503 F...m pulseaudio
  CasperMD5CheckResult: unknown
  CurrentDesktop: ubuntu:GNOME
  Date: Fri Sep  3 18:46:35 2021
  InstallationDate: Installed on 2019-07-17 (779 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 5986:210e Acer, Inc EasyCamera
   Bus 001 Device 004: ID 8087:0a2a Intel Corp. Bluetooth wireless interface
   Bus 001 Device 007: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: LENOVO 81FK
  ProcFB: 0 i915drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-18-generic 
root=UUID=9d963312-3a16-428d-8efd-f1323c6528f1 ro quiet nosplash 
crashkernel=512M-:192M
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-18-generic N/A
   linux-backports-modules-5.11.0-18-generic  N/A
   linux-firmware 1.197.3
  SourcePackage: linux
  UpgradeStatus: Upgraded to hirsute on 2021-06-03 (92 days ago)
  dmi.bios.date: 10/24/2018
  dmi.bios.release: 1.29
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7ZCN29WW
  dmi.board.asset.tag: NO Asset Tag
  dmi.board.name: LNVNB161216
  dmi.board.vendor: LENOVO
  dmi.board.version: SDK0J40709 WIN
  dmi.chassis.asset.tag: NO Asset Tag
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Lenovo ideapad 330-15ICH
  dmi.ec.firmware.release: 1.29
  dmi.modalias: 
dmi:bvnLENOVO:bvr7ZCN29WW:bd10/24/2018:br1.29:efr1.29:svnLENOVO:pn81FK:pvrLenovoideapad330-15ICH:rvnLENOVO:rnLNVNB161216:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad330-15ICH:
  dmi.product.family: ideapad 330-15ICH
  dmi.product.name: 81FK
  dmi.product.sku: LENOVO_MT_81FK_BU_idea_FM_ideapad 330-15ICH
  dmi.product.version: Lenovo ideapad 330-15ICH
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942624/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1942624] Re: NVME "can't change power state from D3Cold to D0 (config space inaccessible)"

[Heitor Alves de Siqueira]

** Also affects: linux-oem-5.14 (Ubuntu)
   Importance: Undecided
   Status: New

** Changed in: linux-oem-5.14 (Ubuntu)
   Status: New => Confirmed

** Also affects: linux (Ubuntu Impish)
   Importance: Undecided
   Status: New

** Also affects: linux-oem-5.14 (Ubuntu Impish)
   Importance: Undecided
   Status: New

** Changed in: linux (Ubuntu)
   Status: Confirmed => Fix Released

** Changed in: linux (Ubuntu Impish)
   Status: New => Confirmed

** Changed in: linux (Ubuntu Impish)
   Importance: Undecided => Medium

** Changed in: linux (Ubuntu Impish)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Also affects: linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Also affects: linux-oem-5.14 (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Changed in: linux (Ubuntu Focal)
   Status: New => Invalid

** Changed in: linux-oem-5.14 (Ubuntu Impish)
   Status: New => Invalid

** Changed in: linux-oem-5.14 (Ubuntu Focal)
   Status: New => Confirmed

** Changed in: linux-oem-5.14 (Ubuntu)
   Importance: Undecided => Medium

** Changed in: linux-oem-5.14 (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: linux-oem-5.14 (Ubuntu)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-oem-5.14 (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Tags added: sts

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-oem-5.14 in Ubuntu.
https://bugs.launchpad.net/bugs/1942624

Title:
  NVME "can't change power state from D3Cold to D0 (config space
  inaccessible)"

Status in linux package in Ubuntu:
  Fix Released
Status in linux-oem-5.14 package in Ubuntu:
  Confirmed
Status in linux source package in Focal:
  Invalid
Status in linux-oem-5.14 source package in Focal:
  Confirmed
Status in linux source package in Impish:
  Confirmed
Status in linux-oem-5.14 source package in Impish:
  Invalid

Bug description:
  Bug with kernels after version 5.11.0-18 on Lenovo Ideapad 330-15ICH.
  The NVME drive with my root partition cannot be mounted at boot with
  an error "can't change power state from D3Cold to D0 (config space
  inaccessible)". I'm willing to help find a root cause if I don't need
  to spent too many hours. All Ubuntu kernels after 5.11.0-18 exhibit
  this bug, but I could boot properly with the official linux kernel
  5.13.0. Thanks a lot for your help

  ProblemType: Bug
  DistroRelease: Ubuntu 21.04
  Package: linux-image-5.11.0-18-generic 5.11.0-18.19
  ProcVersionSignature: Ubuntu 5.11.0-18.19-generic 5.11.17
  Uname: Linux 5.11.0-18-generic x86_64
  ApportVersion: 2.20.11-0ubuntu65.1
  Architecture: amd64
  AudioDevicesInUse:
   USERPID ACCESS COMMAND
   /dev/snd/controlC0:  chris  7503 F pulseaudio
   /dev/snd/pcmC0D0p:   chris  7503 F...m pulseaudio
  CasperMD5CheckResult: unknown
  CurrentDesktop: ubuntu:GNOME
  Date: Fri Sep  3 18:46:35 2021
  InstallationDate: Installed on 2019-07-17 (779 days ago)
  InstallationMedia: Ubuntu 19.04 "Disco Dingo" - Release amd64 (20190416)
  Lsusb:
   Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
   Bus 001 Device 003: ID 5986:210e Acer, Inc EasyCamera
   Bus 001 Device 004: ID 8087:0a2a Intel Corp. Bluetooth wireless interface
   Bus 001 Device 007: ID 1050:0407 Yubico.com Yubikey 4/5 OTP+U2F+CCID
   Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  MachineType: LENOVO 81FK
  ProcFB: 0 i915drmfb
  ProcKernelCmdLine: BOOT_IMAGE=/boot/vmlinuz-5.11.0-18-generic 
root=UUID=9d963312-3a16-428d-8efd-f1323c6528f1 ro quiet nosplash 
crashkernel=512M-:192M
  RelatedPackageVersions:
   linux-restricted-modules-5.11.0-18-generic N/A
   linux-backports-modules-5.11.0-18-generic  N/A
   linux-firmware 1.197.3
  SourcePackage: linux
  UpgradeStatus: Upgraded to hirsute on 2021-06-03 (92 days ago)
  dmi.bios.date: 10/24/2018
  dmi.bios.release: 1.29
  dmi.bios.vendor: LENOVO
  dmi.bios.version: 7ZCN29WW
  dmi.board.asset.tag: NO Asset Tag
  dmi.board.name: LNVNB161216
  dmi.board.vendor: LENOVO
  dmi.board.version: SDK0J40709 WIN
  dmi.chassis.asset.tag: NO Asset Tag
  dmi.chassis.type: 10
  dmi.chassis.vendor: LENOVO
  dmi.chassis.version: Lenovo ideapad 330-15ICH
  dmi.ec.firmware.release: 1.29
  dmi.modalias: 
dmi:bvnLENOVO:bvr7ZCN29WW:bd10/24/2018:br1.29:efr1.29:svnLENOVO:pn81FK:pvrLenovoideapad330-15ICH:rvnLENOVO:rnLNVNB161216:rvrSDK0J40709WIN:cvnLENOVO:ct10:cvrLenovoideapad330-15ICH:
  dmi.product.family: ideapad 330-15ICH
  dmi.product.name: 81FK
  dmi.product.sku: LENOVO_MT_81FK_BU_idea_FM_ideapad 330-15ICH
  dmi.product.version: Lenovo ideapad 330-15ICH
  dmi.sys.vendor: LENOVO

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1942624/+subscriptio

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

** Patch added: "lp1964992-focal.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+attachment/5571326/+files/lp1964992-focal.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] Re: ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

This doesn't seem to be reproducible in Bionic, marking it as fixed
accordingly.

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964992

Title:
  ZFS ignores ARC sizes below allmem/32

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Bionic:
  Fix Released
Status in zfs-linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
  ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
     # cat /etc/modprobe.d/99-zfs-arc.conf
     options zfs zfs_arc_min=536870912
     options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
     # arc_summary | grep -A3 "ARC size"
     ARC size (current):   < 0.1 %1.3 MiB
     Target size (adaptive):   100.0 %   15.7 GiB
     Min size (hard limit):  3.2 %  512.0 MiB
     Max size (high water):   31:1   15.7 GiB

  For a 32GB test system, we should be able to set max ARC sizes below
  1GB.

  [Fix]
  This has been fixed by upstream commit:
   - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

  The commit has been introduced in upstream zfs-2.0.0, so it's needed
  for Bionic and Focal. Releases starting with Impish already have this
  commit by default:

  $ git describe --contains 36a6e2335c45
  zfs-2.0.0-rc1~332
  $ rmadison zfs-linux
   zfs-linux | 0.7.5-1ubuntu15| bionic  | source
   zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
   zfs-linux | 0.8.3-1ubuntu12| focal   | source
   zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
   zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
   zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
   zfs-linux | 2.0.6-1ubuntu2 | impish  | source
   zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
   zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

  [Regression Potential]
  The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1964992/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964992] [NEW] ZFS ignores ARC sizes below allmem/32

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.

[Test Plan]
1. Deploy test system with ZFS storage and 32GB RAM
2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
   # cat /etc/modprobe.d/99-zfs-arc.conf
   options zfs zfs_arc_min=536870912
   options zfs zfs_arc_max=966367641
3. Reboot system
4. Verify ARC sizes through "arc_summary"
   # arc_summary | grep -A3 "ARC size"
   ARC size (current):   < 0.1 %1.3 MiB
   Target size (adaptive):   100.0 %   15.7 GiB
   Min size (hard limit):  3.2 %  512.0 MiB
   Max size (high water):   31:1   15.7 GiB

For a 32GB test system, we should be able to set max ARC sizes below
1GB.

[Fix]
This has been fixed by upstream commit:
 - 36a6e2335c45 "Don't ignore zfs_arc_max below allmem/32"

The commit has been introduced in upstream zfs-2.0.0, so it's needed for
Bionic and Focal. Releases starting with Impish already have this commit
by default:

$ git describe --contains 36a6e2335c45
zfs-2.0.0-rc1~332
$ rmadison zfs-linux
 zfs-linux | 0.7.5-1ubuntu15| bionic  | source
 zfs-linux | 0.7.5-1ubuntu16.12 | bionic-updates  | source
 zfs-linux | 0.8.3-1ubuntu12| focal   | source
 zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
 zfs-linux | 0.8.3-1ubuntu12.13 | focal-updates   | source
 zfs-linux | 0.8.3-1ubuntu12.14 | focal-proposed  | source
 zfs-linux | 2.0.6-1ubuntu2 | impish  | source
 zfs-linux | 2.0.6-1ubuntu2.1   | impish-updates  | source
 zfs-linux | 2.1.2-1ubuntu3 | jammy   | source

[Regression Potential]
The introduced commit essentially removes the limitation of setting ARC 
tunables below allmem/32, and re-arranges the order of how some of the tunables 
are parsed. Regressions would possibly show up as other tunables being ignored 
or not being set correctly due to parsing errors. We should validate whether 
other ARC related tunables are still being set correctly, and whether ZFS is 
using the set values for the ARC memory thresholds.

** Affects: zfs-linux (Ubuntu)
 Importance: Undecided
 Status: Fix Released

** Affects: zfs-linux (Ubuntu Bionic)
     Importance: Medium
 Assignee: Heitor Alves de Siqueira (halves)
 Status: Confirmed

** Affects: zfs-linux (Ubuntu Focal)
     Importance: Medium
 Assignee: Heitor Alves de Siqueira (halves)
 Status: Confirmed


** Tags: sts

** Changed in: zfs-linux (Ubuntu)
   Status: New => Fix Released

** Also affects: zfs-linux (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Also affects: zfs-linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Focal)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Bionic)
   Importance: Undecided => Medium

** Changed in: zfs-linux (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: zfs-linux (Ubuntu Bionic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Description changed:

  [Impact]
- ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32
- threshold. This prevents users from properly restraining ARC sizes, and can
- cause increased memory contention in some systems.
+ ZFS ignores tunable "zfs_arc_max" due to it being below allmem/32 threshold. 
This prevents users from properly restraining ARC sizes, and can cause 
increased memory contention in some systems.
  
  [Test Plan]
  1. Deploy test system with ZFS storage and 32GB RAM
  2. Add ARC tunables to /etc/modprobe.d/99-zfs-arc.conf
-# cat /etc/modprobe.d/99-zfs-arc.conf
-options zfs zfs_arc_min=536870912
-options zfs zfs_arc_max=966367641
+    # cat /etc/modprobe.d/99-zfs-arc.conf
+    options zfs zfs_arc_min=536870912
+    options zfs zfs_arc_max=966367641
  3. Reboot system
  4. Verify ARC sizes through "arc_summary"
-# arc_summary | grep -A3 "ARC size"
-ARC size (current):   < 0.1 %1.3 MiB
-Target size (adaptive):   100.0 %   15.7 GiB
-Min size (hard limit):  3.2 %  512.0 MiB
-Max size (high water):   31:1   15.7 GiB
+    # arc_summary | grep -A3 "ARC size"
+    ARC size (current):   < 0.1 %1.3 MiB
+    Target size (adaptive):   100.0 %   15.7 GiB
+    Min size (hard limit):

[Kernel-packages] [Bug 1964512] Re: Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu Impish)
   Importance: Undecided => High

** Changed in: linux (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: linux (Ubuntu Impish)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Impish)
   Status: New => In Progress

** Changed in: linux (Ubuntu Jammy)
   Status: Confirmed => In Progress

** Changed in: linux (Ubuntu Focal)
   Status: New => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  In Progress
Status in linux source package in Focal:
  In Progress
Status in linux source package in Impish:
  In Progress
Status in linux source package in Jammy:
  In Progress

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1964512] [NEW] Low RX performance for 40G Solarflare NICs

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
* Some 40G Solarflare NICs have low RX performance in some cases, due
  low RX recycle ring size
* RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
* The low fixed sizes can cause a high number of calls to alloc_pages,
  tanking performance for higher link speeds

[Test Plan]
* Users report that iperf3 is sufficient to showcase the bad RX performance
* For some setups, RX performance was around 15Gbps while TX stayed
  consistently above 30Gbps

[Fix]
* This patch sets the RX recycle ring size according to an adapter's
  maximum link speed
* Fix was introduced by commit:
  000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
* --!-- Commit is from net-next --!--

[Regression Potential]
* Regressions would show primarily as performance issues, as we're
  effectively changing ring sizes for all RX traffic
* It's possible to see increased calls to alloc_pages if ring sizes
  aren't being set correctly
* We should look out for excessive memory usage in the sfc driver due to
  the increased ring sizes

** Affects: linux (Ubuntu)
 Importance: High
 Assignee: Heitor Alves de Siqueira (halves)
 Status: Confirmed

** Affects: linux (Ubuntu Focal)
 Importance: Undecided
 Status: New

** Affects: linux (Ubuntu Impish)
 Importance: Undecided
 Status: New

** Affects: linux (Ubuntu Jammy)
 Importance: High
 Assignee: Heitor Alves de Siqueira (halves)
 Status: Confirmed


** Tags: sts

** Also affects: linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu Jammy)
   Importance: High
 Assignee: Heitor Alves de Siqueira (halves)
   Status: Confirmed

** Also affects: linux (Ubuntu Impish)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1964512

Title:
  Low RX performance for 40G Solarflare NICs

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Focal:
  New
Status in linux source package in Impish:
  New
Status in linux source package in Jammy:
  Confirmed

Bug description:
  [Impact]
  * Some 40G Solarflare NICs have low RX performance in some cases, due
low RX recycle ring size
  * RX recycle ring size is either 4096 for IOMMU, 16 for NOIOMMU
  * The low fixed sizes can cause a high number of calls to alloc_pages,
tanking performance for higher link speeds

  [Test Plan]
  * Users report that iperf3 is sufficient to showcase the bad RX performance
  * For some setups, RX performance was around 15Gbps while TX stayed
consistently above 30Gbps

  [Fix]
  * This patch sets the RX recycle ring size according to an adapter's
maximum link speed
  * Fix was introduced by commit:
000fe940e51f "sfc: The size of the RX recycle ring should be more flexible"
  * --!-- Commit is from net-next --!--

  [Regression Potential]
  * Regressions would show primarily as performance issues, as we're
effectively changing ring sizes for all RX traffic
  * It's possible to see increased calls to alloc_pages if ring sizes
aren't being set correctly
  * We should look out for excessive memory usage in the sfc driver due to
the increased ring sizes

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1964512/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1956849] Re: Almost all USB ports suddenly stopped working; unbootable

[Heitor Alves de Siqueira]

@dgatwood would you be able to upload kernel logs and lsusb output from
the -44 kernel? From what you're describing it seems like a possible
issue with host controllers, but it's hard to tell without looking at
what is going in the kernel.

You can use `apport-collect 1956849` to upload relevant data directly to
this LP bug.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1956849

Title:
  Almost all USB ports suddenly stopped working; unbootable

Status in linux package in Ubuntu:
  New

Bug description:
  This package needs to be pulled NOW.  It disables almost all USB-3.0
  and USB-C ports completely.

  Even though I had automatic software updates turned OFF (or so I
  thought), my Mac Pro suddenly got a new kernel when I rebooted it this
  morning:

  Linux macpro-obs 5.11.0-44-generic #48~20.04.2-Ubuntu SMP Tue Dec 14
  15:36:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

  and it contains a P0 showstopper bug.  Upon rebooting, I got dropped
  into the initrd prompt because Linux could not see the external USB
  drive that I'm booting from (WDBAGF5000AGY-WESN).

  I wasted an entire day figuring out what was wrong, and even went so
  far as to order a replacement SSD, planning to rebuild everything from
  scratch, because the drive failed to appear in every single USB port I
  tried.

  Then I discovered that a different SSD didn't work, either.  At that
  point, I realized that something else was wrong, and I kept trying
  until I found one other port that worked.  I was then able to boot and
  get dmesg and lsusb output.

  This kernel update broke not only the built-in ports, but also the
  ports on a generic USB-C PCIe card (Amazon B08PF8XR73).

  Mac Pro built-in USB-3.0(A) ports (2x): working
  Mac Pro built-in USB-C ports (4x): dead
  USB-C PCIe card USB-C ports (2x): dead
  USB-C PCIe card USB-3.0(A) ports (5x): dead

  All devices fail to appear in lsusb when attached to the port,
  including an Apple USB keyboard, an Anker USB-C Ethernet adapter, a WD
  SSD, and a Sandisk SSD.

  I'm going to roll back my kernel to a working kernel, but this package
  needs to be pulled NOW before it affects too many people.  This is too
  catastrophic a bug to wait even a day.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1956849/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

Verified for Focal according to test case from description. A standard
multipass VM with 8 vCPUs configured was able to capture a kernel dump
without errors.

Additionally, since the 5.4 kernel doesn't hit the MMU/reset bug
directly, I've done some additional testing to ensure we didn't
introduce any weird regressions. These scenarios included:

- default crash kernel config
- nested VM crash kernel with multiple CPUs
- nested VM crash kernel with default config
- basic smoke testing of VM up/down through virsh
- basic smoke testing of reboots from within the VM

All of these behaved as expected and showed no issues.

** Tags removed: verification-needed-focal
** Tags added: verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1948862

Title:
  KVM emulation failure when booting into  VM crash kernel with multiple
  CPUs

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Fix Committed
Status in linux source package in Focal:
  Fix Committed

Bug description:
  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.

  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with
  a similar spew as below:

  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a

  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm

  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"

  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger

  [Where problems could occur]
  As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
  Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).

  [Other info]
  This has been fixed by upstream commit:
0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT

  The commit above has been picked up by stable trees up until 5.11, so it's 
only needed in Bionic and Focal (4.15 and 5.4 kernels). There are also two 
follow up commits, which revert the vendor-specific resets:
5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()

  These follow ups have not been picked up in stable trees due to the risk of
  regressions. According to the original fix, they have been introduced 
primarily to aid bisection in case there are workflows relying on the vendor 
resets. As these are not required for the fix and don't conflict with the 
backport, we should leave them out to prevent potential regressions in the 
older kernels.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1948862/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : 

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

For future reference, this bug has been reported in Ubuntu KVM hosts
outside of the specific kdump test scenario (i.e. when running actual
VMs in production-like setups). The "kdump with multiple CPUs" case was
discovered as an alternative that hits the same MMU context bug, without
needing complex hypervisor setups or fancy configuration.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1948862

Title:
  KVM emulation failure when booting into  VM crash kernel with multiple
  CPUs

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Confirmed
Status in linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.

  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with
  a similar spew as below:

  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a

  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm

  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"

  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger

  [Where problems could occur]
  As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
  Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).

  [Other info]
  This has been fixed by upstream commit:
0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT

  The commit above has been picked up by stable trees up until 5.11, so it's 
only needed in Bionic and Focal (4.15 and 5.4 kernels). There are also two 
follow up commits, which revert the vendor-specific resets:
5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()

  These follow ups have not been picked up in stable trees due to the risk of
  regressions. According to the original fix, they have been introduced 
primarily to aid bisection in case there are workflows relying on the vendor 
resets. As these are not required for the fix and don't conflict with the 
backport, we should leave them out to prevent potential regressions in the 
older kernels.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1948862/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu Focal)
   Status: New => Confirmed

** Changed in: linux (Ubuntu Bionic)
   Importance: Undecided => High

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1948862

Title:
  KVM emulation failure when booting into  VM crash kernel with multiple
  CPUs

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Confirmed
Status in linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.

  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with
  a similar spew as below:

  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a

  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm

  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"

  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger

  [Where problems could occur]
  As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
  Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).

  [Other info]
  This has been fixed by upstream commit:
0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT

  The commit above has been picked up by stable trees up until 5.11, so it's 
only needed in Bionic and Focal (4.15 and 5.4 kernels). There are also two 
follow up commits, which revert the vendor-specific resets:
5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()

  These follow ups have not been picked up in stable trees due to the risk of
  regressions. According to the original fix, they have been introduced 
primarily to aid bisection in case there are workflows relying on the vendor 
resets. As these are not required for the fix and don't conflict with the 
backport, we should leave them out to prevent potential regressions in the 
older kernels.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1948862/+subscriptions


-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.
  
  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with a
  similar spew as below:
  
  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a
  
  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm
  
  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"
  
  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger
  
  [Where problems could occur]
  As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
  Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).
  
  [Other info]
  This has been fixed by upstream commit:
-   0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT
+   0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT
  
- And the two follow up commits, which revert the vendor-specific resets:
-   5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
-   61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()
+ The commit above has been picked up by stable trees up until 5.11, so it's 
only needed in Bionic and Focal (4.15 and 5.4 kernels). There are also two 
follow up commits, which revert the vendor-specific resets:
+   5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
+   61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()
  
- These commits have been introduced during the upstream 5.14 and 5.15 release 
candidates. They have been picked up by upstream stable up until 5.11, so 
backports are only needed for Bionic and Focal (4.15 and 5.4 kernels).
- $ git describe --contains 0aa1837533e5 5d2d7e41e3b8 61152cd907d5
- v5.14-rc1~166^2~58
- v5.15-rc1~65^2~119
- v5.15-rc1~65^2~120
+ These follow ups have not been picked up in stable trees due to the risk of
+ regressions. According to the original fix, they have been introduced 
primarily to aid bisection in case there are workflows relying on the vendor 
resets. As these are not required for the fix and don't conflict with the 
backport, we should leave them out to prevent potential regressions in the 
older kernels.

** Changed in: linux (Ubuntu)
   Status: Incomplete => Fix Released

** Also affects: linux (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Also affects: linux (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Changed in: linux (Ubuntu Bionic)
   Status: New => Confirmed

** Changed in: linux (Ubuntu Focal)
   Importance: Undecided => High

** Changed in: linux (Ubuntu Bionic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Focal)
     Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

-- 
You received this bug notification because you are a member o

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.
  
  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with a
  similar spew as below:
  
  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a
  
  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm
  
  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"
  
  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger
  
  [Where problems could occur]
  As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
  Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).
  
  [Other info]
  This has been fixed by upstream commit:
    0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT
  
  And the two follow up commits, which revert the vendor-specific resets:
    5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
    61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()
  
- These commits have been introduced during the upstream 5.14 and 5.15 release 
candidates, and as such should be backported to previous supported kernels.
+ These commits have been introduced during the upstream 5.14 and 5.15 release 
candidates. They have been picked up by upstream stable up until 5.11, so 
backports are only needed for Bionic and Focal (4.15 and 5.4 kernels).
  $ git describe --contains 0aa1837533e5 5d2d7e41e3b8 61152cd907d5
  v5.14-rc1~166^2~58
  v5.15-rc1~65^2~119
  v5.15-rc1~65^2~120

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1948862

Title:
  KVM emulation failure when booting into  VM crash kernel with multiple
  CPUs

Status in linux package in Ubuntu:
  Incomplete

Bug description:
  [Impact]
  When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.

  This happens only when there are multiple enabled CPUs in the crash
  kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
  being used. Due to the vCPU MMU state not being cleaned up correctly,
  the secondary CPUs try to access virtual addresses with a faulty MMU
  context that will result in the emulation failure. This shows up with
  a similar spew as below:

  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000 

[Kernel-packages] [Bug 1948862] Re: KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
- When kexec'ing into a crash kernel with `ncpus` > 1, VMs can raise a KVM
- emulation failure. This will cause the VM to go into the "paused" state, and
- prevents it from being restored without a full VM restart.
+ When kexec'ing into a crash kernel with ncpus > 1, VMs can raise a KVM 
emulation failure. This will cause the VM to go into the "paused" state, and 
prevents it from being restored without a full VM restart.
  
- This happens only when there are multiple enabled CPUs in the crash kernel
- command-line, regardless of whether `nr_cpus` or `maxcpus` is being used. Due 
to
- the vCPU MMU state not being cleaned up correctly, the secondary CPUs try to
- access virtual addresses with a faulty MMU context that will result in the
- emulation failure. This shows up with a similar spew as below:
+ This happens only when there are multiple enabled CPUs in the crash
+ kernel command-line, regardless of whether `nr_cpus` or `maxcpus` is
+ being used. Due to the vCPU MMU state not being cleaned up correctly,
+ the secondary CPUs try to access virtual addresses with a faulty MMU
+ context that will result in the emulation failure. This shows up with a
+ similar spew as below:
  
  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  
  IDT=  
  CR0=6010 CR2= CR3=290b8001 CR4=
  DR0= DR1= DR2= 
DR3=
  DR6=0ff0 DR7=0400
  EFER=
  Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 
0f b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a
  
  [Test Plan]
  1. Boot an Ubuntu guest VM with e.g. multipass:
  $ multipass launch daily:focal -c8 -m16g -n focal-vm
  
  2. Configure guest crash kernel command-line with `nr_cpus=8`:
  ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
  # KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
  KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"
  
  3. Crash guest VM and watch for the KVM emulation failure:
  ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger
  
  [Where problems could occur]
- As we're resetting MMU context on vCPUs, potential regressions would show up 
in
- workloads relying on KVM guests. We should properly test the scenario 
mentioned
- in the bug to make sure secondary CPUs are being cleaned up properly, and that
- no other regressions have been introduced when rebooting or kexec'ing into
- different kernels.
- Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression
- potential should be fairly low and contained to starting/resetting vCPUs
- (i.e. VM start and reboot).
+ As we're resetting MMU context on vCPUs, potential regressions would show up 
in workloads relying on KVM guests. We should properly test the scenario 
mentioned in the bug to make sure secondary CPUs are being cleaned up properly, 
and that no other regressions have been introduced when rebooting or kexec'ing 
into different kernels.
+ Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression 
potential should be fairly low and contained to starting/resetting vCPUs (i.e. 
VM start and reboot).
  
  [Other info]
  This has been fixed by upstream commit:
-   0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT
+   0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT
  
  And the two follow up commits, which revert the vendor-specific resets:
-   5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
-   61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()
+   5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
+   61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()
  
- These commits have been introduced during the upstream 5.14 and 5.15 release
- candidates, and as such should be backported to previous supported kernels.
+ These commits have been introduced during the upstream 5.14 and 5.15 release 
candidates, and as such should be backported to previous supported kernels.
  $ git describe --contains 0aa1837533e5 5d2d7e41e3b8 61152cd907d5
  v5.14-rc1~166^2~58
  v5.15-rc1~65^2~119
  v5.15-rc1~65^2~120

-- 
You received this bug notification because you are a member of Kernel
Packages, 

[Kernel-packages] [Bug 1948862] [NEW] KVM emulation failure when booting into VM crash kernel with multiple CPUs

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
When kexec'ing into a crash kernel with `ncpus` > 1, VMs can raise a KVM
emulation failure. This will cause the VM to go into the "paused" state, and
prevents it from being restored without a full VM restart.

This happens only when there are multiple enabled CPUs in the crash kernel
command-line, regardless of whether `nr_cpus` or `maxcpus` is being used. Due to
the vCPU MMU state not being cleaned up correctly, the secondary CPUs try to
access virtual addresses with a faulty MMU context that will result in the
emulation failure. This shows up with a similar spew as below:

$ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
KVM internal error. Suberror: 1
emulation failure
EAX=de8f EBX= ECX=008f EDX=0600
ESI= EDI= EBP= ESP=f90c
EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =   9300
CS =f000 000f  9b00
SS =de00 000de000  9300
DS =de00 000de000  9300
FS =   9300
GS =   9300
LDT=   8200
TR =   8b00
GDT=  
IDT=  
CR0=6010 CR2= CR3=290b8001 CR4=
DR0= DR1= DR2= 
DR3=
DR6=0ff0 DR7=0400
EFER=
Code=66 83 c4 28 66 5b 66 c3 66 56 66 53 66 52 b1 8f 88 c8 e6 70  71 66 0f 
b6 f0 66 89 f2 67 88 54 24 03 88 c8 e6 70 66 31 db 88 d8 e6 71 66 56 66 68 1a

[Test Plan]
1. Boot an Ubuntu guest VM with e.g. multipass:
$ multipass launch daily:focal -c8 -m16g -n focal-vm

2. Configure guest crash kernel command-line with `nr_cpus=8`:
ubuntu@focal-vm:~$ grep CMDLINE_APPEND /etc/default/kdump-tools
# KDUMP_CMDLINE_APPEND - Additional arguments to append to the command line
KDUMP_CMDLINE_APPEND="reset_devices systemd.unit=kdump-tools-dump.service 
nr_cpus=8 irqpoll nousb ata_piix.prefer_ms_hyperv=0"

3. Crash guest VM and watch for the KVM emulation failure:
ubuntu@focal-vm:~$ echo c | sudo tee /proc/sysrq-trigger

[Where problems could occur]
As we're resetting MMU context on vCPUs, potential regressions would show up in
workloads relying on KVM guests. We should properly test the scenario mentioned
in the bug to make sure secondary CPUs are being cleaned up properly, and that
no other regressions have been introduced when rebooting or kexec'ing into
different kernels.
Since we're adding an MMU reset at kvm_vcpu_reset(), the overall regression
potential should be fairly low and contained to starting/resetting vCPUs
(i.e. VM start and reboot).

[Other info]
This has been fixed by upstream commit:
  0aa1837533e5 KVM: x86: Properly reset MMU context at vCPU RESET/INIT

And the two follow up commits, which revert the vendor-specific resets:
  5d2d7e41e3b8 KVM: SVM: Drop explicit MMU reset at RESET/INIT
  61152cd907d5 KVM: VMX: Remove explicit MMU reset in enter_rmode()

These commits have been introduced during the upstream 5.14 and 5.15 release
candidates, and as such should be backported to previous supported kernels.
$ git describe --contains 0aa1837533e5 5d2d7e41e3b8 61152cd907d5
v5.14-rc1~166^2~58
v5.15-rc1~65^2~119
v5.15-rc1~65^2~120

** Affects: linux (Ubuntu)
 Importance: High
 Assignee: Heitor Alves de Siqueira (halves)
 Status: New


** Tags: sts

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1948862

Title:
  KVM emulation failure when booting into  VM crash kernel with multiple
  CPUs

Status in linux package in Ubuntu:
  New

Bug description:
  [Impact]
  When kexec'ing into a crash kernel with `ncpus` > 1, VMs can raise a KVM
  emulation failure. This will cause the VM to go into the "paused" state, and
  prevents it from being restored without a full VM restart.

  This happens only when there are multiple enabled CPUs in the crash kernel
  command-line, regardless of whether `nr_cpus` or `maxcpus` is being used. Due 
to
  the vCPU MMU state not being cleaned up correctly, the secondary CPUs try to
  access virtual addresses with a faulty MMU context that will result in the
  emulation failure. This shows up with a similar spew as below:

  $ sudo tail -n20 /var/log/libvirt/qemu/focal-vm.log
  KVM internal error. Suberror: 1
  emulation failure
  EAX=de8f EBX= ECX=008f EDX=0600
  ESI= EDI= EBP= ESP=f90c
  EIP=cdb1 EFL=0002 [---] CPL=0 II=0 A20=1 SMM=0 HLT=0
  ES =   9300
  CS =f000 000f  9b00
  SS =de00 000de000  9300
  DS =de00 000de000  9300
  FS =   9300
  GS =   9300
  LDT=   8200
  TR =   8b00
  GDT=  fff

[Kernel-packages] [Bug 1875577] Re: Encrypted swap won't load on 20.04 with zfs root

[Heitor Alves de Siqueira]

Validated with ZFS from focal-proposed, according to test case from description:
ubuntu@z-rotomvm34:~$ dpkg -l | grep zfsutils
ii  zfsutils-linux   0.8.3-1ubuntu12.12
amd64command-line tools to manage OpenZFS filesystems
ubuntu@z-rotomvm34:~$ zfs list
NAME USED  AVAIL REFER  MOUNTPOINT
rpool   2.50G  25.6G  176K  /
rpool/ROOT  2.50G  25.6G  176K  none
rpool/ROOT/zfsroot  2.50G  25.6G 2.50G  /
ubuntu@z-rotomvm34:~$ sudo journalctl -b | grep -i ordering
ubuntu@z-rotomvm34:~$ lsblk -e 7
NAMEMAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda 252:00   30G  0 disk
├─vda1  252:10  512M  0 part  /boot/efi
└─vda2  252:20 29.5G  0 part
nvme0n1 259:00  9.8G  0 disk
└─nvme0n1p1 259:10  9.8G  0 part
  └─swap253:00  9.8G  0 crypt [SWAP]

In addition to the test above, I've also tested the configurations
suggested in the [Test Plan] section. Besides validating the ordering
bug, I've also done basic smoke tests and verified that the ZFS pools
are working as expected.

- Encrypted rootfs on LVM + separate ZFS partitions:
ubuntu@ubuntu-focal:~$ zfs list
NAME   USED  AVAIL REFER  MOUNTPOINT
zfspool492K  4.36G   96K  /mnt/zfspool
zfspool/tank96K  4.36G   96K  /mnt/zfspool/tank
ubuntu@ubuntu-focal:~$ dpkg -l | grep zfsutils
ii  zfsutils-linux   0.8.3-1ubuntu12.12
amd64command-line tools to manage OpenZFS filesystems
ubuntu@ubuntu-focal:~$ zfs list
NAME   USED  AVAIL REFER  MOUNTPOINT
zfspool492K  4.36G   96K  /mnt/zfspool
zfspool/tank96K  4.36G   96K  /mnt/zfspool/tank
ubuntu@ubuntu-focal:~$ sudo journalctl -b | grep -i ordering
ubuntu@ubuntu-focal:~$ lsblk -e7
NAME MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0   11:01 1024M  0 rom
vda  252:00   30G  0 disk
├─vda1   252:10  512M  0 part  /boot/efi
├─vda2   252:201K  0 part
├─vda5   252:50  731M  0 part  /boot
└─vda6   252:60 28.8G  0 part
  └─vda6_crypt   253:00 28.8G  0 crypt
├─vgubuntu--focal-root   253:10 27.8G  0 lvm   /
└─vgubuntu--focal-swap_1 253:20  980M  0 lvm   [SWAP]
vdb  252:16   05G  0 disk
├─vdb1   252:17   05G  0 part
└─vdb9   252:25   08M  0 part

- ZFS on LUKS
ubuntu@z-rotomvm33:~$ dpkg -l | grep zfsutils
ii  zfsutils-linux   0.8.3-1ubuntu12.12
amd64command-line tools to manage OpenZFS filesystems
ubuntu@z-rotomvm33:~$ zfs list
NAME   USED  AVAIL REFER  MOUNTPOINT
zfspool612K  9.20G   96K  /mnt/zfspool
zfspool/tank96K  9.20G   96K  /mnt/zfspool/tank
ubuntu@z-rotomvm33:~$ sudo journalctl -b | grep -i ordering
ubuntu@z-rotomvm33:~$ lsblk -e7
NAMEMAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda 252:00   30G  0 disk
├─vda1  252:10  512M  0 part  /boot/efi
└─vda2  252:20 29.5G  0 part  /
nvme0n1 259:00  9.8G  0 disk
└─nvme0n1p1 259:10  9.8G  0 part
  └─zfspool 253:00  9.8G  0 crypt
ubuntu@z-rotomvm33:~$ cat /etc/crypttab
# 
zfspool /dev/nvme0n1p1 /etc/keyfile luks

- ZFS on dm-raid
ubuntu@z-rotomvm33:~$ dpkg -l | grep zfsutils
ii  zfsutils-linux   0.8.3-1ubuntu12.12
amd64command-line tools to manage OpenZFS filesystems
ubuntu@z-rotomvm33:~$ zfs list
NAME   USED  AVAIL REFER  MOUNTPOINT
zfspool612K  9.20G   96K  /mnt/zfspool
zfspool/tank96K  9.20G   96K  /mnt/zfspool/tank
ubuntu@z-rotomvm33:~$ sudo journalctl -b | grep -i ordering
ubuntu@z-rotomvm33:~$ lsblk -e7
NAMEMAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
vda 252:00   30G  0 disk
├─vda1  252:10  512M  0 part  /boot/efi
└─vda2  252:20 29.5G  0 part  /
nvme0n1 259:00  9.8G  0 disk
└─md127   9:127  0  9.8G  0 raid0
  ├─md127p1 259:10  9.8G  0 part
  └─md127p9 259:208M  0 part


** Tags added: verification-done verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1875577

Title:
  Encrypted swap won't load on 20.04 with zfs root

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Focal:
  Fix Committed

Bug description:
  [Impact]
  Encrypted swap partitions may not load correctly with ZFS root, due to 
ordering cycle on zfs-mount.service.

  [Test Plan]
  1. Install Ubuntu 20.04 using ZFS-on-root
  2. Add encrypted partition to /etc/crypttab:
     swap/dev/nvme0n1p1  /dev/urandom
swap,cipher=aes-xts-plain64,size=256
  3. Add swap 

[Kernel-packages] [Bug 1875577] Re: Encrypted swap won't load on 20.04 with zfs root

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  Encrypted swap partitions may not load correctly with ZFS root, due to 
ordering cycle on zfs-mount.service.
  
  [Test Plan]
  1. Install Ubuntu 20.04 using ZFS-on-root
  2. Add encrypted partition to /etc/crypttab:
-swap/dev/nvme0n1p1  /dev/urandom
swap,cipher=aes-xts-plain64,size=256
+    swap/dev/nvme0n1p1  /dev/urandom
swap,cipher=aes-xts-plain64,size=256
  3. Add swap partition to /etc/fstab:
-/dev/mapper/swapnoneswapsw  0 0
+    /dev/mapper/swapnoneswapsw  0 0
  4. Reboot and check whether swap has loaded correctly, and whether boot logs 
show ordering cycle:
  [6.638228] systemd[1]: systemd-random-seed.service: Found ordering cycle 
on zfs-mount.service/start
  [6.639418] systemd[1]: systemd-random-seed.service: Found dependency on 
zfs-import.target/start
  [6.640474] systemd[1]: systemd-random-seed.service: Found dependency on 
zfs-import-cache.service/start
  [6.641637] systemd[1]: systemd-random-seed.service: Found dependency on 
cryptsetup.target/start
  [6.642734] systemd[1]: systemd-random-seed.service: Found dependency on 
systemd-cryptsetup@swap.service/start
  [6.643951] systemd[1]: systemd-random-seed.service: Found dependency on 
systemd-random-seed.service/start
  [6.645098] systemd[1]: systemd-random-seed.service: Job 
zfs-mount.service/start deleted to break ordering cycle starting with 
systemd-random-seed.service/start
  [ SKIP ] Ordering cycle found, skipping Mount ZFS filesystems
  
  [Where problems could occur]
  Since we're changing the zfs-mount-generator service, regressions could show 
up during mounting of ZFS partitions. We should thoroughly test different 
scenarios of ZFS such as ZFS-on-root, separate ZFS partitions and the presence 
of swap, to make sure all partitions are mounted correctly and no ordering 
cycles are present.
  
+ Below is a list of suggested test scenarios that we should check for 
regressions:
+ 1. ZFS-on-root + encrypted swap (see "Test Plan" section above)
+ 2. Encrypted root + separate ZFS partitions
+ 3. ZFS on LUKS
+ 4. ZFS on dm-raid
+ 
+ Although scenario 4 is usually advised against (ZFS itself should handle
+ RAID), it's a good smoke test to validate that mount order is being
+ handled correctly.
+ 
  [Other Info]
  This has been fixed upstream by the following commits:
  * ec41cafee1da Fix a dependency loop [0]
  * 62663fb7ec19 Fix another dependency loop [1]
  
  The patches above have been introduced in version 2.1.0, with upstream
  backports to zfs-2.0. In Ubuntu, it's present in Groovy and later
  releases, so it's still needed in Focal.
  
  $ rmadison -a source zfs-linux
-  zfs-linux | 0.8.3-1ubuntu12| focal   | source
-  zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
-  zfs-linux | 0.8.3-1ubuntu12.10 | focal-updates   | source
-  zfs-linux | 0.8.4-1ubuntu11| groovy  | source
-  zfs-linux | 0.8.4-1ubuntu11.2  | groovy-updates  | source
-  zfs-linux | 2.0.2-1ubuntu5 | hirsute | source
-  zfs-linux | 2.0.3-8ubuntu5 | impish  | source
+  zfs-linux | 0.8.3-1ubuntu12| focal   | source
+  zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
+  zfs-linux | 0.8.3-1ubuntu12.10 | focal-updates   | source
+  zfs-linux | 0.8.4-1ubuntu11| groovy  | source
+  zfs-linux | 0.8.4-1ubuntu11.2  | groovy-updates  | source
+  zfs-linux | 2.0.2-1ubuntu5 | hirsute | source
+  zfs-linux | 2.0.3-8ubuntu5 | impish  | source
  
  [0] https://github.com/openzfs/zfs/commit/ec41cafee1da
  [1] https://github.com/openzfs/zfs/commit/62663fb7ec19
  
  ORIGINAL DESCRIPTION
  
  
  root@eu1:/var/log# lsb_release -a
  No LSB modules are available.
  Distributor ID:   Ubuntu
  Description:  Ubuntu 20.04 LTS
  Release:  20.04
  Codename: focal
  
  root@eu1:/var/log# apt-cache policy cryptsetup
  cryptsetup:
    Installed: (none)
    Candidate: 2:2.2.2-3ubuntu2
    Version table:
   2:2.2.2-3ubuntu2 500
  500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
  
  OTHER BACKGROUND INFO:
  ==
  
  1. machine has 2 drives. each drive is partitioned into 2 partitions,
  zfs and swap
  
  2. Ubuntu 20.04 installed on ZFS root using debootstrap
  (debootstrap_1.0.118ubuntu1_all)
  
  3. The ZFS root pool is a 2 partition mirror (the first partition of
  each disk)
  
  4. /etc/crypttab is set up as follows:
  
  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802914-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256
  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802933-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256
  
  WHAT I EXPECTED
  ===
  
  I expected machine would reboot and have encrypted swap that used two
  devices under /dev/mapper
  
  WHAT HAPPENED INSTEAD
  

[Kernel-packages] [Bug 1875577] Re: Encrypted swap won't load on 20.04 with zfs root

[Heitor Alves de Siqueira]

** Description changed:

+ [Impact]
+ Encrypted swap partitions may not load correctly with ZFS root, due to 
ordering cycle on zfs-mount.service.
+ 
+ [Test Plan]
+ 1. Install Ubuntu 20.04 using ZFS-on-root
+ 2. Add encrypted partition to /etc/crypttab:
+swap/dev/nvme0n1p1  /dev/urandom
swap,cipher=aes-xts-plain64,size=256
+ 3. Add swap partition to /etc/fstab:
+/dev/mapper/swapnoneswapsw  0 0
+ 4. Reboot and check whether swap has loaded correctly, and whether boot logs 
show ordering cycle:
+ [6.638228] systemd[1]: systemd-random-seed.service: Found ordering cycle 
on zfs-mount.service/start
+ [6.639418] systemd[1]: systemd-random-seed.service: Found dependency on 
zfs-import.target/start
+ [6.640474] systemd[1]: systemd-random-seed.service: Found dependency on 
zfs-import-cache.service/start
+ [6.641637] systemd[1]: systemd-random-seed.service: Found dependency on 
cryptsetup.target/start
+ [6.642734] systemd[1]: systemd-random-seed.service: Found dependency on 
systemd-cryptsetup@swap.service/start
+ [6.643951] systemd[1]: systemd-random-seed.service: Found dependency on 
systemd-random-seed.service/start
+ [6.645098] systemd[1]: systemd-random-seed.service: Job 
zfs-mount.service/start deleted to break ordering cycle starting with 
systemd-random-seed.service/start
+ [ SKIP ] Ordering cycle found, skipping Mount ZFS filesystems
+ 
+ [Where problems could occur]
+ Since we're changing the zfs-mount-generator service, regressions could show 
up during mounting of ZFS partitions. We should thoroughly test different 
scenarios of ZFS such as ZFS-on-root, separate ZFS partitions and the presence 
of swap, to make sure all partitions are mounted correctly and no ordering 
cycles are present.
+ 
+ [Other Info]
+ This has been fixed upstream by the following commits:
+ * ec41cafee1da Fix a dependency loop [0]
+ * 62663fb7ec19 Fix another dependency loop [1]
+ 
+ The patches above have been introduced in version 2.1.0, with upstream
+ backports to zfs-2.0. In Ubuntu, it's present in Groovy and later
+ releases, so it's still needed in Focal.
+ 
+ $ rmadison -a source zfs-linux
+  zfs-linux | 0.8.3-1ubuntu12| focal   | source
+  zfs-linux | 0.8.3-1ubuntu12.9  | focal-security  | source
+  zfs-linux | 0.8.3-1ubuntu12.10 | focal-updates   | source
+  zfs-linux | 0.8.4-1ubuntu11| groovy  | source
+  zfs-linux | 0.8.4-1ubuntu11.2  | groovy-updates  | source
+  zfs-linux | 2.0.2-1ubuntu5 | hirsute | source
+  zfs-linux | 2.0.3-8ubuntu5 | impish  | source
+ 
+ [0] https://github.com/openzfs/zfs/commit/ec41cafee1da
+ [1] https://github.com/openzfs/zfs/commit/62663fb7ec19
+ 
+ ORIGINAL DESCRIPTION
+ 
+ 
  root@eu1:/var/log# lsb_release -a
  No LSB modules are available.
  Distributor ID:   Ubuntu
  Description:  Ubuntu 20.04 LTS
  Release:  20.04
  Codename: focal
  
  root@eu1:/var/log# apt-cache policy cryptsetup
  cryptsetup:
-   Installed: (none)
-   Candidate: 2:2.2.2-3ubuntu2
-   Version table:
-  2:2.2.2-3ubuntu2 500
- 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
+   Installed: (none)
+   Candidate: 2:2.2.2-3ubuntu2
+   Version table:
+  2:2.2.2-3ubuntu2 500
+ 500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
  
  OTHER BACKGROUND INFO:
  ==
  
  1. machine has 2 drives. each drive is partitioned into 2 partitions,
  zfs and swap
  
+ 2. Ubuntu 20.04 installed on ZFS root using debootstrap
+ (debootstrap_1.0.118ubuntu1_all)
  
- 2. Ubuntu 20.04 installed on ZFS root using debootstrap 
(debootstrap_1.0.118ubuntu1_all)
- 
- 
- 3. The ZFS root pool is a 2 partition mirror (the first partition of each 
disk)
- 
+ 3. The ZFS root pool is a 2 partition mirror (the first partition of
+ each disk)
  
  4. /etc/crypttab is set up as follows:
  
  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802914-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256
  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802933-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256
- 
- 
  
  WHAT I EXPECTED
  ===
  
  I expected machine would reboot and have encrypted swap that used two
  devices under /dev/mapper
  
- 
  WHAT HAPPENED INSTEAD
  =
  
- 
- On reboot, swap setup fails with the following messages in /var/log/syslog:
+ On reboot, swap setup fails with the following messages in
+ /var/log/syslog:
  
  Apr 28 17:13:01 eu1 kernel: [5.360793] systemd[1]: cryptsetup.target: 
Found ordering cycle on systemd-cryptsetup@swap.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360795] systemd[1]: cryptsetup.target: 
Found dependency on systemd-random-seed.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360796] systemd[1]: cryptsetup.target: 
Found dependency on zfs-mount.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360797] 

[Kernel-packages] [Bug 1875577] Re: Encrypted swap won't load on 20.04 with zfs root

[Heitor Alves de Siqueira]

** Changed in: zfs-linux (Ubuntu Focal)
   Importance: Undecided => Medium

** Changed in: zfs-linux (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1875577

Title:
  Encrypted swap won't load on 20.04 with zfs root

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Focal:
  Confirmed

Bug description:
  root@eu1:/var/log# lsb_release -a
  No LSB modules are available.
  Distributor ID:   Ubuntu
  Description:  Ubuntu 20.04 LTS
  Release:  20.04
  Codename: focal

  root@eu1:/var/log# apt-cache policy cryptsetup
  cryptsetup:
Installed: (none)
Candidate: 2:2.2.2-3ubuntu2
Version table:
   2:2.2.2-3ubuntu2 500
  500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages

  OTHER BACKGROUND INFO:
  ==

  1. machine has 2 drives. each drive is partitioned into 2 partitions,
  zfs and swap

  
  2. Ubuntu 20.04 installed on ZFS root using debootstrap 
(debootstrap_1.0.118ubuntu1_all)

  
  3. The ZFS root pool is a 2 partition mirror (the first partition of each 
disk)

  
  4. /etc/crypttab is set up as follows:

  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802914-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256
  swap  
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802933-part2
/dev/urandom   swap,cipher=aes-xts-plain64,size=256


  
  WHAT I EXPECTED
  ===

  I expected machine would reboot and have encrypted swap that used two
  devices under /dev/mapper


  WHAT HAPPENED INSTEAD
  =

  
  On reboot, swap setup fails with the following messages in /var/log/syslog:

  Apr 28 17:13:01 eu1 kernel: [5.360793] systemd[1]: cryptsetup.target: 
Found ordering cycle on systemd-cryptsetup@swap.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360795] systemd[1]: cryptsetup.target: 
Found dependency on systemd-random-seed.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360796] systemd[1]: cryptsetup.target: 
Found dependency on zfs-mount.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360797] systemd[1]: cryptsetup.target: 
Found dependency on zfs-load-module.service/start
  Apr 28 17:13:01 eu1 kernel: [5.360798] systemd[1]: cryptsetup.target: 
Found dependency on cryptsetup.target/start
  Apr 28 17:13:01 eu1 kernel: [5.360799] systemd[1]: cryptsetup.target: Job 
systemd-cryptsetup@swap.service/start deleted to break ordering cycle starting 
with cryptsetup.target/start
  . . . . . .
  Apr 28 17:13:01 eu1 kernel: [5.361082] systemd[1]: Unnecessary job for 
/dev/disk/by-id/nvme-SAMSUNG_MZVLB1T0HALR-0_S3W6NX0M802914-part2 was removed

  
  Also, /dev/mapper does not contain any swap devices:

  root@eu1:/var/log# ls -l /dev/mapper
  total 0
  crw--- 1 root root 10, 236 Apr 28 17:13 control
  root@eu1:/var/log#

  
  And top shows no swap:

  MiB Swap:  0.0 total,  0.0 free,  0.0 used.  63153.6 avail
  Mem

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1875577/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1772675] Re: i40e PF reset due to incorrect MDD event

[Heitor Alves de Siqueira]

Verified for Bionic following the same test procedure from comment #13, running 
on the current -proposed kernel:
ubuntu@snorlax:~$ uname -rv
4.15.0-141-generic #145-Ubuntu SMP Wed Mar 24 18:08:07 UTC 2021

I had to adjust the tx_desc_addr probes, subtracting 0x4 from both offsets and 
changing the relevant registers (r15 for i40e, r11 for i40evf). Other probes 
didn't need any changes.
The test results were similar as on Xenial, netdev watchdog works as expected 
and no major issues were encountered with a smoke test.

** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Committed
Status in linux source package in Bionic:
  Fix Committed
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
  - Create 2 VFs on primary NIC
  - Passthrough VF 1 to a Bionic VM
  - Start iperf3 client on VM, going through i40evf interface
  - Start another iperf3 client on host, going through i40e interface
  Both iperf3 clients should be using an external server located on a separate 
host. By loading the kprobe module while iperf3 is running, we should be able 
to raise MDDs more consistently. MDD behaviour can change according to firmware 
version, so we may need to try with different sets of probes. The one with the 
most consistent results seems to be 'corrupt_tx_desc_addr', which corrupts the 
cmd_type_offset_bsz field of the last TX descriptor before the NIC is notified 
of new data.

  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected, and that necessary PF resets will be issued by the netdev watchdog in 
case of any hung queues.

  ==
  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1772675] Re: i40e PF reset due to incorrect MDD event

[Heitor Alves de Siqueira]

Verified for Xenial following the same test procedure from comment #13, running 
on the current -proposed kernel:
ubuntu@snorlax:~/kprobe$ uname -rv
4.4.0-207-generic #239-Ubuntu SMP Thu Mar 25 02:59:26 UTC 2021

The offsets don't seem to have changed on any of the probes, so I've
used the same set that's already uploaded to the bug. The netdev
watchdog kicks in if queues hang, and general test results look good.

** Tags removed: verification-needed-xenial
** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Committed
Status in linux source package in Bionic:
  Fix Committed
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
  - Create 2 VFs on primary NIC
  - Passthrough VF 1 to a Bionic VM
  - Start iperf3 client on VM, going through i40evf interface
  - Start another iperf3 client on host, going through i40e interface
  Both iperf3 clients should be using an external server located on a separate 
host. By loading the kprobe module while iperf3 is running, we should be able 
to raise MDDs more consistently. MDD behaviour can change according to firmware 
version, so we may need to try with different sets of probes. The one with the 
most consistent results seems to be 'corrupt_tx_desc_addr', which corrupts the 
cmd_type_offset_bsz field of the last TX descriptor before the NIC is notified 
of new data.

  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected, and that necessary PF resets will be issued by the netdev watchdog in 
case of any hung queues.

  ==
  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

There's a build failure for zfs-linux on riscv64/focal, but that should
be safe to ignore for this patch. The riscv64 builds seem to have failed
since they were enabled in 0.8.3-1ubuntu11, and are related to an
"Unsupported ISA type" error in the libspl header files. It's likely
that we're missing some compatibility patches from upstream for this
specific architecture.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Focal:
  Fix Committed
Status in zfs-linux source package in Groovy:
  Fix Committed
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

Verified for Xenial, using the ZFS test suite. I don't think Xenial
ships zfs-test, so I pulled it from upstream and ran it against the DKMS
drivers. No new regressions were reported.

ubuntu@z-rotomvm27:~$ dpkg -l | grep zfs-dkms
ii  zfs-dkms  0.6.5.6-0ubuntu29 
  amd64Native OpenZFS filesystem kernel modules for Linux


** Tags added: verification-done-xenial

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Focal:
  Fix Committed
Status in zfs-linux source package in Groovy:
  Fix Committed
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

Verified for Groovy, through the ZFS test suite. No new regressions were
reported.

ubuntu@racah:/usr/share/zfs$ dpkg -l | grep zfs-dkms
ii  zfs-dkms 0.8.4-1ubuntu11.2   
all  OpenZFS filesystem kernel modules for Linux


** Tags added: verification-done-groovy

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Focal:
  Fix Committed
Status in zfs-linux source package in Groovy:
  Fix Committed
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

Verified for Focal, through the ZFS test suite. No new regressions were
reported.

ubuntu@z-rotomvm29:/usr/share/zfs$ dpkg -l | rg zfs-dkms
ii  zfs-dkms 0.8.3-1ubuntu12.7 
all  OpenZFS filesystem kernel modules for Linux


** Tags added: verification-done-focal

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Focal:
  Fix Committed
Status in zfs-linux source package in Groovy:
  Fix Committed
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

Verified for Bionic, through the ZFS test suite. No new regressions were
reported, and results of the test suite are the same as for the previous
version.

ubuntu@z-rotomvm28:/usr/share/zfs$ dpkg -l |grep zfs-dkms
ii  zfs-dkms   0.7.5-1ubuntu16.11   
   all  OpenZFS filesystem kernel modules for Linux

** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  Fix Committed
Status in zfs-linux source package in Bionic:
  Fix Committed
Status in zfs-linux source package in Focal:
  Fix Committed
Status in zfs-linux source package in Groovy:
  Fix Committed
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in 

[Kernel-packages] [Bug 1772675] Re: i40e PF reset due to incorrect MDD event

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.
  
  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.
  
  [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued
  
  An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
  - Create 2 VFs on primary NIC
  - Passthrough VF 1 to a Bionic VM
  - Start iperf3 client on VM, going through i40evf interface
  - Start another iperf3 client on host, going through i40e interface
  Both iperf3 clients should be using an external server located on a separate 
host. By loading the kprobe module while iperf3 is running, we should be able 
to raise MDDs more consistently. MDD behaviour can change according to firmware 
version, so we may need to try with different sets of probes. The one with the 
most consistent results seems to be 'corrupt_tx_desc_addr', which corrupts the 
cmd_type_offset_bsz field of the last TX descriptor before the NIC is notified 
of new data.
  
  [Regression Potential]
- Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.
+ Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected, and that necessary PF resets will be issued by the netdev watchdog in 
case of any hung queues.
  
  ==
  [original description]
  
  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.
  
  See bug 1713553 and bug 1723127 for more details.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
  - Create 2 VFs on primary NIC
  - 

[Kernel-packages] [Bug 1772675] Re: i40e PF reset due to incorrect MDD event

[Heitor Alves de Siqueira]

I'm attaching kprobes for the Xenial kernels. These are based on the 
4.4.0-203/204 versions.
I followed the same test setup for Bionic, as described in the previous 
comment, and also had similar results. The netdev watchdog seems to take good 
care of any hung queues, so in the end PF resets will be issued regardless, if 
necessary.

** Description changed:

  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.
  
  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.
  
- [Test Case]
+ [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued
+ 
+ An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
+ - Create 2 VFs on primary NIC
+ - Passthrough VF 1 to a Bionic VM
+ - Start iperf3 client on VM, going through i40evf interface
+ - Start another iperf3 client on host, going through i40e interface
+ Both iperf3 clients should be using an external server located on a separate 
host. By loading the kprobe module while iperf3 is running, we should be able 
to raise MDDs more consistently. MDD behaviour can change according to firmware 
version, so we may need to try with different sets of probes. The one with the 
most consistent results seems to be 'corrupt_tx_desc_addr', which corrupts the 
cmd_type_offset_bsz field of the last TX descriptor before the NIC is notified 
of new data.
  
  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.
  
  ==
  [original description]
  
  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.
  
  See bug 1713553 and bug 1723127 for more details.

** Attachment added: "probe_tx_xenial.c"
   
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+attachment/5475473/+files/probe_tx_xenial.c

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Procedure]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  An alternative test procedure makes use of the kprobes attached to the LP 
bug. The test setup is as follows:
  - Create 2 VFs on primary NIC
  - Passthrough VF 1 to a Bionic VM
  - Start iperf3 client on VM, going through 

[Kernel-packages] [Bug 1772675] Re: i40e PF reset due to incorrect MDD event

[Heitor Alves de Siqueira]

Given that we're could be changing reset behavior that might be expected
from the firmware, I wrote a quick set of kprobes to force the firmware
to raise MDD events and test out the patched kernel from the PPA.

I tried to force faulty TX descriptors according to "Table 7-138. Tx
Descriptor Validity Checks" in the XL710 Datasheet, under section
"7.6.2.2.1 Interrupt on Misbehavior of VM (Malicious Driver Detection)".
This document is publicly available at Intel's Technical Library site
for this NIC.

The test setup is as follows:
- Create 2 VFs on primary NIC
- Passthrough VF 1 to a Bionic VM
- Start iperf3 client on VM, going through i40evf interface
- Start another iperf3 client on host, going through i40e interface

The iperf3 servers in my testing were running on a separate host, so I
only had clients using the i40e NIC. This was primarily to verify what
the networking and connectivity impact would be if we ran into any MDDs.

After both iperf3 clients were running, I loaded the kprobe modules
according to a specific TX check to validate. Raising MDDs on the VF
turned out to be pretty trivial, and most of the i40e probes also work
on i40evf. MDDs on the PF were a bit more tricky to get, but I had good
results with corrupting the final TX descriptor's cmd_type_offset_bsz
field. As this happens right before the driver notifies the NIC about
the new data, it should force the firmware to raise the MDD event, as
opposed to us "manually" triggering it from the driver. This has the
benefit of keeping things consistent from the firmware's point of view,
as in the end it is the one responsible for detecting and notifying the
kernel about those events.

The primary point with this test was to verify whether we could leave
the NIC in an inconsistent state, by avoiding or delaying the PF reset.
The results were promising, and should hopefully give some more data on
the value of the upstream patch.

When raising MDDs on the VF, the firmware correctly slaps the
appropriate queues and schedules any resets as required. This is the
same behavior as before. With the test kernel however, we don't issue
any resets to the PF, so the iperf3 tests continue running uninterrupted
as desired.

When raising MDDs on the PF, we don't issue any resets anymore and
depending on what probe was used, connectivity will stop momentarily.
The netdev watchdog kicks in shortly afterwards, and issues a PF reset
as appropriate, and network connectivity resumes. This confirms that
even with the upstream patch any hung queues that don't reset
immediately will recover afterwards, as the queue watchdogs will take
care of those. This is consistent with the upstream behavior, and the
kernel logs look similar as to the one below:

[  573.279608] NETDEV WATCHDOG: ens1f1 (i40e): transmit queue 1 timed out
[  573.279652] WARNING: CPU: 14 PID: 0 at 
/build/linux-lqvoqZ/linux-4.15.0/net/sched/sch_generic.c:323 
dev_watchdog+0x221/0x230
[  573.279659] Modules linked in: vhost_net vhost tap vfio_pci vfio_virqfd 
vfio_iommu_type1 vfio i40evf xt_CHECKSUM iptable_mangle ipt_MASQUERADE 
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 xt_tcpudp 
ebtable_filter devlink ebtables nls_iso8859_1 intel_rapl sb_edac 
x86_pkg_temp_thermal intel_powerclamp ipmi_ssif coretemp intel_cstate 
intel_rapl_perf lpc_ich hpilo ipmi_si ipmi_devintf ipmi_msghandler shpchp 
ioatdma acpi_power_meter mac_hid sch_fq_codel kvm_intel kvm irqbypass 
iptable_filter ip6table_filter ip6_tables br_netfilter bridge stp llc 
arp_tables ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi 
scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 
raid456 async_raid6_recov
[  573.279726]  async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c 
raid1 raid0 multipath linear ses enclosure crct10dif_pclmul crc32_pclmul 
ghash_clmulni_intel pcbc mgag200 i2c_algo_bit ttm aesni_intel aes_x86_64 
crypto_simd glue_helper cryptd drm_kms_helper syscopyarea ixgbe sysfillrect 
sysimgblt fb_sys_fops dca i40e drm tg3 ptp nvme hpsa pps_core nvme_core mdio 
scsi_transport_sas wmi [last unloaded: probe_tx_desc]
[  573.279756] CPU: 14 PID: 0 Comm: swapper/14 Tainted: G   OE
4.15.0-137-generic #141+TEST298651v20210225b1-Ubuntu
[  573.279757] Hardware name: HP ProLiant DL360 Gen9, BIOS P89 05/06/2015
[  573.279763] RIP: 0010:dev_watchdog+0x221/0x230
[  573.279764] RSP: 0018:8f28bf183e58 EFLAGS: 00010286
[  573.279766] RAX:  RBX: 0001 RCX: 083f
[  573.279767] RDX:  RSI: 00f6 RDI: 083f
[  573.279769] RBP: 8f28bf183e88 R08: 0694 R09: 0004
[  573.279770] R10: 8f28bf183ee0 R11: 0001 R12: 0040
[  573.279772] R13: 8f2827c69000 R14: 8f2827c69478 R15: 8f2827fa4f40
[  573.279774] FS:  () GS:8f28bf18() 

[Kernel-packages] [Bug 1772675] Re: Intel i40e PF reset due to incorrect MDD detection

[Heitor Alves de Siqueira]

** Summary changed:

- Intel i40e PF reset due to incorrect MDD detection (continues...again...)
+ Intel i40e PF reset due to incorrect MDD detection

** Summary changed:

- Intel i40e PF reset due to incorrect MDD detection
+ i40e PF reset due to incorrect MDD event

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Case]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.

  ==
  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1772675] Re: Intel i40e PF reset due to incorrect MDD detection (continues...again...)

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.
  
  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.
  
  [Test Case]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued
  
  [Regression Potential]
- Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking.
+ Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.
  
  ==
  [original description]
  
  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.
  
  See bug 1713553 and bug 1723127 for more details.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  i40e PF reset due to incorrect MDD event

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Case]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.

  ==
  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1772675] Re: Intel i40e PF reset due to incorrect MDD detection (continues...again...)

[Heitor Alves de Siqueira]

** Description changed:

- [impact]
+ [Impact]
+ The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.
  
- The i40e driver sometimes causes a "malicious device" event that the
- firmware detects, which causes the firmware to reset the nic, causing an
- interruption in the network connection - which can cause further
- problems, e.g. if the interface is in a bond; the reset will at least
- cause a temporary interruption in network traffic.
+ [Fix]
+ In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.
  
- [fix]
- 
- The fix for this is currently unknown.  As the "MDD event" is generated
- by the i40e firmware, and is completely undocumented, there is no way to
- tell what the i40e driver did to cause the MDD event.
- 
- [test case]
- 
- the bug is unfortunately very difficult to reproduce, but as shown in
- this (and previous) bug comments, some users of the i40e have traffic
- that can consistently reproduce the problem (although usually on the
- order of days, or longer, to reproduce). Reproducing is easily detected,
- as the nw traffic will be interrupted and the system logs will contain a
- message like:
- 
+ [Test Case]
+ The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
+ Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued
  
- [regression potential]
+ [Regression Potential]
+ Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking.
  
- unknown since the specific fix is unknown.
- 
+ ==
  [original description]
  
  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.
  
  See bug 1713553 and bug 1723127 for more details.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  Intel i40e PF reset due to incorrect MDD detection
  (continues...again...)

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [Impact]
  The i40e driver sometimes causes a "malicious device" event that the firmware 
detects, which causes the firmware to reset the NIC, causing an interruption in 
the network connection - which can cause further problems, e.g. if the 
interface is in a bond; the reset will at least cause a temporary interruption 
in network traffic.

  [Fix]
  In the case of MDD events issued for the PF, they are usually the result of a 
misconfigured TX descriptor and not due to "bad" actions in the VFs. We don't 
need to issue a reset to the whole NIC, TX hang checks should handle those if 
necessary.

  [Test Case]
  The bug is unfortunately difficult to reproduce, as there's no detailed 
documentation on how the i40e firmware detects and raises MDDs. We have seen 
reports of this happening in Xenial and Bionic, for workloads stressing i40e 
bonds in LACP mode.
  Reproducing is easily detected, as the network traffic will be interrupted 
and the system logs will contain a message like:
  i40e :02:00.1: TX driver issue detected, PF reset issued

  [Regression Potential]
  Since we're removing resets for the NIC, regressions could show up as issues 
in connectivity after the MDD events are raised. If the firmware expects the 
whole NIC to reset, we could see TX/RX hangs and general unresponsiveness in 
networking. The potential for this should however be fairly low, as this patch 
has been present since kernel 5.2 and hasn't seen any fixes or regressions 
upstream. Basic smoke tests also showed that the driver continues working as 
expected.

  ==
  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely
  
  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
  
  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40
  
  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to have
  two iput() threads racing for the same inode: one of them scheduled
  async and the other executed synchronously as part of the writeback
  path. If the writeback thread tries to evict the inode while the async
  thread is running, it might re-enter the block layer for the same inode
  due to ZFS counters being in an inconsistent state. This then causes the
  kworker thread to stall the writeback, which in turn prevents the
  transaction group sync to complete and locks other ZFS threads.
  
  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]
  
  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].
  
  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527
  
  [Regression Potential]
- The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress in the sync and 
evict paths.
+ The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, 
causing write hangs and eventually stalling all ZFS-backed operations 
indefinitely. We should monitor heavy I/O workloads that put a lot of stress in 
the sync and evict paths to exercise the new changes.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Patch added: "lp1916486-xenial.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1916486/+attachment/5471918/+files/lp1916486-xenial.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Patch added: "lp1916486-groovy.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1916486/+attachment/5471921/+files/lp1916486-groovy.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Patch added: "lp1916486-focal.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1916486/+attachment/5471920/+files/lp1916486-focal.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Patch added: "lp1916486-bionic.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1916486/+attachment/5471919/+files/lp1916486-bionic.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  Fix Released
Status in zfs-linux package in Debian:
  New

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527

  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress in the sync and 
evict paths.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1916486/+subscriptions

-- 

[Kernel-packages] [Bug 1772675] Re: Intel i40e PF reset due to incorrect MDD detection (continues...again...)

[Heitor Alves de Siqueira]

I've pushed some test kernels for X/B to a public PPA assigned to this
bug [0], hopefully it'll make validating the upstream patch a bit
easier.

[0] https://launchpad.net/~halves/+archive/ubuntu/lp1772675

** Changed in: linux (Ubuntu Xenial)
   Importance: Low => High

** Changed in: linux (Ubuntu Bionic)
   Importance: Low => High

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  Intel i40e PF reset due to incorrect MDD detection
  (continues...again...)

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [impact]

  The i40e driver sometimes causes a "malicious device" event that the
  firmware detects, which causes the firmware to reset the nic, causing
  an interruption in the network connection - which can cause further
  problems, e.g. if the interface is in a bond; the reset will at least
  cause a temporary interruption in network traffic.

  [fix]

  The fix for this is currently unknown.  As the "MDD event" is
  generated by the i40e firmware, and is completely undocumented, there
  is no way to tell what the i40e driver did to cause the MDD event.

  [test case]

  the bug is unfortunately very difficult to reproduce, but as shown in
  this (and previous) bug comments, some users of the i40e have traffic
  that can consistently reproduce the problem (although usually on the
  order of days, or longer, to reproduce). Reproducing is easily
  detected, as the nw traffic will be interrupted and the system logs
  will contain a message like:

  i40e :02:00.1: TX driver issue detected, PF reset issued

  [regression potential]

  unknown since the specific fix is unknown.

  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Changed in: zfs-linux (Ubuntu Hirsute)
   Status: In Progress => Fix Released

** Changed in: zfs-linux (Ubuntu Hirsute)
 Assignee: Heitor Alves de Siqueira (halves) => (unassigned)

** Description changed:

  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely
  
  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6
  
  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40
  
  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to have
  two iput() threads racing for the same inode: one of them scheduled
  async and the other executed synchronously as part of the writeback
  path. If the writeback thread tries to evict the inode while the async
  thread is running, it might re-enter the block layer for the same inode
  due to ZFS counters being in an inconsistent state. This then causes the
  kworker thread to stall the writeback, which in turn prevents the
  transaction group sync to complete and locks other ZFS threads.
  
  This is fixed by the upstream commit:
- - Fix zrele race in zrele_async that can cause hang (2921ad6cba54) [0]
+ - Fix zrele race in zrele_async that can cause hang (43eaef6de817) [0]
  
  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].
  
  [0] https://github.com/openzfs/zfs/pull/11530
  [1] https://github.com/openzfs/zfs/issues/11527
  
  [Regression Potential]
  The patch has been tested in the ZFS test suite and in production 
environments, so the potential for further regressions should be fairly 
controlled. Potential regressions might arise in the ZFS writeback path, so we 
should monitor heavy I/O workloads that put a lot of stress in the sync and 
evict paths.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  Fix Released
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status 

[Kernel-packages] [Bug 1916486] Re: zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: New => In Progress

** Changed in: zfs-linux (Ubuntu Xenial)
   Status: New => In Progress

** Changed in: zfs-linux (Ubuntu Focal)
   Status: New => In Progress

** Changed in: zfs-linux (Ubuntu Groovy)
   Status: New => In Progress

** Changed in: zfs-linux (Ubuntu Hirsute)
   Status: Confirmed => In Progress

** Bug watch added: Debian Bug tracker #983331
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983331

** Also affects: zfs-linux (Debian) via
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983331
   Importance: Unknown
   Status: Unknown

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1916486

Title:
  zfs_zrele_async can cause txg sync deadlocks

Status in zfs-linux package in Ubuntu:
  In Progress
Status in zfs-linux source package in Xenial:
  In Progress
Status in zfs-linux source package in Bionic:
  In Progress
Status in zfs-linux source package in Focal:
  In Progress
Status in zfs-linux source package in Groovy:
  In Progress
Status in zfs-linux source package in Hirsute:
  In Progress
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  [Impact]
  TXG sync stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we can see hung task timeouts in the kernel logs 
due to a transaction group deadlock. Userspace process will hang and display 
stack traces similar to the one below:
  [49181.619711] clnt_server D0 21699  28868 0x0320
  [49181.619715] Call Trace:
  [49181.619725]  __schedule+0x24e/0x880
  [49181.619730]  schedule+0x2c/0x80
  [49181.619750]  cv_wait_common+0x11e/0x140 [spl]
  [49181.619763]  ? wait_woken+0x80/0x80
  [49181.619775]  __cv_wait+0x15/0x20 [spl]
  [49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
  [49181.619884]  ? _cond_resched+0x19/0x40
  [49181.619887]  ? mutex_lock+0x12/0x40
  [49181.619959]  zil_commit+0x17/0x20 [zfs]
  [49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
  [49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
  [49181.620100]  vfs_fsync_range+0x51/0xb0
  [49181.620105]  do_fsync+0x3d/0x70
  [49181.620109]  SyS_fsync+0x10/0x20
  [49181.620114]  do_syscall_64+0x73/0x130
  [49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

  We also might see a kworker thread blocking in the zfs writeback/evict path:
  [49181.881570] kworker/u17:3   D0  4915  2 0x8000
  [49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
  [49181.881577] Call Trace:
  [49181.881580]  __schedule+0x24e/0x880
  [49181.881582]  ? atomic_t_wait+0x60/0x60
  [49181.881584]  schedule+0x2c/0x80
  [49181.881588]  bit_wait+0x11/0x60
  [49181.881592]  __wait_on_bit+0x4c/0x90
  [49181.881596]  ? atomic_t_wait+0x60/0x60
  [49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
  [49181.881601]  ? bit_waitqueue+0x40/0x40
  [49181.881605]  inode_wait_for_writeback+0x26/0x40
  [49181.881609]  evict+0xb5/0x1a0
  [49181.881611]  iput+0x19c/0x230
  [49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
  [49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
  [49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
  [49181.881752]  zil_commit+0x17/0x20 [zfs]
  [49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
  [49181.881787]  do_writepages+0x4b/0xe0
  [49181.881790]  __writeback_single_inode+0x45/0x350
  [49181.881792]  ? __writeback_single_inode+0x45/0x350
  [49181.881794]  writeback_sb_inodes+0x1d7/0x530
  [49181.881796]  wb_writeback+0xfb/0x300
  [49181.881799]  wb_workfn+0xad/0x400
  [49181.881800]  ? wb_workfn+0xad/0x400
  [49181.881803]  ? __switch_to_asm+0x35/0x70
  [49181.881809]  process_one_work+0x1de/0x420
  [49181.881811]  worker_thread+0x32/0x410
  [49181.881813]  kthread+0x121/0x140
  [49181.881815]  ? process_one_work+0x420/0x420
  [49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
  [49181.881819]  ret_from_fork+0x35/0x40

  This is caused by a race between ZFS writeback and evict threads,
  usually during a transaction group sync operation. It's possible to
  have two iput() threads racing for the same inode: one of them
  scheduled async and the other executed synchronously as part of the
  writeback path. If the writeback thread tries to evict the inode while
  the async thread is running, it might re-enter the block layer for the
  same inode due to ZFS counters being in an inconsistent state. This
  then causes the kworker thread to stall the writeback, which in turn
  prevents the transaction group sync to complete and locks other ZFS
  threads.

  This is fixed by the upstream commit:
  - Fix zrele race in zrele_async that can cause hang (2921ad6cba54) [0]

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually 

[Kernel-packages] [Bug 1916486] [NEW] zfs_zrele_async can cause txg sync deadlocks

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
TXG sync stalls, causing ZFS workloads to hang indefinitely

[Description]
For certain ZFS workloads, we can see hung task timeouts in the kernel logs due 
to a transaction group deadlock. Userspace process will hang and display stack 
traces similar to the one below:
[49181.619711] clnt_server D0 21699  28868 0x0320
[49181.619715] Call Trace:
[49181.619725]  __schedule+0x24e/0x880
[49181.619730]  schedule+0x2c/0x80
[49181.619750]  cv_wait_common+0x11e/0x140 [spl]
[49181.619763]  ? wait_woken+0x80/0x80
[49181.619775]  __cv_wait+0x15/0x20 [spl]
[49181.619872]  zil_commit.part.14+0x80/0x8c0 [zfs]
[49181.619884]  ? _cond_resched+0x19/0x40
[49181.619887]  ? mutex_lock+0x12/0x40
[49181.619959]  zil_commit+0x17/0x20 [zfs]
[49181.620026]  zfs_fsync+0x77/0xe0 [zfs]
[49181.620093]  zpl_fsync+0x68/0xa0 [zfs]
[49181.620100]  vfs_fsync_range+0x51/0xb0
[49181.620105]  do_fsync+0x3d/0x70
[49181.620109]  SyS_fsync+0x10/0x20
[49181.620114]  do_syscall_64+0x73/0x130
[49181.620119]  entry_SYSCALL_64_after_hwframe+0x41/0xa6

We also might see a kworker thread blocking in the zfs writeback/evict path:
[49181.881570] kworker/u17:3   D0  4915  2 0x8000
[49181.881576] Workqueue: writeback wb_workfn (flush-zfs-10)
[49181.881577] Call Trace:
[49181.881580]  __schedule+0x24e/0x880
[49181.881582]  ? atomic_t_wait+0x60/0x60
[49181.881584]  schedule+0x2c/0x80
[49181.881588]  bit_wait+0x11/0x60
[49181.881592]  __wait_on_bit+0x4c/0x90
[49181.881596]  ? atomic_t_wait+0x60/0x60
[49181.881599]  __inode_wait_for_writeback+0xb9/0xf0
[49181.881601]  ? bit_waitqueue+0x40/0x40
[49181.881605]  inode_wait_for_writeback+0x26/0x40
[49181.881609]  evict+0xb5/0x1a0
[49181.881611]  iput+0x19c/0x230
[49181.881648]  zfs_iput_async+0x1d/0x80 [zfs]
[49181.881682]  zfs_get_data+0x1d4/0x2a0 [zfs]
[49181.881718]  zil_commit.part.14+0x640/0x8c0 [zfs]
[49181.881752]  zil_commit+0x17/0x20 [zfs]
[49181.881784]  zpl_writepages+0xd5/0x160 [zfs]
[49181.881787]  do_writepages+0x4b/0xe0
[49181.881790]  __writeback_single_inode+0x45/0x350
[49181.881792]  ? __writeback_single_inode+0x45/0x350
[49181.881794]  writeback_sb_inodes+0x1d7/0x530
[49181.881796]  wb_writeback+0xfb/0x300
[49181.881799]  wb_workfn+0xad/0x400
[49181.881800]  ? wb_workfn+0xad/0x400
[49181.881803]  ? __switch_to_asm+0x35/0x70
[49181.881809]  process_one_work+0x1de/0x420
[49181.881811]  worker_thread+0x32/0x410
[49181.881813]  kthread+0x121/0x140
[49181.881815]  ? process_one_work+0x420/0x420
[49181.881817]  ? kthread_create_worker_on_cpu+0x70/0x70
[49181.881819]  ret_from_fork+0x35/0x40

This is caused by a race between ZFS writeback and evict threads,
usually during a transaction group sync operation. It's possible to have
two iput() threads racing for the same inode: one of them scheduled
async and the other executed synchronously as part of the writeback
path. If the writeback thread tries to evict the inode while the async
thread is running, it might re-enter the block layer for the same inode
due to ZFS counters being in an inconsistent state. This then causes the
kworker thread to stall the writeback, which in turn prevents the
transaction group sync to complete and locks other ZFS threads.

This is fixed by the upstream commit:
- Fix zrele race in zrele_async that can cause hang (2921ad6cba54) [0]

[Test Case]
Being a race condition, this issue has been hard to reproduce consistently. 
This has been reported on heavy I/O workloads, mixing file creation and 
deletion. We have some reports both from upstream and from Ubuntu users that 
this is usually reproducible on e.g. heavy SQL workloads or on complex 
ccache-enabled builds [1].

[0] https://github.com/openzfs/zfs/pull/11530
[1] https://github.com/openzfs/zfs/issues/11527

[Regression Potential]
The patch has been tested in the ZFS test suite and in production environments, 
so the potential for further regressions should be fairly controlled. Potential 
regressions might arise in the ZFS writeback path, so we should monitor heavy 
I/O workloads that put a lot of stress in the sync and evict paths.

** Affects: zfs-linux (Ubuntu)
 Importance: Critical
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress

** Affects: zfs-linux (Ubuntu Xenial)
 Importance: High
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress

** Affects: zfs-linux (Ubuntu Bionic)
 Importance: Critical
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress

** Affects: zfs-linux (Ubuntu Focal)
 Importance: Critical
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress

** Affects: zfs-linux (Ubuntu Groovy)
 Importance: Critical
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress

** Affects: zfs-linux (Ubuntu Hirsute)
 Importance: Critical
 Assignee: Heitor Alves de Siqueira (halves)
 Status: In Progress


** Tags: sts

** Description changed:

+ [Impact

[Kernel-packages] [Bug 1772675] Re: Intel i40e PF reset due to incorrect MDD detection (continues...again...)

[Heitor Alves de Siqueira]

The patch [0] is present in Ubuntu releases starting with Focal, so it
should be fixed for that and later releases (including current bionic-
hwe). I'll look into backporting and testing this for Xenial and Bionic
GA.

$ git log --oneline -1 a1df906c5be7
  a1df906c5be7 i40e: change behavior on PF in response to MDD event
$ git describe --contains a1df906c5be7
  v5.2-rc1~133^2~57^2~9

[0] https://git.kernel.org/linus/a1df906c5be7

** Changed in: linux (Ubuntu)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Xenial)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Cosmic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu Bionic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux (Ubuntu)
   Status: Expired => Fix Released

** Changed in: linux (Ubuntu Xenial)
   Status: Expired => In Progress

** Changed in: linux (Ubuntu Bionic)
   Status: Expired => In Progress

** Changed in: linux (Ubuntu Cosmic)
   Status: Expired => In Progress

** Tags added: sts

** Changed in: linux (Ubuntu Cosmic)
   Status: In Progress => Won't Fix

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1772675

Title:
  Intel i40e PF reset due to incorrect MDD detection
  (continues...again...)

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  In Progress
Status in linux source package in Bionic:
  In Progress
Status in linux source package in Cosmic:
  Won't Fix

Bug description:
  [impact]

  The i40e driver sometimes causes a "malicious device" event that the
  firmware detects, which causes the firmware to reset the nic, causing
  an interruption in the network connection - which can cause further
  problems, e.g. if the interface is in a bond; the reset will at least
  cause a temporary interruption in network traffic.

  [fix]

  The fix for this is currently unknown.  As the "MDD event" is
  generated by the i40e firmware, and is completely undocumented, there
  is no way to tell what the i40e driver did to cause the MDD event.

  [test case]

  the bug is unfortunately very difficult to reproduce, but as shown in
  this (and previous) bug comments, some users of the i40e have traffic
  that can consistently reproduce the problem (although usually on the
  order of days, or longer, to reproduce). Reproducing is easily
  detected, as the nw traffic will be interrupted and the system logs
  will contain a message like:

  i40e :02:00.1: TX driver issue detected, PF reset issued

  [regression potential]

  unknown since the specific fix is unknown.

  [original description]

  This is a continuation from bug 1713553 and then bug 1723127; a patch
  was added in the first bug and then the second bug, to attempt to fix
  this, and it may have helped reduce the issue but appears not to have
  fixed it, based on more reports.

  See bug 1713553 and bug 1723127 for more details.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1772675/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1840043] Re: bcache: Performance degradation when querying priority_stats

[Heitor Alves de Siqueira]

@pponnuvel stat(2) doesn't perform a read(2) on the file, so it won't
call the show() method for priority_stats (which was the problematic
call for this specific sysfs attribute).

In any case, this should be fixed in all supported Ubuntu series, so
even read(2) shouldn't be an issue anymore. Please let us know if you
see any performance regressions related to this change!

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1840043

Title:
  bcache: Performance degradation when querying priority_stats

Status in Linux:
  Fix Released
Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Fix Released
Status in linux source package in Bionic:
  Fix Released
Status in linux source package in Disco:
  Fix Released
Status in linux source package in Eoan:
  Fix Released

Bug description:
  [Impact]
  Querying bcache's priority_stats attribute in sysfs causes severe performance 
degradation for read/write workloads and occasional system stalls

  [Test Case]
  Note: As the sorting step has the most noticeable performance impact, the 
test case below pins a workload and the sysfs query to the same CPU. CPU 
contention issues still occur without any pinning, this just removes the 
scheduling factor of landing in different CPUs and affecting different tasks.

  1) Start a read/write workload on the bcache device with e.g. fio or dd, 
pinned to a certain CPU:
  # taskset 0x10 dd if=/dev/zero of=/dev/bcache0 bs=4k status=progress

  2) Start a sysfs query loop for the priority_stats attribute pinned to the 
same CPU:
  # for i in {1..10}; do taskset 0x10 cat 
/sys/fs/bcache/*/cache0/priority_stats > /dev/null; done

  3) Monitor the read/write workload for any performance impact

  [Fix]
  To fix CPU contention and performance impact, a cond_resched() call is 
introduced in the priority_stats sort comparison.

  [Regression Potential]
  Regression potential is low, as the change is confined to the priority_stats 
sysfs query. In cases where frequent queries to bcache priority_stats take 
place (e.g. node_exporter), the impact should be more noticeable as those could 
now take a bit longer to complete. A regression due to this patch would most 
likely show up as a performance degradation in bcache-focused workloads.

  --

  [Description]
  In the latest bcache drivers, there's a sysfs attribute that calculates 
bucket priority statistics in /sys/fs/bcache/*/cache0/priority_stats. Querying 
this file has a big performance impact on tasks that run in the same CPU, and 
also affects read/write performance of the bcache device itself.

  This is due to the way the driver calculates the stats: the bcache
  buckets are locked and iterated through, collecting information about
  each individual bucket. An array of nbucket elements is constructed
  and sorted afterwards, which can cause very high CPU contention in
  cases of larger bcache setups.

  From our tests, the sorting step of the priority_stats query causes
  the most expressive performance reduction, as it can hinder tasks that
  are not even doing any bcache IO. If a task is "unlucky" to be
  scheduled in the same CPU as the sysfs query, its performance will be
  harshly reduced as both compete for CPU time. We've had users report
  systems stalls of up to ~6s due to this, as a result from monitoring
  tools that query the priority_stats periodically (e.g. Prometheus Node
  Exporter from [0]). These system stalls have triggered several other
  issues such as ceph-mon re-elections, problems in percona-cluster and
  general network stalls, so the impact is not isolated to bcache IO
  workloads.

  An example benchmark can be seen in [1], where the read performance on
  a bcache device suffered quite heavily (going from ~40k IOPS to ~4k
  IOPS due to priority_stats). Other comparison charts are found under
  [2].

  [0] https://github.com/prometheus/node_exporter
  [1] 
https://people.canonical.com/~halves/priority_stats/read/4k-iops-2Dsmooth.png
  [2] https://people.canonical.com/~halves/priority_stats/

To manage notifications about this bug go to:
https://bugs.launchpad.net/linux/+bug/1840043/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1908264] Re: Disable Atari partition support for linux-aws

[Heitor Alves de Siqueira]

Patches sent to the kernel-team list:
https://lists.ubuntu.com/archives/kernel-team/2020-December/115633.html

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1908264

Title:
  Disable Atari partition support for linux-aws

Status in linux-aws package in Ubuntu:
  In Progress
Status in linux-aws source package in Xenial:
  In Progress
Status in linux-aws source package in Bionic:
  In Progress
Status in linux-aws source package in Focal:
  In Progress
Status in linux-aws source package in Groovy:
  In Progress
Status in linux-aws source package in Hirsute:
  In Progress

Bug description:
  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

  [Regression Potential]
  There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and is unlikely to be used in AWS instances, the regression potential should be 
very low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1908264/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1908264] Re: Disable Atari partition support for linux-aws

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.
  
  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html
  
  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)
  
  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu
  
  [Regression Potential]
- There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.
+ There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and is unlikely to be used in AWS instances, the regression potential should be 
very low.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1908264

Title:
  Disable Atari partition support for linux-aws

Status in linux-aws package in Ubuntu:
  In Progress
Status in linux-aws source package in Xenial:
  In Progress
Status in linux-aws source package in Bionic:
  In Progress
Status in linux-aws source package in Focal:
  In Progress
Status in linux-aws source package in Groovy:
  In Progress
Status in linux-aws source package in Hirsute:
  In Progress

Bug description:
  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

  [Regression Potential]
  There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and is unlikely to be used in AWS instances, the regression potential should be 
very low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1908264/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1908264] Re: Disable Atari partition support for linux-aws

[Heitor Alves de Siqueira]

** Changed in: linux-aws (Ubuntu Xenial)
   Status: New => Confirmed

** Changed in: linux-aws (Ubuntu Bionic)
   Status: New => Confirmed

** Changed in: linux-aws (Ubuntu Focal)
   Status: New => Confirmed

** Changed in: linux-aws (Ubuntu Groovy)
   Status: New => Confirmed

** Changed in: linux-aws (Ubuntu Xenial)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-aws (Ubuntu Focal)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-aws (Ubuntu Bionic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-aws (Ubuntu Focal)
   Status: Confirmed => In Progress

** Changed in: linux-aws (Ubuntu Hirsute)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-aws (Ubuntu Hirsute)
   Status: Confirmed => In Progress

** Changed in: linux-aws (Ubuntu Groovy)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: linux-aws (Ubuntu Groovy)
   Status: Confirmed => In Progress

** Changed in: linux-aws (Ubuntu Bionic)
   Status: Confirmed => In Progress

** Changed in: linux-aws (Ubuntu Xenial)
   Status: Confirmed => In Progress

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1908264

Title:
  Disable Atari partition support for linux-aws

Status in linux-aws package in Ubuntu:
  In Progress
Status in linux-aws source package in Xenial:
  In Progress
Status in linux-aws source package in Bionic:
  In Progress
Status in linux-aws source package in Focal:
  In Progress
Status in linux-aws source package in Groovy:
  In Progress
Status in linux-aws source package in Hirsute:
  In Progress

Bug description:
  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

  [Regression Potential]
  There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1908264/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1908264] Re: Disable Atari partition support for linux-aws

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
- Atari partition tables have very broad specifications, and can spontaneously
- show up on uncleared disks with "garbage" data. There have been reports of
- misinterpreted AHDI partition tables before (see [0] and [1]), usually as an
- unintended side-effect of using non-zeroed disks. Users of this specific
- partition scheme are unlikely to be on AWS instances, so we should disable 
them.
+ Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.
  
  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html
  
  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)
  
  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu
  
  [Regression Potential]
- There could be unexpected dependencies on Atari partitions that are impacted 
by
- this change. Considering Atari hardware hasn't been produced in a long time, 
and
- are unlikely to be used in AWS instances, the regression potential should be
- very low.
+ There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1908264

Title:
  Disable Atari partition support for linux-aws

Status in linux-aws package in Ubuntu:
  Confirmed
Status in linux-aws source package in Xenial:
  New
Status in linux-aws source package in Bionic:
  New
Status in linux-aws source package in Focal:
  New
Status in linux-aws source package in Groovy:
  New
Status in linux-aws source package in Hirsute:
  Confirmed

Bug description:
  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

  [Regression Potential]
  There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1908264/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1908264] [NEW] Disable Atari partition support for linux-aws

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

[0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
[1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

[Test Case]
Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

[Fix]
Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

[Regression Potential]
There could be unexpected dependencies on Atari partitions that are impacted by 
this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.

** Affects: linux-aws (Ubuntu)
 Importance: High
 Status: Confirmed

** Affects: linux-aws (Ubuntu Xenial)
 Importance: Undecided
 Status: New

** Affects: linux-aws (Ubuntu Bionic)
 Importance: Undecided
 Status: New

** Affects: linux-aws (Ubuntu Focal)
 Importance: Undecided
 Status: New

** Affects: linux-aws (Ubuntu Groovy)
 Importance: Undecided
 Status: New

** Affects: linux-aws (Ubuntu Hirsute)
 Importance: High
 Status: Confirmed


** Tags: sts

** Also affects: linux-aws (Ubuntu Hirsute)
   Importance: High
   Status: Confirmed

** Also affects: linux-aws (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Also affects: linux-aws (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Also affects: linux-aws (Ubuntu Focal)
   Importance: Undecided
   Status: New

** Also affects: linux-aws (Ubuntu Groovy)
   Importance: Undecided
   Status: New

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux-aws in Ubuntu.
https://bugs.launchpad.net/bugs/1908264

Title:
  Disable Atari partition support for linux-aws

Status in linux-aws package in Ubuntu:
  Confirmed
Status in linux-aws source package in Xenial:
  New
Status in linux-aws source package in Bionic:
  New
Status in linux-aws source package in Focal:
  New
Status in linux-aws source package in Groovy:
  New
Status in linux-aws source package in Hirsute:
  Confirmed

Bug description:
  [Impact]
  Atari partition tables have very broad specifications, and can spontaneously 
show up on uncleared disks with "garbage" data. There have been reports of 
misinterpreted AHDI partition tables before (see [0] and [1]), usually as an 
unintended side-effect of using non-zeroed disks. Users of this specific 
partition scheme are unlikely to be on AWS instances, so we should disable them.

  [0] 
https://lore.kernel.org/linux-block/1467736712-11825-1-git-send-email-kris...@linux.vnet.ibm.com/
  [1] https://mail-index.netbsd.org/port-atari/1995/11/19/0001.html

  [Test Case]
  Ensure that CONFIG_ATARI_PARTITION is not set in /boot/config-$(uname -r)

  [Fix]
  Unset CONFIG_ATARI_PARTITION in debian.aws/config/config.common.ubuntu

  [Regression Potential]
  There could be unexpected dependencies on Atari partitions that are impacted 
by this change. Considering Atari hardware hasn't been produced in a long time, 
and are unlikely to be used in AWS instances, the regression potential should 
be very low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux-aws/+bug/1908264/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1852432] Re: i40e: Setting VF MAC address causes General Protection Fault

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu)
   Status: Confirmed => Fix Released

** Changed in: linux (Ubuntu Focal)
   Status: Confirmed => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1852432

Title:
  i40e: Setting VF MAC address causes General Protection Fault

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Bionic:
  Fix Released
Status in linux source package in Disco:
  Fix Released
Status in linux source package in Eoan:
  Fix Released
Status in linux source package in Focal:
  Fix Released

Bug description:
  [Impact]
   * Creating SR-IOV enabled VMs in Openstack can sometimes trigger the GPF and 
leave system unusable

  [Test Case]
   * Continuously spin up VFs and set MAC address with e.g. ifconfig

  [Fix]
   * The fix updates the VSI pointer passed down to i40e_set_vf_mac function() 
if the adapter is still in reset, preventing the GPF.

  [Regression Potential]
   * Regression potential should be low, as we're now updating the VSI using 
the ID stored in the VF pointer
   * Regressions could arise from issues in VF creation or reset, as that would 
corrupt the new VSI pointer
   * Patch was validated and tested in a production environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1852432/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1869229] Re: Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

[Heitor Alves de Siqueira]

For reference, the kernel spew of the BUG_ON:

[   78.354129] kernel BUG at 
/home/ubuntu/xenial-aws/drivers/nvme/host/pci.c:619!
[   78.357297] invalid opcode:  [#1] SMP
[   78.359613] Modules linked in: dm_snapshot dm_bufio xfs ppdev serio_raw 
parport_pc 8250_fintek parport i2c_piix4 ib_iser rdma_cm iw_cm ib_cm ib_sa 
ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 
autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor 
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear crct10dif_pclmul 
crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul 
glue_helper ablk_helper cryptd ena
[   78.387878] CPU: 0 PID: 1687 Comm: mount Not tainted 4.4.0-1105-aws #116
[   78.390837] Hardware name: Amazon EC2 c5d.large/, BIOS 1.0 10/16/2017
[   78.393692] task: 8800bb155400 ti: 8800b93bc000 task.ti: 
8800b93bc000
[   78.396973] RIP: 0010:[]  [] 
nvme_queue_rq+0x8c6/0xa60
[   78.400787] RSP: 0018:8800b93bf7c8  EFLAGS: 00010286
[   78.403151] RAX: 0078 RBX: 1000 RCX: 1000
[   78.406276] RDX:  RSI: 0246 RDI: 
[   78.409390] RBP: 8800b93bf8a8 R08: 8800b916c700 R09: 1000
[   78.412518] R10: 0001ec00 R11: 8800b8e3 R12: fc00
[   78.417056] R13: 0010 R14: fc00 R15: 35fd5000
[   78.421581] FS:  7f30fe043840() GS:880130a0() 
knlGS:
[   78.427884] CS:  0010 DS:  ES:  CR0: 80050033
[   78.431827] CR2: 7f57d4057889 CR3: 35974000 CR4: 00360670
[   78.436322] DR0:  DR1:  DR2: 
[   78.440821] DR3:  DR6: fffe0ff0 DR7: 0400
[   78.445316] Stack:
[   78.447706]  880036009480 880036009700 8800b7782800 
0ff8
[   78.454583]  8800b8e30420 8800360a9400 8801fc00 
8800b7697b00
[   78.461462]  88011000 8800b8e3 88003604c000 
0001ffc00400
[   78.468332] Call Trace:
[   78.470921]  [] blk_mq_make_request+0x407/0x550
[   78.475001]  [] generic_make_request+0x114/0x2d0
[   78.479110]  [] ? bvec_alloc+0x91/0x100
[   78.482936]  [] submit_bio+0x76/0x160
[   78.486680]  [] _xfs_buf_ioapply+0x2e4/0x4a0 [xfs]
[   78.490866]  [] ? wake_up_q+0x70/0x70
[   78.494601]  [] ? xfs_bwrite+0x24/0x60 [xfs]
[   78.498583]  [] xfs_buf_submit_wait+0x5d/0x230 [xfs]
[   78.502861]  [] xfs_bwrite+0x24/0x60 [xfs]
[   78.506785]  [] xlog_bwrite+0x7f/0x100 [xfs]
[   78.510787]  [] xlog_write_log_records+0x1a4/0x230 [xfs]
[   78.515192]  [] xlog_clear_stale_blocks+0xb7/0x1b0 [xfs]
[   78.519596]  [] ? xlog_bread+0x3f/0x50 [xfs]
[   78.523588]  [] xlog_find_tail+0x2db/0x3b0 [xfs]
[   78.527705]  [] xlog_recover+0x2d/0x160 [xfs]
[   78.531720]  [] xfs_log_mount+0xdb/0x2a0 [xfs]
[   78.535767]  [] xfs_mountfs+0x4f3/0x870 [xfs]
[   78.539788]  [] ? xfs_mru_cache_create+0x12b/0x180 [xfs]
[   78.544197]  [] xfs_fs_fill_super+0x3bb/0x4e0 [xfs]
[   78.548400]  [] mount_bdev+0x270/0x2c0
[   78.552169]  [] ? xfs_parseargs+0xab0/0xab0 [xfs]
[   78.556338]  [] xfs_fs_mount+0x15/0x20 [xfs]
[   78.560337]  [] mount_fs+0x3d/0x170
[   78.564091]  [] ? __alloc_percpu+0x15/0x20
[   78.568060]  [] vfs_kern_mount+0x67/0x110
[   78.571971]  [] do_mount+0x25f/0xda0
[   78.575692]  [] ? mntput+0x24/0x40
[   78.579334]  [] ? __kmalloc_track_caller+0x1b6/0x250
[   78.583595]  [] ? __fput+0x193/0x230
[   78.587296]  [] ? memdup_user+0x42/0x70
[   78.59]  [] SyS_mount+0x9f/0x100
[   78.594804]  [] entry_SYSCALL_64_fastpath+0x22/0xcb
[   78.599029] Code: 11 e3 e3 ff 44 8b 95 50 ff ff ff 48 89 85 68 ff ff ff 4c 
8b 48 10 44 8b 58 18 8b 95 58 ff ff ff 8b 8d 60 ff ff ff e9 0a fd ff ff <0f> 0b 
48 8b 73 68 48 8b bd 70 ff ff ff e8 58 c5 e2 ff 83 f8 01
[   78.625198] RIP  [] nvme_queue_rq+0x8c6/0xa60
[   78.629410]  RSP 
[   78.632442] ---[ end trace de20412ccd13806e ]---

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1869229

Title:
  Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Confirmed

Bug description:
  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.

  Upstream commit 729204ef49ec ("block: relax check on sg gap")
  introduced a way to merge bios if they are physically contiguous. This
  can lead to issues if one rq starts with a non-aligned buffer, as it
  can cause the merged segment to end in an unaligned virtual boundary.
  In some AWS instances, it's possible to craft such a request when
  attempting to mount LVM snapshots using xfs. This will then cause a
  kernel spew due to a BUG_ON in nvme_setup_prps(), which checks if
  dma_len is aligned to the 

[Kernel-packages] [Bug 1869229] Re: Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

[Heitor Alves de Siqueira]

Patches sent out to kernel-team list:

https://lists.ubuntu.com/archives/kernel-team/2020-March/108551.html
https://lists.ubuntu.com/archives/kernel-team/2020-March/108552.html

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1869229

Title:
  Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Confirmed

Bug description:
  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.

  Upstream commit 729204ef49ec ("block: relax check on sg gap")
  introduced a way to merge bios if they are physically contiguous. This
  can lead to issues if one rq starts with a non-aligned buffer, as it
  can cause the merged segment to end in an unaligned virtual boundary.
  In some AWS instances, it's possible to craft such a request when
  attempting to mount LVM snapshots using xfs. This will then cause a
  kernel spew due to a BUG_ON in nvme_setup_prps(), which checks if
  dma_len is aligned to the page size.

  [Fix]
  Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") prevents requests that begin with an unaligned buffer from being 
merged.

  [Test Case]
  This has been verified on AWS with c5d.large instances:

  1) Prepare the LVM device + snapshot
  $ sudo vgcreate vg0 /dev/nvme1n1
  $ sudo lvcreate -L5G -n data0 vg0
  $ sudo mkfs.xfs /dev/vg0/data0
  $ sudo mount /dev/vg0/data0 /mnt
  $ sudo touch /mnt/test
  $ sudo touch /mnt/test2
  $ sudo ls /mnt
  $ sudo umount /mnt
  $ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap

  2) Attempting to mount the previously created snapshot results in the Oops:
  $ sudo mount /dev/vg0/data0_snap /mnt
  Segmentation fault (core dumped)

  [Regression Potential]
  The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
  The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1869229/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1869229] Re: Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

[Heitor Alves de Siqueira]

** Description changed:

  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.
  
  Upstream commit 729204ef49ec ("block: relax check on sg gap") introduced
  a way to merge bios if they are physically contiguous. This can lead to
  issues if one rq starts with a non-aligned buffer, as it can cause the
  merged segment to end in an unaligned virtual boundary. In some AWS
  instances, it's possible to craft such a request when attempting to
  mount LVM snapshots using xfs. This will then cause a kernel spew due to
  a BUG_ON in nvme_setup_prps(), which checks if dma_len is aligned to the
  page size.
  
  [Fix]
- Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") disallows requests that begin with an unaligned buffer from being 
merged.
+ Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") prevents requests that begin with an unaligned buffer from being 
merged.
  
  [Test Case]
  This has been verified on AWS with c5d.large instances:
  
  1) Prepare the LVM device + snapshot
  $ sudo vgcreate vg0 /dev/nvme1n1
  $ sudo lvcreate -L5G -n data0 vg0
  $ sudo mkfs.xfs /dev/vg0/data0
  $ sudo mount /dev/vg0/data0 /mnt
  $ sudo touch /mnt/test
  $ sudo touch /mnt/test2
  $ sudo ls /mnt
  $ sudo umount /mnt
  $ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap
  
  2) Attempting to mount the previously created snapshot results in the Oops:
- $ sudo mount /dev/vg0/data0_snap /mnt 
+ $ sudo mount /dev/vg0/data0_snap /mnt
  Segmentation fault (core dumped)
  
  [Regression Potential]
  The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
  The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1869229

Title:
  Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Confirmed

Bug description:
  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.

  Upstream commit 729204ef49ec ("block: relax check on sg gap")
  introduced a way to merge bios if they are physically contiguous. This
  can lead to issues if one rq starts with a non-aligned buffer, as it
  can cause the merged segment to end in an unaligned virtual boundary.
  In some AWS instances, it's possible to craft such a request when
  attempting to mount LVM snapshots using xfs. This will then cause a
  kernel spew due to a BUG_ON in nvme_setup_prps(), which checks if
  dma_len is aligned to the page size.

  [Fix]
  Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") prevents requests that begin with an unaligned buffer from being 
merged.

  [Test Case]
  This has been verified on AWS with c5d.large instances:

  1) Prepare the LVM device + snapshot
  $ sudo vgcreate vg0 /dev/nvme1n1
  $ sudo lvcreate -L5G -n data0 vg0
  $ sudo mkfs.xfs /dev/vg0/data0
  $ sudo mount /dev/vg0/data0 /mnt
  $ sudo touch /mnt/test
  $ sudo touch /mnt/test2
  $ sudo ls /mnt
  $ sudo umount /mnt
  $ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap

  2) Attempting to mount the previously created snapshot results in the Oops:
  $ sudo mount /dev/vg0/data0_snap /mnt
  Segmentation fault (core dumped)

  [Regression Potential]
  The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
  The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1869229/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1869229] Re: Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

[Heitor Alves de Siqueira]

** Changed in: linux (Ubuntu Xenial)
   Status: New => Confirmed

** Changed in: linux (Ubuntu Xenial)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1869229

Title:
  Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  Confirmed

Bug description:
  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.

  Upstream commit 729204ef49ec ("block: relax check on sg gap")
  introduced a way to merge bios if they are physically contiguous. This
  can lead to issues if one rq starts with a non-aligned buffer, as it
  can cause the merged segment to end in an unaligned virtual boundary.
  In some AWS instances, it's possible to craft such a request when
  attempting to mount LVM snapshots using xfs. This will then cause a
  kernel spew due to a BUG_ON in nvme_setup_prps(), which checks if
  dma_len is aligned to the page size.

  [Fix]
  Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") disallows requests that begin with an unaligned buffer from being 
merged.

  [Test Case]
  This has been verified on AWS with c5d.large instances:

  1) Prepare the LVM device + snapshot
  $ sudo vgcreate vg0 /dev/nvme1n1
  $ sudo lvcreate -L5G -n data0 vg0
  $ sudo mkfs.xfs /dev/vg0/data0
  $ sudo mount /dev/vg0/data0 /mnt
  $ sudo touch /mnt/test
  $ sudo touch /mnt/test2
  $ sudo ls /mnt
  $ sudo umount /mnt
  $ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap

  2) Attempting to mount the previously created snapshot results in the Oops:
  $ sudo mount /dev/vg0/data0_snap /mnt 
  Segmentation fault (core dumped)

  [Regression Potential]
  The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
  The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1869229/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1869229] [NEW] Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

[Heitor Alves de Siqueira]

Public bug reported:

[Impact]
When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in nvme 
driver.

Upstream commit 729204ef49ec ("block: relax check on sg gap") introduced
a way to merge bios if they are physically contiguous. This can lead to
issues if one rq starts with a non-aligned buffer, as it can cause the
merged segment to end in an unaligned virtual boundary. In some AWS
instances, it's possible to craft such a request when attempting to
mount LVM snapshots using xfs. This will then cause a kernel spew due to
a BUG_ON in nvme_setup_prps(), which checks if dma_len is aligned to the
page size.

[Fix]
Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") disallows requests that begin with an unaligned buffer from being 
merged.

[Test Case]
This has been verified on AWS with c5d.large instances:

1) Prepare the LVM device + snapshot
$ sudo vgcreate vg0 /dev/nvme1n1
$ sudo lvcreate -L5G -n data0 vg0
$ sudo mkfs.xfs /dev/vg0/data0
$ sudo mount /dev/vg0/data0 /mnt
$ sudo touch /mnt/test
$ sudo touch /mnt/test2
$ sudo ls /mnt
$ sudo umount /mnt
$ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap

2) Attempting to mount the previously created snapshot results in the Oops:
$ sudo mount /dev/vg0/data0_snap /mnt 
Segmentation fault (core dumped)

[Regression Potential]
The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

** Affects: linux (Ubuntu)
 Importance: Undecided
 Status: Fix Released

** Affects: linux (Ubuntu Xenial)
 Importance: Undecided
 Status: New


** Tags: sts

** Changed in: linux (Ubuntu)
     Assignee: Heitor Alves de Siqueira (halves) => (unassigned)

** Also affects: linux (Ubuntu Xenial)
   Importance: Undecided
   Status: New

** Changed in: linux (Ubuntu)
   Status: New => Fix Released

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1869229

Title:
  Mounting LVM snapshots with xfs can hit kernel BUG in nvme driver

Status in linux package in Ubuntu:
  Fix Released
Status in linux source package in Xenial:
  New

Bug description:
  [Impact]
  When mounting LVM snapshots using xfs, it's possible to hit a BUG_ON() in 
nvme driver.

  Upstream commit 729204ef49ec ("block: relax check on sg gap")
  introduced a way to merge bios if they are physically contiguous. This
  can lead to issues if one rq starts with a non-aligned buffer, as it
  can cause the merged segment to end in an unaligned virtual boundary.
  In some AWS instances, it's possible to craft such a request when
  attempting to mount LVM snapshots using xfs. This will then cause a
  kernel spew due to a BUG_ON in nvme_setup_prps(), which checks if
  dma_len is aligned to the page size.

  [Fix]
  Upstream commit 5a8d75a1b8c9 ("block: fix bio_will_gap() for first bvec with 
offset") disallows requests that begin with an unaligned buffer from being 
merged.

  [Test Case]
  This has been verified on AWS with c5d.large instances:

  1) Prepare the LVM device + snapshot
  $ sudo vgcreate vg0 /dev/nvme1n1
  $ sudo lvcreate -L5G -n data0 vg0
  $ sudo mkfs.xfs /dev/vg0/data0
  $ sudo mount /dev/vg0/data0 /mnt
  $ sudo touch /mnt/test
  $ sudo touch /mnt/test2
  $ sudo ls /mnt
  $ sudo umount /mnt
  $ sudo lvcreate -l100%FREE -s /dev/vg0/data0 -n data0_snap

  2) Attempting to mount the previously created snapshot results in the Oops:
  $ sudo mount /dev/vg0/data0_snap /mnt 
  Segmentation fault (core dumped)

  [Regression Potential]
  The fix prevents some bios from being merged, so it can have a performance 
impact in certain scenarios. The patch only targets misaligned segments, so the 
impact should be less noticeable in the general case.
  The commit is also present in mainline kernels since 4.13, and hasn't been 
changed significantly, so potential for other regressions should be low.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1869229/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1856084] Re: Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

** Patch added: "lp1856084-focal.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+attachment/5312221/+files/lp1856084-focal.debdiff

** Tags removed: sts-sponsor

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Disco:
  Confirmed
Status in zfs-linux source package in Eoan:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
  [72742.132266]  mutex_lock+0x2f/0x40
  [72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
  [72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
  [72742.152622]  ? mutex_lock+0x12/0x40
  [72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
  [72742.171450]  zil_commit+0xde/0x150 [zfs]
  [72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
  [72742.175044]  zpl_fsync+0x80/0x110 [zfs]
  [72742.191690]  vfs_fsync_range+0x51/0xb0
  [72742.193876]  do_fsync+0x3d/0x70
  [72742.195126]  SyS_fsync+0x10/0x20
  [72742.211059]  do_syscall_64+0x73/0x130
  [72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

  It's possible to hit this issue due to a race between the ZFS evict
  and writeback threads. If the z_iput task is trying to evict a znode
  that's currently sitting in the writeback thread, both will "livelock"
  each other and stall the ZIO pipeline, causing other ZFS operations
  (such as zil_commit) to hang indefinitely.

  This has been documented and fixed upstream in PR#9583 [0]. We need to
  pull two fixes from upstream: the first one fixes the zfs_zget() issue
  in the writeback thread, while the second fixes a regression on
  O_TMPFILE descriptors caused by the first one.

  Upstream patches:
   - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
   - Check for unlinked znodes after igrab() (0c46813805f4)

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
The racing window between evict() and the ZFS writeback thread is quite strict, 
but users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

  [Regression Potential]
  These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
  Additional regressions would likely cause issues with the ZFS 
writeback/commit and IO pipeline, so they should be spotted fairly quickly.

  [0] https://github.com/zfsonlinux/zfs/pull/9583
  [1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
  [2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1852432] Re: i40e: Setting VF MAC address causes General Protection Fault

[Heitor Alves de Siqueira]

Verified on disco with the following test case:

$ uname -r
5.0.0-38-generic
$ echo 64 | sudo tee /sys/class/net/ens1f0/device/sriov_numvfs && virsh 
attach-interface iov hostdev :08:02.6 --managed --live
64
Interface attached successfully

** Tags removed: verification-needed-disco
** Tags added: verification-done-disco

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1852432

Title:
  i40e: Setting VF MAC address causes General Protection Fault

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Bionic:
  Fix Committed
Status in linux source package in Disco:
  Fix Committed
Status in linux source package in Eoan:
  Fix Committed
Status in linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
   * Creating SR-IOV enabled VMs in Openstack can sometimes trigger the GPF and 
leave system unusable

  [Test Case]
   * Continuously spin up VFs and set MAC address with e.g. ifconfig

  [Fix]
   * The fix updates the VSI pointer passed down to i40e_set_vf_mac function() 
if the adapter is still in reset, preventing the GPF.

  [Regression Potential]
   * Regression potential should be low, as we're now updating the VSI using 
the ID stored in the VF pointer
   * Regressions could arise from issues in VF creation or reset, as that would 
corrupt the new VSI pointer
   * Patch was validated and tested in a production environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1852432/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1856084] Re: Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

** Patch added: "lp1856084-eoan.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+attachment/5312045/+files/lp1856084-eoan.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Disco:
  Confirmed
Status in zfs-linux source package in Eoan:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
  [72742.132266]  mutex_lock+0x2f/0x40
  [72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
  [72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
  [72742.152622]  ? mutex_lock+0x12/0x40
  [72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
  [72742.171450]  zil_commit+0xde/0x150 [zfs]
  [72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
  [72742.175044]  zpl_fsync+0x80/0x110 [zfs]
  [72742.191690]  vfs_fsync_range+0x51/0xb0
  [72742.193876]  do_fsync+0x3d/0x70
  [72742.195126]  SyS_fsync+0x10/0x20
  [72742.211059]  do_syscall_64+0x73/0x130
  [72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

  It's possible to hit this issue due to a race between the ZFS evict
  and writeback threads. If the z_iput task is trying to evict a znode
  that's currently sitting in the writeback thread, both will "livelock"
  each other and stall the ZIO pipeline, causing other ZFS operations
  (such as zil_commit) to hang indefinitely.

  This has been documented and fixed upstream in PR#9583 [0]. We need to
  pull two fixes from upstream: the first one fixes the zfs_zget() issue
  in the writeback thread, while the second fixes a regression on
  O_TMPFILE descriptors caused by the first one.

  Upstream patches:
   - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
   - Check for unlinked znodes after igrab() (0c46813805f4)

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
The racing window between evict() and the ZFS writeback thread is quite strict, 
but users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

  [Regression Potential]
  These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
  Additional regressions would likely cause issues with the ZFS 
writeback/commit and IO pipeline, so they should be spotted fairly quickly.

  [0] https://github.com/zfsonlinux/zfs/pull/9583
  [1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
  [2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1856084] Re: Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

** Patch added: "lp1856084-disco.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+attachment/5312044/+files/lp1856084-disco.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Disco:
  Confirmed
Status in zfs-linux source package in Eoan:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
  [72742.132266]  mutex_lock+0x2f/0x40
  [72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
  [72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
  [72742.152622]  ? mutex_lock+0x12/0x40
  [72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
  [72742.171450]  zil_commit+0xde/0x150 [zfs]
  [72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
  [72742.175044]  zpl_fsync+0x80/0x110 [zfs]
  [72742.191690]  vfs_fsync_range+0x51/0xb0
  [72742.193876]  do_fsync+0x3d/0x70
  [72742.195126]  SyS_fsync+0x10/0x20
  [72742.211059]  do_syscall_64+0x73/0x130
  [72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

  It's possible to hit this issue due to a race between the ZFS evict
  and writeback threads. If the z_iput task is trying to evict a znode
  that's currently sitting in the writeback thread, both will "livelock"
  each other and stall the ZIO pipeline, causing other ZFS operations
  (such as zil_commit) to hang indefinitely.

  This has been documented and fixed upstream in PR#9583 [0]. We need to
  pull two fixes from upstream: the first one fixes the zfs_zget() issue
  in the writeback thread, while the second fixes a regression on
  O_TMPFILE descriptors caused by the first one.

  Upstream patches:
   - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
   - Check for unlinked znodes after igrab() (0c46813805f4)

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
The racing window between evict() and the ZFS writeback thread is quite strict, 
but users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

  [Regression Potential]
  These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
  Additional regressions would likely cause issues with the ZFS 
writeback/commit and IO pipeline, so they should be spotted fairly quickly.

  [0] https://github.com/zfsonlinux/zfs/pull/9583
  [1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
  [2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1856084] Re: Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

** Tags added: sts-sponsor

** Patch added: "lp1856084-bionic.debdiff"
   
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+attachment/5312043/+files/lp1856084-bionic.debdiff

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Disco:
  Confirmed
Status in zfs-linux source package in Eoan:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
  [72742.132266]  mutex_lock+0x2f/0x40
  [72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
  [72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
  [72742.152622]  ? mutex_lock+0x12/0x40
  [72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
  [72742.171450]  zil_commit+0xde/0x150 [zfs]
  [72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
  [72742.175044]  zpl_fsync+0x80/0x110 [zfs]
  [72742.191690]  vfs_fsync_range+0x51/0xb0
  [72742.193876]  do_fsync+0x3d/0x70
  [72742.195126]  SyS_fsync+0x10/0x20
  [72742.211059]  do_syscall_64+0x73/0x130
  [72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

  It's possible to hit this issue due to a race between the ZFS evict
  and writeback threads. If the z_iput task is trying to evict a znode
  that's currently sitting in the writeback thread, both will "livelock"
  each other and stall the ZIO pipeline, causing other ZFS operations
  (such as zil_commit) to hang indefinitely.

  This has been documented and fixed upstream in PR#9583 [0]. We need to
  pull two fixes from upstream: the first one fixes the zfs_zget() issue
  in the writeback thread, while the second fixes a regression on
  O_TMPFILE descriptors caused by the first one.

  Upstream patches:
   - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
   - Check for unlinked znodes after igrab() (0c46813805f4)

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
The racing window between evict() and the ZFS writeback thread is quite strict, 
but users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

  [Regression Potential]
  These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
  Additional regressions would likely cause issues with the ZFS 
writeback/commit and IO pipeline, so they should be spotted fairly quickly.

  [0] https://github.com/zfsonlinux/zfs/pull/9583
  [1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
  [2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1856084] Re: Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

** Also affects: zfs-linux (Ubuntu Focal)
   Importance: Medium
 Assignee: Heitor Alves de Siqueira (halves)
   Status: Confirmed

** Also affects: zfs-linux (Ubuntu Disco)
   Importance: Undecided
   Status: New

** Also affects: zfs-linux (Ubuntu Bionic)
   Importance: Undecided
   Status: New

** Also affects: zfs-linux (Ubuntu Eoan)
   Importance: Undecided
   Status: New

** Changed in: zfs-linux (Ubuntu Bionic)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Disco)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Bionic)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu Eoan)
   Status: New => Confirmed

** Changed in: zfs-linux (Ubuntu Disco)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

** Changed in: zfs-linux (Ubuntu Eoan)
 Assignee: (unassigned) => Heitor Alves de Siqueira (halves)

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux source package in Bionic:
  Confirmed
Status in zfs-linux source package in Disco:
  Confirmed
Status in zfs-linux source package in Eoan:
  Confirmed
Status in zfs-linux source package in Focal:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
  [72742.132266]  mutex_lock+0x2f/0x40
  [72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
  [72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
  [72742.152622]  ? mutex_lock+0x12/0x40
  [72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
  [72742.171450]  zil_commit+0xde/0x150 [zfs]
  [72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
  [72742.175044]  zpl_fsync+0x80/0x110 [zfs]
  [72742.191690]  vfs_fsync_range+0x51/0xb0
  [72742.193876]  do_fsync+0x3d/0x70
  [72742.195126]  SyS_fsync+0x10/0x20
  [72742.211059]  do_syscall_64+0x73/0x130
  [72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

  It's possible to hit this issue due to a race between the ZFS evict
  and writeback threads. If the z_iput task is trying to evict a znode
  that's currently sitting in the writeback thread, both will "livelock"
  each other and stall the ZIO pipeline, causing other ZFS operations
  (such as zil_commit) to hang indefinitely.

  This has been documented and fixed upstream in PR#9583 [0]. We need to
  pull two fixes from upstream: the first one fixes the zfs_zget() issue
  in the writeback thread, while the second fixes a regression on
  O_TMPFILE descriptors caused by the first one.

  Upstream patches:
   - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
   - Check for unlinked znodes after igrab() (0c46813805f4)

  [Test Case]
  Being a race condition, this issue has been hard to reproduce consistently. 
The racing window between evict() and the ZFS writeback thread is quite strict, 
but users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

  [Regression Potential]
  These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
  Additional regressions would likely cause issues with the ZFS 
writeback/commit and IO pipeline, so they should be spotted fairly quickly.

  [0] https://github.com/zfsonlinux/zfs/pull/9583
  [1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
  [2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1856084/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp

[Kernel-packages] [Bug 1852432] Re: i40e: Setting VF MAC address causes General Protection Fault

[Heitor Alves de Siqueira]

Verified on Bionic with the following test case:
$ echo 64 | sudo tee /sys/class/net/ens1f0/device/sriov_numvfs && virsh 
attach-interface valuable-bluefish hostdev :08:02.6 --managed --live

On 4.15.0-72.81, I get the following GPF:
[119044.656412] general protection fault:  [#1] SMP PTI
[119044.656455] Modules linked in: i40evf vfio_pci vfio_virqfd vfio_iommu_type1 
vfio vhost_net vhost tap ebtable_filter ebtables devlink ip6t
able_filter ip6_tables kvm_intel binfmt_misc ipt_REJECT nf_reject_ipv4 
xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_CHECKSUM xt_comm
ent xt_tcpudp iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
nf_nat_ipv4 nf_nat nf_conntrack iptable_filter bridge stp llc dummy
 ixgbevf ipmi_ssif nls_iso8859_1 intel_rapl sb_edac x86_pkg_temp_thermal 
intel_powerclamp coretemp kvm irqbypass intel_cstate intel_rapl_perf
 lpc_ich hpilo ioatdma shpchp ipmi_si ipmi_devintf ipmi_msghandler mac_hid 
acpi_power_meter sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 
btrfs zstd_compress raid10
[119044.656929]  raid456 async_raid6_recov async_memcpy async_pq async_xor 
async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear mgag2
00 i2c_algo_bit crct10dif_pclmul crc32_pclmul ttm ghash_clmulni_intel ses pcbc 
enclosure drm_kms_helper aesni_intel syscopyarea aes_x86_64 cr
ypto_simd sysfillrect glue_helper cryptd sysimgblt ixgbe fb_sys_fops i40e tg3 
dca drm nvme ptp hpsa pps_core nvme_core mdio scsi_transport_sa
s wmi [last unloaded: kvm_intel]
[119044.657204] CPU: 11 PID: 10300 Comm: libvirtd Not tainted 4.15.0-72-generic 
#81-Ubuntu
[119044.657255] Hardware name: HP ProLiant DL360 Gen9, BIOS P89 05/06/2015
[119044.657310] RIP: 0010:i40e_sync_vsi_filters+0x95/0xd00 [i40e]
[119044.657349] RSP: 0018:bcc5479fb718 EFLAGS: 00010202
[119044.657385] RAX: 77b7ed74a738c96d RBX: 9aad58382000 RCX: 

[119044.657431] RDX: 0001 RSI: fe01 RDI: 
9aad58382000
[119044.657477] RBP: bcc5479fb7b8 R08:  R09: 
3494
[119044.657523] R10:  R11: d9714ff8ecd4 R12: 
9aad58382000
[119044.657569] R13: 9aad384662a0 R14: 9aad58382a28 R15: 
9aad67b1643c
[119044.657616] FS:  7fd7aef27700() GS:9aad7f0c() 
knlGS:
[119044.657668] CS:  0010 DS:  ES:  CR0: 80050033
[119044.657706] CR2: 7fd79001c058 CR3: 001029f30001 CR4: 
001626e0
[119044.657752] Call Trace:
[119044.657778]  ? del_timer_sync+0x45/0x50
[119044.657809]  ? __next_timer_interrupt+0xe0/0xe0
[119044.657851]  i40e_ndo_set_vf_mac+0x109/0x2b0 [i40e]
[119044.657890]  do_setlink+0x8a5/0xed0
[119044.657919]  ? kmalloc_large_node+0x3b/0x60
[119044.657951]  ? security_sock_rcv_skb+0x41/0x60
[119044.657986]  rtnl_setlink+0xdc/0x130
[119044.658017]  rtnetlink_rcv_msg+0x221/0x2b0
[119044.658049]  ? aa_label_sk_perm+0x129/0x140
[119044.658081]  ? _cond_resched+0x19/0x40
[119044.658110]  ? rtnl_calcit.isra.25+0x110/0x110
[119044.658142]  netlink_rcv_skb+0x54/0x130
[119044.658172]  rtnetlink_rcv+0x15/0x20
[119044.658198]  netlink_unicast+0x19e/0x240
[119044.658227]  netlink_sendmsg+0x2d1/0x3d0
[119044.658258]  sock_sendmsg+0x3e/0x50
[119044.659986]  ___sys_sendmsg+0x2a0/0x2f0
[119044.661666]  ? aa_label_sk_perm+0x129/0x140
[119044.663304]  ? _raw_spin_unlock_bh+0x1e/0x20
[119044.664914]  __sys_sendmsg+0x54/0x90
[119044.666469]  ? __sys_sendmsg+0x54/0x90
[119044.668027]  SyS_sendmsg+0x12/0x20
[119044.669602]  do_syscall_64+0x73/0x130
[119044.671139]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[119044.672643] RIP: 0033:0x7fd7bbcab607
[119044.674095] RSP: 002b:7fd7aef263d0 EFLAGS: 0293 ORIG_RAX: 
002e
[119044.675548] RAX: ffda RBX: 0016 RCX: 
7fd7bbcab607
[119044.676968] RDX:  RSI: 7fd7aef26430 RDI: 
0016
[119044.678355] RBP: 7fd7aef26430 R08:  R09: 

[119044.679718] R10: 7fd7aef264bc R11: 0293 R12: 

[119044.681012] R13: 7fd7aef26430 R14: 7fd790012a70 R15: 
0016
[119044.682258] Code: e8 91 38 15 e5 f0 41 0f ba 2c 24 02 72 e8 48 8b 83 40 0f 
00 00 c7 45 8c 00 00 00 00 48 89 85 78 ff ff ff 48 8b 03 48 85
 c0 74 17 <8b> 80 30 02 00 00 8b b3 0c 02 00 00 31 c6 89 83 0c 02 00 00 89
[119044.685125] RIP: i40e_sync_vsi_filters+0x95/0xd00 [i40e] RSP: 
bcc5479fb718
[119044.686379] ---[ end trace f85bde9c06a1d01a ]---

On 4.15.0-73-generic:
$ echo 64 | sudo tee /sys/class/net/ens1f0/device/sriov_numvfs && virsh 
attach-interface valuable-bluefish hostdev :08:02
.6 --managed --live
64
Interface attached successfully
$ uname -r
4.15.0-73-generic

** Tags removed: verification-needed-bionic
** Tags added: verification-done-bionic

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in 

[Kernel-packages] [Bug 1856084] [NEW] Livelock between ZFS evict and writeback threads

[Heitor Alves de Siqueira]

Public bug reported:

Livelock between ZFS evict and writeback threads

[Impact]
ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

[Description]
For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

[72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this 
message.
[72742.070429] mysqld  D0  5713   2881 0x0320
[72742.073220] Call Trace:
[72742.075305]  __schedule+0x24e/0x880
[72742.090436]  schedule+0x2c/0x80
[72742.090438]  schedule_preempt_disabled+0xe/0x10
[72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
[72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
[72742.090555]  __mutex_lock_slowpath+0x13/0x20
[72742.115374]  ? __mutex_lock_slowpath+0x13/0x20
[72742.132266]  mutex_lock+0x2f/0x40
[72742.134207]  zil_commit_impl+0x1b0/0x1b30 [zfs]
[72742.150428]  ? spl_kmem_alloc+0x115/0x180 [spl]
[72742.152622]  ? mutex_lock+0x12/0x40
[72742.154819]  ? zfs_refcount_add_many+0x9a/0x100 [zfs]
[72742.171450]  zil_commit+0xde/0x150 [zfs]
[72742.173687]  zfs_fsync+0x77/0xe0 [zfs]
[72742.175044]  zpl_fsync+0x80/0x110 [zfs]
[72742.191690]  vfs_fsync_range+0x51/0xb0
[72742.193876]  do_fsync+0x3d/0x70
[72742.195126]  SyS_fsync+0x10/0x20
[72742.211059]  do_syscall_64+0x73/0x130
[72742.214078]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2

It's possible to hit this issue due to a race between the ZFS evict and
writeback threads. If the z_iput task is trying to evict a znode that's
currently sitting in the writeback thread, both will "livelock" each
other and stall the ZIO pipeline, causing other ZFS operations (such as
zil_commit) to hang indefinitely.

This has been documented and fixed upstream in PR#9583 [0]. We need to
pull two fixes from upstream: the first one fixes the zfs_zget() issue
in the writeback thread, while the second fixes a regression on
O_TMPFILE descriptors caused by the first one.

Upstream patches:
 - Break out of zfs_zget early if unlinked znode (41e1aa2a06f8)
 - Check for unlinked znodes after igrab() (0c46813805f4)

[Test Case]
Being a race condition, this issue has been hard to reproduce consistently. The 
racing window between evict() and the ZFS writeback thread is quite strict, but 
users have reported this to show up after some hours of running 
LXD-containerized mySQL workloads.

[Regression Potential]
These patches have been tested both in the ZFS test suite and in production 
environments, so the potential for further regressions should be low.
Additional regressions would likely cause issues with the ZFS writeback/commit 
and IO pipeline, so they should be spotted fairly quickly.

[0] https://github.com/zfsonlinux/zfs/pull/9583
[1] https://github.com/zfsonlinux/zfs/commit/41e1aa2a06f8
[2] https://github.com/zfsonlinux/zfs/commit/0c46813805f4

** Affects: zfs-linux (Ubuntu)
 Importance: Undecided
 Assignee: Heitor Alves de Siqueira (halves)
 Status: Confirmed

** Affects: zfs-linux (Debian)
 Importance: Unknown
 Status: Unknown


** Tags: sts

** Bug watch added: Debian Bug tracker #946610
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946610

** Also affects: zfs-linux (Debian) via
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=946610
   Importance: Unknown
   Status: Unknown

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to zfs-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1856084

Title:
  Livelock between ZFS evict and writeback threads

Status in zfs-linux package in Ubuntu:
  Confirmed
Status in zfs-linux package in Debian:
  Unknown

Bug description:
  Livelock between ZFS evict and writeback threads

  [Impact]
  ZIO pipeline stalls, causing ZFS workloads to hang indefinitely

  [Description]
  For certain ZFS workloads, we start seeing hung task timeouts in the kernel 
logs due to zil_commit() stalling. This is due to zfs_zget() not detecting 
whether a znode has been marked for deletion before attempting to access it, 
causing a constant "retry loop" in zfs_get_data() if that znode has been 
unlinked already. An example of the stack traces follows:

  [72742.051703] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables 
this message.
  [72742.070429] mysqld  D0  5713   2881 0x0320
  [72742.073220] Call Trace:
  [72742.075305]  __schedule+0x24e/0x880
  [72742.090436]  schedule+0x2c/0x80
  [72742.090438]  schedule_preempt_disabled+0xe/0x10
  [72742.090441]  __mutex_lock.isra.5+0x276/0x4e0
  [72742.090547]  ? dmu_tx_destroy+0x105/0x130 [zfs]
  [72742.090555]  __mutex_lock_slowpath+0x13/0x20
  [72742.115374]  ? __mutex_lock_slowpath+0x13/0x2

[Kernel-packages] [Bug 1852432] Re: i40e: Setting VF MAC address causes General Protection Fault

[Heitor Alves de Siqueira]

** No longer affects: linux (Ubuntu Xenial)

** Description changed:

  [Impact]
-  * Creating SR-IOV enabled VMs in Openstack can sometimes trigger the GPF and 
leave system unusable
+  * Creating SR-IOV enabled VMs in Openstack can sometimes trigger the GPF and 
leave system unusable
  
  [Test Case]
-  * Continuously spin up VFs and set MAC address with e.g. ifconfig
+  * Continuously spin up VFs and set MAC address with e.g. ifconfig
  
  [Fix]
-  * The fix updates the VSI pointer passed down to i40e_set_vf_mac function() 
if the adapter is still in reset, preventing the GPF.
+  * The fix updates the VSI pointer passed down to i40e_set_vf_mac function() 
if the adapter is still in reset, preventing the GPF.
  
  [Regression Potential]
-  * Regression potential should be low, as we're now updating the VSI using 
the ID stored in the VF pointer
-  * Regressions could arise from issues in VF creation or reset, as that would 
corrupt the new VSI pointer
-  * Patch was validated and tested in a production environment
- 
- [Other]
-  * Fix was introduced in v5.4-rc1, so all previous kernels are affected
+  * Regression potential should be low, as we're now updating the VSI using 
the ID stored in the VF pointer
+  * Regressions could arise from issues in VF creation or reset, as that would 
corrupt the new VSI pointer
+  * Patch was validated and tested in a production environment

-- 
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1852432

Title:
  i40e: Setting VF MAC address causes General Protection Fault

Status in linux package in Ubuntu:
  Confirmed
Status in linux source package in Bionic:
  Confirmed
Status in linux source package in Disco:
  Confirmed
Status in linux source package in Eoan:
  Confirmed
Status in linux source package in Focal:
  Confirmed

Bug description:
  [Impact]
   * Creating SR-IOV enabled VMs in Openstack can sometimes trigger the GPF and 
leave system unusable

  [Test Case]
   * Continuously spin up VFs and set MAC address with e.g. ifconfig

  [Fix]
   * The fix updates the VSI pointer passed down to i40e_set_vf_mac function() 
if the adapter is still in reset, preventing the GPF.

  [Regression Potential]
   * Regression potential should be low, as we're now updating the VSI using 
the ID stored in the VF pointer
   * Regressions could arise from issues in VF creation or reset, as that would 
corrupt the new VSI pointer
   * Patch was validated and tested in a production environment

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1852432/+subscriptions

-- 
Mailing list: https://launchpad.net/~kernel-packages
Post to : kernel-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~kernel-packages
More help   : https://help.launchpad.net/ListHelp