[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
This bug was fixed in the package linux-bluefield - 5.4.0-1070.76
---
linux-bluefield (5.4.0-1070.76) focal; urgency=medium
* focal/linux-bluefield: 5.4.0-1070.76 -proposed tracker (LP:
#2030986)
* EFI pstore not cleared on boot (LP: #1978079)
- [Config] bluefield: Enable CONFIG_EFI_VARS_PSTORE as build-in
- [Config] bluefield: Remove efi-pstore from modules list
* Focal update: v5.4.246 upstream stable release (LP: #2028981)
- [Config] bluefield: updateconfigs for SCSI_DPT_I2O
- SAUCE: xfrm: Check if_id in xfrm full offload
* rshim console truncates dmesg output due to tmfifo issue (LP: #2028197)
- SAUCE: mlxbf-tmfifo.c: Fix rhsim console w/ truncated dmesg output
* mlxbf-gige: Fix kernel panic after reboot (LP: #2030765)
- SAUCE: mlxbf-gige: Fix kernel panic after reboot (part 1/2)
- SAUCE: mlxbf-gige: Fix kernel panic after reboot (part 2/2)
[ Ubuntu: 5.4.0-162.179 ]
* focal/linux: 5.4.0-162.179 -proposed tracker (LP: #2031128)
* libgnutls report "trap invalid opcode" when trying to install packages over
https (LP: #2031093)
- [Config]: disable CONFIG_GDS_FORCE_MITIGATION
[ Ubuntu: 5.4.0-160.177 ]
* focal/linux: 5.4.0-160.177 -proposed tracker (LP: #2031017)
* Fix boot test warning for log_check "CPU: 0 PID: 0 at
arch/x86/kernel/fpu/xstate.c:878 get_xsave_addr+0x98/0xb0" (LP: #2031022)
- x86/pkeys: Revert a5eff7259790 ("x86/pkeys: Add PKRU value to
init_fpstate")
[ Ubuntu: 5.4.0-158.175 ]
* focal/linux: 5.4.0-158.175 -proposed tracker (LP: #2030466)
* CVE-2022-40982
- x86/mm: Initialize text poking earlier
- x86/mm: fix poking_init() for Xen PV guests
- x86/mm: Use mm_alloc() in poking_init()
- mm: Move mm_cachep initialization to mm_init()
- init: Provide arch_cpu_finalize_init()
- x86/cpu: Switch to arch_cpu_finalize_init()
- ARM: cpu: Switch to arch_cpu_finalize_init()
- sparc/cpu: Switch to arch_cpu_finalize_init()
- um/cpu: Switch to arch_cpu_finalize_init()
- init: Remove check_bugs() leftovers
- init: Invoke arch_cpu_finalize_init() earlier
- init, x86: Move mem_encrypt_init() into arch_cpu_finalize_init()
- x86/fpu: Remove cpuinfo argument from init functions
- x86/fpu: Mark init functions __init
- x86/fpu: Move FPU initialization into arch_cpu_finalize_init()
- x86/xen: Fix secondary processors' FPU initialization
- x86/speculation: Add Gather Data Sampling mitigation
- x86/speculation: Add force option to GDS mitigation
- x86/speculation: Add Kconfig option for GDS
- KVM: Add GDS_NO support to KVM
- Documentation/x86: Fix backwards on/off logic about YMM support
- [Config]: Enable CONFIG_ARCH_HAS_CPU_FINALIZE_INIT and
CONFIG_GDS_FORCE_MITIGATION
* CVE-2023-3609
- net/sched: cls_u32: Fix reference counter leak leading to overflow
* CVE-2023-20593
- x86/cpu/amd: Move the errata checking functionality up
- x86/cpu/amd: Add a Zenbleed fix
* CVE-2023-3611
- net/sched: sch_qfq: account for stab overhead in qfq_enqueue
* stacked overlay file system mounts that have chroot() called against them
appear to be getting locked (by the kernel most likely?) (LP: #2016398)
- SAUCE: overlayfs: fix reference count mismatch
* Focal update: v5.4.246 upstream stable release (LP: #2028981)
- RDMA/efa: Fix unsupported page sizes in device
- RDMA/bnxt_re: Enable SRIOV VF support on Broadcom's 57500 adapter series
- RDMA/bnxt_re: Refactor queue pair creation code
- RDMA/bnxt_re: Fix return value of bnxt_re_process_raw_qp_pkt_rx
- iommu/rockchip: Fix unwind goto issue
- iommu/amd: Don't block updates to GATag if guest mode is on
- dmaengine: pl330: rename _start to prevent build error
- net/mlx5: fw_tracer, Fix event handling
- netrom: fix info-leak in nr_write_internal()
- af_packet: Fix data-races of pkt_sk(sk)->num.
- amd-xgbe: fix the false linkup in xgbe_phy_status
- mtd: rawnand: ingenic: fix empty stub helper definitions
- af_packet: do not use READ_ONCE() in packet_bind()
- tcp: deny tcp_disconnect() when threads are waiting
- tcp: Return user_mss for TCP_MAXSEG in CLOSE/LISTEN state if user_mss set
- net/sched: sch_ingress: Only create under TC_H_INGRESS
- net/sched: sch_clsact: Only create under TC_H_CLSACT
- net/sched: Reserve TC_H_INGRESS (TC_H_CLSACT) for ingress (clsact) Qdiscs
- net/sched: Prohibit regrafting ingress or clsact Qdiscs
- net: sched: fix NULL pointer dereference in mq_attach
- ocfs2/dlm: move BITS_TO_BYTES() to bitops.h for wider use
- net/netlink: fix NETLINK_LIST_MEMBERSHIPS length report
- udp6: Fix race condition in udp6_sendmsg & connect
- net: dsa: mv88e6xxx: Increase wait after reset deactivation
- mtd: rawnand: marvell: ensure timing values are written
- mtd: rawnand: marvell: don't set the NAND frequency select
- watchdog: menz069_wdt: fix watchdog
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
root@bu-lab26v-oob:~# cat /etc/mlnx-release DOCA_2.2.0_BSP_4.2.1_Ubuntu_20.04-2.sru.5.4.0-1070 root@bu-lab26v-oob:~# uname -a Linux bu-lab26v-oob 5.4.0-1070-bluefield #76-Ubuntu SMP PREEMPT Wed Aug 30 16:56:35 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux root@bu-lab26v-oob:~# cat /sys/module/pstore/parameters/backend efi root@bu-lab26v-oob:~# systemctl status systemd-pstore.service ● systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Tue 2023-09-19 14:16:32 UTC; 1 day 1h ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Sep 19 14:16:28 localhost systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. Sep 19 14:16:32 localhost systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. root@bu-lab26v-oob:~# echo 1 > /proc/sys/kernel/sysrq root@bu-lab26v-oob:~# echo 1 > /proc/sys/kernel/panic root@bu-lab26v-oob:~# echo "c" > /proc/sysrq-trigger *system rebooted* root@bu-lab26v-oob:~# ls /sys/fs/pstore root@bu-lab26v-oob:~# ls /var/lib/systemd/pstore 169522441 169522442 root@bu-lab26v-oob:~# systemctl status systemd-pstore.service ● systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: active (exited) since Wed 2023-09-20 15:41:29 UTC; 56s ago Docs: man:systemd-pstore(8) Process: 485 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS) Main PID: 485 (code=exited, status=0/SUCCESS) Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441409001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441409001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441408001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441408001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441407001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441407001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441406001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441406001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441405001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441405001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441304001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441304001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441303001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441303001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441302001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441302001 Sep 20 15:41:29 localhost systemd-pstore[485]: PStore dmesg-efi-169522441301001 moved to /var/lib/systemd/pstore/169522441/dmesg-efi-169522441301001 Sep 20 15:41:29 localhost systemd[1]: Finished Platform Persistent Storage Archival. ** Tags removed: verification-needed-focal-linux-bluefield ** Tags added: verification-done-focal-linux-bluefield -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in linux-bluefield package in Ubuntu: New Status in systemd package in Ubuntu: Fix Released Status in linux-bluefield source package in Focal: Fix Committed Status in systemd source package in Focal: Fix Released Status in systemd source package in Impish: Won't Fix Status in linux-bluefield source package in Jammy: Fix Committed Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service;
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
This bug is awaiting verification that the linux-bluefield/5.4.0-1070.76 kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-done- focal-linux-bluefield'. If the problem still exists, change the tag 'verification-needed-focal-linux-bluefield' to 'verification-failed- focal-linux-bluefield'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: kernel-spammed-focal-linux-bluefield-v2 verification-needed-focal-linux-bluefield -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in linux-bluefield package in Ubuntu: New Status in systemd package in Ubuntu: Fix Released Status in linux-bluefield source package in Focal: Fix Committed Status in systemd source package in Focal: Fix Released Status in systemd source package in Impish: Won't Fix Status in linux-bluefield source package in Jammy: Fix Committed Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Changed in: linux-bluefield (Ubuntu Focal) Status: New => Fix Committed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in linux-bluefield package in Ubuntu: New Status in systemd package in Ubuntu: Fix Released Status in linux-bluefield source package in Focal: Fix Committed Status in systemd source package in Focal: Fix Released Status in systemd source package in Impish: Won't Fix Status in linux-bluefield source package in Jammy: Fix Committed Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Test Plan] In order to be able to reproduce this issue, the system must have EFI- backed pstore. To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend` If it says `efi`, the steps below are applicable. Otherwise, find an environment that has EFI backed pstore. # Enable the pstore service. This service is supposed to move the data in
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Also affects: linux-bluefield (Ubuntu) Importance: Undecided Status: New ** No longer affects: linux-bluefield (Ubuntu Impish) ** No longer affects: linux-bluefield (Ubuntu Kinetic) ** Changed in: linux-bluefield (Ubuntu) Assignee: (unassigned) => William Tu (wtu) ** Changed in: linux-bluefield (Ubuntu Focal) Assignee: (unassigned) => William Tu (wtu) ** Changed in: linux-bluefield (Ubuntu Jammy) Assignee: (unassigned) => William Tu (wtu) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in linux-bluefield package in Ubuntu: New Status in systemd package in Ubuntu: Fix Released Status in linux-bluefield source package in Focal: New Status in systemd source package in Focal: Fix Released Status in systemd source package in Impish: Won't Fix Status in linux-bluefield source package in Jammy: New Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Test Plan] In order
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
This bug was fixed in the package systemd - 245.4-4ubuntu3.18 --- systemd (245.4-4ubuntu3.18) focal; urgency=medium [ Nick Rosbrook ] * core: make sure we don't get confused when setting TERM for a tty fd (LP: #1959475) File: debian/patches/lp1959475-core-make-sure-we-don-t-get-confused-when-setting-TERM-fo.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=b10c6853050dde26665caf3b15444d768d2bc498 * shared/calendarspec: when mktime() moves us backwards, jump forward (LP: #1966800) File: debian/patches/lp1966800-shared-calendarspec-when-mktime-moves-us-backwards-jump-f.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=1f063541e44f6ff1a6904676d4264a2e49a09594 * network: do not remove localhost address (LP: #1979951) File: debian/patches/lp1979951-network-do-not-remove-localhost-address.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=87f872b8c5451f353601fb606e7fd7a479217cef * units: remove the restart limit on the modprobe@.service (LP: #1982462) File: debian/patches/lp1982462-units-remove-the-restart-limit-on-the-modprobe-.service.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=68353ffaf3539e6a58ef62a8b50850f56eae29ea [ Mustafa Kemal Gilor ] * d/p/lp1978079-efi-pstore-not-cleared-on-boot.patch: pstore: Run after modules are loaded. Thanks to Alexander Graf . (LP: #1978079) Author: Mustafa Kemal Gilor File: debian/patches/lp1978079-efi-pstore-not-cleared-on-boot.patch https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=6e60756f2079d6408abdb967127a1d9b9a0eba8c -- Nick Rosbrook Wed, 31 Aug 2022 11:27:33 -0400 ** Changed in: systemd (Ubuntu Focal) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Released Status in systemd source package in Impish: Won't Fix Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
Verification done for focal: - Environment - ubuntu@crustle:~$ uname -a Linux crustle 5.4.0-125-generic #141-Ubuntu SMP Wed Aug 10 13:42:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux ubuntu@crustle:~$ cat /etc/os-release NAME="Ubuntu" VERSION="20.04.4 LTS (Focal Fossa)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 20.04.4 LTS" VERSION_ID="20.04" HOME_URL="https://www.ubuntu.com/; SUPPORT_URL="https://help.ubuntu.com/; BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/; PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy; VERSION_CODENAME=focal UBUNTU_CODENAME=focal ubuntu@crustle:~$ systemd --version systemd 245 (245.4-4ubuntu3.18) +PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid root@crustle:/home/ubuntu# cat /sys/module/pstore/parameters/backend efi - - Test steps - See [Test Plan] - - Result - Verification : OK root@crustle:/home/ubuntu# echo 1 > /proc/sys/kernel/sysrq root@crustle:/home/ubuntu# echo 1 > /proc/sys/kernel/panic root@crustle:/home/ubuntu# echo "c" > /proc/sysrq-trigger * system reboots * root@crustle:/home/ubuntu# ls /sys/fs/pstore root@crustle:/home/ubuntu# ls /var/lib/systemd/pstore 166271364 root@crustle:/home/ubuntu# systemctl status systemd-pstore ● systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: active (exited) since Fri 2022-09-09 08:57:29 UTC; 3min 10s ago Docs: man:systemd-pstore(8) Process: 639 ExecStart=/lib/systemd/systemd-pstore (code=exited, status=0/SUCCESS) Main PID: 639 (code=exited, status=0/SUCCESS) Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364709001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364709001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364708001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364708001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364607001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364607001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364606001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364606001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364605001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364605001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364604001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364604001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364603001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364603001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364602001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364602001 Sep 09 08:57:29 crustle systemd-pstore[639]: PStore dmesg-efi-166271364601001 moved to /var/lib/systemd/pstore/166271364/dmesg-efi-166271364601001 Sep 09 08:57:29 crustle systemd[1]: Finished Platform Persistent Storage Archival. ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Status in systemd source package in Impish: Won't Fix Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
The focal autopkgtest regressions were all resolved by retries. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Status in systemd source package in Impish: Won't Fix Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Test Plan] In order to be able to reproduce this issue, the system must have EFI- backed pstore. To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend` If it says `efi`, the steps below are applicable. Otherwise, find an environment that has EFI backed pstore. # Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore # to the `/var/lib/systemd/pstore` path on boot. systemctl enable systemd-pstore.service # (or can be vendor enabled) # Crash the kernel echo 1 > /proc/sys/kernel/sysrq echo 1 >
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Merge proposal linked: https://code.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/+git/systemd/+merge/429490 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Status in systemd source package in Impish: Won't Fix Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. [Test Plan] In order to be able to reproduce this issue, the system must have EFI- backed pstore. To check which kind of backend that pstore, use `cat /sys/module/pstore/parameters/backend` If it says `efi`, the steps below are applicable. Otherwise, find an environment that has EFI backed pstore. # Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore # to the `/var/lib/systemd/pstore` path on boot. systemctl enable systemd-pstore.service # (or can be vendor enabled) # Crash
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
Hello Alexander, or anyone else affected, Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.18 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-focal. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: systemd (Ubuntu Focal) Status: In Progress => Fix Committed ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Focal: Fix Committed Status in systemd source package in Impish: Won't Fix Status in systemd source package in Jammy: Fix Released Status in systemd source package in Kinetic: Fix Released Bug description: [Impact] Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r--
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Description changed: + [Impact] + Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival - Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) - Active: inactive (dead) - Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago - └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met -Docs: man:systemd-pstore(8) + Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) + Active: inactive (dead) + Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago + └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met + Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. + + [Test Plan] + + In order to be able to reproduce this issue, the system must have EFI- + backed pstore. + + To check which kind of backend that pstore, use `cat + /sys/module/pstore/parameters/backend` + + If it says `efi`, the steps below are applicable. Otherwise, find an + environment that has EFI backed pstore. + + # Enable the pstore service. This service is supposed to move the data in /sys/fs/pstore + # to the `/var/lib/systemd/pstore` path on boot. + systemctl enable systemd-pstore.service # (or can be vendor enabled) + + # Crash the kernel + echo 1 > /proc/sys/kernel/sysrq + echo 1 > /proc/sys/kernel/panic # this is usually set to zero, causing kernel to loop over the panic and freeze + echo "c" > /proc/sysrq-trigger + + # The system will reboot itself. Check `/sys/fs/pstore` path first: + ls
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Tags added: ubuntu-sponsors -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Tags added: seg sts sts-sponsor -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Merge proposal linked: https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424958 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Merge proposal linked: https://code.launchpad.net/~mustafakemalgilor/ubuntu/+source/systemd/+git/systemd/+merge/424938 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Tags removed: rls-ff-incoming rls-jj-incoming -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Tags added: fr-2484 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Changed in: systemd (Ubuntu) Assignee: (unassigned) => Mustafa Kemal Gilor (mustafakemalgilor) ** Changed in: systemd (Ubuntu) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: In Progress Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
** Tags added: rls-ff-incoming rls-jj-incoming -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: New Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
The upstream pull request has been accepted, please backport the patch to (at least) Ubuntu 20.04 and 22.04. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: New Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
Systemd pull request: https://github.com/systemd/systemd/pull/23682 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: New Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1978079] Re: EFI pstore not cleared on boot
This systemd issue describes the problem above as well: https://github.com/systemd/systemd/issues/18540 ** Bug watch added: github.com/systemd/systemd/issues #18540 https://github.com/systemd/systemd/issues/18540 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/1978079 Title: EFI pstore not cleared on boot Status in systemd package in Ubuntu: New Bug description: Systemd has a systemd-pstore component that scans the pstore on boot and if non-empty, takes all previously created dumps, transfers them into its journal and removes the pstore elements. This is very important on UEFI systems, which only have a limited amount of space for variables. In Ubuntu, the kernel is configured with CONFIG_EFI_VARS_PSTORE=m which means the EFI pstore support gets loaded dynamically. In all of my boots, this dynamic module loading happened *after* systemd tried to check for pstore variables. So systemd-pstore never starts and never clears the UEFI variable store. I see this happening in AWS on Graviton instances, which eventually run out of space to store the dumps. On real hardware, this behavior may lead to unbootable systems. ``` $ systemctl status systemd-pstore ○ systemd-pstore.service - Platform Persistent Storage Archival Loaded: loaded (/lib/systemd/system/systemd-pstore.service; enabled; vendor preset: enabled) Active: inactive (dead) Condition: start condition failed at Thu 2022-06-09 09:11:41 UTC; 29min ago └─ ConditionDirectoryNotEmpty=/sys/fs/pstore was not met Docs: man:systemd-pstore(8) Jun 09 09:11:41 ip-172-31-0-61 systemd[1]: Condition check resulted in Platform Persistent Storage Archival being skipped. $ ls -la /sys/fs/pstore total 0 drwxr-x--- 2 root root0 Jun 9 09:11 . drwxr-xr-x 8 root root0 Jun 9 09:11 .. -r--r--r-- 1 root root 1803 Jun 9 09:07 dmesg-efi-165476562001001 -r--r--r-- 1 root root 1777 Jun 9 09:07 dmesg-efi-165476562002001 -r--r--r-- 1 root root 1773 Jun 9 09:07 dmesg-efi-165476562003001 -r--r--r-- 1 root root 1815 Jun 9 09:07 dmesg-efi-165476562004001 -r--r--r-- 1 root root 1826 Jun 9 09:07 dmesg-efi-165476562005001 -r--r--r-- 1 root root 1754 Jun 9 09:07 dmesg-efi-165476562006001 -r--r--r-- 1 root root 1821 Jun 9 09:07 dmesg-efi-165476562007001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562008001 -r--r--r-- 1 root root 1729 Jun 9 09:07 dmesg-efi-165476562009001 -r--r--r-- 1 root root 1819 Jun 9 09:07 dmesg-efi-165476562010001 -r--r--r-- 1 root root 1767 Jun 9 09:07 dmesg-efi-165476562011001 -r--r--r-- 1 root root 1775 Jun 9 09:07 dmesg-efi-165476562012001 -r--r--r-- 1 root root 1802 Jun 9 09:07 dmesg-efi-165476562013001 -r--r--r-- 1 root root 1812 Jun 9 09:07 dmesg-efi-165476562014001 -r--r--r-- 1 root root 1764 Jun 9 09:07 dmesg-efi-165476562015001 -r--r--r-- 1 root root 1795 Jun 9 09:11 dmesg-efi-165476589801001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589802001 -r--r--r-- 1 root root 1683 Jun 9 09:11 dmesg-efi-165476589803001 -r--r--r-- 1 root root 1785 Jun 9 09:11 dmesg-efi-165476589804001 -r--r--r-- 1 root root 1771 Jun 9 09:11 dmesg-efi-165476589805001 -r--r--r-- 1 root root 1797 Jun 9 09:11 dmesg-efi-165476589806001 -r--r--r-- 1 root root 1805 Jun 9 09:11 dmesg-efi-165476589807001 -r--r--r-- 1 root root 1781 Jun 9 09:11 dmesg-efi-165476589808001 -r--r--r-- 1 root root 1806 Jun 9 09:11 dmesg-efi-165476589809001 -r--r--r-- 1 root root 1821 Jun 9 09:11 dmesg-efi-165476589810001 -r--r--r-- 1 root root 1763 Jun 9 09:11 dmesg-efi-165476589811001 -r--r--r-- 1 root root 1783 Jun 9 09:11 dmesg-efi-165476589812001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589813001 -r--r--r-- 1 root root 1788 Jun 9 09:11 dmesg-efi-165476589814001 -r--r--r-- 1 root root 1786 Jun 9 09:11 dmesg-efi-165476589815001 ``` This problem affects (at least) Ubuntu 20.04 and 22.04. A quick fix would be to configure CONFIG_EFI_VARS_PSTORE=y so that it's always available. A long term fix would make systemd rescan the directory after all module probing settled. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1978079/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
