[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
I think neither rsyslog, nor cups, or sssd, are the correct packages for this bug, but I agree that spreading out the investigation over two different bugs is not good. I'll mark https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2064088 as a duplicate, and add cups and sssd tasks to this one for tracking purposes, but I suspect the fix will be elsewhere in the end. ** Also affects: sssd (Ubuntu) Importance: Undecided Status: New ** Also affects: cups (Ubuntu) Importance: Undecided Status: New ** Summary changed: - rsyslog service timeout on noble numbat + Services fail to start in noble deployed with TPM+FDE ** Changed in: sssd (Ubuntu) Status: New => Confirmed ** Changed in: cups (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: Services fail to start in noble deployed with TPM+FDE Status in cups package in Ubuntu: Confirmed Status in rsyslog package in Ubuntu: Confirmed Status in sssd package in Ubuntu: Confirmed Bug description: What's known so far: - 24.04 desktop deployed with TPM+FDE shows this bug - services confined with apparmor that need to access something in /run/systemd (like the notify socket) fail to do so, even if the apparmor profile is in complain mode. And the apparmor profile does already have rules to allow that access - only after running aa-disable can the service start fine - paths logged by the apparmor DENIED or ALLOWED messages are missing the "/run" prefix from "/run/systemd/..". - When we add rules to the profile using "/systemd/" (i.e., also dropping the /run prefix), then it works - other access in /run/systemd/ are also blocked, but the most noticeable one is the notify mechanism - comment #2 also states that azure CVM images are also impacted - comment #4 has instructions on how to create such a VM locally with LXD vms Original description follows: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/cups/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
Do you think we should mark #2064088 as a duplicate of this (or vice- versa), if we're confident this is the same underlying issue? There are some outstanding questions for me on that bug, but it might make sense to focus our comments in one place going forward. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
Andreas and I spent some time this afternoon investigating this issue. Here are our findings. First, we noticed that the paths being reported by apparmor on dmesg appear to be relative to /run. This is just an impression, though: I believe that, for some reason, apparmor/systemd/something-else is actually seeing the paths as "/systemd/notify" instead of "/run/systemd/notify". Therefore, we decided to try to list those paths inside the apparmor profile, like: /systemd/journal/dev-log rwkl, /systemd/notify rwkl, Note that we're using "rwkl" just because we don't want to deal with limiting the scope of each access. After adding these paths to /etc/apparmor.d/usr.sbin.rsyslogd and reloading the profile, the service can finally be (re)started. This indicates that there's a discrepancy between the paths seeing by apparmor/systemd/Linux and those seeing by the userspace application. With that in mind, our next idea was to try to use "systemd-run" to mimic what's happening with rsyslogd. This could help us determine which component is problematic, but unfortunately we were unable to make the failure happen. We tried many combinations of commands; some of them are listed below: # Try to "ls" the notify socket using different paths systemd-run -p AppArmorProfile=rsyslogd ls /run/systemd/notify systemd-run -p AppArmorProfile=rsyslogd ls /systemd/notify # Likewise, but running the command using the syslog user systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /run/systemd/notify systemd-run --uid 102 -p AppArmorProfile=rsyslogd ls /systemd/notify Strangely, "ls" was able to properly list the contents of /run/systemd/notify on both cases (which it shouldn't, because the apparmor profile doesn't allow it). It also reported that "/systemd/notify", which is correct but unexpected (because we thought that systemd might be the problematic component which doesn't use "/run" in the paths). We also double checked and confirmed that the processes started by "systemd-run" have "systemd" as their parent, so in theory we should have seen the same problem here. There is also the fact that these file accesses are being denied even when the apparmor profile is running in complain mode. AFAIU, this shouldn't happen. Unless apparmor is affecting the path resolution that happens when the service tries to connect to the socket, effectively mangling the final path... but that would be very weird, I believe. Either way, it is unclear: 1) Why we're seeing these "partial" paths in the logs. 2) Why these accesses are being denied even when the apparmor profile is in complain mode. 3) Why "systemd-run" can't seem to reproduce the problem. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
So far it's rsyslog, cups, and sssd (just confirmed via https://bugs.launchpad.net/ubuntu/+source/sssd/+bug/2064088). -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
I've just found that the cups.service is also experiencing the same behaviour. Again it has the service type 'notify'. I suspect other services using this type will have the same problem. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
Here's how to reproduce this in a LXD VM: Download Ubuntu 24.04 Desktop image into ~/Downloads Import the ISO $ lxc storage volume import default ~/Downloads/ubuntu-24.04-desktop-amd64.iso 24.04-desktop --type=iso Prepare a LXD VM $ lxc init --empty --vm lxd-noble-fde -c limits.memory=6GiB -c limits.cpu=4 -d root,size=32GiB $ lxc config device add lxd-noble-fde iso-volume disk pool=default source=24.04-desktop boot.priority=10 $ lxc config device add lxd-noble-fde tpm tpm $ lxc start --console=vga lxd-noble-fde Go through the manual install process but at the "Disk Setup" step, select "Erase disk and install Ubuntu" and click "Advanced features...". Select "Enable hardware-backed full disk encryption" then click "OK" Once the installation is done, force the LXD VM to stop $ lxc stop --force lxd-noble-fde Remove the ISO $ lxc config device remove lxd-noble-fde iso-volume Start the VM back $ lxc start lxd-noble-fde Once logged in, rsyslog should eventually fail to start and the same Apparmor denials should show up in `journalctl -k`. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
Status changed to 'Confirmed' because the bug affects multiple users. ** Changed in: rsyslog (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 2064096] Re: rsyslog service timeout on noble numbat
Azure CVM images are impacted by the same issue. I see on #2064088 that this is an tpm-backed FDE system. So I think it's the same problem here if those desktop images use an systemd-based initramfs. For now I suspect that the issue is due to systemd starting services and setting up UNIX sockets (eg /run/systemd/journal/dev-log, /run/systemd/notify and others) before doing the pivot_root and reexec. Then, when apparmor tries to resolve the path of the peer socket it fails here[1] I believe. [1] https://git.launchpad.net/~ubuntu- kernel/ubuntu/+source/linux/+git/noble/tree/fs/d_path.c#n125 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to rsyslog in Ubuntu. https://bugs.launchpad.net/bugs/2064096 Title: rsyslog service timeout on noble numbat Status in rsyslog package in Ubuntu: Confirmed Bug description: This might be related to #2064088 The rsyslog service is continually timing out and restarting. If I use a service drop-in file and change the 'Type' from 'notify' to 'simple', the service starts and appears to work normally. In the journal, I can see the attached apparmor errors. I can't make sense of them, but if it's a similar issue to #2064088, then I suspect apparmor is preventing the systemd notify function from alerting systemd that the service is up and running. ProblemType: Bug DistroRelease: Ubuntu 24.04 Package: rsyslog 8.2312.0-3ubuntu9 ProcVersionSignature: Ubuntu 6.8.0-31.31-generic 6.8.1 Uname: Linux 6.8.0-31-generic x86_64 ApportVersion: 2.28.1-0ubuntu2 Architecture: amd64 CasperMD5CheckMismatches: ./boot/grub/grub.cfg CasperMD5CheckResult: fail CurrentDesktop: ubuntu:GNOME Date: Mon Apr 29 10:37:46 2024 ProcEnviron: LANG=en_GB.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=xterm-256color XDG_RUNTIME_DIR= SourcePackage: rsyslog UpgradeStatus: No upgrade log present (probably fresh install) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rsyslog/+bug/2064096/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
