[systemd-devel] systemctl hangs with 249.7 systemd in yocto Honister release
Hi all, We are runing OpenBMC 2.11.0 release which is based on yocto Honister and systemd 249.7 release, and we find systemctl hangs occasionally. The phenomenon is just like the below issue: https://github.com/openbmc/openbmc/issues/1097 systemctl command will always return timeout and dmesg will continue to pop up below message: |systemd-journald[445]: Failed to send WATCHDOG=1 notification message: Transport endpoint is not connected| Is there any known issue which will cause this problem? Or do you have any suggestion on how to debug? For the above bug, I don't see any resolution before it was closed. Thanks, Heyi
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On 10. 11. 2022. 10:20, Greg KH wrote: On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote: On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: Dear Sirs, When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled CONFIG_DEBUG_KMEMLEAK=y and the result showed an unreferenced object in kworker process: # cat /sys/kernel/debug/kmemleak unreferenced object 0xa01dabff6100 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 unreferenced object 0xa01dabff6ec0 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [ ] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 # Please fing the build config and lshw output attached. dmesg is useless, as it is filled with events like: [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792 [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" kernel memory leaks, potentially exposing race conditions or other more serious bug. How is a memory leak a race condition? The bug is now also confirmed and now manifested also in the Ubuntu 22.04 LTS jammy 6.1-rc4 build. Here is the kmemleak output: unreferenced object 0x9242b13b3980 (size 64): comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) hex dump (first 32 bytes): 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... backtrace: [ ] __kmem_cache_alloc_node+0x380/0x4e0 [<2b17af47>] kmalloc_node_trace+0x27/0xa0 [<4c09eee5>] xhci_alloc_command+0x6e/0x180 This is a totally different backtrace from above, how are they related? This looks like a potential xhci issue. Can you use 'git bisect' to track down the offending change that caused this? thanks, greg k-h Hello, Greg, Thorsten! After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels. The bisect shows the problem appeared before 4.17, so unless I find what is causing black screen when booting pre-4.17 kernels, it's a no-go ... :( Thanks, Mirsad -- Mirsad Goran Todorovac Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu -- System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb, Republic of Croatia The European Union
Re: [systemd-devel] RFC: Passing on initial client user in systemd-userdbd
Hi, > in theory, I have implemented that now, but I somehow cannot make > userdbd send credentials on, because it does not seem to call > varlink_write at all. I am a bit lost with that, because as I > understand the code, all parts of userdbd and homed should be using > varlink_write when calling methods on remote services and when > replying. Yet, attaching gdb to userdbd and setting a breakpoint in > varlink_write does not break… OK, wait… that seems to be a red herring. The interaction between the multiplexer and the sockets in /run/systemd/userdb does not seem to work at all in the container spawned by mkosi. -nik signature.asc Description: PGP signature
Re: [systemd-devel] RFC: Passing on initial client user in systemd-userdbd
Hi Lennart, in theory, I have implemented that now, but I somehow cannot make userdbd send credentials on, because it does not seem to call varlink_write at all. I am a bit lost with that, because as I understand the code, all parts of userdbd and homed should be using varlink_write when calling methods on remote services and when replying. Yet, attaching gdb to userdbd and setting a breakpoint in varlink_write does not break… Can you give me a hint on what part of the picture I might be missing? Cheers, Nik signature.asc Description: PGP signature
Re: [systemd-devel] RFC: Passing on initial client user in systemd-userdbd
On Mo, 28.11.22 16:27, Dominik George (n...@naturalnet.de) wrote: > Hi, > > > You don't have to send that really, the kernel will implicitly attach it > > automatically whenever the sender's credentials change. Thus, a > > receiver can safely assume that the ucred remains the same as the > > SO_PEERCRED data until it receives a new SCM_CREDENTIALS that says > > otherwise. > > > > You want to send SCM_CREDENTIALS explicitly only when you actively try > > to impersonate someone else. > > I'm not convinced of that. Of course, sending the creds if it does not > differ from the process running would be sufficient, but doing it in > all cases removes a lot of complexity. >From the receiving side there's very little difference in behaviour: the kernel will automatically send this stuff *anyway* if needed. hence on the receiving side in the Varlink object just add a new "struct ucred" field that stores the last SCM_CREDENTIALS ucred that was received. Update it whenever a new SCM_CREDENTIALS is received. It will look the exact same way from the receiving side if on the sending side the kernel sent it automatically because the sender's uid changed or if the sender appended it explicitly because it felt like it, for example because it wants to impersonate someone. > Sending SCM_CREDENTIALS selectively would mean we would have to > introduce a distinction between systemd-userdbd acting as multiplexer > and not doing so, which would require moving quite a bit of code > around that is now neatly generic. userdbd should always impersonate the client it received the request on. What I am saying is that regular varlink client code (i.e. not userdbd, not an impersonator) should not bother with this at all, since the kernel well attach this info anyway if needed. Only impersonators need to attach SCM_CREDENTIALS explicitly, and userdb should be one of these impersonators. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] RFC: Passing on initial client user in systemd-userdbd
Hi, > You don't have to send that really, the kernel will implicitly attach it > automatically whenever the sender's credentials change. Thus, a > receiver can safely assume that the ucred remains the same as the > SO_PEERCRED data until it receives a new SCM_CREDENTIALS that says > otherwise. > > You want to send SCM_CREDENTIALS explicitly only when you actively try > to impersonate someone else. I'm not convinced of that. Of course, sending the creds if it does not differ from the process running would be sufficient, but doing it in all cases removes a lot of complexity. Sending SCM_CREDENTIALS selectively would mean we would have to introduce a distinction between systemd-userdbd acting as multiplexer and not doing so, which would require moving quite a bit of code around that is now neatly generic. On the other hand, the only downside of always sending the credentials is a few bytes overhead on the wire. In my opinion, saving htese few bytes does not justify the added complexity by making a distinction between the code paths. What do you think? > In the varlink API please report the SCM_CREDENTIALS ucred seperately > from the SO_PEERCRED though (i.e. from the current ucreds we already > store). For various purposes it is interesting to know the identity of > the process initiating the connection, if it's different from the > process actually sending messages over it. Sure! Cheers, Nik signature.asc Description: PGP signature
Re: [systemd-devel] Prevent firmware from falling back to next EFI boot option on secure boot failure?
Wanted to follow up on this--was away from my desk because of holidays in the US. I did open an issue on github: https://github.com/systemd/systemd/issues/25548 Thank you for your help here, Lennart. Thanks, --Daniel On Wed, Nov 23, 2022 at 12:32 PM Lennart Poettering wrote: > > On Mi, 23.11.22 17:56, Lennart Poettering (lenn...@poettering.net) wrote: > > > > If this is a bug, I'd be willing to attempt a pull request submission > > > if a suggested fix is given. Overall we like the functionality > > > sd-boot provides and the integration with systemd, but this is likely > > > a hard requirement for our use case. > > > > Yes please file an issue on github first, and this does sound a lot > > like something we should fix, hence a PR that addresses this would be > > more than welcome, too. > > BTW, I think we should treat an EFI binary like a system we can't boot > as per the boot assessment logic. i.e. whenever we fail to invoke a > binary (regardless if the reason is the security check or something > else), then we should count down it's counters, and then stop using it > once it hits zero. > > i.e. i think this should hook into the logic described in > https://systemd.io/AUTOMATIC_BOOT_ASSESSMENT > > Lennart > > -- > Lennart Poettering, Berlin
Re: [systemd-devel] RFC: Passing on initial client user in systemd-userdbd
On Mo, 28.11.22 00:14, Dominik George (n...@naturalnet.de) wrote: 65;6800;1c > Hi, > > > The approach brings me a bit farther away from being able to implement it > > myself, but not too far I guess ;). > > I've spent some time reading the userdb code now, and it actually > seems pretty easy to do. > > Here's my rough plan: > > 1. In src/userdb/userdbd-manager.c manager_startup(), set teh > SO_PASSCRED socket option > 2. In src/shared/varlink.c, change the behaviour in two places: > - In varlink_read, use recvmsg to read the SCM_CREDENTIALS >message and, if we get one and its uid is valid, store the >ucred in the varlink struct and set its ucred_acquired to truw > - In varlink_write, always send an SCM_CREDENTIALS message — >if ucred_acquired is true on the varlink object, send this >ucred struct' if it is false, send an empty message to use >our real credentials You don't have to send that really, the kernel will implicitly attach it automatically whenever the sender's credentials change. Thus, a receiver can safely assume that the ucred remains the same as the SO_PEERCRED data until it receives a new SCM_CREDENTIALS that says otherwise. You want to send SCM_CREDENTIALS explicitly only when you actively try to impersonate someone else. > Given that all userdbd services in systemd, including the multiplexer, > use the same code, this should be all there is to it to enable the > discussed behaviour in systemd, and downstream service implementations > could start using it. > > If yhere is nothing fundamentally wrong with my assessment, I'll give > the implementation a shot. Sounds great! Happy to review a PR for that. In the varlink API please report the SCM_CREDENTIALS ucred seperately from the SO_PEERCRED though (i.e. from the current ucreds we already store). For various purposes it is interesting to know the identity of the process initiating the connection, if it's different from the process actually sending messages over it. Lennart -- Lennart Poettering, Berlin
