Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On Tue, Nov 29, 2022 at 04:35:10AM +0100, Mirsad Goran Todorovac wrote: > On 10. 11. 2022. 10:20, Greg KH wrote: > > On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote: > > > On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: > > > > > > > Dear Sirs, > > > > > > > > When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled > > > > CONFIG_DEBUG_KMEMLEAK=y > > > > and the result showed an unreferenced object in kworker process: > > > > > > > > # cat /sys/kernel/debug/kmemleak > > > > unreferenced object 0xa01dabff6100 (size 16): > > > > comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) > > > > hex dump (first 16 bytes): > > > > 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... > > > > backtrace: > > > > [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 > > > > [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 > > > > [<05472512>] kstrdup+0x36/0x70 > > > > [<2f797ac4>] kstrdup_const+0x28/0x30 > > > > [] kvasprintf_const+0x78/0xa0 > > > > [ ] kobject_set_name_vargs+0x23/0xa0 > > > > [<4158a6c0>] dev_set_name+0x53/0x70 > > > > [<1a120541>] memstick_check+0xff/0x384 [memstick] > > > > [<122bb894>] process_one_work+0x214/0x3f0 > > > > [ ] worker_thread+0x34/0x3d0 > > > > [<02409855>] kthread+0xed/0x120 > > > > [<7b02b4a3>] ret_from_fork+0x1f/0x30 > > > > unreferenced object 0xa01dabff6ec0 (size 16): > > > > comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) > > > > hex dump (first 16 bytes): > > > > 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... > > > > backtrace: > > > > [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 > > > > [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 > > > > [<05472512>] kstrdup+0x36/0x70 > > > > [<2f797ac4>] kstrdup_const+0x28/0x30 > > > > [ ] kvasprintf_const+0x78/0xa0 > > > > [ ] kobject_set_name_vargs+0x23/0xa0 > > > > [<4158a6c0>] dev_set_name+0x53/0x70 > > > > [<1a120541>] memstick_check+0xff/0x384 [memstick] > > > > [<122bb894>] process_one_work+0x214/0x3f0 > > > > [ ] worker_thread+0x34/0x3d0 > > > > [<02409855>] kthread+0xed/0x120 > > > > [<7b02b4a3>] ret_from_fork+0x1f/0x30 > > > > # > > > > > > > > Please fing the build config and lshw output attached. > > > > > > > > dmesg is useless, as it is filled with events like: > > > > > > > > [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 > > > > [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > > > [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: > > > > 458762 > > > > [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 > > > > [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > > > [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: > > > > 458762 > > > > [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 > > > > [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > > > [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: > > > > 458792 > > > > [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 > > > > [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > > > > > This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" > > > kernel memory leaks, potentially exposing race conditions or other more > > > serious bug. > > > > How is a memory leak a race condition? > > > > > The bug is now also confirmed and now manifested also in the Ubuntu 22.04 > > > LTS jammy 6.1-rc4 build. > > > > > > Here is the kmemleak output: > > > > > > unreferenced object 0x9242b13b3980 (size 64): > > > comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) > > > hex dump (first 32 bytes): > > > 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... > > > 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... > > > backtrace: > > > [ ] __kmem_cache_alloc_node+0x380/0x4e0 > > > [<2b17af47>] kmalloc_node_trace+0x27/0xa0 > > > [<4c09eee5>] xhci_alloc_command+0x6e/0x180 > > > > This is a totally different backtrace from above, how are they related? > > > > This looks like a potential xhci issue. Can you use 'git bisect' to > > track down the offending change that caused this? > > > > thanks, > > > > greg k-h > > Hello, Greg, Thorsten! > > After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels. > The bisect shows the problem appeared before 4.17, so unless I find what is > causing black screen when booting pre-4.17
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On 10. 11. 2022. 10:20, Greg KH wrote: On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote: On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: Dear Sirs, When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled CONFIG_DEBUG_KMEMLEAK=y and the result showed an unreferenced object in kworker process: # cat /sys/kernel/debug/kmemleak unreferenced object 0xa01dabff6100 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 unreferenced object 0xa01dabff6ec0 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [ ] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 # Please fing the build config and lshw output attached. dmesg is useless, as it is filled with events like: [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792 [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" kernel memory leaks, potentially exposing race conditions or other more serious bug. How is a memory leak a race condition? The bug is now also confirmed and now manifested also in the Ubuntu 22.04 LTS jammy 6.1-rc4 build. Here is the kmemleak output: unreferenced object 0x9242b13b3980 (size 64): comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) hex dump (first 32 bytes): 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... backtrace: [ ] __kmem_cache_alloc_node+0x380/0x4e0 [<2b17af47>] kmalloc_node_trace+0x27/0xa0 [<4c09eee5>] xhci_alloc_command+0x6e/0x180 This is a totally different backtrace from above, how are they related? This looks like a potential xhci issue. Can you use 'git bisect' to track down the offending change that caused this? thanks, greg k-h Hello, Greg, Thorsten! After multiple attempts, my box's UEFI refuses to run pre-4.17 kernels. The bisect shows the problem appeared before 4.17, so unless I find what is causing black screen when booting pre-4.17 kernels, it's a no-go ... :( Thanks, Mirsad -- Mirsad Goran Todorovac Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu -- System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb, Republic of Croatia The European Union
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On 10.11.2022. 10:20, Greg KH wrote: On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote: On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: Dear Sirs, When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled CONFIG_DEBUG_KMEMLEAK=y and the result showed an unreferenced object in kworker process: # cat /sys/kernel/debug/kmemleak unreferenced object 0xa01dabff6100 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 unreferenced object 0xa01dabff6ec0 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [ ] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 # Please fing the build config and lshw output attached. dmesg is useless, as it is filled with events like: [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792 [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" kernel memory leaks, potentially exposing race conditions or other more serious bug. How is a memory leak a race condition? Probably I connected it with spurious memleak in thermald that occurs only under the torture tests. The bug is now also confirmed and now manifested also in the Ubuntu 22.04 LTS jammy 6.1-rc4 build. Here is the kmemleak output: unreferenced object 0x9242b13b3980 (size 64): comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) hex dump (first 32 bytes): 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... backtrace: [ ] __kmem_cache_alloc_node+0x380/0x4e0 [<2b17af47>] kmalloc_node_trace+0x27/0xa0 [<4c09eee5>] xhci_alloc_command+0x6e/0x180 This is a totally different backtrace from above, how are they related? This looks like a potential xhci issue. Can you use 'git bisect' to track down the offending change that caused this? thanks, greg k-h Hi, Greg, I saw this email only now. I will try to bisect the bug. I was trying to bisect the first instance that occurs only on AlmaLinux and it is introduced before 4.17. Here is the progress of the bisect on AlmaLinux 8.6 (a.k.a. Centos 8): mtodorov@domac:~/linux/kernel/linux_stable$ git bisect log git bisect start # bad: [f0c4d9fc9cc9462659728d168387191387e903cc] Linux 6.1-rc4 git bisect bad f0c4d9fc9cc9462659728d168387191387e903cc # bad: [fbd56ddcecab5a3623a89c8e941fdbcc55b41045] Linux 6.0.1 git bisect bad fbd56ddcecab5a3623a89c8e941fdbcc55b41045 # bad: [7e18e42e4b280c85b76967a9106a13ca61c16179] Linux 6.0-rc4 git bisect bad 7e18e42e4b280c85b76967a9106a13ca61c16179 # bad: [568035b01cfb107af8d2e4bd2fb9aea22cf5b868] Linux 6.0-rc1 git bisect bad 568035b01cfb107af8d2e4bd2fb9aea22cf5b868 # bad: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On 10.11.2022. 9:28, Thorsten Leemhuis wrote: This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" kernel memory leaks, potentially exposing race conditions or other more serious bug. Maybe, but that sadly is also true for a lot of other known issues, for example those in this quite long list: https://syzkaller.appspot.com/upstream#open It would help if you could pinpoint the problem, then we know who should look into this. You CCed me and the regression list, so I assume it's a regression. Hence: Could you try to bisect it? Ciao, Thorsten Hi, Thorsten! I am taking this task, however, it might not be a relatively easy bisect like the past two or three (probably were beginner's luck). It appears to predate 4.19: $ head -13 memleak-kwork-4.19.0-08-84df9525b0c2-memlk-menu-al.log unreferenced object 0x8ff917c79c40 (size 16): comm "kworker/u12:4", pid 422, jiffies 4294672757 (age 380.342s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<4f7bc99c>] kobject_set_name_vargs+0x1e/0x90 [] dev_set_name+0x57/0x70 [<1fd9518e>] memstick_check+0x99/0x330 [memstick] [<8c797991>] process_one_work+0x1a7/0x3a0 [<405c5133>] worker_thread+0x30/0x390 [<60a58c2e>] kthread+0x112/0x130 [<4b138871>] ret_from_fork+0x35/0x40 [<29f3e0aa>] 0x $ Cheers, Mirsad -- Mirsad Todorovac System engineer Faculty of Graphic Arts | Academy of Fine Arts University of Zagreb Republic of Croatia, the European Union -- Sistem inženjer Grafički fakultet | Akademija likovnih umjetnosti Sveučilište u Zagrebu
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On Thu, Nov 10, 2022 at 05:57:57AM +0100, Mirsad Goran Todorovac wrote: > On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: > > > Dear Sirs, > > > > When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled > > CONFIG_DEBUG_KMEMLEAK=y > > and the result showed an unreferenced object in kworker process: > > > > # cat /sys/kernel/debug/kmemleak > > unreferenced object 0xa01dabff6100 (size 16): > > comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) > > hex dump (first 16 bytes): > > 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... > > backtrace: > > [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 > > [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 > > [<05472512>] kstrdup+0x36/0x70 > > [<2f797ac4>] kstrdup_const+0x28/0x30 > > [] kvasprintf_const+0x78/0xa0 > > [ ] kobject_set_name_vargs+0x23/0xa0 > > [<4158a6c0>] dev_set_name+0x53/0x70 > > [<1a120541>] memstick_check+0xff/0x384 [memstick] > > [<122bb894>] process_one_work+0x214/0x3f0 > > [ ] worker_thread+0x34/0x3d0 > > [<02409855>] kthread+0xed/0x120 > > [<7b02b4a3>] ret_from_fork+0x1f/0x30 > > unreferenced object 0xa01dabff6ec0 (size 16): > > comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) > > hex dump (first 16 bytes): > > 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... > > backtrace: > > [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 > > [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 > > [<05472512>] kstrdup+0x36/0x70 > > [<2f797ac4>] kstrdup_const+0x28/0x30 > > [ ] kvasprintf_const+0x78/0xa0 > > [ ] kobject_set_name_vargs+0x23/0xa0 > > [<4158a6c0>] dev_set_name+0x53/0x70 > > [<1a120541>] memstick_check+0xff/0x384 [memstick] > > [<122bb894>] process_one_work+0x214/0x3f0 > > [ ] worker_thread+0x34/0x3d0 > > [<02409855>] kthread+0xed/0x120 > > [<7b02b4a3>] ret_from_fork+0x1f/0x30 > > # > > > > Please fing the build config and lshw output attached. > > > > dmesg is useless, as it is filled with events like: > > > > [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 > > [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 > > [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 > > [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 > > [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 > > [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792 > > [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 > > [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 > > This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" > kernel memory leaks, potentially exposing race conditions or other more > serious bug. How is a memory leak a race condition? > The bug is now also confirmed and now manifested also in the Ubuntu 22.04 > LTS jammy 6.1-rc4 build. > > Here is the kmemleak output: > > unreferenced object 0x9242b13b3980 (size 64): > comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) > hex dump (first 32 bytes): > 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... > 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... > backtrace: > [ ] __kmem_cache_alloc_node+0x380/0x4e0 > [<2b17af47>] kmalloc_node_trace+0x27/0xa0 > [<4c09eee5>] xhci_alloc_command+0x6e/0x180 This is a totally different backtrace from above, how are they related? This looks like a potential xhci issue. Can you use 'git bisect' to track down the offending change that caused this? thanks, greg k-h
Re: [systemd-devel] BUG: kworker + systemd-udevd memory leaks found in 6.1.0-rc4
On 04. 11. 2022. 11:40, Mirsad Goran Todorovac wrote: Dear Sirs, When building a RPM 6.1.0-rc3 for AlmaLinux 8.6, I have enabled CONFIG_DEBUG_KMEMLEAK=y and the result showed an unreferenced object in kworker process: # cat /sys/kernel/debug/kmemleak unreferenced object 0xa01dabff6100 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894771 (age 5284.956s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 unreferenced object 0xa01dabff6ec0 (size 16): comm "kworker/u12:4", pid 400, jiffies 4294894774 (age 5284.944s) hex dump (first 16 bytes): 6d 65 6d 73 74 69 63 6b 30 00 00 00 00 00 00 00 memstick0... backtrace: [<9ff951f6>] __kmem_cache_alloc_node+0x380/0x4e0 [<451f4268>] __kmalloc_node_track_caller+0x55/0x150 [<05472512>] kstrdup+0x36/0x70 [<2f797ac4>] kstrdup_const+0x28/0x30 [ ] kvasprintf_const+0x78/0xa0 [ ] kobject_set_name_vargs+0x23/0xa0 [<4158a6c0>] dev_set_name+0x53/0x70 [<1a120541>] memstick_check+0xff/0x384 [memstick] [<122bb894>] process_one_work+0x214/0x3f0 [ ] worker_thread+0x34/0x3d0 [<02409855>] kthread+0xed/0x120 [<7b02b4a3>] ret_from_fork+0x1f/0x30 # Please fing the build config and lshw output attached. dmesg is useless, as it is filled with events like: [ 6068.996120] evbug: Event. Dev: input4, Type: 1, Code: 31, Value: 0 [ 6068.996121] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.124145] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.124149] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 1 [ 6069.124150] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.196003] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458762 [ 6069.196007] evbug: Event. Dev: input4, Type: 1, Code: 34, Value: 0 [ 6069.196009] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 [ 6069.788129] evbug: Event. Dev: input4, Type: 4, Code: 4, Value: 458792 [ 6069.788133] evbug: Event. Dev: input4, Type: 1, Code: 28, Value: 1 [ 6069.788135] evbug: Event. Dev: input4, Type: 0, Code: 0, Value: 0 This bug is confirmed in 6.1-rc4, among the "thermald" and "systemd-dev" kernel memory leaks, potentially exposing race conditions or other more serious bug. The bug is now also confirmed and now manifested also in the Ubuntu 22.04 LTS jammy 6.1-rc4 build. Here is the kmemleak output: unreferenced object 0x9242b13b3980 (size 64): comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) hex dump (first 32 bytes): 80 8b a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 B... 20 86 a0 f0 42 92 ff ff 00 00 00 00 00 00 00 00 ...B... backtrace: [ ] __kmem_cache_alloc_node+0x380/0x4e0 [<2b17af47>] kmalloc_node_trace+0x27/0xa0 [<4c09eee5>] xhci_alloc_command+0x6e/0x180 [<99436a99>] xhci_alloc_command_with_ctx+0x1d/0x60 [<70f45e17>] xhci_change_max_exit_latency+0x2e/0x1c0 [<1b13cf46>] xhci_disable_usb3_lpm_timeout+0x77/0xb0 [ ] usb_disable_link_state+0x53/0xd0 [<68febd40>] usb_disable_lpm+0x82/0xc0 [ ] usb_unlocked_disable_lpm+0x2d/0x50 [<61a59294>] usb_disable_device+0x12f/0x250 [<44e2299d>] usb_set_configuration+0x65c/0x9a0 [<1af32470>] driver_set_config_work+0x78/0xa0 [<320216c3>] process_one_work+0x214/0x3f0 [<694e932f>] worker_thread+0x34/0x3d0 [<02e30775>] kthread+0xed/0x120 [<512e8066>] ret_from_fork+0x1f/0x30 unreferenced object 0x9242f0a08620 (size 32): comm "kworker/5:3", pid 43106, jiffies 4305052439 (age 71828.792s) hex dump (first 32 bytes): 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 30 86 a0 f0 42 92 ff ff 30 86 a0 f0 42 92 ff ff 0...B...0...B... backtrace: [ ] __kmem_cache_alloc_node+0x380/0x4e0 [<2b17af47>] kmalloc_node_trace+0x27/0xa0 [<459908cf>] xhci_alloc_command+0xe6/0x180 [<99436a99>] xhci_alloc_command_with_ctx+0x1d/0x60 [<70f45e17>] xhci_change_max_exit_latency+0x2e/0x1c0 [<1b13cf46>]
