Re: /sys/kernel/debug/kmemleak empty despite kmemleak reports
Dear Catalin, Am 13.07.20 um 20:27 schrieb Catalin Marinas: On Thu, Jul 09, 2020 at 11:08:52PM +0200, Paul Menzel wrote: Am 09.07.20 um 19:57 schrieb Catalin Marinas: On Thu, Jul 09, 2020 at 04:37:10PM +0200, Paul Menzel wrote: Despite Linux 5.8-rc4 reporting memory leaks on the IBM POWER 8 S822LC, the file does not contain more information. $ dmesg […] > [48662.953323] perf: interrupt took too long (2570 > 2500), lowering kernel.perf_event_max_sample_rate to 77750 [48854.810636] perf: interrupt took too long (3216 > 3212), lowering kernel.perf_event_max_sample_rate to 62000 [52300.044518] perf: interrupt took too long (4244 > 4020), lowering kernel.perf_event_max_sample_rate to 47000 [52751.373083] perf: interrupt took too long (5373 > 5305), lowering kernel.perf_event_max_sample_rate to 37000 [53354.000363] perf: interrupt took too long (6793 > 6716), lowering kernel.perf_event_max_sample_rate to 29250 [53850.215606] perf: interrupt took too long (8672 > 8491), lowering kernel.perf_event_max_sample_rate to 23000 [57542.266099] perf: interrupt took too long (10940 > 10840), lowering kernel.perf_event_max_sample_rate to 18250 [57559.645404] perf: interrupt took too long (13714 > 13675), lowering kernel.perf_event_max_sample_rate to 14500 [61608.697728] Can't find PMC that caused IRQ [71774.463111] kmemleak: 12 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [92372.044785] process '@/usr/bin/gnatmake-5' started with executable stack [92849.380672] FS-Cache: Loaded [92849.417269] FS-Cache: Netfs 'nfs' registered for caching [92849.595974] NFS: Registering the id_resolver key type [92849.596000] Key type id_resolver registered [92849.596000] Key type id_legacy registered [101808.079143] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [106904.323471] Can't find PMC that caused IRQ [129416.391456] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [158171.604221] kmemleak: 34 new suspected memory leaks (see /sys/kernel/debug/kmemleak) $ sudo cat /sys/kernel/debug/kmemleak When they are no longer present, they are most likely false positives. How can this be? Shouldn’t the false positive also be logged in `/sys/kernel/debug/kmemleak`? Sorry, I wasn't clear. It can be a transient false positive. At a subsequent scan, kmemleak found pointer referring the previously reported objects and no longer shows them. Interesting. Is it possible to print a message in that case to avoid confusion? Was this triggered during boot? Or under some workload? From the timestamps it looks like under some load. Was it during boot? I put a delay of 60s to avoid this but, depending on the platform, it can still trigger. No, it happened after several hours of runtime. Kind regards, Paul
Re: /sys/kernel/debug/kmemleak empty despite kmemleak reports
On Thu, Jul 09, 2020 at 11:08:52PM +0200, Paul Menzel wrote: > Am 09.07.20 um 19:57 schrieb Catalin Marinas: > > On Thu, Jul 09, 2020 at 04:37:10PM +0200, Paul Menzel wrote: > > > Despite Linux 5.8-rc4 reporting memory leaks on the IBM POWER 8 S822LC, > > > the > > > file does not contain more information. > > > > > > > $ dmesg > > > > […] > [48662.953323] perf: interrupt took too long (2570 > 2500), > > > > lowering kernel.perf_event_max_sample_rate to 77750 > > > > [48854.810636] perf: interrupt took too long (3216 > 3212), lowering > > > > kernel.perf_event_max_sample_rate to 62000 > > > > [52300.044518] perf: interrupt took too long (4244 > 4020), lowering > > > > kernel.perf_event_max_sample_rate to 47000 > > > > [52751.373083] perf: interrupt took too long (5373 > 5305), lowering > > > > kernel.perf_event_max_sample_rate to 37000 > > > > [53354.000363] perf: interrupt took too long (6793 > 6716), lowering > > > > kernel.perf_event_max_sample_rate to 29250 > > > > [53850.215606] perf: interrupt took too long (8672 > 8491), lowering > > > > kernel.perf_event_max_sample_rate to 23000 > > > > [57542.266099] perf: interrupt took too long (10940 > 10840), lowering > > > > kernel.perf_event_max_sample_rate to 18250 > > > > [57559.645404] perf: interrupt took too long (13714 > 13675), lowering > > > > kernel.perf_event_max_sample_rate to 14500 > > > > [61608.697728] Can't find PMC that caused IRQ > > > > [71774.463111] kmemleak: 12 new suspected memory leaks (see > > > > /sys/kernel/debug/kmemleak) > > > > [92372.044785] process '@/usr/bin/gnatmake-5' started with executable > > > > stack > > > > [92849.380672] FS-Cache: Loaded > > > > [92849.417269] FS-Cache: Netfs 'nfs' registered for caching > > > > [92849.595974] NFS: Registering the id_resolver key type > > > > [92849.596000] Key type id_resolver registered > > > > [92849.596000] Key type id_legacy registered > > > > [101808.079143] kmemleak: 1 new suspected memory leaks (see > > > > /sys/kernel/debug/kmemleak) > > > > [106904.323471] Can't find PMC that caused IRQ > > > > [129416.391456] kmemleak: 1 new suspected memory leaks (see > > > > /sys/kernel/debug/kmemleak) > > > > [158171.604221] kmemleak: 34 new suspected memory leaks (see > > > > /sys/kernel/debug/kmemleak) > > > > $ sudo cat /sys/kernel/debug/kmemleak > > > > When they are no longer present, they are most likely false positives. > > How can this be? Shouldn’t the false positive also be logged in > `/sys/kernel/debug/kmemleak`? Sorry, I wasn't clear. It can be a transient false positive. At a subsequent scan, kmemleak found pointer referring the previously reported objects and no longer shows them. > > Was this triggered during boot? Or under some workload? > > From the timestamps it looks like under some load. Was it during boot? I put a delay of 60s to avoid this but, depending on the platform, it can still trigger. -- Catalin
Re: /sys/kernel/debug/kmemleak empty despite kmemleak reports
Dear Catalin, Am 09.07.20 um 19:57 schrieb Catalin Marinas: On Thu, Jul 09, 2020 at 04:37:10PM +0200, Paul Menzel wrote: Despite Linux 5.8-rc4 reporting memory leaks on the IBM POWER 8 S822LC, the file does not contain more information. $ dmesg […] > [48662.953323] perf: interrupt took too long (2570 > 2500), lowering kernel.perf_event_max_sample_rate to 77750 [48854.810636] perf: interrupt took too long (3216 > 3212), lowering kernel.perf_event_max_sample_rate to 62000 [52300.044518] perf: interrupt took too long (4244 > 4020), lowering kernel.perf_event_max_sample_rate to 47000 [52751.373083] perf: interrupt took too long (5373 > 5305), lowering kernel.perf_event_max_sample_rate to 37000 [53354.000363] perf: interrupt took too long (6793 > 6716), lowering kernel.perf_event_max_sample_rate to 29250 [53850.215606] perf: interrupt took too long (8672 > 8491), lowering kernel.perf_event_max_sample_rate to 23000 [57542.266099] perf: interrupt took too long (10940 > 10840), lowering kernel.perf_event_max_sample_rate to 18250 [57559.645404] perf: interrupt took too long (13714 > 13675), lowering kernel.perf_event_max_sample_rate to 14500 [61608.697728] Can't find PMC that caused IRQ [71774.463111] kmemleak: 12 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [92372.044785] process '@/usr/bin/gnatmake-5' started with executable stack [92849.380672] FS-Cache: Loaded [92849.417269] FS-Cache: Netfs 'nfs' registered for caching [92849.595974] NFS: Registering the id_resolver key type [92849.596000] Key type id_resolver registered [92849.596000] Key type id_legacy registered [101808.079143] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [106904.323471] Can't find PMC that caused IRQ [129416.391456] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [158171.604221] kmemleak: 34 new suspected memory leaks (see /sys/kernel/debug/kmemleak) $ sudo cat /sys/kernel/debug/kmemleak When they are no longer present, they are most likely false positives. How can this be? Shouldn’t the false positive also be logged in `/sys/kernel/debug/kmemleak`? Was this triggered during boot? Or under some workload? From the timestamps it looks like under some load. Kind regards, Paul
Re: /sys/kernel/debug/kmemleak empty despite kmemleak reports
On Thu, Jul 09, 2020 at 04:37:10PM +0200, Paul Menzel wrote: > Despite Linux 5.8-rc4 reporting memory leaks on the IBM POWER 8 S822LC, the > file does not contain more information. > > > $ dmesg > > […] > [48662.953323] perf: interrupt took too long (2570 > 2500), > > lowering > kernel.perf_event_max_sample_rate to 77750 > > [48854.810636] perf: interrupt took too long (3216 > 3212), lowering > > kernel.perf_event_max_sample_rate to 62000 > > [52300.044518] perf: interrupt took too long (4244 > 4020), lowering > > kernel.perf_event_max_sample_rate to 47000 > > [52751.373083] perf: interrupt took too long (5373 > 5305), lowering > > kernel.perf_event_max_sample_rate to 37000 > > [53354.000363] perf: interrupt took too long (6793 > 6716), lowering > > kernel.perf_event_max_sample_rate to 29250 > > [53850.215606] perf: interrupt took too long (8672 > 8491), lowering > > kernel.perf_event_max_sample_rate to 23000 > > [57542.266099] perf: interrupt took too long (10940 > 10840), lowering > > kernel.perf_event_max_sample_rate to 18250 > > [57559.645404] perf: interrupt took too long (13714 > 13675), lowering > > kernel.perf_event_max_sample_rate to 14500 > > [61608.697728] Can't find PMC that caused IRQ > > [71774.463111] kmemleak: 12 new suspected memory leaks (see > > /sys/kernel/debug/kmemleak) > > [92372.044785] process '@/usr/bin/gnatmake-5' started with executable stack > > [92849.380672] FS-Cache: Loaded > > [92849.417269] FS-Cache: Netfs 'nfs' registered for caching > > [92849.595974] NFS: Registering the id_resolver key type > > [92849.596000] Key type id_resolver registered > > [92849.596000] Key type id_legacy registered > > [101808.079143] kmemleak: 1 new suspected memory leaks (see > > /sys/kernel/debug/kmemleak) > > [106904.323471] Can't find PMC that caused IRQ > > [129416.391456] kmemleak: 1 new suspected memory leaks (see > > /sys/kernel/debug/kmemleak) > > [158171.604221] kmemleak: 34 new suspected memory leaks (see > > /sys/kernel/debug/kmemleak) > > $ sudo cat /sys/kernel/debug/kmemleak When they are no longer present, they are most likely false positives. Was this triggered during boot? Or under some workload? -- Catalin
/sys/kernel/debug/kmemleak empty despite kmemleak reports
Dear Linux folks, Despite Linux 5.8-rc4 reporting memory leaks on the IBM POWER 8 S822LC, the file does not contain more information. $ dmesg […] > [48662.953323] perf: interrupt took too long (2570 > 2500), lowering kernel.perf_event_max_sample_rate to 77750 [48854.810636] perf: interrupt took too long (3216 > 3212), lowering kernel.perf_event_max_sample_rate to 62000 [52300.044518] perf: interrupt took too long (4244 > 4020), lowering kernel.perf_event_max_sample_rate to 47000 [52751.373083] perf: interrupt took too long (5373 > 5305), lowering kernel.perf_event_max_sample_rate to 37000 [53354.000363] perf: interrupt took too long (6793 > 6716), lowering kernel.perf_event_max_sample_rate to 29250 [53850.215606] perf: interrupt took too long (8672 > 8491), lowering kernel.perf_event_max_sample_rate to 23000 [57542.266099] perf: interrupt took too long (10940 > 10840), lowering kernel.perf_event_max_sample_rate to 18250 [57559.645404] perf: interrupt took too long (13714 > 13675), lowering kernel.perf_event_max_sample_rate to 14500 [61608.697728] Can't find PMC that caused IRQ [71774.463111] kmemleak: 12 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [92372.044785] process '@/usr/bin/gnatmake-5' started with executable stack [92849.380672] FS-Cache: Loaded [92849.417269] FS-Cache: Netfs 'nfs' registered for caching [92849.595974] NFS: Registering the id_resolver key type [92849.596000] Key type id_resolver registered [92849.596000] Key type id_legacy registered [101808.079143] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [106904.323471] Can't find PMC that caused IRQ [129416.391456] kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak) [158171.604221] kmemleak: 34 new suspected memory leaks (see /sys/kernel/debug/kmemleak) $ sudo cat /sys/kernel/debug/kmemleak $ Kind regards, Paul
