Re: lockdep warning while booting POWER9 PowerNV
powerpc: define arch_is_kernel_initmem_freed() for lockdep
Under certain circumstances, we hit a warning in lockdep_register_key:
if (WARN_ON_ONCE(static_obj(key)))
return;
This occurs when the key falls into initmem that has since been freed
and can now be reused. This has been observed on boot, and under memory
pressure.
Define arch_is_kernel_initmem_freed(), which allows lockdep to correctly
identify this memory as dynamic.
This fixes a bug picked up by the powerpc64 syzkaller instance where we
hit the WARN via alloc_netdev_mqs.
Link: https://github.com/linuxppc/issues/issues/284
Link: https://lore.kernel.org/linuxppc-dev/87ef0vpfbc@mpe.ellerman.id.au/
Reported-by: Qian Cai
Reported-by: ppc syzbot c/o Andrew Donnellan
Commit-message-by: Daniel Axtens
---
The ppc64 syzkaller link is probably not stable enough to go into
the git history forever, but fwiw:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
---
arch/powerpc/include/asm/sections.h | 14 ++
1 file changed, 14 insertions(+)
diff --git a/arch/powerpc/include/asm/sections.h
b/arch/powerpc/include/asm/sections.h
index 5a9b6eb651b6..d19871763ed4 100644
--- a/arch/powerpc/include/asm/sections.h
+++ b/arch/powerpc/include/asm/sections.h
@@ -5,8 +5,22 @@
#include
#include
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
#include
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+ if (!init_mem_is_free)
+ return 0;
+
+ return addr >= (unsigned long)__init_begin &&
+ addr < (unsigned long)__init_end;
+}
+
extern char __head_end[];
#ifdef __powerpc64__
Re: lockdep warning while booting POWER9 PowerNV
On 25 November 2019 12:12:01 pm AEDT, Daniel Axtens wrote:
>Hi Michael,
>
Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH)
>would generate
a warning in lockdep_register_key() at,
if (WARN_ON_ONCE(static_obj(key)))
because
key = 0xc19ad118
&_stext = 0xc000
&_end = 0xc49d
i.e., it will cause static_obj() returns 1.
>>>
>>> (back from a trip)
>>>
>>> Hi Qian,
>>>
>>> Does this mean that on POWER9 it can happen that a dynamically
>allocated
>>> object has an address that falls between &_stext and &_end?
>>
>> I thought that was true on all arches due to initmem, but seems not.
>>
>> I guess we have the same problem as s390 and we need to define
>> arch_is_kernel_initmem_freed().
>>
>> Qian, can you try this:
>>
>> diff --git a/arch/powerpc/include/asm/sections.h
>b/arch/powerpc/include/asm/sections.h
>> index 4a1664a8658d..616b1b7b7e52 100644
>> --- a/arch/powerpc/include/asm/sections.h
>> +++ b/arch/powerpc/include/asm/sections.h
>> @@ -5,8 +5,22 @@
>>
>> #include
>> #include
>> +
>> +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
>> +
>> #include
>>
>> +extern bool init_mem_is_free;
>> +
>> +static inline int arch_is_kernel_initmem_freed(unsigned long addr)
>> +{
>> +if (!init_mem_is_free)
>> +return 0;
>> +
>> +return addr >= (unsigned long)__init_begin &&
>> +addr < (unsigned long)__init_end;
>> +}
>> +
>> extern char __head_end[];
>>
>> #ifdef __powerpc64__
>>
>
>This also fixes the following syzkaller bug:
>https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
>https://github.com/linuxppc/issues/issues/284
>
>Would you like me to do up a nice commit message for it?
That'd be great, thanks.
cheers
Re: lockdep warning while booting POWER9 PowerNV
Hi Michael,
>>> Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would
>>> generate
>>> a warning in lockdep_register_key() at,
>>>
>>> if (WARN_ON_ONCE(static_obj(key)))
>>>
>>> because
>>>
>>> key = 0xc19ad118
>>> &_stext = 0xc000
>>> &_end = 0xc49d
>>>
>>> i.e., it will cause static_obj() returns 1.
>>
>> (back from a trip)
>>
>> Hi Qian,
>>
>> Does this mean that on POWER9 it can happen that a dynamically allocated
>> object has an address that falls between &_stext and &_end?
>
> I thought that was true on all arches due to initmem, but seems not.
>
> I guess we have the same problem as s390 and we need to define
> arch_is_kernel_initmem_freed().
>
> Qian, can you try this:
>
> diff --git a/arch/powerpc/include/asm/sections.h
> b/arch/powerpc/include/asm/sections.h
> index 4a1664a8658d..616b1b7b7e52 100644
> --- a/arch/powerpc/include/asm/sections.h
> +++ b/arch/powerpc/include/asm/sections.h
> @@ -5,8 +5,22 @@
>
> #include
> #include
> +
> +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
> +
> #include
>
> +extern bool init_mem_is_free;
> +
> +static inline int arch_is_kernel_initmem_freed(unsigned long addr)
> +{
> + if (!init_mem_is_free)
> + return 0;
> +
> + return addr >= (unsigned long)__init_begin &&
> + addr < (unsigned long)__init_end;
> +}
> +
> extern char __head_end[];
>
> #ifdef __powerpc64__
>
This also fixes the following syzkaller bug:
https://syzkaller-ppc64.appspot.com/bug?id=cfdf75cd985012d0124cd41e6fa095d33e7d0f6b
https://github.com/linuxppc/issues/issues/284
Would you like me to do up a nice commit message for it?
Regards,
Daniel
>
> cheers
Re: lockdep warning while booting POWER9 PowerNV
> On Sep 4, 2019, at 11:55 PM, Michael Ellerman wrote:
>
> Bart Van Assche writes:
>> On 8/30/19 2:13 PM, Qian Cai wrote:
>>> https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
>>>
>>> Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would
>>> generate
>>> a warning in lockdep_register_key() at,
>>>
>>> if (WARN_ON_ONCE(static_obj(key)))
>>>
>>> because
>>>
>>> key = 0xc19ad118
>>> &_stext = 0xc000
>>> &_end = 0xc49d
>>>
>>> i.e., it will cause static_obj() returns 1.
>>
>> (back from a trip)
>>
>> Hi Qian,
>>
>> Does this mean that on POWER9 it can happen that a dynamically allocated
>> object has an address that falls between &_stext and &_end?
>
> I thought that was true on all arches due to initmem, but seems not.
>
> I guess we have the same problem as s390 and we need to define
> arch_is_kernel_initmem_freed().
>
> Qian, can you try this:
>
> diff --git a/arch/powerpc/include/asm/sections.h
> b/arch/powerpc/include/asm/sections.h
> index 4a1664a8658d..616b1b7b7e52 100644
> --- a/arch/powerpc/include/asm/sections.h
> +++ b/arch/powerpc/include/asm/sections.h
> @@ -5,8 +5,22 @@
>
> #include
> #include
> +
> +#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
> +
> #include
>
> +extern bool init_mem_is_free;
> +
> +static inline int arch_is_kernel_initmem_freed(unsigned long addr)
> +{
> + if (!init_mem_is_free)
> + return 0;
> +
> + return addr >= (unsigned long)__init_begin &&
> + addr < (unsigned long)__init_end;
> +}
> +
> extern char __head_end[];
>
> #ifdef __powerpc64__
>
Michael, this fix is also needed as it starts to trigger another one of those
where the allocated
memory is from initmem.
[ 31.326825] key = c19049a0
[ 31.326862] stext = c000, end = c70e
[ 31.326907] init_start = c0c7, init_end = c20f
[ 31.325021] WARNING: CPU: 0 PID: 5 at kernel/locking/lockdep.c:1121
lockdep_register_key+0xb4/0x340
[ 31.325061] Modules linked in: tg3(+) ahci(+) libahci libata mdio libphy
firmware_class dm_mirror dm_region_hash dm_log dm_mod
[ 31.325128] CPU: 0 PID: 5 Comm: kworker/0:0 Not tainted
5.4.0-rc8-next-20191120+ #4
[ 31.325190] Workqueue: events work_for_cpu_fn
[ 31.325215] NIP: c01a23a4 LR: c075eccc CTR:
[ 31.325257] REGS: c0002e72f4c0 TRAP: 0700 Not tainted
(5.4.0-rc8-next-20191120+)
[ 31.325320] MSR: 9282b033 CR:
48000c20 XER: 2004
[ 31.325392] CFAR: c01a233c IRQMASK: 0
GPR00: c075eccc c0002e72f750 c2cff500
c70df500
GPR04: c014beb01990 c19042b8
GPR08: c0425e28
c00c04761020
GPR12: c70e c0002e5214f8
c01ffca018c8
GPR16: c01ffca018e4 c01ffca01c80 c01ffca018d0
c0002e6e3e48
GPR20: c2cbf500 c0002e520080 c01ffca05408
c0002e6e3e00
GPR24: c07d36d0 0005 0005
c1904000
GPR28: c70e c000 c19049a0
c0002e72f7f0
[ 31.325765] NIP [c01a23a4] lockdep_register_key+0xb4/0x340
[ 31.325809] LR [c075eccc] alloc_netdev_mqs+0x15c/0x500
[ 31.325848] Call Trace:
[ 31.325886] [c0002e72f750] [0005] 0x5 (unreliable)
[ 31.325930] [c0002e72f7f0] [c075eccc]
alloc_netdev_mqs+0x15c/0x500
[ 31.325984] [c0002e72f8d0] [c07d37f0]
alloc_etherdev_mqs+0x60/0x90
[ 31.326047] [c0002e72f910] [c0080f150110] tg3_init_one+0x108/0x1d00
[tg3]
[ 31.326098] [c0002e72fac0] [c0633b48] local_pci_probe+0x78/0x100
[ 31.326143] [c0002e72fb50] [c0134b60] work_for_cpu_fn+0x40/0x70
[ 31.326190] [c0002e72fb80] [c013927c]
process_one_work+0x3ac/0x710
[ 31.326221] [c0002e72fc70] [c0138d90]
process_scheduled_works+0x60/0xa0
[ 31.326274] [c0002e72fcb0] [c0139ba4] worker_thread+0x344/0x4a0
[ 31.326317] [c0002e72fda0] [c0142f68] kthread+0x1b8/0x1e0
[ 31.326363] [c0002e72fe20] [c000b748]
ret_from_kernel_thread+0x5c/0x74
[ 31.326412] Instruction dump:
[ 31.326448] 2823 418200a0 7fc3f378 48191fd9 6000 70630001 41810018
7fc3f378
[ 31.326510] 4807fe25 6000 70630001 40810060 <0fe0> 3c62fffc 8883fa2f
70840001
[ 31.326573] irq event stamp: 806
[ 31.326617] hardirqs last enabled at (805): []
_raw_write_unlock_irqrestore+0x5c/0xc0
[ 31.32] hardirqs last disabled at (806): []
program_check_common+0x21c/0x230
[ 31.326710] softirqs last enabled at (0): []
copy_process+0x688/0x1850
[ 31.326752] softirqs last disabled at (0): [<>] 0x0
[ 31.326791]
Re: lockdep warning while booting POWER9 PowerNV
On Thu, 2019-09-05 at 13:55 +1000, Michael Ellerman wrote:
> Bart Van Assche writes:
> > On 8/30/19 2:13 PM, Qian Cai wrote:
> > > https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
> > >
> > > Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would
> > > generate
> > > a warning in lockdep_register_key() at,
> > >
> > > if (WARN_ON_ONCE(static_obj(key)))
> > >
> > > because
> > >
> > > key = 0xc19ad118
> > > &_stext = 0xc000
> > > &_end = 0xc49d
> > >
> > > i.e., it will cause static_obj() returns 1.
> >
> > (back from a trip)
> >
> > Hi Qian,
> >
> > Does this mean that on POWER9 it can happen that a dynamically allocated
> > object has an address that falls between &_stext and &_end?
>
> I thought that was true on all arches due to initmem, but seems not.
>
> I guess we have the same problem as s390 and we need to define
> arch_is_kernel_initmem_freed().
Actually, it is in the .bss section. The commit 2d4f567103ff ("KVM: PPC:
Introduce kvm_tmp framework") adds kvm_tmp[] into the .bss section and then free
the rest of unused spaces back to the page allocator.
kernel_init
kvm_guest_init
kvm_free_tmp
free_reserved_area
free_unref_page
free_unref_page_prepare
Later, alloc_workqueue() happens to allocate some pages from there, and triggers
the warning. Not sure what the best way to solve this.
Re: lockdep warning while booting POWER9 PowerNV
Bart Van Assche writes:
> On 8/30/19 2:13 PM, Qian Cai wrote:
>> https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config
>>
>> Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would
>> generate
>> a warning in lockdep_register_key() at,
>>
>> if (WARN_ON_ONCE(static_obj(key)))
>>
>> because
>>
>> key = 0xc19ad118
>> &_stext = 0xc000
>> &_end = 0xc49d
>>
>> i.e., it will cause static_obj() returns 1.
>
> (back from a trip)
>
> Hi Qian,
>
> Does this mean that on POWER9 it can happen that a dynamically allocated
> object has an address that falls between &_stext and &_end?
I thought that was true on all arches due to initmem, but seems not.
I guess we have the same problem as s390 and we need to define
arch_is_kernel_initmem_freed().
Qian, can you try this:
diff --git a/arch/powerpc/include/asm/sections.h
b/arch/powerpc/include/asm/sections.h
index 4a1664a8658d..616b1b7b7e52 100644
--- a/arch/powerpc/include/asm/sections.h
+++ b/arch/powerpc/include/asm/sections.h
@@ -5,8 +5,22 @@
#include
#include
+
+#define arch_is_kernel_initmem_freed arch_is_kernel_initmem_freed
+
#include
+extern bool init_mem_is_free;
+
+static inline int arch_is_kernel_initmem_freed(unsigned long addr)
+{
+ if (!init_mem_is_free)
+ return 0;
+
+ return addr >= (unsigned long)__init_begin &&
+ addr < (unsigned long)__init_end;
+}
+
extern char __head_end[];
#ifdef __powerpc64__
cheers
Re: lockdep warning while booting POWER9 PowerNV
On 8/30/19 2:13 PM, Qian Cai wrote: https://raw.githubusercontent.com/cailca/linux-mm/master/powerpc.config Once in a while, booting an IBM POWER9 PowerNV system (8335-GTH) would generate a warning in lockdep_register_key() at, if (WARN_ON_ONCE(static_obj(key))) because key = 0xc19ad118 &_stext = 0xc000 &_end = 0xc49d i.e., it will cause static_obj() returns 1. (back from a trip) Hi Qian, Does this mean that on POWER9 it can happen that a dynamically allocated object has an address that falls between &_stext and &_end? Since I am not familiar with POWER9 nor have access to such a system, can you propose a patch? Thanks, Bart.
