Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
Christian Brauner writes:
> On Thu, Oct 17, 2019 at 09:00:48AM +1100, Michael Ellerman wrote:
>> Christian Brauner writes:
>> > On Wed, Oct 16, 2019 at 11:27:32PM +1100, Michael Ellerman wrote:
>> >> On a machine with a 64K PAGE_SIZE, the nested for loops in
>> >> test_check_nonzero_user() can lead to soft lockups, eg:
>> >>
>> >> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
>> >> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
>> >> virtio_balloon ip_tables x_tables autofs4
>> >> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
>> >> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
>> >> ...
>> >> NIP __might_sleep+0x20/0xc0
>> >> LR __might_fault+0x40/0x60
>> >> Call Trace:
>> >> check_zeroed_user+0x12c/0x200
>> >> test_user_copy_init+0x67c/0x1210 [test_user_copy]
>> >> do_one_initcall+0x60/0x340
>> >> do_init_module+0x7c/0x2f0
>> >> load_module+0x2d94/0x30e0
>> >> __do_sys_finit_module+0xc8/0x150
>> >> system_call+0x5c/0x68
>> >>
>> >> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
>> >> tweak it to only scan a 1024 byte region, but make it cross the
>> >> page boundary.
>> >>
>> >> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
>> >> Suggested-by: Aleksa Sarai
>> >> Signed-off-by: Michael Ellerman
>> >
>> > With Aleksa's Reviewed-by I've picked this up:
>> > https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=copy_struct_from_user
>>
>> Thanks. Are you planning to send that to Linus for v5.4 or v5.5 ?
>
> This looks like a pretty straight bugfix to me since it's clearly
> causing an issue for you on power so v5.4-rc4 is what I'd aim for. I
> just want it to be in linux-next until tomorrow.
I see it in mainine now, thanks!
cheers
Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
On Thu, Oct 17, 2019 at 09:00:48AM +1100, Michael Ellerman wrote:
> Christian Brauner writes:
> > On Wed, Oct 16, 2019 at 11:27:32PM +1100, Michael Ellerman wrote:
> >> On a machine with a 64K PAGE_SIZE, the nested for loops in
> >> test_check_nonzero_user() can lead to soft lockups, eg:
> >>
> >> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
> >> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
> >> virtio_balloon ip_tables x_tables autofs4
> >> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
> >> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
> >> ...
> >> NIP __might_sleep+0x20/0xc0
> >> LR __might_fault+0x40/0x60
> >> Call Trace:
> >> check_zeroed_user+0x12c/0x200
> >> test_user_copy_init+0x67c/0x1210 [test_user_copy]
> >> do_one_initcall+0x60/0x340
> >> do_init_module+0x7c/0x2f0
> >> load_module+0x2d94/0x30e0
> >> __do_sys_finit_module+0xc8/0x150
> >> system_call+0x5c/0x68
> >>
> >> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
> >> tweak it to only scan a 1024 byte region, but make it cross the
> >> page boundary.
> >>
> >> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
> >> Suggested-by: Aleksa Sarai
> >> Signed-off-by: Michael Ellerman
> >
> > With Aleksa's Reviewed-by I've picked this up:
> > https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=copy_struct_from_user
>
> Thanks. Are you planning to send that to Linus for v5.4 or v5.5 ?
This looks like a pretty straight bugfix to me since it's clearly
causing an issue for you on power so v5.4-rc4 is what I'd aim for. I
just want it to be in linux-next until tomorrow.
Christian
Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
Christian Brauner writes:
> On Wed, Oct 16, 2019 at 11:27:32PM +1100, Michael Ellerman wrote:
>> On a machine with a 64K PAGE_SIZE, the nested for loops in
>> test_check_nonzero_user() can lead to soft lockups, eg:
>>
>> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
>> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
>> virtio_balloon ip_tables x_tables autofs4
>> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
>> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
>> ...
>> NIP __might_sleep+0x20/0xc0
>> LR __might_fault+0x40/0x60
>> Call Trace:
>> check_zeroed_user+0x12c/0x200
>> test_user_copy_init+0x67c/0x1210 [test_user_copy]
>> do_one_initcall+0x60/0x340
>> do_init_module+0x7c/0x2f0
>> load_module+0x2d94/0x30e0
>> __do_sys_finit_module+0xc8/0x150
>> system_call+0x5c/0x68
>>
>> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
>> tweak it to only scan a 1024 byte region, but make it cross the
>> page boundary.
>>
>> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
>> Suggested-by: Aleksa Sarai
>> Signed-off-by: Michael Ellerman
>
> With Aleksa's Reviewed-by I've picked this up:
> https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=copy_struct_from_user
Thanks. Are you planning to send that to Linus for v5.4 or v5.5 ?
cheers
Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
On Wed, Oct 16, 2019 at 11:27:32PM +1100, Michael Ellerman wrote:
> On a machine with a 64K PAGE_SIZE, the nested for loops in
> test_check_nonzero_user() can lead to soft lockups, eg:
>
> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
> virtio_balloon ip_tables x_tables autofs4
> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
> ...
> NIP __might_sleep+0x20/0xc0
> LR __might_fault+0x40/0x60
> Call Trace:
> check_zeroed_user+0x12c/0x200
> test_user_copy_init+0x67c/0x1210 [test_user_copy]
> do_one_initcall+0x60/0x340
> do_init_module+0x7c/0x2f0
> load_module+0x2d94/0x30e0
> __do_sys_finit_module+0xc8/0x150
> system_call+0x5c/0x68
>
> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
> tweak it to only scan a 1024 byte region, but make it cross the
> page boundary.
>
> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
> Suggested-by: Aleksa Sarai
> Signed-off-by: Michael Ellerman
With Aleksa's Reviewed-by I've picked this up:
https://git.kernel.org/pub/scm/linux/kernel/git/brauner/linux.git/log/?h=copy_struct_from_user
Thanks!
Christian
Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
On Wed, Oct 16, 2019 at 11:27:32PM +1100, Michael Ellerman wrote:
> On a machine with a 64K PAGE_SIZE, the nested for loops in
> test_check_nonzero_user() can lead to soft lockups, eg:
>
> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
> virtio_balloon ip_tables x_tables autofs4
> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
> ...
> NIP __might_sleep+0x20/0xc0
> LR __might_fault+0x40/0x60
> Call Trace:
> check_zeroed_user+0x12c/0x200
> test_user_copy_init+0x67c/0x1210 [test_user_copy]
> do_one_initcall+0x60/0x340
> do_init_module+0x7c/0x2f0
> load_module+0x2d94/0x30e0
> __do_sys_finit_module+0xc8/0x150
> system_call+0x5c/0x68
>
> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
> tweak it to only scan a 1024 byte region, but make it cross the
> page boundary.
>
> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
> Suggested-by: Aleksa Sarai
> Signed-off-by: Michael Ellerman
Thanks!
Acked-by: Christian Brauner
Re: [PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
On 2019-10-16, Michael Ellerman wrote:
> On a machine with a 64K PAGE_SIZE, the nested for loops in
> test_check_nonzero_user() can lead to soft lockups, eg:
>
> watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
> Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
> virtio_balloon ip_tables x_tables autofs4
> CPU: 4 PID: 611 Comm: modprobe Tainted: G L
> 5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
> ...
> NIP __might_sleep+0x20/0xc0
> LR __might_fault+0x40/0x60
> Call Trace:
> check_zeroed_user+0x12c/0x200
> test_user_copy_init+0x67c/0x1210 [test_user_copy]
> do_one_initcall+0x60/0x340
> do_init_module+0x7c/0x2f0
> load_module+0x2d94/0x30e0
> __do_sys_finit_module+0xc8/0x150
> system_call+0x5c/0x68
>
> Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
> tweak it to only scan a 1024 byte region, but make it cross the
> page boundary.
>
> Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
> Suggested-by: Aleksa Sarai
> Signed-off-by: Michael Ellerman
Thanks Michael.
Reviewed-by: Aleksa Sarai
> ---
> lib/test_user_copy.c | 22 +++---
> 1 file changed, 19 insertions(+), 3 deletions(-)
>
> v2: Rework calculation to just use PAGE_SIZE directly.
> Rebase onto Christian's tree.
>
> diff --git a/lib/test_user_copy.c b/lib/test_user_copy.c
> index ad2372727b1b..5ff04d8fe971 100644
> --- a/lib/test_user_copy.c
> +++ b/lib/test_user_copy.c
> @@ -47,9 +47,25 @@ static bool is_zeroed(void *from, size_t size)
> static int test_check_nonzero_user(char *kmem, char __user *umem, size_t
> size)
> {
> int ret = 0;
> - size_t start, end, i;
> - size_t zero_start = size / 4;
> - size_t zero_end = size - zero_start;
> + size_t start, end, i, zero_start, zero_end;
> +
> + if (test(size < 2 * PAGE_SIZE, "buffer too small"))
> + return -EINVAL;
> +
> + /*
> + * We want to cross a page boundary to exercise the code more
> + * effectively. We also don't want to make the size we scan too large,
> + * otherwise the test can take a long time and cause soft lockups. So
> + * scan a 1024 byte region across the page boundary.
> + */
> + size = 1024;
> + start = PAGE_SIZE - (size / 2);
> +
> + kmem += start;
> + umem += start;
> +
> + zero_start = size / 4;
> + zero_end = size - zero_start;
>
> /*
>* We conduct a series of check_nonzero_user() tests on a block of
> --
> 2.21.0
>
--
Aleksa Sarai
Senior Software Engineer (Containers)
SUSE Linux GmbH
[PATCH v2] usercopy: Avoid soft lockups in test_check_nonzero_user()
On a machine with a 64K PAGE_SIZE, the nested for loops in
test_check_nonzero_user() can lead to soft lockups, eg:
watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [modprobe:611]
Modules linked in: test_user_copy(+) vmx_crypto gf128mul crc32c_vpmsum
virtio_balloon ip_tables x_tables autofs4
CPU: 4 PID: 611 Comm: modprobe Tainted: G L
5.4.0-rc1-gcc-8.2.0-1-gf5a1a536fa14-dirty #1151
...
NIP __might_sleep+0x20/0xc0
LR __might_fault+0x40/0x60
Call Trace:
check_zeroed_user+0x12c/0x200
test_user_copy_init+0x67c/0x1210 [test_user_copy]
do_one_initcall+0x60/0x340
do_init_module+0x7c/0x2f0
load_module+0x2d94/0x30e0
__do_sys_finit_module+0xc8/0x150
system_call+0x5c/0x68
Even with a 4K PAGE_SIZE the test takes multiple seconds. Instead
tweak it to only scan a 1024 byte region, but make it cross the
page boundary.
Fixes: f5a1a536fa14 ("lib: introduce copy_struct_from_user() helper")
Suggested-by: Aleksa Sarai
Signed-off-by: Michael Ellerman
---
lib/test_user_copy.c | 22 +++---
1 file changed, 19 insertions(+), 3 deletions(-)
v2: Rework calculation to just use PAGE_SIZE directly.
Rebase onto Christian's tree.
diff --git a/lib/test_user_copy.c b/lib/test_user_copy.c
index ad2372727b1b..5ff04d8fe971 100644
--- a/lib/test_user_copy.c
+++ b/lib/test_user_copy.c
@@ -47,9 +47,25 @@ static bool is_zeroed(void *from, size_t size)
static int test_check_nonzero_user(char *kmem, char __user *umem, size_t size)
{
int ret = 0;
- size_t start, end, i;
- size_t zero_start = size / 4;
- size_t zero_end = size - zero_start;
+ size_t start, end, i, zero_start, zero_end;
+
+ if (test(size < 2 * PAGE_SIZE, "buffer too small"))
+ return -EINVAL;
+
+ /*
+* We want to cross a page boundary to exercise the code more
+* effectively. We also don't want to make the size we scan too large,
+* otherwise the test can take a long time and cause soft lockups. So
+* scan a 1024 byte region across the page boundary.
+*/
+ size = 1024;
+ start = PAGE_SIZE - (size / 2);
+
+ kmem += start;
+ umem += start;
+
+ zero_start = size / 4;
+ zero_end = size - zero_start;
/*
* We conduct a series of check_nonzero_user() tests on a block of
--
2.21.0
