Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 10:32 -0600, Jens Axboe wrote: > > Sure, but s/get_cpu_var/get_cpu_ptr first. > > Doh. Ditto, box OTOH spotted the booboo instantly :) -Mike
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 10:32 -0600, Jens Axboe wrote: > > Sure, but s/get_cpu_var/get_cpu_ptr first. > > Doh. Ditto, box OTOH spotted the booboo instantly :) -Mike
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 10:26 AM, Mike Galbraith wrote:
> On Tue, 2017-05-09 at 09:45 -0600, Jens Axboe wrote:
>
>>> Is it from this_cpu_ptr() in blk_stat_add()?
>>
>> Yeah.
>
> So why is this complaining, doesn't rcu_read_lock() disable
> preemption?
Ah, I guess it doesn't if PREEMPT_RCU is set. How about the
below?
>>>
>>> Should do it. I was about to run LTP (where it turned up) again
>>> anyway, I'll add this. No news is good news, what you should hear.
>>
>> Thanks, let me know, so I can add your tested-by (or whatever you
>> prefer).
>
> Sure, but s/get_cpu_var/get_cpu_ptr first.
Doh.
diff --git a/block/blk-stat.c b/block/blk-stat.c
index 6c2f40940439..c52356d90fe3 100644
--- a/block/blk-stat.c
+++ b/block/blk-stat.c
@@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
rcu_read_lock();
list_for_each_entry_rcu(cb, >stats->callbacks, list) {
- if (blk_stat_is_active(cb)) {
- bucket = cb->bucket_fn(rq);
- if (bucket < 0)
- continue;
- stat = _cpu_ptr(cb->cpu_stat)[bucket];
- __blk_stat_add(stat, value);
- }
+ if (!blk_stat_is_active(cb))
+ continue;
+
+ bucket = cb->bucket_fn(rq);
+ if (bucket < 0)
+ continue;
+
+ stat = _cpu_ptr(cb->cpu_stat)[bucket];
+ __blk_stat_add(stat, value);
+ put_cpu_ptr(cb->cpu_stat);
}
rcu_read_unlock();
}
--
Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 10:26 AM, Mike Galbraith wrote:
> On Tue, 2017-05-09 at 09:45 -0600, Jens Axboe wrote:
>
>>> Is it from this_cpu_ptr() in blk_stat_add()?
>>
>> Yeah.
>
> So why is this complaining, doesn't rcu_read_lock() disable
> preemption?
Ah, I guess it doesn't if PREEMPT_RCU is set. How about the
below?
>>>
>>> Should do it. I was about to run LTP (where it turned up) again
>>> anyway, I'll add this. No news is good news, what you should hear.
>>
>> Thanks, let me know, so I can add your tested-by (or whatever you
>> prefer).
>
> Sure, but s/get_cpu_var/get_cpu_ptr first.
Doh.
diff --git a/block/blk-stat.c b/block/blk-stat.c
index 6c2f40940439..c52356d90fe3 100644
--- a/block/blk-stat.c
+++ b/block/blk-stat.c
@@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
rcu_read_lock();
list_for_each_entry_rcu(cb, >stats->callbacks, list) {
- if (blk_stat_is_active(cb)) {
- bucket = cb->bucket_fn(rq);
- if (bucket < 0)
- continue;
- stat = _cpu_ptr(cb->cpu_stat)[bucket];
- __blk_stat_add(stat, value);
- }
+ if (!blk_stat_is_active(cb))
+ continue;
+
+ bucket = cb->bucket_fn(rq);
+ if (bucket < 0)
+ continue;
+
+ stat = _cpu_ptr(cb->cpu_stat)[bucket];
+ __blk_stat_add(stat, value);
+ put_cpu_ptr(cb->cpu_stat);
}
rcu_read_unlock();
}
--
Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 09:45 -0600, Jens Axboe wrote: > > > > > > Is it from this_cpu_ptr() in blk_stat_add()? > > > > > > > > > > Yeah. > > > > > > > > So why is this complaining, doesn't rcu_read_lock() disable > > > > preemption? > > > > > > Ah, I guess it doesn't if PREEMPT_RCU is set. How about the > > > below? > > > > Should do it. I was about to run LTP (where it turned up) again > > anyway, I'll add this. No news is good news, what you should hear. > > Thanks, let me know, so I can add your tested-by (or whatever you > prefer). Sure, but s/get_cpu_var/get_cpu_ptr first. -MIke
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 09:45 -0600, Jens Axboe wrote: > > > > > > Is it from this_cpu_ptr() in blk_stat_add()? > > > > > > > > > > Yeah. > > > > > > > > So why is this complaining, doesn't rcu_read_lock() disable > > > > preemption? > > > > > > Ah, I guess it doesn't if PREEMPT_RCU is set. How about the > > > below? > > > > Should do it. I was about to run LTP (where it turned up) again > > anyway, I'll add this. No news is good news, what you should hear. > > Thanks, let me know, so I can add your tested-by (or whatever you > prefer). Sure, but s/get_cpu_var/get_cpu_ptr first. -MIke
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:40 AM, Mike Galbraith wrote: > On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote: >> On 05/09/2017 09:13 AM, Jens Axboe wrote: >>> On 05/09/2017 09:04 AM, Mike Galbraith wrote: On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: > On 05/09/2017 12:07 AM, Mike Galbraith wrote: >> Hi Jens, >> >> I was about to fix up this splat.. >> >> [ 445.022141] loop: module loaded >> [ 445.078116] BUG: using smp_processor_id() in preemptible >> [] code: loop0/3801 >> [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 >> [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G >> E 4.12.0-default #40 >> [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G] >> -/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010 >> [ 445.108564] Call Trace: >> [ 445.111016] dump_stack+0x65/0x89 >> [ 445.114330] check_preemption_disabled+0xde/0xf0 >> [ 445.118945] debug_smp_processor_id+0x17/0x20 >> [ 445.123300] blk_stat_add+0xb0/0x130 >> [ 445.126876] __blk_mq_complete_request+0xb5/0x150 >> [ 445.131575] blk_mq_complete_request+0x16/0x20 >> [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] >> [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 >> [ 445.144816] ? finish_task_switch+0x85/0x270 >> [ 445.149085] ? __schedule+0x291/0x8c0 >> [ 445.152747] kthread_worker_fn+0xc2/0x1d0 >> [ 445.156754] kthread+0x114/0x150 >> [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 >> [ 445.164424] ? kthread_park+0x60/0x60 >> [ 445.168085] ret_from_fork+0x2c/0x40 > > Is it from this_cpu_ptr() in blk_stat_add()? Yeah. >>> >>> So why is this complaining, doesn't rcu_read_lock() disable >>> preemption? >> >> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below? > > Should do it. I was about to run LTP (where it turned up) again > anyway, I'll add this. No news is good news, what you should hear. Thanks, let me know, so I can add your tested-by (or whatever you prefer). -- Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:40 AM, Mike Galbraith wrote: > On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote: >> On 05/09/2017 09:13 AM, Jens Axboe wrote: >>> On 05/09/2017 09:04 AM, Mike Galbraith wrote: On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: > On 05/09/2017 12:07 AM, Mike Galbraith wrote: >> Hi Jens, >> >> I was about to fix up this splat.. >> >> [ 445.022141] loop: module loaded >> [ 445.078116] BUG: using smp_processor_id() in preemptible >> [] code: loop0/3801 >> [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 >> [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G >> E 4.12.0-default #40 >> [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G] >> -/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010 >> [ 445.108564] Call Trace: >> [ 445.111016] dump_stack+0x65/0x89 >> [ 445.114330] check_preemption_disabled+0xde/0xf0 >> [ 445.118945] debug_smp_processor_id+0x17/0x20 >> [ 445.123300] blk_stat_add+0xb0/0x130 >> [ 445.126876] __blk_mq_complete_request+0xb5/0x150 >> [ 445.131575] blk_mq_complete_request+0x16/0x20 >> [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] >> [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 >> [ 445.144816] ? finish_task_switch+0x85/0x270 >> [ 445.149085] ? __schedule+0x291/0x8c0 >> [ 445.152747] kthread_worker_fn+0xc2/0x1d0 >> [ 445.156754] kthread+0x114/0x150 >> [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 >> [ 445.164424] ? kthread_park+0x60/0x60 >> [ 445.168085] ret_from_fork+0x2c/0x40 > > Is it from this_cpu_ptr() in blk_stat_add()? Yeah. >>> >>> So why is this complaining, doesn't rcu_read_lock() disable >>> preemption? >> >> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below? > > Should do it. I was about to run LTP (where it turned up) again > anyway, I'll add this. No news is good news, what you should hear. Thanks, let me know, so I can add your tested-by (or whatever you prefer). -- Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote:
> On 05/09/2017 09:13 AM, Jens Axboe wrote:
> > On 05/09/2017 09:04 AM, Mike Galbraith wrote:
> > > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
> > > > On 05/09/2017 12:07 AM, Mike Galbraith wrote:
> > > > > Hi Jens,
> > > > >
> > > > > I was about to fix up this splat..
> > > > >
> > > > > [ 445.022141] loop: module loaded
> > > > > [ 445.078116] BUG: using smp_processor_id() in preemptible
> > > > > [] code: loop0/3801
> > > > > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20
> > > > > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G
> > > > > E 4.12.0-default #40
> > > > > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]
> > > > > -/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010
> > > > > [ 445.108564] Call Trace:
> > > > > [ 445.111016] dump_stack+0x65/0x89
> > > > > [ 445.114330] check_preemption_disabled+0xde/0xf0
> > > > > [ 445.118945] debug_smp_processor_id+0x17/0x20
> > > > > [ 445.123300] blk_stat_add+0xb0/0x130
> > > > > [ 445.126876] __blk_mq_complete_request+0xb5/0x150
> > > > > [ 445.131575] blk_mq_complete_request+0x16/0x20
> > > > > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop]
> > > > > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40
> > > > > [ 445.144816] ? finish_task_switch+0x85/0x270
> > > > > [ 445.149085] ? __schedule+0x291/0x8c0
> > > > > [ 445.152747] kthread_worker_fn+0xc2/0x1d0
> > > > > [ 445.156754] kthread+0x114/0x150
> > > > > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0
> > > > > [ 445.164424] ? kthread_park+0x60/0x60
> > > > > [ 445.168085] ret_from_fork+0x2c/0x40
> > > >
> > > > Is it from this_cpu_ptr() in blk_stat_add()?
> > >
> > > Yeah.
> >
> > So why is this complaining, doesn't rcu_read_lock() disable
> > preemption?
>
> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?
Should do it. I was about to run LTP (where it turned up) again
anyway, I'll add this. No news is good news, what you should hear.
> diff --git a/block/blk-stat.c b/block/blk-stat.c
> index 6c2f40940439..022f5925a427 100644
> --- a/block/blk-stat.c
> +++ b/block/blk-stat.c
> @@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
>
> rcu_read_lock();
> list_for_each_entry_rcu(cb, >stats->callbacks, list) {
> - if (blk_stat_is_active(cb)) {
> - bucket = cb->bucket_fn(rq);
> - if (bucket < 0)
> - continue;
> - stat = _cpu_ptr(cb->cpu_stat)[bucket];
> - __blk_stat_add(stat, value);
> - }
> + if (!blk_stat_is_active(cb))
> + continue;
> +
> + bucket = cb->bucket_fn(rq);
> + if (bucket < 0)
> + continue;
> +
> + stat = _cpu_var(cb->cpu_stat)[bucket];
> + __blk_stat_add(stat, value);
> + put_cpu_var(cb->cpu_stat);
> }
> rcu_read_unlock();
> }
>
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 09:15 -0600, Jens Axboe wrote:
> On 05/09/2017 09:13 AM, Jens Axboe wrote:
> > On 05/09/2017 09:04 AM, Mike Galbraith wrote:
> > > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
> > > > On 05/09/2017 12:07 AM, Mike Galbraith wrote:
> > > > > Hi Jens,
> > > > >
> > > > > I was about to fix up this splat..
> > > > >
> > > > > [ 445.022141] loop: module loaded
> > > > > [ 445.078116] BUG: using smp_processor_id() in preemptible
> > > > > [] code: loop0/3801
> > > > > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20
> > > > > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: G
> > > > > E 4.12.0-default #40
> > > > > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]
> > > > > -/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010
> > > > > [ 445.108564] Call Trace:
> > > > > [ 445.111016] dump_stack+0x65/0x89
> > > > > [ 445.114330] check_preemption_disabled+0xde/0xf0
> > > > > [ 445.118945] debug_smp_processor_id+0x17/0x20
> > > > > [ 445.123300] blk_stat_add+0xb0/0x130
> > > > > [ 445.126876] __blk_mq_complete_request+0xb5/0x150
> > > > > [ 445.131575] blk_mq_complete_request+0x16/0x20
> > > > > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop]
> > > > > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40
> > > > > [ 445.144816] ? finish_task_switch+0x85/0x270
> > > > > [ 445.149085] ? __schedule+0x291/0x8c0
> > > > > [ 445.152747] kthread_worker_fn+0xc2/0x1d0
> > > > > [ 445.156754] kthread+0x114/0x150
> > > > > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0
> > > > > [ 445.164424] ? kthread_park+0x60/0x60
> > > > > [ 445.168085] ret_from_fork+0x2c/0x40
> > > >
> > > > Is it from this_cpu_ptr() in blk_stat_add()?
> > >
> > > Yeah.
> >
> > So why is this complaining, doesn't rcu_read_lock() disable
> > preemption?
>
> Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?
Should do it. I was about to run LTP (where it turned up) again
anyway, I'll add this. No news is good news, what you should hear.
> diff --git a/block/blk-stat.c b/block/blk-stat.c
> index 6c2f40940439..022f5925a427 100644
> --- a/block/blk-stat.c
> +++ b/block/blk-stat.c
> @@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
>
> rcu_read_lock();
> list_for_each_entry_rcu(cb, >stats->callbacks, list) {
> - if (blk_stat_is_active(cb)) {
> - bucket = cb->bucket_fn(rq);
> - if (bucket < 0)
> - continue;
> - stat = _cpu_ptr(cb->cpu_stat)[bucket];
> - __blk_stat_add(stat, value);
> - }
> + if (!blk_stat_is_active(cb))
> + continue;
> +
> + bucket = cb->bucket_fn(rq);
> + if (bucket < 0)
> + continue;
> +
> + stat = _cpu_var(cb->cpu_stat)[bucket];
> + __blk_stat_add(stat, value);
> + put_cpu_var(cb->cpu_stat);
> }
> rcu_read_unlock();
> }
>
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:13 AM, Jens Axboe wrote:
> On 05/09/2017 09:04 AM, Mike Galbraith wrote:
>> On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
>>> On 05/09/2017 12:07 AM, Mike Galbraith wrote:
Hi Jens,
I was about to fix up this splat..
[ 445.022141] loop: module loaded
[ 445.078116] BUG: using smp_processor_id() in preemptible []
code: loop0/3801
[ 445.085873] caller is debug_smp_processor_id+0x17/0x20
[ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE
4.12.0-default #40
[ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698
, BIOS -[D6E150AUS-1.10]- 12/15/2010
[ 445.108564] Call Trace:
[ 445.111016] dump_stack+0x65/0x89
[ 445.114330] check_preemption_disabled+0xde/0xf0
[ 445.118945] debug_smp_processor_id+0x17/0x20
[ 445.123300] blk_stat_add+0xb0/0x130
[ 445.126876] __blk_mq_complete_request+0xb5/0x150
[ 445.131575] blk_mq_complete_request+0x16/0x20
[ 445.136020] loop_queue_work+0x5f/0xaa0 [loop]
[ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40
[ 445.144816] ? finish_task_switch+0x85/0x270
[ 445.149085] ? __schedule+0x291/0x8c0
[ 445.152747] kthread_worker_fn+0xc2/0x1d0
[ 445.156754] kthread+0x114/0x150
[ 445.159983] ? __kthread_init_worker+0xb0/0xb0
[ 445.164424] ? kthread_park+0x60/0x60
[ 445.168085] ret_from_fork+0x2c/0x40
>>>
>>> Is it from this_cpu_ptr() in blk_stat_add()?
>>
>> Yeah.
>
> So why is this complaining, doesn't rcu_read_lock() disable
> preemption?
Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?
diff --git a/block/blk-stat.c b/block/blk-stat.c
index 6c2f40940439..022f5925a427 100644
--- a/block/blk-stat.c
+++ b/block/blk-stat.c
@@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
rcu_read_lock();
list_for_each_entry_rcu(cb, >stats->callbacks, list) {
- if (blk_stat_is_active(cb)) {
- bucket = cb->bucket_fn(rq);
- if (bucket < 0)
- continue;
- stat = _cpu_ptr(cb->cpu_stat)[bucket];
- __blk_stat_add(stat, value);
- }
+ if (!blk_stat_is_active(cb))
+ continue;
+
+ bucket = cb->bucket_fn(rq);
+ if (bucket < 0)
+ continue;
+
+ stat = _cpu_var(cb->cpu_stat)[bucket];
+ __blk_stat_add(stat, value);
+ put_cpu_var(cb->cpu_stat);
}
rcu_read_unlock();
}
--
Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:13 AM, Jens Axboe wrote:
> On 05/09/2017 09:04 AM, Mike Galbraith wrote:
>> On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote:
>>> On 05/09/2017 12:07 AM, Mike Galbraith wrote:
Hi Jens,
I was about to fix up this splat..
[ 445.022141] loop: module loaded
[ 445.078116] BUG: using smp_processor_id() in preemptible []
code: loop0/3801
[ 445.085873] caller is debug_smp_processor_id+0x17/0x20
[ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE
4.12.0-default #40
[ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698
, BIOS -[D6E150AUS-1.10]- 12/15/2010
[ 445.108564] Call Trace:
[ 445.111016] dump_stack+0x65/0x89
[ 445.114330] check_preemption_disabled+0xde/0xf0
[ 445.118945] debug_smp_processor_id+0x17/0x20
[ 445.123300] blk_stat_add+0xb0/0x130
[ 445.126876] __blk_mq_complete_request+0xb5/0x150
[ 445.131575] blk_mq_complete_request+0x16/0x20
[ 445.136020] loop_queue_work+0x5f/0xaa0 [loop]
[ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40
[ 445.144816] ? finish_task_switch+0x85/0x270
[ 445.149085] ? __schedule+0x291/0x8c0
[ 445.152747] kthread_worker_fn+0xc2/0x1d0
[ 445.156754] kthread+0x114/0x150
[ 445.159983] ? __kthread_init_worker+0xb0/0xb0
[ 445.164424] ? kthread_park+0x60/0x60
[ 445.168085] ret_from_fork+0x2c/0x40
>>>
>>> Is it from this_cpu_ptr() in blk_stat_add()?
>>
>> Yeah.
>
> So why is this complaining, doesn't rcu_read_lock() disable
> preemption?
Ah, I guess it doesn't if PREEMPT_RCU is set. How about the below?
diff --git a/block/blk-stat.c b/block/blk-stat.c
index 6c2f40940439..022f5925a427 100644
--- a/block/blk-stat.c
+++ b/block/blk-stat.c
@@ -96,13 +96,16 @@ void blk_stat_add(struct request *rq)
rcu_read_lock();
list_for_each_entry_rcu(cb, >stats->callbacks, list) {
- if (blk_stat_is_active(cb)) {
- bucket = cb->bucket_fn(rq);
- if (bucket < 0)
- continue;
- stat = _cpu_ptr(cb->cpu_stat)[bucket];
- __blk_stat_add(stat, value);
- }
+ if (!blk_stat_is_active(cb))
+ continue;
+
+ bucket = cb->bucket_fn(rq);
+ if (bucket < 0)
+ continue;
+
+ stat = _cpu_var(cb->cpu_stat)[bucket];
+ __blk_stat_add(stat, value);
+ put_cpu_var(cb->cpu_stat);
}
rcu_read_unlock();
}
--
Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:04 AM, Mike Galbraith wrote: > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: >> On 05/09/2017 12:07 AM, Mike Galbraith wrote: >>> Hi Jens, >>> >>> I was about to fix up this splat.. >>> >>> [ 445.022141] loop: module loaded >>> [ 445.078116] BUG: using smp_processor_id() in preemptible [] >>> code: loop0/3801 >>> [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 >>> [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE >>> 4.12.0-default #40 >>> [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , >>> BIOS -[D6E150AUS-1.10]- 12/15/2010 >>> [ 445.108564] Call Trace: >>> [ 445.111016] dump_stack+0x65/0x89 >>> [ 445.114330] check_preemption_disabled+0xde/0xf0 >>> [ 445.118945] debug_smp_processor_id+0x17/0x20 >>> [ 445.123300] blk_stat_add+0xb0/0x130 >>> [ 445.126876] __blk_mq_complete_request+0xb5/0x150 >>> [ 445.131575] blk_mq_complete_request+0x16/0x20 >>> [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] >>> [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 >>> [ 445.144816] ? finish_task_switch+0x85/0x270 >>> [ 445.149085] ? __schedule+0x291/0x8c0 >>> [ 445.152747] kthread_worker_fn+0xc2/0x1d0 >>> [ 445.156754] kthread+0x114/0x150 >>> [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 >>> [ 445.164424] ? kthread_park+0x60/0x60 >>> [ 445.168085] ret_from_fork+0x2c/0x40 >> >> Is it from this_cpu_ptr() in blk_stat_add()? > > Yeah. So why is this complaining, doesn't rcu_read_lock() disable preemption? -- Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 09:04 AM, Mike Galbraith wrote: > On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: >> On 05/09/2017 12:07 AM, Mike Galbraith wrote: >>> Hi Jens, >>> >>> I was about to fix up this splat.. >>> >>> [ 445.022141] loop: module loaded >>> [ 445.078116] BUG: using smp_processor_id() in preemptible [] >>> code: loop0/3801 >>> [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 >>> [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE >>> 4.12.0-default #40 >>> [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , >>> BIOS -[D6E150AUS-1.10]- 12/15/2010 >>> [ 445.108564] Call Trace: >>> [ 445.111016] dump_stack+0x65/0x89 >>> [ 445.114330] check_preemption_disabled+0xde/0xf0 >>> [ 445.118945] debug_smp_processor_id+0x17/0x20 >>> [ 445.123300] blk_stat_add+0xb0/0x130 >>> [ 445.126876] __blk_mq_complete_request+0xb5/0x150 >>> [ 445.131575] blk_mq_complete_request+0x16/0x20 >>> [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] >>> [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 >>> [ 445.144816] ? finish_task_switch+0x85/0x270 >>> [ 445.149085] ? __schedule+0x291/0x8c0 >>> [ 445.152747] kthread_worker_fn+0xc2/0x1d0 >>> [ 445.156754] kthread+0x114/0x150 >>> [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 >>> [ 445.164424] ? kthread_park+0x60/0x60 >>> [ 445.168085] ret_from_fork+0x2c/0x40 >> >> Is it from this_cpu_ptr() in blk_stat_add()? > > Yeah. So why is this complaining, doesn't rcu_read_lock() disable preemption? -- Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: > On 05/09/2017 12:07 AM, Mike Galbraith wrote: > > Hi Jens, > > > > I was about to fix up this splat.. > > > > [ 445.022141] loop: module loaded > > [ 445.078116] BUG: using smp_processor_id() in preemptible [] > > code: loop0/3801 > > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 > > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE > > 4.12.0-default #40 > > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , > > BIOS -[D6E150AUS-1.10]- 12/15/2010 > > [ 445.108564] Call Trace: > > [ 445.111016] dump_stack+0x65/0x89 > > [ 445.114330] check_preemption_disabled+0xde/0xf0 > > [ 445.118945] debug_smp_processor_id+0x17/0x20 > > [ 445.123300] blk_stat_add+0xb0/0x130 > > [ 445.126876] __blk_mq_complete_request+0xb5/0x150 > > [ 445.131575] blk_mq_complete_request+0x16/0x20 > > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] > > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 > > [ 445.144816] ? finish_task_switch+0x85/0x270 > > [ 445.149085] ? __schedule+0x291/0x8c0 > > [ 445.152747] kthread_worker_fn+0xc2/0x1d0 > > [ 445.156754] kthread+0x114/0x150 > > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 > > [ 445.164424] ? kthread_park+0x60/0x60 > > [ 445.168085] ret_from_fork+0x2c/0x40 > > Is it from this_cpu_ptr() in blk_stat_add()? Yeah. > > ..but then noticed that __blk_mq_run_hw_queue() is also called under > > get_cpu(), but recently grew a might_sleep(), so maybe wants some block > > layer eyeballs. > > That part is fine. The might sleep depends on BLOCKING being set in > the hardware queue flags. And if that is set, then we don't call > it under get_cpu(). Ah, right. -Mike
Re: get/put_cpu() usage in block/blk-mq.c
On Tue, 2017-05-09 at 08:53 -0600, Jens Axboe wrote: > On 05/09/2017 12:07 AM, Mike Galbraith wrote: > > Hi Jens, > > > > I was about to fix up this splat.. > > > > [ 445.022141] loop: module loaded > > [ 445.078116] BUG: using smp_processor_id() in preemptible [] > > code: loop0/3801 > > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 > > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE > > 4.12.0-default #40 > > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , > > BIOS -[D6E150AUS-1.10]- 12/15/2010 > > [ 445.108564] Call Trace: > > [ 445.111016] dump_stack+0x65/0x89 > > [ 445.114330] check_preemption_disabled+0xde/0xf0 > > [ 445.118945] debug_smp_processor_id+0x17/0x20 > > [ 445.123300] blk_stat_add+0xb0/0x130 > > [ 445.126876] __blk_mq_complete_request+0xb5/0x150 > > [ 445.131575] blk_mq_complete_request+0x16/0x20 > > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] > > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 > > [ 445.144816] ? finish_task_switch+0x85/0x270 > > [ 445.149085] ? __schedule+0x291/0x8c0 > > [ 445.152747] kthread_worker_fn+0xc2/0x1d0 > > [ 445.156754] kthread+0x114/0x150 > > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 > > [ 445.164424] ? kthread_park+0x60/0x60 > > [ 445.168085] ret_from_fork+0x2c/0x40 > > Is it from this_cpu_ptr() in blk_stat_add()? Yeah. > > ..but then noticed that __blk_mq_run_hw_queue() is also called under > > get_cpu(), but recently grew a might_sleep(), so maybe wants some block > > layer eyeballs. > > That part is fine. The might sleep depends on BLOCKING being set in > the hardware queue flags. And if that is set, then we don't call > it under get_cpu(). Ah, right. -Mike
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 12:07 AM, Mike Galbraith wrote: > Hi Jens, > > I was about to fix up this splat.. > > [ 445.022141] loop: module loaded > [ 445.078116] BUG: using smp_processor_id() in preemptible [] code: > loop0/3801 > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE > 4.12.0-default #40 > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , > BIOS -[D6E150AUS-1.10]- 12/15/2010 > [ 445.108564] Call Trace: > [ 445.111016] dump_stack+0x65/0x89 > [ 445.114330] check_preemption_disabled+0xde/0xf0 > [ 445.118945] debug_smp_processor_id+0x17/0x20 > [ 445.123300] blk_stat_add+0xb0/0x130 > [ 445.126876] __blk_mq_complete_request+0xb5/0x150 > [ 445.131575] blk_mq_complete_request+0x16/0x20 > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 > [ 445.144816] ? finish_task_switch+0x85/0x270 > [ 445.149085] ? __schedule+0x291/0x8c0 > [ 445.152747] kthread_worker_fn+0xc2/0x1d0 > [ 445.156754] kthread+0x114/0x150 > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 > [ 445.164424] ? kthread_park+0x60/0x60 > [ 445.168085] ret_from_fork+0x2c/0x40 Is it from this_cpu_ptr() in blk_stat_add()? > ..but then noticed that __blk_mq_run_hw_queue() is also called under > get_cpu(), but recently grew a might_sleep(), so maybe wants some block > layer eyeballs. That part is fine. The might sleep depends on BLOCKING being set in the hardware queue flags. And if that is set, then we don't call it under get_cpu(). -- Jens Axboe
Re: get/put_cpu() usage in block/blk-mq.c
On 05/09/2017 12:07 AM, Mike Galbraith wrote: > Hi Jens, > > I was about to fix up this splat.. > > [ 445.022141] loop: module loaded > [ 445.078116] BUG: using smp_processor_id() in preemptible [] code: > loop0/3801 > [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 > [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE > 4.12.0-default #40 > [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , > BIOS -[D6E150AUS-1.10]- 12/15/2010 > [ 445.108564] Call Trace: > [ 445.111016] dump_stack+0x65/0x89 > [ 445.114330] check_preemption_disabled+0xde/0xf0 > [ 445.118945] debug_smp_processor_id+0x17/0x20 > [ 445.123300] blk_stat_add+0xb0/0x130 > [ 445.126876] __blk_mq_complete_request+0xb5/0x150 > [ 445.131575] blk_mq_complete_request+0x16/0x20 > [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] > [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 > [ 445.144816] ? finish_task_switch+0x85/0x270 > [ 445.149085] ? __schedule+0x291/0x8c0 > [ 445.152747] kthread_worker_fn+0xc2/0x1d0 > [ 445.156754] kthread+0x114/0x150 > [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 > [ 445.164424] ? kthread_park+0x60/0x60 > [ 445.168085] ret_from_fork+0x2c/0x40 Is it from this_cpu_ptr() in blk_stat_add()? > ..but then noticed that __blk_mq_run_hw_queue() is also called under > get_cpu(), but recently grew a might_sleep(), so maybe wants some block > layer eyeballs. That part is fine. The might sleep depends on BLOCKING being set in the hardware queue flags. And if that is set, then we don't call it under get_cpu(). -- Jens Axboe
get/put_cpu() usage in block/blk-mq.c
Hi Jens, I was about to fix up this splat.. [ 445.022141] loop: module loaded [ 445.078116] BUG: using smp_processor_id() in preemptible [] code: loop0/3801 [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE 4.12.0-default #40 [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010 [ 445.108564] Call Trace: [ 445.111016] dump_stack+0x65/0x89 [ 445.114330] check_preemption_disabled+0xde/0xf0 [ 445.118945] debug_smp_processor_id+0x17/0x20 [ 445.123300] blk_stat_add+0xb0/0x130 [ 445.126876] __blk_mq_complete_request+0xb5/0x150 [ 445.131575] blk_mq_complete_request+0x16/0x20 [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 [ 445.144816] ? finish_task_switch+0x85/0x270 [ 445.149085] ? __schedule+0x291/0x8c0 [ 445.152747] kthread_worker_fn+0xc2/0x1d0 [ 445.156754] kthread+0x114/0x150 [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 [ 445.164424] ? kthread_park+0x60/0x60 [ 445.168085] ret_from_fork+0x2c/0x40 ..but then noticed that __blk_mq_run_hw_queue() is also called under get_cpu(), but recently grew a might_sleep(), so maybe wants some block layer eyeballs. -Mike
get/put_cpu() usage in block/blk-mq.c
Hi Jens, I was about to fix up this splat.. [ 445.022141] loop: module loaded [ 445.078116] BUG: using smp_processor_id() in preemptible [] code: loop0/3801 [ 445.085873] caller is debug_smp_processor_id+0x17/0x20 [ 445.091016] CPU: 7 PID: 3801 Comm: loop0 Tainted: GE 4.12.0-default #40 [ 445.098838] Hardware name: IBM System x3550 M3 -[7944K3G]-/69Y5698 , BIOS -[D6E150AUS-1.10]- 12/15/2010 [ 445.108564] Call Trace: [ 445.111016] dump_stack+0x65/0x89 [ 445.114330] check_preemption_disabled+0xde/0xf0 [ 445.118945] debug_smp_processor_id+0x17/0x20 [ 445.123300] blk_stat_add+0xb0/0x130 [ 445.126876] __blk_mq_complete_request+0xb5/0x150 [ 445.131575] blk_mq_complete_request+0x16/0x20 [ 445.136020] loop_queue_work+0x5f/0xaa0 [loop] [ 445.140461] ? _raw_spin_unlock_irq+0x21/0x40 [ 445.144816] ? finish_task_switch+0x85/0x270 [ 445.149085] ? __schedule+0x291/0x8c0 [ 445.152747] kthread_worker_fn+0xc2/0x1d0 [ 445.156754] kthread+0x114/0x150 [ 445.159983] ? __kthread_init_worker+0xb0/0xb0 [ 445.164424] ? kthread_park+0x60/0x60 [ 445.168085] ret_from_fork+0x2c/0x40 ..but then noticed that __blk_mq_run_hw_queue() is also called under get_cpu(), but recently grew a might_sleep(), so maybe wants some block layer eyeballs. -Mike
