On Thu, Mar 07, 2024 at 01:52:15PM -0500, Steven Rostedt wrote:
> On Thu, 7 Mar 2024 16:57:33 +
> Mark Rutland wrote:
>
> > * Use rcu_tasks_trace to synchronize updates?
>
> Yes. I think I wanted both. The above to make sure it covers all cases
> where something could be preempted, and a
On Thu, 7 Mar 2024 16:57:33 +
Mark Rutland wrote:
> * Use rcu_tasks_trace to synchronize updates?
Yes. I think I wanted both. The above to make sure it covers all cases
where something could be preempted, and a case for those covered when RCU
isn't watching (which always has preemption
On Thu, Mar 07, 2024 at 04:57:33PM +, Mark Rutland wrote:
> On Mon, Mar 04, 2024 at 04:16:01AM -0500, Joel Fernandes wrote:
> > On 3/2/2024 8:01 PM, Joel Fernandes wrote:
> > Case 1: For !CONFIG_DYNAMIC_FTRACE update of ftrace_trace_function
> >
> > This config is itself expected to be slow.
Hi Joel, Paul, Steve,
On Mon, Mar 04, 2024 at 04:16:01AM -0500, Joel Fernandes wrote:
> Hi Paul,
>
> On 3/2/2024 8:01 PM, Joel Fernandes wrote:
> >> As you noted, one thing that Ankur's series changes is that preemption
> >> can occur anywhere that it is not specifically disabled in kernels
> >>
On Tue, 5 Mar 2024 09:53:42 -0800
"Paul E. McKenney" wrote:
> > Case 1: For !CONFIG_DYNAMIC_FTRACE update of ftrace_trace_function
> >
> > This config is itself expected to be slow. However seeing what it does, it
> > is
> > trying to make sure the global function pointer
On Tue, Mar 05, 2024 at 07:57:40PM +, Mark Rutland wrote:
> On Tue, Mar 05, 2024 at 09:53:42AM -0800, Paul E. McKenney wrote:
> > On Mon, Mar 04, 2024 at 04:16:01AM -0500, Joel Fernandes wrote:
> > > Hi Paul,
> >
> > Thank you, Joel!
> >
> > > On 3/2/2024 8:01 PM, Joel Fernandes wrote:
> > >
On Tue, Mar 05, 2024 at 09:53:42AM -0800, Paul E. McKenney wrote:
> On Mon, Mar 04, 2024 at 04:16:01AM -0500, Joel Fernandes wrote:
> > Hi Paul,
>
> Thank you, Joel!
>
> > On 3/2/2024 8:01 PM, Joel Fernandes wrote:
> > >> As you noted, one thing that Ankur's series changes is that preemption
> >
On Mon, Mar 04, 2024 at 04:16:01AM -0500, Joel Fernandes wrote:
> Hi Paul,
Thank you, Joel!
> On 3/2/2024 8:01 PM, Joel Fernandes wrote:
> >> As you noted, one thing that Ankur's series changes is that preemption
> >> can occur anywhere that it is not specifically disabled in kernels
> >> built
Hi Paul,
On 3/2/2024 8:01 PM, Joel Fernandes wrote:
>> As you noted, one thing that Ankur's series changes is that preemption
>> can occur anywhere that it is not specifically disabled in kernels
>> built with CONFIG_PREEMPT_NONE=y or CONFIG_PREEMPT_VOLUNTARY=y. This in
>> turn changes Tasks
Hi Paul,
On 3/2/2024 8:01 PM, Joel Fernandes wrote:
>> As you noted, one thing that Ankur's series changes is that preemption
>> can occur anywhere that it is not specifically disabled in kernels
>> built with CONFIG_PREEMPT_NONE=y or CONFIG_PREEMPT_VOLUNTARY=y. This in
>> turn changes Tasks
On Sat, Mar 2, 2024 at 7:25 PM Paul E. McKenney wrote:
>
> On Fri, Mar 01, 2024 at 09:24:15PM -0500, Joel Fernandes wrote:
> > (Shrinking CC a bit)
> >
> > On Thu, Feb 29, 2024 at 1:29 PM Paul E. McKenney wrote:
> > >
> > > On Thu, Feb 29, 2024 at 12:41:55PM -0500, Joel Fernandes wrote:
> > > >
On Fri, Mar 01, 2024 at 09:24:15PM -0500, Joel Fernandes wrote:
> (Shrinking CC a bit)
>
> On Thu, Feb 29, 2024 at 1:29 PM Paul E. McKenney wrote:
> >
> > On Thu, Feb 29, 2024 at 12:41:55PM -0500, Joel Fernandes wrote:
> > > > On Feb 29, 2024, at 11:57 AM, Paul E. McKenney
> > > > wrote:
> > >
(Shrinking CC a bit)
On Thu, Feb 29, 2024 at 1:29 PM Paul E. McKenney wrote:
>
> On Thu, Feb 29, 2024 at 12:41:55PM -0500, Joel Fernandes wrote:
> > > On Feb 29, 2024, at 11:57 AM, Paul E. McKenney wrote:
> > > On Thu, Feb 29, 2024 at 09:21:48AM -0500, Joel Fernandes wrote:
> > >>> On
On Thu, Feb 29, 2024 at 12:41:55PM -0500, Joel Fernandes wrote:
> > On Feb 29, 2024, at 11:57 AM, Paul E. McKenney wrote:
> > On Thu, Feb 29, 2024 at 09:21:48AM -0500, Joel Fernandes wrote:
> >>> On 2/28/2024 5:58 PM, Paul E. McKenney wrote:
> >>> On Wed, Feb 28, 2024 at 02:48:44PM -0800, Alexei
> On Feb 29, 2024, at 11:57 AM, Paul E. McKenney wrote:
>
> On Thu, Feb 29, 2024 at 09:21:48AM -0500, Joel Fernandes wrote:
>>
>>
>>> On 2/28/2024 5:58 PM, Paul E. McKenney wrote:
>>> On Wed, Feb 28, 2024 at 02:48:44PM -0800, Alexei Starovoitov wrote:
On Wed, Feb 28, 2024 at 2:31 PM
On Thu, Feb 29, 2024 at 09:21:48AM -0500, Joel Fernandes wrote:
>
>
> On 2/28/2024 5:58 PM, Paul E. McKenney wrote:
> > On Wed, Feb 28, 2024 at 02:48:44PM -0800, Alexei Starovoitov wrote:
> >> On Wed, Feb 28, 2024 at 2:31 PM Steven Rostedt wrote:
> >>>
> >>> On Wed, 28 Feb 2024 14:19:11 -0800
>
On 2/28/2024 5:58 PM, Paul E. McKenney wrote:
> On Wed, Feb 28, 2024 at 02:48:44PM -0800, Alexei Starovoitov wrote:
>> On Wed, Feb 28, 2024 at 2:31 PM Steven Rostedt wrote:
>>>
>>> On Wed, 28 Feb 2024 14:19:11 -0800
>>> "Paul E. McKenney" wrote:
>>>
>>
>> Well, to your initial point,
Hi Eric,
On Tue, Feb 27, 2024 at 10:44 AM Eric Dumazet wrote:
>
> Hmm
> Why napi_busy_loop() does not have a similar problem ?
>
I just tried and can reproduce similar behavior on sk busy poll.
However, the interesting thing is, this can happen if I set a super
high polling interval but just
On Wed, Feb 28, 2024 at 02:48:44PM -0800, Alexei Starovoitov wrote:
> On Wed, Feb 28, 2024 at 2:31 PM Steven Rostedt wrote:
> >
> > On Wed, 28 Feb 2024 14:19:11 -0800
> > "Paul E. McKenney" wrote:
> >
> > > > >
> > > > > Well, to your initial point, cond_resched() does eventually invoke
> > > >
On Wed, Feb 28, 2024 at 05:33:07PM -0500, Steven Rostedt wrote:
> On Wed, 28 Feb 2024 14:19:11 -0800
> "Paul E. McKenney" wrote:
>
> > > >
> > > > Well, to your initial point, cond_resched() does eventually invoke
> > > > preempt_schedule_common(), so you are quite correct that as far as
> > >
On Wed, Feb 28, 2024 at 2:31 PM Steven Rostedt wrote:
>
> On Wed, 28 Feb 2024 14:19:11 -0800
> "Paul E. McKenney" wrote:
>
> > > >
> > > > Well, to your initial point, cond_resched() does eventually invoke
> > > > preempt_schedule_common(), so you are quite correct that as far as
> > > > Tasks
On Wed, 28 Feb 2024 14:19:11 -0800
"Paul E. McKenney" wrote:
> > >
> > > Well, to your initial point, cond_resched() does eventually invoke
> > > preempt_schedule_common(), so you are quite correct that as far as
> > > Tasks RCU is concerned, cond_resched() is not a quiescent state.
> >
> >
On Wed, Feb 28, 2024 at 05:10:43PM -0500, Joel Fernandes wrote:
>
>
> > On Feb 28, 2024, at 4:52 PM, Paul E. McKenney wrote:
> >
> > On Wed, Feb 28, 2024 at 04:27:47PM -0500, Joel Fernandes wrote:
> >>
> >>
> On Feb 28, 2024, at 4:13 PM, Paul E. McKenney wrote:
> >>>
> >>> On Wed,
> On Feb 28, 2024, at 4:52 PM, Paul E. McKenney wrote:
>
> On Wed, Feb 28, 2024 at 04:27:47PM -0500, Joel Fernandes wrote:
>>
>>
On Feb 28, 2024, at 4:13 PM, Paul E. McKenney wrote:
>>>
>>> On Wed, Feb 28, 2024 at 03:14:34PM -0500, Joel Fernandes wrote:
> On Wed, Feb 28, 2024
On Wed, Feb 28, 2024 at 04:27:47PM -0500, Joel Fernandes wrote:
>
>
> > On Feb 28, 2024, at 4:13 PM, Paul E. McKenney wrote:
> >
> > On Wed, Feb 28, 2024 at 03:14:34PM -0500, Joel Fernandes wrote:
> >>> On Wed, Feb 28, 2024 at 12:18 PM Paul E. McKenney
> >>> wrote:
> >>>
> >>> On Wed, Feb
> On Feb 28, 2024, at 4:13 PM, Paul E. McKenney wrote:
>
> On Wed, Feb 28, 2024 at 03:14:34PM -0500, Joel Fernandes wrote:
>>> On Wed, Feb 28, 2024 at 12:18 PM Paul E. McKenney
>>> wrote:
>>>
>>> On Wed, Feb 28, 2024 at 10:37:51AM -0600, Yan Zhai wrote:
On Wed, Feb 28, 2024 at 9:37
On Wed, Feb 28, 2024 at 03:14:34PM -0500, Joel Fernandes wrote:
> On Wed, Feb 28, 2024 at 12:18 PM Paul E. McKenney wrote:
> >
> > On Wed, Feb 28, 2024 at 10:37:51AM -0600, Yan Zhai wrote:
> > > On Wed, Feb 28, 2024 at 9:37 AM Joel Fernandes
> > > wrote:
> > > > Also optionally, I wonder if
On Wed, Feb 28, 2024 at 12:18 PM Paul E. McKenney wrote:
>
> On Wed, Feb 28, 2024 at 10:37:51AM -0600, Yan Zhai wrote:
> > On Wed, Feb 28, 2024 at 9:37 AM Joel Fernandes
> > wrote:
> > > Also optionally, I wonder if calling rcu_tasks_qs() directly is better
> > > (for documentation if anything)
On Wed, Feb 28, 2024 at 09:48:42AM -0600, Yan Zhai wrote:
> On Wed, Feb 28, 2024 at 9:10 AM Paul E. McKenney wrote:
> >
> > On Wed, Feb 28, 2024 at 12:50:53PM +0100, Toke Høiland-Jørgensen wrote:
> > > "Paul E. McKenney" writes:
> > >
> > > > On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric
On Wed, Feb 28, 2024 at 10:37:51AM -0600, Yan Zhai wrote:
> On Wed, Feb 28, 2024 at 9:37 AM Joel Fernandes wrote:
> > Also optionally, I wonder if calling rcu_tasks_qs() directly is better
> > (for documentation if anything) since the issue is Tasks RCU specific. Also
> > code comment above the
On Wed, Feb 28, 2024 at 9:37 AM Joel Fernandes wrote:
> Also optionally, I wonder if calling rcu_tasks_qs() directly is better
> (for documentation if anything) since the issue is Tasks RCU specific. Also
> code comment above the rcu_softirq_qs() call about cond_resched() not taking
> care of
On Wed, Feb 28, 2024 at 9:35 AM Jakub Kicinski wrote:
>
> On Wed, 28 Feb 2024 07:15:42 -0800 Paul E. McKenney wrote:
> > > > Another complication is that although CONFIG_PREEMPT_RT kernels are
> > > > built with CONFIG_PREEMPT_RCU, the reverse is not always the case.
> > > > And if we are not
On Wed, Feb 28, 2024 at 9:10 AM Paul E. McKenney wrote:
>
> On Wed, Feb 28, 2024 at 12:50:53PM +0100, Toke Høiland-Jørgensen wrote:
> > "Paul E. McKenney" writes:
> >
> > > On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
> > >> On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
> >
On 2/27/2024 1:32 PM, Paul E. McKenney wrote:
> On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
>> On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
>>> We noticed task RCUs being blocked when threaded NAPIs are very busy in
>>> production: detaching any BPF tracing programs, i.e.
On Wed, 28 Feb 2024 07:15:42 -0800 Paul E. McKenney wrote:
> > > Another complication is that although CONFIG_PREEMPT_RT kernels are
> > > built with CONFIG_PREEMPT_RCU, the reverse is not always the case.
> > > And if we are not repolling, don't we have a high probability of doing
> > > a
On Wed, Feb 28, 2024 at 06:43:43AM -0800, Jakub Kicinski wrote:
> On Tue, 27 Feb 2024 20:42:24 -0800 Paul E. McKenney wrote:
> > On Tue, Feb 27, 2024 at 07:10:01PM -0800, Jakub Kicinski wrote:
> > > On Tue, 27 Feb 2024 10:32:22 -0800 Paul E. McKenney wrote:
> > > > The theory is that PREEMPT_RCU
On Wed, Feb 28, 2024 at 12:50:53PM +0100, Toke Høiland-Jørgensen wrote:
> "Paul E. McKenney" writes:
>
> > On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
> >> On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
> >> >
> >> > We noticed task RCUs being blocked when threaded NAPIs are
On Tue, 27 Feb 2024 20:42:24 -0800 Paul E. McKenney wrote:
> On Tue, Feb 27, 2024 at 07:10:01PM -0800, Jakub Kicinski wrote:
> > On Tue, 27 Feb 2024 10:32:22 -0800 Paul E. McKenney wrote:
> > > The theory is that PREEMPT_RCU kernels have preemption, and get their
> > > quiescent states that way.
"Paul E. McKenney" writes:
> On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
>> On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
>> >
>> > We noticed task RCUs being blocked when threaded NAPIs are very busy in
>> > production: detaching any BPF tracing programs, i.e. removing a
On Tue, Feb 27, 2024 at 07:10:01PM -0800, Jakub Kicinski wrote:
> On Tue, 27 Feb 2024 10:32:22 -0800 Paul E. McKenney wrote:
> > > > + if (!IS_ENABLED(CONFIG_PREEMPT_RT))
> > > > + rcu_softirq_qs();
> > > > +
> > > >
On Tue, 27 Feb 2024 10:32:22 -0800 Paul E. McKenney wrote:
> > > + if (!IS_ENABLED(CONFIG_PREEMPT_RT))
> > > + rcu_softirq_qs();
> > > +
> > > local_bh_enable();
> > >
> > > if (!repoll)
> >
> >
On Tue, Feb 27, 2024 at 03:22:57PM -0600, Yan Zhai wrote:
> On Tue, Feb 27, 2024 at 12:32 PM Paul E. McKenney wrote:
> >
> > On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
> > > On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
> > > >
> > > > We noticed task RCUs being blocked when
On Tue, Feb 27, 2024 at 12:32 PM Paul E. McKenney wrote:
>
> On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
> > On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
> > >
> > > We noticed task RCUs being blocked when threaded NAPIs are very busy in
> > > production: detaching any BPF
On Tue, Feb 27, 2024 at 10:44 AM Eric Dumazet wrote:
>
> Hmm
> Why napi_busy_loop() does not have a similar problem ?
>
That's a good question. Let me try if I can repro this on a busy loop
as well, since the structure seems very alike.
> It is unclear why rcu_all_qs() in __cond_resched() is
On Tue, Feb 27, 2024 at 05:44:17PM +0100, Eric Dumazet wrote:
> On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
> >
> > We noticed task RCUs being blocked when threaded NAPIs are very busy in
> > production: detaching any BPF tracing programs, i.e. removing a ftrace
> > trampoline, will simply
On Tue, Feb 27, 2024 at 4:44 PM Yan Zhai wrote:
>
> We noticed task RCUs being blocked when threaded NAPIs are very busy in
> production: detaching any BPF tracing programs, i.e. removing a ftrace
> trampoline, will simply block for very long in rcu_tasks_wait_gp. This
> ranges from hundreds of
We noticed task RCUs being blocked when threaded NAPIs are very busy in
production: detaching any BPF tracing programs, i.e. removing a ftrace
trampoline, will simply block for very long in rcu_tasks_wait_gp. This
ranges from hundreds of seconds to even an hour, severely harming any
observability
47 matches
Mail list logo
