Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-18 Thread Google
On Thu, 18 Apr 2024 12:10:59 +0100 Jonthan Haslam wrote: > Hi Masami, > > > > > OK, then I'll push this to for-next at this moment. > > > > Please share if you have a good idea for the batch interface which can > > > > be > > > > backported. I guess it should involve updating userspace changes

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-18 Thread Jonthan Haslam
Hi Masami, > > > OK, then I'll push this to for-next at this moment. > > > Please share if you have a good idea for the batch interface which can be > > > backported. I guess it should involve updating userspace changes too. > > > > Did you (or anyone else) need anything more from me on this one

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-11 Thread Jonthan Haslam
> > > OK, then I'll push this to for-next at this moment. > > > Please share if you have a good idea for the batch interface which can be > > > backported. I guess it should involve updating userspace changes too. > > > > Did you (or anyone else) need anything more from me on this one so that it

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-10 Thread Google
On Wed, 10 Apr 2024 11:38:11 +0100 Jonthan Haslam wrote: > Hi Masami, > > > > > Which is why I was asking to land this patch as is, as it relieves the > > > > scalability pains in production and is easy to backport to old > > > > kernels. And then we can work on batched APIs and switch to

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-10 Thread Jonthan Haslam
Hi Masami, > > > Which is why I was asking to land this patch as is, as it relieves the > > > scalability pains in production and is easy to backport to old > > > kernels. And then we can work on batched APIs and switch to per-CPU rw > > > semaphore. > > OK, then I'll push this to for-next at

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-04 Thread Jonthan Haslam
> > Things to note about the results: > > > > - The results are slightly variable so don't get too caught up on > > individual thread count - it's the trend that is important. > > - In terms of throughput with this specific benchmark a *very* macro view > > is that the RW spinlock provides

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-03 Thread Andrii Nakryiko
On Wed, Apr 3, 2024 at 4:05 AM Jonthan Haslam wrote: > > > > > > Given the discussion around per-cpu rw semaphore and need for > > > > > (internal) batched attachment API for uprobes, do you think you can > > > > > apply this patch as is for now? We can then gain initial improvements > > > > > in

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-04-03 Thread Jonthan Haslam
> > > > Given the discussion around per-cpu rw semaphore and need for > > > > (internal) batched attachment API for uprobes, do you think you can > > > > apply this patch as is for now? We can then gain initial improvements > > > > in scalability that are also easy to backport, and Jonathan will

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-29 Thread Andrii Nakryiko
On Fri, Mar 29, 2024 at 5:36 PM Masami Hiramatsu wrote: > > On Fri, 29 Mar 2024 10:33:57 -0700 > Andrii Nakryiko wrote: > > > On Wed, Mar 27, 2024 at 5:45 PM Andrii Nakryiko > > wrote: > > > > > > On Wed, Mar 27, 2024 at 5:18 PM Masami Hiramatsu > > > wrote: > > > > > > > > On Wed, 27 Mar

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-29 Thread Google
On Fri, 29 Mar 2024 10:33:57 -0700 Andrii Nakryiko wrote: > On Wed, Mar 27, 2024 at 5:45 PM Andrii Nakryiko > wrote: > > > > On Wed, Mar 27, 2024 at 5:18 PM Masami Hiramatsu > > wrote: > > > > > > On Wed, 27 Mar 2024 17:06:01 + > > > Jonthan Haslam wrote: > > > > > > > > > Masami, > > >

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-29 Thread Andrii Nakryiko
On Wed, Mar 27, 2024 at 5:45 PM Andrii Nakryiko wrote: > > On Wed, Mar 27, 2024 at 5:18 PM Masami Hiramatsu wrote: > > > > On Wed, 27 Mar 2024 17:06:01 + > > Jonthan Haslam wrote: > > > > > > > Masami, > > > > > > > > > > Given the discussion around per-cpu rw semaphore and need for > > > >

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-27 Thread Andrii Nakryiko
On Wed, Mar 27, 2024 at 5:18 PM Masami Hiramatsu wrote: > > On Wed, 27 Mar 2024 17:06:01 + > Jonthan Haslam wrote: > > > > > Masami, > > > > > > > > Given the discussion around per-cpu rw semaphore and need for > > > > (internal) batched attachment API for uprobes, do you think you can > > >

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-27 Thread Google
On Wed, 27 Mar 2024 17:06:01 + Jonthan Haslam wrote: > > > Masami, > > > > > > Given the discussion around per-cpu rw semaphore and need for > > > (internal) batched attachment API for uprobes, do you think you can > > > apply this patch as is for now? We can then gain initial improvements

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-27 Thread Jonthan Haslam
> > Masami, > > > > Given the discussion around per-cpu rw semaphore and need for > > (internal) batched attachment API for uprobes, do you think you can > > apply this patch as is for now? We can then gain initial improvements > > in scalability that are also easy to backport, and Jonathan will

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-26 Thread Google
On Mon, 25 Mar 2024 19:04:59 + Jonthan Haslam wrote: > Hi Masami, > > > > This change has been tested against production workloads that exhibit > > > significant contention on the spinlock and an almost order of magnitude > > > reduction for mean uprobe execution time is observed (28 -> 3.5

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-26 Thread Google
On Tue, 26 Mar 2024 09:01:47 -0700 Andrii Nakryiko wrote: > On Sun, Mar 24, 2024 at 8:03 PM Masami Hiramatsu wrote: > > > > On Thu, 21 Mar 2024 07:57:35 -0700 > > Jonathan Haslam wrote: > > > > > Active uprobes are stored in an RB tree and accesses to this tree are > > > dominated by read

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-26 Thread Andrii Nakryiko
On Sun, Mar 24, 2024 at 8:03 PM Masami Hiramatsu wrote: > > On Thu, 21 Mar 2024 07:57:35 -0700 > Jonathan Haslam wrote: > > > Active uprobes are stored in an RB tree and accesses to this tree are > > dominated by read operations. Currently these accesses are serialized by > > a spinlock but this

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-26 Thread Jonthan Haslam
> > > Have you considered/measured per-CPU RW semaphores? > > > > No I hadn't but thanks hugely for suggesting it! In initial measurements > > it seems to be between 20-100% faster than the RW spinlocks! Apologies for > > all the exclamation marks but I'm very excited. I'll do some more testing >

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-25 Thread Andrii Nakryiko
On Mon, Mar 25, 2024 at 12:12 PM Jonthan Haslam wrote: > > Hi Ingo, > > > > This change has been tested against production workloads that exhibit > > > significant contention on the spinlock and an almost order of magnitude > > > reduction for mean uprobe execution time is observed (28 -> 3.5 >

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-25 Thread Jonthan Haslam
Hi Ingo, > > This change has been tested against production workloads that exhibit > > significant contention on the spinlock and an almost order of magnitude > > reduction for mean uprobe execution time is observed (28 -> 3.5 microsecs). > > Have you considered/measured per-CPU RW semaphores?

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-25 Thread Jonthan Haslam
Hi Masami, > > This change has been tested against production workloads that exhibit > > significant contention on the spinlock and an almost order of magnitude > > reduction for mean uprobe execution time is observed (28 -> 3.5 microsecs). > > Looks good to me. > > Acked-by: Masami Hiramatsu

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-25 Thread Google
On Thu, 21 Mar 2024 07:57:35 -0700 Jonathan Haslam wrote: > Active uprobes are stored in an RB tree and accesses to this tree are > dominated by read operations. Currently these accesses are serialized by > a spinlock but this leads to enormous contention when large numbers of > threads are

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-23 Thread Ingo Molnar
* Jonathan Haslam wrote: > Active uprobes are stored in an RB tree and accesses to this tree are > dominated by read operations. Currently these accesses are serialized by > a spinlock but this leads to enormous contention when large numbers of > threads are executing active probes. > > This

Re: [PATCH] uprobes: reduce contention on uprobes_tree access

2024-03-21 Thread Andrii Nakryiko
On Thu, Mar 21, 2024 at 7:57 AM Jonathan Haslam wrote: > > Active uprobes are stored in an RB tree and accesses to this tree are > dominated by read operations. Currently these accesses are serialized by > a spinlock but this leads to enormous contention when large numbers of > threads are