Hi Shrikanth, Thanks for the quick response and for looping in Paul.
> This is true only if user selected PREEMPT_DYNAMIC option i think. Yes, exactly. The issue is that CONFIG_PREEMPT_DYNAMIC=y has been in the Fedora/RHEL kernel config since kernel 5.16 (2021). It was silently ignored on ppc64le until your 6.16 patch added HAVE_PREEMPT_DYNAMIC_KEY, so this is the first time PREEMPT_RCU actually takes effect for all Fedora/RHEL ppc64le users. > Does your preemption mode remain the same in two cases? Verified. On 6.15-rc6 (before PREEMPT_DYNAMIC takes effect): CONFIG_PREEMPT_VOLUNTARY=y (no CONFIG_PREEMPT_DYNAMIC, no /sys/kernel/debug/sched/preempt) On both 7.1 and 7.2-rc4 (with PREEMPT_DYNAMIC active): CONFIG_PREEMPT_LAZY=y CONFIG_PREEMPT_DYNAMIC=y CONFIG_PREEMPT_RCU=y cat /sys/kernel/debug/sched/preempt: "full (lazy)" So the runtime preemption mode is the same on 7.1 and 7.2. The difference between 6.15 and 7.1+ is that the preemption model changed from voluntary (static) to full/lazy (dynamic), which is what pulls in PREEMPT_RCU. > Weak memory model would need barriers irrespective of > HAVE_PREEMPT_DYNAMIC_CALL or HAVE_PREEMPT_DYNAMIC_KEY. > Static key too is expected to minimal cost. You're right, I should clarify -- the overhead is not in the static key mechanism itself. The cost comes from __rcu_read_lock() and __rcu_read_unlock() which are called when CONFIG_PREEMPT_RCU=y. These need real memory barriers (lwsync/isync) on ppc64le regardless of whether the dynamic mechanism uses keys or calls. The real question is: why does enabling PREEMPT_RCU cost ~33% on ppc64le but only ~3% on x86_64? The answer is that x86_64's TSO memory model makes the barriers in __rcu_read_lock/__rcu_read_unlock essentially free, while ppc64le's weak ordering requires explicit lwsync/isync instructions, which are expensive when called thousands of times per second in the SELinux AVC hot path. I also have new data from SELinux isolation testing that helps quantify this. On kernel 7.1 (which already has PREEMPT_RCU=y): SELinux mode kill bogo-ops/sec vs enforcing ------------ ----------------- ------------ Enforcing 69,107 baseline Permissive 70,248 +1.6% Disabled 93,566 +35.4% Permissive ~ enforcing confirms the overhead is not in SELinux policy evaluation. Disabling SELinux removes the rcu_read_lock/unlock call sites in the AVC path and recovers most of the performance -- but still leaves a ~13% gap vs 6.12 (no PREEMPT_RCU), which is the base cost of PREEMPT_RCU in the non-SELinux parts of the kill() path. So the core issue is: on ppc64le, CONFIG_PREEMPT_RCU makes rcu_read_lock/unlock significantly more expensive, and the kill() syscall path hits them very heavily through SELinux AVC lookups. Thanks, Jirka On Mon, Jul 27, 2026 at 6:19 AM Shrikanth Hegde <[email protected]> wrote: > > +cc paul for any further/RCU insights. > > On 7/27/26 12:03 AM, Jirka Hladky wrote: > > Hi Shrikanth, Christophe, > > > > Hi Jirka, thanks for the report. > > > I'm seeing a significant performance regression on ppc64le after this > > patch landed in 6.16, caused by CONFIG_PREEMPT_RCU becoming active > > once HAVE_PREEMPT_DYNAMIC_KEY is selected. > > > > This is true only if user selected PREEMPT_DYNAMIC option i think. > > config PREEMPT_RCU > bool > default y if (PREEMPT || PREEMPT_RT || PREEMPT_DYNAMIC) > select TREE_RCU > > > > Benchmark: stress-ng kill stressor (tight kill() syscall loop), > > single thread, POWER10 LPAR (8 vCPUs, 1 core SMT-8). > > > > Bisected across Fedora ELN kernel builds on ppc64le: > > > > kernel CONFIG_PREEMPT_RCU kill bogo-ops/sec > > --- 6.15-rc6 (eln148) no 103,207 > > 6.16 (eln150) yes 70,281 (-32%) > > 6.18 (eln154) yes 72,552 (-30%) > > > > Does your preemption mode remain the same in two cases? > > > For comparison, x86_64 (AMD EPYC 9355P) with the same config change > > shows only a 2.8% regression: > > > > 6.12 x86_64 37,436 > > 7.2 x86_64 36,392 (-2.8%) > > > > perf report shows the overhead comes from rcu_read_lock/unlock in the > > SELinux AVC path (check_kill_permission -> security_task_kill -> > > selinux_task_kill -> avc_has_perm -> avc_lookup): > > > > Function 6.15 (no PREEMPT_RCU) 6.16 (PREEMPT_RCU) > > --- avc_lookup 15.23% 24.79% > > __rcu_read_lock ~0% 4.52% > > __rcu_read_unlock ~0% 4.17% > > selinux_task_kill 6.23% 7.35% > > audit_signal_info* 0.94% 3.59% > > > > On x86_64, rcu_read_lock/unlock are cheap thanks to static calls > > (HAVE_PREEMPT_DYNAMIC_CALL). On ppc64le with the KEY-based > > implementation, the weak memory model requires real barriers > > (lwsync/isync) making each RCU read-side critical section > > significantly more expensive. > > Weak memory model would need barriers irrespective of > HAVE_PREEMPT_DYNAMIC_CALL > or HAVE_PREEMPT_DYNAMIC_KEY. That's my assumption. I will look > more into it. Also i don't know much about PREEMPT_RCU. So might take a while. > > > > > This aligns with Christophe's earlier review comment that > > HAVE_PREEMPT_DYNAMIC_CALL should be more performant. Would > > implementing static calls for ppc64 be feasible to close this gap? > > > > Static key too is expected to minimal cost. There maybe more into this. > > > Test details: > > - Machine: IBM POWER10 (pvr 0080 0200), pHyp virtualization > > - stress-ng 0.21.03, gcc 14.3.1, glibc 2.39 > > - Tuned profile: virtual-guest > > - SELinux: enforcing (permissive recovers only ~7%) > > > > Happy to run additional tests if needed. > > > > > -- -Jirka
