From: Soumya AR <[email protected]> gimple_inexpensive_call_p considers every internal call inexpensive. This leads to the following issues:
- Unrolling can duplicate a loop containing IFN_ATOMIC_FETCH_MINMAX before gimple-isel, which may then lower each copy into its own CAS loop. - pass_ch turns 'while (!CAS) ...' into 'if (!CAS) do ... while (!CAS)'. Pattern-matching then only replaces the CAS inside the loop, leaving a redundant CAS behind. Following Richi's suggestion to handle IFNs through combined_fn (https://gcc.gnu.org/pipermail/gcc-patches/2026-August/729213.html), gimple_inexpensive_call_p now has a check for gimple_call_combined_fn and returns false for CFN_ATOMIC_COMPARE_EXCHANGE and CFN_ATOMIC_FETCH_MINMAX. After doing this, tree_estimate_probability_bb ICEs on internal calls, since its call heuristic assumes every expensive call has a callee. Therefore, the call heuristic now skips internal calls. Bootstrapped and regression tested on aarch64-linux-gnu and x86_64-linux-gnu. Cross-compiled and regression tested for arm-linux-gnueabihf-armv7-a and aarch64-linux-gnu without LSE. OK for mainline? gcc/ChangeLog: * gimple.cc (gimple_inexpensive_call_p): Return false for IFN_ATOMIC_COMPARE_EXCHANGE and IFN_ATOMIC_FETCH_MINMAX. * predict.cc (tree_estimate_probability_bb): Skip internal calls. gcc/testsuite/ChangeLog: * gcc.dg/atomic-fetch-minmax-unroll.c: New test. * gcc.dg/tree-ssa/cas-to-minmax-2.c: Check that no CAS is left after the rewrite. Signed-off-by: Soumya AR <[email protected]> --- gcc/gimple.cc | 9 +++++++- gcc/predict.cc | 1 + .../gcc.dg/atomic-fetch-minmax-unroll.c | 22 +++++++++++++++++++ .../gcc.dg/tree-ssa/cas-to-minmax-2.c | 1 + 4 files changed, 32 insertions(+), 1 deletion(-) create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-unroll.c diff --git a/gcc/gimple.cc b/gcc/gimple.cc index 2295f353472b..feda355cdbeb 100644 --- a/gcc/gimple.cc +++ b/gcc/gimple.cc @@ -3564,7 +3564,14 @@ bool gimple_inexpensive_call_p (gcall *stmt) { if (gimple_call_internal_p (stmt)) - return true; + switch (gimple_call_combined_fn (stmt)) + { + case CFN_ATOMIC_COMPARE_EXCHANGE: + case CFN_ATOMIC_FETCH_MINMAX: + return false; + default: + return true; + } tree decl = gimple_call_fndecl (stmt); if (decl && is_inexpensive_builtin (decl)) return true; diff --git a/gcc/predict.cc b/gcc/predict.cc index 9bcecc024934..27d7f9e772be 100644 --- a/gcc/predict.cc +++ b/gcc/predict.cc @@ -3257,6 +3257,7 @@ tree_estimate_probability_bb (basic_block bb, bool local_only) { gimple *stmt = gsi_stmt (bi); if (is_gimple_call (stmt) + && !gimple_call_internal_p (stmt) && !gimple_inexpensive_call_p (as_a <gcall *> (stmt)) /* Constant and pure calls are hardly used to signalize something exceptional. */ diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-unroll.c b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-unroll.c new file mode 100644 index 000000000000..1b6284b5f1f9 --- /dev/null +++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-unroll.c @@ -0,0 +1,22 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -fdump-tree-cunroll" } */ + +int +repeat_fetch_min (int *ptr, int value) +{ + int total = 0; + for (int i = 0; i < 8; ++i) + total += __atomic_fetch_min (ptr, value + i, __ATOMIC_RELAXED); + return total; +} + +int +repeat_fetch_max (int *ptr, int value) +{ + int total = 0; + for (int i = 0; i < 8; ++i) + total += __atomic_fetch_max (ptr, value + i, __ATOMIC_RELAXED); + return total; +} + +/* { dg-final { scan-tree-dump-times "ATOMIC_FETCH_MINMAX" 2 "cunroll" } } */ diff --git a/gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c b/gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c index 0f7f4a9e1645..c896f48f247c 100644 --- a/gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c +++ b/gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c @@ -73,4 +73,5 @@ min_builtin (int *ptr, int val) /* { dg-final { scan-tree-dump "__atomic_compare_exchange_4" "dom2" } } */ /* { dg-final { scan-tree-dump "min_builtin: rewrote loop \[0-9\]+ to atomic MIN" "cas-to-atomic-op" } } */ +/* { dg-final { scan-tree-dump-not "ATOMIC_COMPARE_EXCHANGE" "cas-to-atomic-op" } } */ /* { dg-final { scan-tree-dump-not "__atomic_compare_exchange" "cas-to-atomic-op" } } */ -- 2.43.0
