From: Soumya AR <[email protected]> Sending a fresh version of this series after some small fixes and cleanup.
v4: https://gcc.gnu.org/pipermail/gcc-patches/2026-October/734825.html v5 1/5 and 2/5 were posted inline in reply to v4. Changes since v4: [1/5] - gimple-isel no longer repeats the IR walk. - Added diagnostics for const pointers. - Mark builtin arguments as read, so building the IFN directly no longer causes bogus unused-but-set warnings. [2/5] - Shifted can_expand_atomic_minmax to this patch (was incorrectly addded in an earlier one.) - Fixed a possible ICE when expanding a fetch min/max with an unused result on a target that only provides the atomic_fetch_<op> pattern. - Documented the new min/max optabs in md.texi. [3/5] - atomic-minmax.c now passes -march=armv8-a+nolse explicitly. Soumya AR (5): middle-end: Add support for atomic fetch min/max builtins via CAS lowering expand: Expand IFN_ATOMIC_FETCH_MINMAX to target-specific optabs aarch64: Add backend support for atomic fetch min/max operations middle-end: Pattern match CAS loops to IFN_ATOMIC_FETCH_MINMAX middle-end: Treat atomic CAS and fetch min/max IFNs as expensive calls gcc/Makefile.in | 2 + gcc/atomic-ifn-lowering.cc | 472 +++++++ gcc/atomic-ifn-lowering.h | 26 + gcc/builtins.cc | 72 ++ gcc/builtins.h | 1 + gcc/c-family/c-common.cc | 110 +- gcc/c-family/c-common.h | 1 + gcc/config/aarch64/aarch64-protos.h | 4 + gcc/config/aarch64/aarch64.cc | 66 + gcc/config/aarch64/atomics.md | 54 +- gcc/config/aarch64/iterators.md | 30 +- gcc/doc/extend.texi | 18 + gcc/doc/md.texi | 36 + gcc/gimple-isel.cc | 7 +- gcc/gimple.cc | 9 +- gcc/internal-fn.cc | 8 + gcc/internal-fn.def | 1 + gcc/optabs.cc | 38 +- gcc/optabs.def | 24 + gcc/optabs.h | 2 + gcc/passes.def | 1 + gcc/predict.cc | 1 + gcc/sync-builtins.def | 7 + .../c-c++-common/atomic-fetch-minmax-const.c | 8 + .../c-c++-common/atomic-fetch-minmax-unused.c | 28 + .../g++.dg/eh/atomic-fetch-minmax-nce.C | 18 + .../g++.dg/eh/atomic-fetch-minmax-no-nce.C | 20 + .../template/builtin-atomic-overloads6.C | 26 +- .../template/builtin-atomic-overloads7.C | 17 +- .../g++.dg/tree-ssa/cas-to-minmax-1.C | 48 + .../g++.dg/tree-ssa/cas-to-minmax-2.C | 31 + .../gcc.dg/atomic-fetch-minmax-bad-types.c | 14 + .../gcc.dg/atomic-fetch-minmax-bitint.c | 34 + .../gcc.dg/atomic-fetch-minmax-ifn.c | 16 + .../gcc.dg/atomic-fetch-minmax-lower.c | 26 + .../gcc.dg/atomic-fetch-minmax-pointers.c | 16 + .../gcc.dg/atomic-fetch-minmax-sign-cast.c | 38 + .../atomic-fetch-minmax-type-mismatch.c | 31 + .../gcc.dg/atomic-fetch-minmax-unroll.c | 22 + gcc/testsuite/gcc.dg/atomic-op-1.c | 242 ++++ gcc/testsuite/gcc.dg/atomic-op-2.c | 242 ++++ gcc/testsuite/gcc.dg/atomic-op-3.c | 242 ++++ gcc/testsuite/gcc.dg/atomic-op-4.c | 242 ++++ gcc/testsuite/gcc.dg/atomic-op-5.c | 244 ++++ .../gcc.dg/tree-ssa/cas-to-minmax-1.c | 28 + .../gcc.dg/tree-ssa/cas-to-minmax-2.c | 77 ++ .../gcc.dg/tree-ssa/cas-to-minmax-3.c | 29 + .../gcc.dg/tree-ssa/cas-to-minmax-4.c | 189 +++ .../gcc.dg/tree-ssa/cas-to-minmax-run-1.c | 212 +++ .../gcc.target/aarch64/atomic-minmax-lse.c | 122 ++ .../gcc.target/aarch64/atomic-minmax-nolse.c | 200 +++ .../gcc.target/aarch64/atomic-minmax.c | 128 ++ .../gcc.target/aarch64/atomic-minmax.x | 186 +++ gcc/tree-cas-to-atomic-op.cc | 1142 +++++++++++++++++ gcc/tree-pass.h | 1 + libgcc/config/aarch64/lse.S | 65 +- libgcc/config/aarch64/t-lse | 3 +- 57 files changed, 4957 insertions(+), 20 deletions(-) create mode 100644 gcc/atomic-ifn-lowering.cc create mode 100644 gcc/atomic-ifn-lowering.h create mode 100644 gcc/testsuite/c-c++-common/atomic-fetch-minmax-const.c create mode 100644 gcc/testsuite/c-c++-common/atomic-fetch-minmax-unused.c create mode 100644 gcc/testsuite/g++.dg/eh/atomic-fetch-minmax-nce.C create mode 100644 gcc/testsuite/g++.dg/eh/atomic-fetch-minmax-no-nce.C create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-1.C create mode 100644 gcc/testsuite/g++.dg/tree-ssa/cas-to-minmax-2.C create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-bitint.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-pointers.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-unroll.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-1.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-2.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-3.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-4.c create mode 100644 gcc/testsuite/gcc.dg/tree-ssa/cas-to-minmax-run-1.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax-lse.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax-nolse.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax.c create mode 100644 gcc/testsuite/gcc.target/aarch64/atomic-minmax.x create mode 100644 gcc/tree-cas-to-atomic-op.cc Range-diff against v5: 1: be38021d5fe6 ! 1: a9ad92974254 middle-end: Add support for atomic fetch min/max builtins via CAS lowering @@ Commit message atomic fetch min/max. (resolve_overloaded_builtin): Resolve atomic min/max builtins to IFN_ATOMIC_FETCH_MINMAX. + * c-common.h (mark_exp_read): Declare. gcc/testsuite/ChangeLog: + * c-c++-common/atomic-fetch-minmax-const.c: New test. + * c-c++-common/atomic-fetch-minmax-unused.c: New test. * g++.dg/eh/atomic-fetch-minmax-nce.C: New test. * g++.dg/eh/atomic-fetch-minmax-no-nce.C: New test. * g++.dg/template/builtin-atomic-overloads6.C: Extend for fetch @@ gcc/atomic-ifn-lowering.cc (new) + loop_bb: + current_value = PHI<loaded_value, observed_value> + desired_value = op (current_value, value) -+ cas_succeeded = CAS (ptr, &expected_value, desired_value) -+ observed_value = expected_value // CAS overwrote it on failure ++ cas_result = .ATOMIC_COMPARE_EXCHANGE (ptr, current_value, ++ desired_value, ...) ++ observed_value = REALPART_EXPR <cas_result> ++ cas_succeeded = IMAGPART_EXPR <cas_result> + if (cas_succeeded) goto exit_bb else goto loop_bb + + exit_bb: @@ gcc/atomic-ifn-lowering.cc (new) +#include "atomic-ifn-lowering.h" + +/* Descriptor for an atomic-fetch IFN that this code can lower. -+ can_expand_directly returns true when the backend can expand the IFN via a -+ direct optab, in which case we leave the call for the RTL expander. lower -+ replaces the IFN call with a CAS loop. */ ++ lower replaces the IFN call with a CAS loop. */ + +struct atomic_op_lowering +{ @@ gcc/atomic-ifn-lowering.cc (new) + : insert_type_conversion (gsi_loop, newval, atomic_type); +} + -+/* Return true if the backend can expand an atomic fetch MIN (if IS_MIN) or -+ MAX on DATATYPE directly via a target optab. UNUSED_RESULT is true if -+ nothing uses the fetched value. */ -+ -+bool -+can_expand_atomic_fetch_minmax_p (tree datatype, bool is_min, -+ bool unused_result) -+{ -+ if (!flag_inline_atomics) -+ return false; -+ -+ bool is_signed = !TYPE_UNSIGNED (datatype); -+ machine_mode mode = TYPE_MODE (datatype); -+ -+ optab fetch_op_optab -+ = is_signed ? (is_min ? atomic_fetch_smin_optab : atomic_fetch_smax_optab) -+ : (is_min ? atomic_fetch_umin_optab : atomic_fetch_umax_optab); -+ -+ if (optab_handler (fetch_op_optab, mode) != CODE_FOR_nothing) -+ return true; -+ -+ /* If nothing uses the result, the no-result optab also suffices. MIN/MAX -+ has no op_fetch form, so we do not need those optabs. */ -+ if (unused_result) -+ { -+ optab no_result_optab -+ = is_signed ? (is_min ? atomic_smin_optab : atomic_smax_optab) -+ : (is_min ? atomic_umin_optab : atomic_umax_optab); -+ if (direct_optab_handler (no_result_optab, mode) != CODE_FOR_nothing) -+ return true; -+ } -+ -+ return false; -+} -+ -+/* Return true when the backend can expand an IFN_ATOMIC_FETCH_MINMAX call -+ directly via a target optab. */ -+ -+static bool -+can_expand_atomic_minmax (gcall *call) -+{ -+ tree datatype = TREE_TYPE (gimple_call_arg (call, 1)); -+ tree_code cmp_code -+ = (tree_code) tree_to_uhwi (gimple_call_arg (call, 2)); -+ gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); -+ return can_expand_atomic_fetch_minmax_p (datatype, cmp_code == MIN_EXPR, -+ gimple_call_lhs (call) == NULL_TREE); -+} -+ +/* Lower an IFN_ATOMIC_FETCH_MINMAX call to a CAS loop. */ + +static void @@ gcc/atomic-ifn-lowering.cc (new) + lower_via_cas_loop (gsi, emit_minmax_desired_value); +} + -+/* Atomic-fetch IFNs that we lower when the target lacks a direct optab. */ ++/* Atomic-fetch IFNs supported by this lowering code. */ + +static const atomic_op_lowering atomic_op_table[] = { + { IFN_ATOMIC_FETCH_MINMAX, lower_minmax_call }, @@ gcc/atomic-ifn-lowering.cc (new) + { + if (dump_file) + { -+ fprintf (dump_file, "Lowering atomic-fetch IFN to CAS loop:\n "); ++ fprintf (dump_file, "Lowering atomic IFN:\n "); + print_gimple_stmt (dump_file, call, 0, TDF_SLIM); + } + @@ gcc/atomic-ifn-lowering.cc (new) +} + +unsigned int -+lower_atomic_ifn (void) ++lower_atomic_ifn (const vec<gcall *> &atomic_calls) +{ -+ basic_block bb; + bool changed = false; + -+ FOR_EACH_BB_FN (bb, cfun) -+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);) -+ { -+ if (maybe_lower_atomic_ifn (&gsi)) -+ { -+ bb = gsi_bb (gsi); -+ changed = true; -+ continue; -+ } -+ -+ gsi_next (&gsi); -+ } ++ for (gcall *call : atomic_calls) ++ { ++ gimple_stmt_iterator gsi = gsi_for_stmt (call); ++ changed |= maybe_lower_atomic_ifn (&gsi); ++ } + + if (!changed) + return 0; @@ gcc/atomic-ifn-lowering.h (new) +#ifndef GCC_ATOMIC_IFN_LOWERING_H +#define GCC_ATOMIC_IFN_LOWERING_H + -+extern unsigned int lower_atomic_ifn (void); ++extern unsigned int lower_atomic_ifn (const vec<gcall *> &); + +#endif /* GCC_ATOMIC_IFN_LOWERING_H */ @@ gcc/c-family/c-common.cc: resolve_overloaded_builtin (location_t loc, tree funct params); } -+ /* Atomic fetch min/max builtins are implemented through an IFN -+ that will be either lowered to a CAS loop or expanded via a -+ direct optab, depending on the target. */ ++ /* Implement atomic fetch min/max builtins through an IFN. */ + if (orig_code == BUILT_IN_ATOMIC_FETCH_MIN + || orig_code == BUILT_IN_ATOMIC_FETCH_MAX) + { @@ gcc/c-family/c-common.cc: resolve_overloaded_builtin (location_t loc, tree funct + function); + return error_mark_node; + } ++ tree ptr_type = TREE_TYPE (first_param); ++ if (TYPE_READONLY (TREE_TYPE (ptr_type))) ++ { ++ if (c_dialect_cxx ()) ++ { ++ if (!complain) ++ return error_mark_node; ++ permerror (loc, "invalid conversion from %qT to %qT", ++ ptr_type, build_pointer_type (datatype)); ++ } ++ else ++ pedwarn (loc, OPT_Wdiscarded_qualifiers, ++ "passing argument 1 of %qE discards %qv qualifier " ++ "from pointer target type", ++ function, TYPE_QUAL_CONST); ++ } ++ ++ /* Building the IFN directly does not mark the arguments as read, ++ so do it here to avoid bogus -Wunused-but-set-* warnings. */ ++ for (tree arg : *params) ++ mark_exp_read (arg); + + tree_code op_code + = (orig_code == BUILT_IN_ATOMIC_FETCH_MIN) ? MIN_EXPR : MAX_EXPR; @@ gcc/c-family/c-common.cc: resolve_overloaded_builtin (location_t loc, tree funct NULL); if (result == error_mark_node) + ## gcc/c-family/c-common.h ## +@@ gcc/c-family/c-common.h: extern tree build_real_imag_expr (location_t, enum tree_code, tree); + extern tree build_unary_op (location_t, enum tree_code, tree, bool); + extern tree build_binary_op (location_t, enum tree_code, tree, tree, bool); + extern tree perform_integral_promotions (tree); ++extern void mark_exp_read (tree); + + /* These functions must be defined by each front-end which implements + a variant of the C language. They are used by port files. */ + ## gcc/doc/extend.texi ## @@ gcc/doc/extend.texi: alignment. A value of 0 indicates typical alignment should be used. The compiler may also ignore this parameter. @@ gcc/doc/extend.texi: alignment. A value of 0 indicates typical alignment should +@defbuiltin{@var{type} __atomic_fetch_min (@var{type} *@var{ptr}, @var{type} @var{val}, int @var{memorder})} +@defbuiltinx{@var{type} __atomic_fetch_max (@var{type} *@var{ptr}, @var{type} @var{val}, int @var{memorder})} +These built-in functions perform an atomic minimum or maximum operation, and -+return the value that had previously been in @code{*@var{ptr}}. Operations -+on pointer arguments are performed as if the operands were of -+the @code{uintptr_t} type. That is, they are not scaled by the size of -+the type to which the pointer points. Integer comparisons use the signedness of -+the type pointed to by @var{ptr}. ++return the value that had previously been in @code{*@var{ptr}}. Minimum ++and maximum comparisons are signed or unsigned according to the type ++pointed to by @var{ptr}. Operations on pointer arguments are performed as ++if the operands were of the @code{uintptr_t} type. That is, they are not ++scaled by the size of the type to which the pointer points. + +@smallexample +@{ tmp = *ptr; *ptr = tmp < val ? tmp : val; return tmp; @} // min @@ gcc/gimple-isel.cc: along with GCC; see the file COPYING3. If not see /* Expand all ARRAY_REF(VIEW_CONVERT_EXPR) gimple assignments into calls to @@ gcc/gimple-isel.cc: pass_gimple_isel::execute (struct function *fun) + gimple_stmt_iterator gsi; + basic_block bb; + bool cfg_changed = false; ++ auto_vec<gcall *> atomic_calls; + if (optimize) + enable_ranger (fun); + +@@ gcc/gimple-isel.cc: pass_gimple_isel::execute (struct function *fun) + + if (gcall *call = dyn_cast <gcall*>(*gsi)) + { ++ if (gimple_call_internal_p (call, IFN_ATOMIC_FETCH_MINMAX)) ++ atomic_calls.safe_push (call); + gimple_isel_builtin_call (call, &gsi); + continue; + } +@@ gcc/gimple-isel.cc: pass_gimple_isel::execute (struct function *fun) if (optimize) disable_ranger (fun); - return cfg_changed ? TODO_cleanup_cfg : 0; -+ return lower_atomic_ifn () | (cfg_changed ? TODO_cleanup_cfg : 0); ++ return lower_atomic_ifn (atomic_calls) ++ | (cfg_changed ? TODO_cleanup_cfg : 0); } } // anon namespace @@ gcc/sync-builtins.def: DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_IS_LOCK_FREE, DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_THREAD_FENCE, "__atomic_thread_fence", + ## gcc/testsuite/c-c++-common/atomic-fetch-minmax-const.c (new) ## +@@ ++/* { dg-do compile } */ ++ ++int ++fetch_min_const (const int *p, int v) ++{ ++ return __atomic_fetch_min (p, v, __ATOMIC_RELAXED); /* { dg-warning "passing argument 1 of .__atomic_fetch_min. discards .const. qualifier from pointer target type" "" { target c } } */ ++ /* { dg-error "invalid conversion from .const int\\*. to .int\\*." "" { target c++ } .-1 } */ ++} + + ## gcc/testsuite/c-c++-common/atomic-fetch-minmax-unused.c (new) ## +@@ ++/* { dg-do compile } */ ++/* { dg-options "-Wall -Wextra" } */ ++ ++int mem_int; ++ ++void ++fetch_min_ptr (int **src) ++{ ++ int *p; ++ p = *src; ++ __atomic_fetch_min (p, 1, __ATOMIC_RELAXED); ++} ++ ++void ++fetch_max_value (int *src) ++{ ++ int v; ++ v = *src; ++ __atomic_fetch_max (&mem_int, v, __ATOMIC_RELAXED); ++} ++ ++void ++fetch_min_memorder (int *src) ++{ ++ int o; ++ o = *src; ++ __atomic_fetch_min (&mem_int, 1, o); ++} + ## gcc/testsuite/g++.dg/eh/atomic-fetch-minmax-nce.C (new) ## @@ +// { dg-do compile } @@ gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c (new) + return min_result + max_result; +} + -+/* { dg-final { scan-tree-dump-times "Lowering atomic-fetch IFN to CAS loop" 4 "isel" } } */ ++/* { dg-final { scan-tree-dump-times "Lowering atomic IFN" 4 "isel" } } */ +/* { dg-final { scan-tree-dump-times "\\.ATOMIC_COMPARE_EXCHANGE" 4 "isel" } } */ ## gcc/testsuite/gcc.dg/atomic-fetch-minmax-pointers.c (new) ## 2: cc919885cf40 ! 2: 936c767ac848 expand: Expand IFN_ATOMIC_FETCH_MINMAX to target-specific optabs @@ Commit message * atomic-ifn-lowering.h (can_expand_atomic_fetch_minmax_p): Declare. * builtins.cc (expand_ifn_atomic_fetch_minmax): New. * builtins.h (expand_ifn_atomic_fetch_minmax): Declare. + * doc/md.texi (atomic_<op><mode>, atomic_fetch_<op><mode>) + (atomic_<op>_fetch<mode>, sync_<op><mode>, sync_old_<op><mode>) + (sync_new_<op><mode>): Document the smin, smax, umin and umax + variants. * internal-fn.cc (expand_ATOMIC_FETCH_MINMAX): Call expand_ifn_atomic_fetch_minmax. * optabs.cc (get_atomic_op_for_code): Handle SMIN, SMAX, UMIN, @@ gcc/atomic-ifn-lowering.cc: After lowering the CFG looks somewhat like: #include "fold-const.h" #include "builtins.h" #include "internal-fn.h" -@@ gcc/atomic-ifn-lowering.cc: After lowering the CFG looks somewhat like: + #include "atomic-ifn-lowering.h" + + /* Descriptor for an atomic-fetch IFN that this code can lower. +- lower replaces the IFN call with a CAS loop. */ ++ can_expand_directly returns true when the backend can expand the IFN via a ++ direct optab, in which case we leave the call for the RTL expander. lower ++ replaces the IFN call with a CAS loop. */ + struct atomic_op_lowering { internal_fn ifn; @@ gcc/atomic-ifn-lowering.cc: After lowering the CFG looks somewhat like: void (*lower) (gimple_stmt_iterator *); }; -@@ gcc/atomic-ifn-lowering.cc: can_expand_atomic_fetch_minmax_p (tree datatype, bool is_min, - bool is_signed = !TYPE_UNSIGNED (datatype); - machine_mode mode = TYPE_MODE (datatype); +@@ gcc/atomic-ifn-lowering.cc: emit_minmax_desired_value (gimple_stmt_iterator *gsi_loop, tree current_value, + : insert_type_conversion (gsi_loop, newval, atomic_type); + } ++/* Return true if the backend can expand an atomic fetch MIN (if IS_MIN) or ++ MAX on DATATYPE directly via a target optab. UNUSED_RESULT is true if ++ nothing uses the fetched value. */ ++ ++bool ++can_expand_atomic_fetch_minmax_p (tree datatype, bool is_min, ++ bool unused_result) ++{ ++ if (!flag_inline_atomics) ++ return false; ++ ++ bool is_signed = !TYPE_UNSIGNED (datatype); ++ machine_mode mode = TYPE_MODE (datatype); ++ + if (!can_atomic_load_p (mode)) + return false; + - optab fetch_op_optab - = is_signed ? (is_min ? atomic_fetch_smin_optab : atomic_fetch_smax_optab) - : (is_min ? atomic_fetch_umin_optab : atomic_fetch_umax_optab); ++ optab fetch_op_optab ++ = is_signed ? (is_min ? atomic_fetch_smin_optab : atomic_fetch_smax_optab) ++ : (is_min ? atomic_fetch_umin_optab : atomic_fetch_umax_optab); ++ ++ if (optab_handler (fetch_op_optab, mode) != CODE_FOR_nothing) ++ return true; ++ ++ /* If nothing uses the result, the no-result optab also suffices. MIN/MAX ++ has no op_fetch form, so we do not need those optabs. */ ++ if (unused_result) ++ { ++ optab no_result_optab ++ = is_signed ? (is_min ? atomic_smin_optab : atomic_smax_optab) ++ : (is_min ? atomic_umin_optab : atomic_umax_optab); ++ if (direct_optab_handler (no_result_optab, mode) != CODE_FOR_nothing) ++ return true; ++ } ++ ++ return false; ++} ++ ++/* Return true when the backend can expand an IFN_ATOMIC_FETCH_MINMAX call ++ directly via a target optab. */ ++ ++static bool ++can_expand_atomic_minmax (gcall *call) ++{ ++ tree datatype = TREE_TYPE (gimple_call_arg (call, 1)); ++ tree_code cmp_code ++ = (tree_code) tree_to_uhwi (gimple_call_arg (call, 2)); ++ gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR); ++ return can_expand_atomic_fetch_minmax_p (datatype, cmp_code == MIN_EXPR, ++ gimple_call_lhs (call) == NULL_TREE); ++} ++ + /* Lower an IFN_ATOMIC_FETCH_MINMAX call to a CAS loop. */ + + static void @@ gcc/atomic-ifn-lowering.cc: lower_minmax_call (gimple_stmt_iterator *gsi) - /* Atomic-fetch IFNs that we lower when the target lacks a direct optab. */ + lower_via_cas_loop (gsi, emit_minmax_desired_value); + } + +-/* Atomic-fetch IFNs supported by this lowering code. */ ++/* Atomic-fetch IFNs that we lower when the target lacks a direct optab. */ static const atomic_op_lowering atomic_op_table[] = { - { IFN_ATOMIC_FETCH_MINMAX, lower_minmax_call }, @@ gcc/atomic-ifn-lowering.cc: maybe_lower_atomic_ifn (gimple_stmt_iterator *gsi) + if (dump_file) { - fprintf (dump_file, "Lowering atomic-fetch IFN to CAS loop:\n "); + fprintf (dump_file, "Lowering atomic IFN:\n "); ## gcc/atomic-ifn-lowering.h ## @@ #define GCC_ATOMIC_IFN_LOWERING_H - extern unsigned int lower_atomic_ifn (void); + extern unsigned int lower_atomic_ifn (const vec<gcall *> &); +extern bool can_expand_atomic_fetch_minmax_p (tree, bool, bool); #endif /* GCC_ATOMIC_IFN_LOWERING_H */ @@ gcc/builtins.cc: expand_builtin_atomic_compare_exchange (machine_mode mode, tree + if (!result) + gcc_unreachable (); + ++ if (target == const0_rtx) ++ return; ++ + if (temp_target != target) + convert_move (SUBREG_REG (target), result, !is_signed); + else if (result != target) @@ gcc/builtins.h: extern void expand_builtin_trap (void); scalar_mode, machine_mode, tree, rtx); + ## gcc/c-family/c-common.cc ## +@@ gcc/c-family/c-common.cc: resolve_overloaded_builtin (location_t loc, tree function, + params); + } + +- /* Implement atomic fetch min/max builtins through an IFN. */ ++ /* Atomic fetch min/max builtins are implemented through an IFN ++ that will be either lowered to a CAS loop or expanded via a ++ direct optab, depending on the target. */ + if (orig_code == BUILT_IN_ATOMIC_FETCH_MIN + || orig_code == BUILT_IN_ATOMIC_FETCH_MAX) + { + + ## gcc/doc/md.texi ## +@@ gcc/doc/md.texi: interruptible locking. + @mdindex sync_and@var{mode} + @mdindex sync_xor@var{mode} + @mdindex sync_nand@var{mode} ++@mdindex sync_smin@var{mode} ++@mdindex sync_smax@var{mode} ++@mdindex sync_umin@var{mode} ++@mdindex sync_umax@var{mode} + @item @samp{sync_add@var{mode}}, @samp{sync_sub@var{mode}} + @itemx @samp{sync_ior@var{mode}}, @samp{sync_and@var{mode}} + @itemx @samp{sync_xor@var{mode}}, @samp{sync_nand@var{mode}} ++@itemx @samp{sync_smin@var{mode}}, @samp{sync_smax@var{mode}} ++@itemx @samp{sync_umin@var{mode}}, @samp{sync_umax@var{mode}} + These patterns emit code for an atomic operation on memory. + Operand 0 is the memory on which the atomic operation is performed. + Operand 1 is the second operand to the binary operator. +@@ gcc/doc/md.texi: from a compare-and-swap operation, if defined. + @mdindex sync_old_and@var{mode} + @mdindex sync_old_xor@var{mode} + @mdindex sync_old_nand@var{mode} ++@mdindex sync_old_smin@var{mode} ++@mdindex sync_old_smax@var{mode} ++@mdindex sync_old_umin@var{mode} ++@mdindex sync_old_umax@var{mode} + @item @samp{sync_old_add@var{mode}}, @samp{sync_old_sub@var{mode}} + @itemx @samp{sync_old_ior@var{mode}}, @samp{sync_old_and@var{mode}} + @itemx @samp{sync_old_xor@var{mode}}, @samp{sync_old_nand@var{mode}} ++@itemx @samp{sync_old_smin@var{mode}}, @samp{sync_old_smax@var{mode}} ++@itemx @samp{sync_old_umin@var{mode}}, @samp{sync_old_umax@var{mode}} + These patterns emit code for an atomic operation on memory, + and return the value that the memory contained before the operation. + Operand 0 is the result value, operand 1 is the memory on which the +@@ gcc/doc/md.texi: from a compare-and-swap operation, if defined. + @mdindex sync_new_and@var{mode} + @mdindex sync_new_xor@var{mode} + @mdindex sync_new_nand@var{mode} ++@mdindex sync_new_smin@var{mode} ++@mdindex sync_new_smax@var{mode} ++@mdindex sync_new_umin@var{mode} ++@mdindex sync_new_umax@var{mode} + @item @samp{sync_new_add@var{mode}}, @samp{sync_new_sub@var{mode}} + @itemx @samp{sync_new_ior@var{mode}}, @samp{sync_new_and@var{mode}} + @itemx @samp{sync_new_xor@var{mode}}, @samp{sync_new_nand@var{mode}} ++@itemx @samp{sync_new_smin@var{mode}}, @samp{sync_new_smax@var{mode}} ++@itemx @samp{sync_new_umin@var{mode}}, @samp{sync_new_umax@var{mode}} + These patterns are like their @code{sync_old_@var{op}} counterparts, + except that they return the value that exists in the memory location + after the operation, rather than before the operation. +@@ gcc/doc/md.texi: compare and swap loop. + @mdindex atomic_and@var{mode} + @mdindex atomic_xor@var{mode} + @mdindex atomic_nand@var{mode} ++@mdindex atomic_smin@var{mode} ++@mdindex atomic_smax@var{mode} ++@mdindex atomic_umin@var{mode} ++@mdindex atomic_umax@var{mode} + @item @samp{atomic_add@var{mode}}, @samp{atomic_sub@var{mode}} + @itemx @samp{atomic_or@var{mode}}, @samp{atomic_and@var{mode}} + @itemx @samp{atomic_xor@var{mode}}, @samp{atomic_nand@var{mode}} ++@itemx @samp{atomic_smin@var{mode}}, @samp{atomic_smax@var{mode}} ++@itemx @samp{atomic_umin@var{mode}}, @samp{atomic_umax@var{mode}} + These patterns emit code for an atomic operation on memory with memory + model semantics. Operand 0 is the memory on which the atomic operation is + performed. Operand 1 is the second operand to the binary operator. +@@ gcc/doc/md.texi: none of these are available a compare-and-swap loop will be used. + @mdindex atomic_fetch_and@var{mode} + @mdindex atomic_fetch_xor@var{mode} + @mdindex atomic_fetch_nand@var{mode} ++@mdindex atomic_fetch_smin@var{mode} ++@mdindex atomic_fetch_smax@var{mode} ++@mdindex atomic_fetch_umin@var{mode} ++@mdindex atomic_fetch_umax@var{mode} + @item @samp{atomic_fetch_add@var{mode}}, @samp{atomic_fetch_sub@var{mode}} + @itemx @samp{atomic_fetch_or@var{mode}}, @samp{atomic_fetch_and@var{mode}} + @itemx @samp{atomic_fetch_xor@var{mode}}, @samp{atomic_fetch_nand@var{mode}} ++@itemx @samp{atomic_fetch_smin@var{mode}}, @samp{atomic_fetch_smax@var{mode}} ++@itemx @samp{atomic_fetch_umin@var{mode}}, @samp{atomic_fetch_umax@var{mode}} + These patterns emit code for an atomic operation on memory with memory + model semantics, and return the original value. Operand 0 is an output + operand which contains the value of the memory location before the +@@ gcc/doc/md.texi: loop will be used. + @mdindex atomic_and_fetch@var{mode} + @mdindex atomic_xor_fetch@var{mode} + @mdindex atomic_nand_fetch@var{mode} ++@mdindex atomic_smin_fetch@var{mode} ++@mdindex atomic_smax_fetch@var{mode} ++@mdindex atomic_umin_fetch@var{mode} ++@mdindex atomic_umax_fetch@var{mode} + @item @samp{atomic_add_fetch@var{mode}}, @samp{atomic_sub_fetch@var{mode}} + @itemx @samp{atomic_or_fetch@var{mode}}, @samp{atomic_and_fetch@var{mode}} + @itemx @samp{atomic_xor_fetch@var{mode}}, @samp{atomic_nand_fetch@var{mode}} ++@itemx @samp{atomic_smin_fetch@var{mode}}, @samp{atomic_smax_fetch@var{mode}} ++@itemx @samp{atomic_umin_fetch@var{mode}}, @samp{atomic_umax_fetch@var{mode}} + These patterns emit code for an atomic operation on memory with memory + model semantics and return the result after the operation is performed. + Operand 0 is an output operand which contains the value after the + ## gcc/internal-fn.cc ## @@ gcc/internal-fn.cc: expand_ATOMIC_XOR_FETCH_CMP_0 (internal_fn, gcall *call) /* Expand atomic fetch minmax. */ 3: 070bc3c69213 ! 3: 25e8a0f8c8f3 aarch64: Add backend support for atomic fetch min/max operations @@ gcc/testsuite/gcc.target/aarch64/atomic-minmax-nolse.c (new) ## gcc/testsuite/gcc.target/aarch64/atomic-minmax.c (new) ## @@ +/* { dg-do run } */ -+/* { dg-additional-options "--save-temps -moutline-atomics" } */ ++/* { dg-additional-options "--save-temps -march=armv8-a+nolse -moutline-atomics" } */ +/* { dg-final { check-function-bodies "**" "" } } */ + +#include "atomic-minmax.x" 4: 2e8c434bf82d = 4: 829ea1a7ab9b middle-end: Pattern match CAS loops to IFN_ATOMIC_FETCH_MINMAX 5: 0f73d0970650 = 5: 92f1ea01340a middle-end: Treat atomic CAS and fetch min/max IFNs as expensive calls -- 2.43.0
