From: Soumya AR <[email protected]>
This patch introduces support for atomic fetch min/max operations through a new
internal function and gimple lowering pass. To do this, the following 2 generic
builtins are added.
- __atomic_fetch_min
- __atomic_fetch_max
Note that no op_fetch variants are introduced for min/max. Other atomic ops
expose op_fetch builtins to back operator overloads in libstdc++ (operator+=,
operator-=, etc.); min/max have no operator they could be overloaded onto, so
we can skip implementing these interfaces.
----
In the initial drafts for this patch, the lowering was min/max specific, but
since we plan to support more atomic operations this way (FP fetch_add/sub,
parts of reduction etc), this has been rewritten to hopefully make it easier
for future operations to pick up.
Adding a new fetch IFN to this pass now simply requires a single entry in
atomic_op_table[] and implementing its two callbacks (more on this below).
----
Builtins resolve in the frontend to a single internal function,
IFN_ATOMIC_FETCH_MINMAX (ptr, value, memorder, is_min, zero_of_datatype)
where OP_CODE is the gimple tree_code (MIN_EXPR or MAX_EXPR) selecting min vs
max, and ZERO_OF_DATATYPE is a zero constant whose TREE_TYPE carries the
operation's intended datatype (including signedness).
Encoding the datatype on a typed zero constant rather than deriving it from
the pointer in the lowering pass is helpful because gimplification strips
casts that it considers useless_type_conversion_p. For example,
unsigned val;
int do_minmax (int x)
{
return __atomic_fetch_min ((int *) &val, x, 0);
}
Here, unsigned* -> int* can be optimized away as useless, leading to an
unsigned min happening, which is not the intention.
The constant is built from TYPE_MAIN_VARIANT of the pointee so qualifiers like
volatile do not ride on this token. Those stay on the pointer arg, which the
lowering pass passes through unchanged into the synthesised load and CAS.
----
The lowering pass is structured around the idea that every atomic fetch_op can
be expressed as a CAS loop where only a small slice of the loop body differs per
op:
loaded_value = __atomic_load_N (ptr, RELAXED);
loop:
current_value = PHI <loaded_value (entry), observed_value (latch)>;
desired_value = op (current_value, value); <-- op-specific
expected_value = current_value;
cas_succeeded = __atomic_compare_exchange_N (ptr, &expected_value,
desired_value, false,
memorder, RELAXED);
observed_value = expected_value;
if (!cas_succeeded) goto loop;
exit:
lhs = current_value; <-- op-specific
Following this observation, lower_via_cas_loop implements two op-specific
callbacks:
- emit_desired_value produces desired_value = op (current_value, value).
For min/max, this inserts a MIN_EXPR / MAX_EXPR.
- emit_lhs_value produces the value assigned to the IFN's LHS in the exit
block. For min/max we follow fetch_op semantics, so it just emits
current_value. Other ops can return desired_value instead for op_fetch
variants.
----
This pass runs after IPA so that the optab query sees the correct backend.
That matters under offloading, where we can compile for either host or target,
and querying optabs pre-IPA would not give us the right backend. See
https://gcc.gnu.org/pipermail/gcc-patches/2025-November/699630.html
for the relevant discussion.
----
The pass is gated on the cfun->calls_atomic_ifn bit that the frontend sets when
it emits the IFN. This bit neds to be propogated through:
- tree-inline
- move_sese_region_to_fn
- LTO in/out streamer
I was able to catch these cases when testing OpenMP offloading (to verify that
the lowering pass picks the right backend for both host and target). Would
appreciate pointers on whether it should be propogated somewhere else that I'm
missing.
----
The diagnostics in the frontend are gated on 'complain', so we can emit a
substitution failure instead of a hard error in a SFINAE environment.
This is also tested in builtin-atomic-overloads6/7.C
This is what sync_resolve_params uses as well, but it's written to only parse
fndecls, so we have to do diagnostics separately here.
----
Bootstrapped and regression tested on aarch64-linux-gnu and x86_64-linux-gnu,
no regression. Cross-compiled and regression tested under qemu for
arm-linux-gnueabihf-armv7-a and aarch64-linux-gnu without LSE.
OK for trunk?
Signed-off-by: Soumya AR <[email protected]>
gcc/ChangeLog:
* Makefile.in: Add atomic-ifn-lowering.o.
* function.h (struct function): Add calls_atomic_ifn bit.
* internal-fn.cc (expand_ATOMIC_FETCH_MINMAX): Empty defintion.
* internal-fn.def (ATOMIC_FETCH_MINMAX): New internal function.
* lto-streamer-in.cc (input_struct_function_base): Stream in the
calls_atomic_ifn bit.
* lto-streamer-out.cc (output_struct_function_base): Stream it out.
* passes.def: Add pass_lower_atomic_ifn after IPA.
* sync-builtins.def (BUILT_IN_ATOMIC_FETCH_MIN): New builtin.
(BUILT_IN_ATOMIC_FETCH_MAX): New builtin.
* tree-cfg.cc (move_sese_region_to_fn): Propagate calls_atomic_ifn
when moving an SESE region into a different function.
* tree-inline.cc (expand_call_inline): Propagate calls_atomic_ifn
from callee to caller.
* tree-pass.h (make_pass_lower_atomic_ifn): Declare.
* atomic-ifn-lowering.cc: New file.
gcc/c-family/ChangeLog:
* c-common.cc (resolve_overloaded_builtin): Lower atomic min/max
builtins to IFN_ATOMIC_FETCH_MINMAX.
gcc/testsuite/ChangeLog:
* g++.dg/template/builtin-atomic-overloads6.C: Extend for
fetch_min/max.
* g++.dg/template/builtin-atomic-overloads7.C: Likewise.
* gcc.dg/atomic-op-1.c: Likewise.
* gcc.dg/atomic-op-2.c: Likewise.
* gcc.dg/atomic-op-3.c: Likewise.
* gcc.dg/atomic-op-4.c: Likewise.
* gcc.dg/atomic-op-5.c: Likewise.
* gcc.dg/atomic-fetch-minmax-bad-types.c: New test.
* gcc.dg/atomic-fetch-minmax-ifn.c: New test.
* gcc.dg/atomic-fetch-minmax-lower.c: New test.
* gcc.dg/atomic-fetch-minmax-sign-cast.c: New test.
* gcc.dg/atomic-fetch-minmax-type-mismatch.c: New test.
---
gcc/Makefile.in | 1 +
gcc/atomic-ifn-lowering.cc | 432 ++++++++++++++++++
gcc/c-family/c-common.cc | 67 ++-
gcc/function.h | 4 +
gcc/internal-fn.cc | 8 +
gcc/internal-fn.def | 1 +
gcc/lto-streamer-in.cc | 1 +
gcc/lto-streamer-out.cc | 1 +
gcc/passes.def | 1 +
gcc/sync-builtins.def | 7 +
.../template/builtin-atomic-overloads6.C | 33 +-
.../template/builtin-atomic-overloads7.C | 23 +-
.../gcc.dg/atomic-fetch-minmax-bad-types.c | 17 +
.../gcc.dg/atomic-fetch-minmax-ifn.c | 16 +
.../gcc.dg/atomic-fetch-minmax-lower.c | 17 +
.../gcc.dg/atomic-fetch-minmax-sign-cast.c | 38 ++
.../atomic-fetch-minmax-type-mismatch.c | 29 ++
gcc/testsuite/gcc.dg/atomic-op-1.c | 243 +++++++++-
gcc/testsuite/gcc.dg/atomic-op-2.c | 243 +++++++++-
gcc/testsuite/gcc.dg/atomic-op-3.c | 243 +++++++++-
gcc/testsuite/gcc.dg/atomic-op-4.c | 243 +++++++++-
gcc/testsuite/gcc.dg/atomic-op-5.c | 245 +++++++++-
gcc/tree-cfg.cc | 3 +
gcc/tree-inline.cc | 1 +
gcc/tree-pass.h | 1 +
25 files changed, 1907 insertions(+), 11 deletions(-)
create mode 100644 gcc/atomic-ifn-lowering.cc
create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c
create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c
create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c
create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c
create mode 100644 gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 0863a72c7d3..bcd5059d00c 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1421,6 +1421,7 @@ OBJS = \
alias.o \
alloc-pool.o \
asm-toplevel.o \
+ atomic-ifn-lowering.o \
auto-inc-dec.o \
auto-profile.o \
bb-reorder.o \
diff --git a/gcc/atomic-ifn-lowering.cc b/gcc/atomic-ifn-lowering.cc
new file mode 100644
index 00000000000..77129a1ca7e
--- /dev/null
+++ b/gcc/atomic-ifn-lowering.cc
@@ -0,0 +1,432 @@
+/* Lower atomic-fetch internal functions to a Compare-and-Swap (CAS) loop.
+ Copyright The GNU Toolchain Authors.
+
+ This file is part of GCC.
+
+ GCC is free software; you can redistribute it and/or modify it
+ under the terms of the GNU General Public License as published by
+ the Free Software Foundation; either version 3, or (at your option)
+ any later version.
+
+ GCC is distributed in the hope that it will be useful, but
+ WITHOUT ANY WARRANTY; without even the implied warranty of
+ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ General Public License for more details.
+
+ You should have received a copy of the GNU General Public License
+ along with GCC; see the file COPYING3. If not see
+ <http://www.gnu.org/licenses/>.
+
+This pass lowers atomic-fetch IFNs to a load + CAS loop:
+
+ current_value = atomic_load (ptr); // initial observation
+ do {
+ desired_value = op (current_value, value);
+ // CAS writes desired_value if memory still holds current_value;
+ // otherwise it refreshes current_value with the actual contents
+ // and the loop retries.
+ } while (!atomic_compare_exchange (ptr, ¤t_value, desired_value));
+ // LHS gets current_value (if fetch_op) or desired_value (if op_fetch).
+
+After lowering the CFG looks somewhat like:
+
+ before_bb:
+ loaded_value = atomic_load (ptr)
+ goto loop_bb
+
+ loop_bb:
+ current_value = PHI<loaded_value, observed_value>
+ desired_value = op (current_value, value)
+ cas_succeeded = CAS (ptr, &expected_value, desired_value)
+ observed_value = expected_value // CAS overwrote it on failure
+ if (cas_succeeded) goto exit_bb else goto loop_bb
+
+ exit_bb:
+ // LHS = current_value or desired_value, depending on the IFN
+*/
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "backend.h"
+#include "target.h"
+#include "tree.h"
+#include "gimple.h"
+#include "gimple-iterator.h"
+#include "cfghooks.h"
+#include "cfgloop.h"
+#include "tree-pass.h"
+#include "gimple-pretty-print.h"
+#include "memmodel.h"
+#include "ssa.h"
+#include "optabs-query.h"
+#include "fold-const.h"
+#include "builtins.h"
+#include "internal-fn.h"
+
+/* Descriptor for a single atomic-fetch IFN this pass knows how to lower.
+ LOWER is called to replace the IFN call: it must remove the call from
+ the gimple, typically by emitting a CAS loop via lower_via_cas_loop
+ with IFN-specific callbacks. */
+
+struct atomic_op_lowering
+{
+ internal_fn ifn;
+ void (*lower) (gcall *);
+};
+
+/* Helper to convert between types and insert into the gimple sequence.
+ Returns the SSA name holding the converted value. */
+
+static tree
+insert_type_conversion (gimple_stmt_iterator *gsi, tree src_value,
+ tree dest_type, enum tree_code convert_code = NOP_EXPR,
+ bool insert_after = true)
+{
+ if (TREE_TYPE (src_value) == dest_type)
+ return src_value;
+
+ tree result = make_ssa_name (dest_type);
+ gassign *convert_stmt = gimple_build_assign (result, convert_code,
src_value);
+
+ if (insert_after)
+ gsi_insert_after (gsi, convert_stmt, GSI_NEW_STMT);
+ else
+ gsi_insert_before (gsi, convert_stmt, GSI_SAME_STMT);
+
+ return result;
+}
+
+/* Lower the atomic-fetch IFN call STMT to a load + CAS loop. The op-specific
+ work is supplied via two callbacks: EMIT_DESIRED_VALUE produces the value
+ CAS will try to store, and EMIT_LHS_VALUE produces the value to assign to
+ STMT's LHS (ie. the IFN's LHS). */
+
+static void
+lower_via_cas_loop (gcall *stmt,
+ tree (*emit_desired_value) (gimple_stmt_iterator *, tree,
+ gcall *),
+ tree (*emit_lhs_value) (tree, tree, gimple_stmt_iterator *))
+{
+ gimple_stmt_iterator iter = gsi_for_stmt (stmt);
+
+ tree ptr = gimple_call_arg (stmt, 0);
+ tree memorder = gimple_call_arg (stmt, 2);
+
+ tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4));
+
+ /* Build the atomic builtins we need. */
+ int size_log2 = exact_log2 (tree_to_uhwi (TYPE_SIZE_UNIT (datatype)));
+ built_in_function load_builtin_fn
+ = (built_in_function) ((int) BUILT_IN_ATOMIC_LOAD_N + size_log2 + 1);
+ tree load_fn = builtin_decl_explicit (load_builtin_fn);
+
+ /* Atomic operations expect unsigned values as arguments. Calculate the
+ corresponding unsigned type. */
+ tree unsigned_type = unsigned_type_for (datatype);
+
+ /* Insert atomic load before the IFN. We will be using the IFN as a split
+ point later to insert the loop body. Since the load is not part of the
+ loop body, we insert it before the IFN. */
+ tree loaded_value = make_ssa_name (unsigned_type);
+ gcall *load_call
+ = gimple_build_call (load_fn, 2, ptr,
+ build_int_cst (integer_type_node, MEMMODEL_RELAXED));
+ gimple_call_set_lhs (load_call, loaded_value);
+ gsi_insert_before (&iter, load_call, GSI_SAME_STMT);
+
+ /* Save LHS before removing the IFN. Need it later to assign the result. */
+ tree lhs = gimple_call_lhs (stmt);
+
+ /* Build the loop body. We will be inserting this after the split point. */
+ gimple_seq loop_body = NULL;
+ gimple_stmt_iterator gsi_loop = gsi_last (loop_body);
+
+ /* Create a SSA name for the loop variable. This will be the PHI node in the
+ loop body. */
+ tree current_value = make_ssa_name (unsigned_type);
+
+ /* Emit the op-specific gimple producing the value CAS will store. */
+ tree desired_value = emit_desired_value (&gsi_loop, current_value, stmt);
+
+ /* Emit the CAS as IFN_ATOMIC_COMPARE_EXCHANGE. The IFN takes 'expected'
+ by value (no addressable local needed) and returns a complex
+ (REALPART = observed expected, IMAGPART = success flag). On targets
+ without a direct CAS optab the IFN expander automatically falls back to
+ the __atomic_compare_exchange_N builtin call, so emitting unconditionally
+ is safe. Arg 3 is a flag encoding (weak ? 256 : 0) + size_in_bytes; we
+ never emit weak here, so the flag is just the size. */
+ int flag = int_size_in_bytes (datatype);
+ tree complex_type = build_complex_type (unsigned_type);
+ tree cas_result = make_ssa_name (complex_type);
+ gcall *cas_call
+ = gimple_build_call_internal (IFN_ATOMIC_COMPARE_EXCHANGE, 6,
+ ptr, /* address to update */
+ current_value, /* expected, by value */
+ desired_value, /* desired value */
+ build_int_cst (integer_type_node, flag),
+ memorder, /* success order */
+ build_int_cst (integer_type_node,
+ MEMMODEL_RELAXED));
+ gimple_call_set_lhs (cas_call, cas_result);
+ gsi_insert_after (&gsi_loop, cas_call, GSI_NEW_STMT);
+
+ /* Extract the observed expected (REALPART) and the success flag (IMAGPART)
+ from the complex result. The IFN returns _Complex unsigned_type, so both
+ REALPART_EXPR and IMAGPART_EXPR produce values of unsigned_type, not bool.
+ Cast it back to bool for the exit gcond. */
+ tree observed_value = make_ssa_name (unsigned_type);
+ gsi_insert_after (&gsi_loop,
+ gimple_build_assign (observed_value, REALPART_EXPR,
+ build1 (REALPART_EXPR, unsigned_type,
+ cas_result)),
+ GSI_NEW_STMT);
+
+ tree cas_succeeded_raw = make_ssa_name (unsigned_type);
+ gsi_insert_after (&gsi_loop,
+ gimple_build_assign (cas_succeeded_raw, IMAGPART_EXPR,
+ build1 (IMAGPART_EXPR, unsigned_type,
+ cas_result)),
+ GSI_NEW_STMT);
+
+ tree cas_succeeded = make_ssa_name (boolean_type_node);
+ gsi_insert_after (&gsi_loop,
+ gimple_build_assign (cas_succeeded, NOP_EXPR,
+ cas_succeeded_raw),
+ GSI_NEW_STMT);
+
+ /* Add conditional to end of loop body. The conditional statement currently
+ has no successors; we add loop_bb and exit_bb as successors below. */
+ gcond *loop_cond = gimple_build_cond (NE_EXPR, cas_succeeded,
+ build_int_cst (boolean_type_node, 0),
+ NULL_TREE, NULL_TREE);
+ gsi_insert_after (&gsi_loop, loop_cond, GSI_NEW_STMT);
+
+ /* Get the previous statement in the basic block. We use it to split the
+ block after the atomic load. Currently the iter points to the IFN; the
+ bb looks like:
+
+ <previous_code>
+ |
+ atomic_load (ptr)
+ |
+ IFN_ATOMIC_FETCH_MINMAX <- gsi
+ |
+ <following_code>
+
+ We remove the IFN and replace it with the loop body. Split at the IFN,
+ then split the resultant edge to insert the loop body, resulting in:
+
+ before_bb (contains <previous_code> and atomic_load (ptr))
+ |
+ loop_bb (contains the MIN/MAX operation and the CAS operation)
+ |
+ after_bb (contains <following_code>)
+ */
+
+ gimple_stmt_iterator prev_gsi = iter;
+ gsi_prev (&prev_gsi);
+
+ /* Remove the IFN. */
+ gsi_remove (&iter, true);
+
+ edge e = split_block (gsi_bb (prev_gsi), gsi_stmt (prev_gsi));
+ basic_block exit_bb = e->dest;
+ basic_block loop_bb = split_edge (e);
+
+ /* Create PHI node for oldval. One edge will come from the predecessor
+ (loaded_value), and the other will come from the loop back
+ (observed_value). */
+ gphi *loop_phi = create_phi_node (current_value, loop_bb);
+ add_phi_arg (loop_phi, loaded_value, single_pred_edge (loop_bb),
+ UNKNOWN_LOCATION);
+
+ /* Insert the loop body into loop_bb. */
+ set_bb_seq (loop_bb, loop_body);
+
+ /* Set the basic block for all statements in the loop body to ensure they
+ are correctly assigned to loop_bb. */
+ for (gimple_stmt_iterator si = gsi_start_bb (loop_bb); !gsi_end_p (si);
+ gsi_next (&si))
+ gimple_set_bb (gsi_stmt (si), loop_bb);
+
+ /* loop_bb currently has a fallthrough edge to exit_bb, due to split_edge.
+ We need to remove it and create two conditional edges. */
+ remove_edge (single_succ_edge (loop_bb));
+ edge true_edge = make_edge (loop_bb, exit_bb, EDGE_TRUE_VALUE);
+ edge false_edge = make_edge (loop_bb, loop_bb, EDGE_FALSE_VALUE);
+
+ false_edge->probability = profile_probability::unlikely ();
+ true_edge->probability = false_edge->probability.invert ();
+
+ /* We've introduced a new back-edge, flag the loop tree for fixup. */
+ if (current_loops)
+ loops_state_set (LOOPS_NEED_FIXUP);
+
+ /* Add the loop-back phi argument. */
+ add_phi_arg (loop_phi, observed_value, false_edge, UNKNOWN_LOCATION);
+
+ if (lhs)
+ {
+ gimple_stmt_iterator exit_gsi = gsi_after_labels (exit_bb);
+ tree lhs_value = emit_lhs_value (datatype, current_value, &exit_gsi);
+ gsi_insert_before (&exit_gsi, gimple_build_assign (lhs, lhs_value),
+ GSI_SAME_STMT);
+ }
+}
+
+/* Emit gimple computing MIN/MAX (CURRENT_VALUE, value) at GSI_LOOP for the
+ IFN_ATOMIC_FETCH_MINMAX call STMT. CURRENT_VALUE is this iteration's
+ loaded value in the unsigned type. Returns an SSA name (in the same
+ unsigned type) holding the value CAS will try to store. */
+
+static tree
+emit_minmax_desired_value (gimple_stmt_iterator *gsi_loop, tree current_value,
+ gcall *stmt)
+{
+ tree value = gimple_call_arg (stmt, 1);
+ tree datatype = TREE_TYPE (gimple_call_arg (stmt, 4));
+ /* Arg 3 of the IFN is the tree_code for the operation (MIN_EXPR or
+ MAX_EXPR), encoded as an integer constant. */
+ tree_code cmp_code
+ = (tree_code) tree_to_uhwi (gimple_call_arg (stmt, 3));
+ gcc_assert (cmp_code == MIN_EXPR || cmp_code == MAX_EXPR);
+ tree unsigned_type = unsigned_type_for (datatype);
+
+ /* __atomic_load_N and __atomic_compare_exchange_N always operate in an
+ unsigned mode, so CURRENT_VALUE arrives from the load unsigned. But
+ MIN_EXPR / MAX_EXPR have semantics that depend on their operands' signs,
+ so for a signed DATATYPE we have to cast CURRENT_VALUE back to DATATYPE
+ (otherwise MIN/MAX would do an unsigned compare on the raw bits and
+ give the wrong answer for negative values).
+
+ Avoid using TYPE_UNSIGNED for this: it returns true for pointers even
+ though unsigned_type_for gives a different TREE_TYPE, so the check
+ would skip a real cast and leave the MIN/MAX operands mismatched. */
+ tree oldval_typed
+ = useless_type_conversion_p (datatype, TREE_TYPE (current_value))
+ ? current_value
+ : insert_type_conversion (gsi_loop, current_value, datatype);
+
+ /* Compute newval = MIN/MAX (oldval_typed, value) in DATATYPE. */
+ tree newval = make_ssa_name (datatype);
+ gassign *minmax_assign
+ = gimple_build_assign (newval, cmp_code, oldval_typed, value);
+ gsi_insert_after (gsi_loop, minmax_assign, GSI_NEW_STMT);
+
+ /* Cast newval back to UNSIGNED_TYPE so CAS can store it. */
+ return useless_type_conversion_p (unsigned_type, datatype)
+ ? newval
+ : insert_type_conversion (gsi_loop, newval, unsigned_type);
+}
+
+/* Return the SSA name to assign to the IFN's LHS for IFN_ATOMIC_FETCH_MINMAX.
+ CURRENT_VALUE (the loaded unsigned value on CAS success) should be cast
+ back to DATATYPE if it differs, with any cast inserted at EXIT_GSI. */
+
+static tree
+emit_minmax_lhs_value (tree datatype, tree current_value,
+ gimple_stmt_iterator *exit_gsi)
+{
+ return useless_type_conversion_p (datatype, TREE_TYPE (current_value))
+ ? current_value
+ : insert_type_conversion (exit_gsi, current_value, datatype,
+ NOP_EXPR, false);
+}
+
+/* Lower an IFN_ATOMIC_FETCH_MINMAX call to a CAS loop. */
+
+static void
+lower_minmax_call (gcall *call)
+{
+ lower_via_cas_loop (call, emit_minmax_desired_value, emit_minmax_lhs_value);
+}
+
+/* Registry of atomic-fetch IFNs this pass lowers. */
+
+static const atomic_op_lowering atomic_op_table[] = {
+ { IFN_ATOMIC_FETCH_MINMAX, lower_minmax_call },
+};
+
+static unsigned int
+lower_atomic_ifn (void)
+{
+ basic_block bb;
+ bool changed = false;
+
+ /* Avoid lowering atomic IFN calls during the BB walk because lowering
+ can split basic blocks and corrupt the iterator. Collect the calls
+ and their matching descriptors first, then lower them one by one. */
+ auto_vec<std::pair<gcall *, const atomic_op_lowering *> > to_lower;
+ FOR_EACH_BB_FN (bb, cfun)
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb); !gsi_end_p (gsi);
+ gsi_next (&gsi))
+ {
+ gcall *call = dyn_cast<gcall *> (gsi_stmt (gsi));
+ if (!call)
+ continue;
+ for (const atomic_op_lowering &op : atomic_op_table)
+ if (gimple_call_internal_p (call, op.ifn))
+ {
+ to_lower.safe_push ({ call, &op });
+ break;
+ }
+ }
+
+ for (auto &[call, op] : to_lower)
+ {
+ if (dump_file)
+ {
+ fprintf (dump_file, "Lowering atomic-fetch IFN to CAS loop:\n ");
+ print_gimple_stmt (dump_file, call, 0, TDF_SLIM);
+ }
+ op->lower (call);
+ changed = true;
+ }
+
+ return changed ? TODO_update_ssa | TODO_cleanup_cfg : 0;
+}
+
+static bool
+gate_lower_atomic_ifn (function *fn)
+{
+ /* Skip the pass for functions that never constructed an atomic IFN. */
+ return fn->calls_atomic_ifn;
+}
+
+namespace {
+
+const pass_data pass_data_lower_atomic_ifn = {
+ GIMPLE_PASS, /* type */
+ "lower_atomic_ifn", /* name */
+ OPTGROUP_NONE, /* optinfo_flags */
+ TV_NONE, /* tv_id */
+ (PROP_cfg | PROP_ssa), /* properties_required */
+ 0, /* properties_provided */
+ 0, /* properties_destroyed */
+ 0, /* todo_flags_start */
+ 0 /* todo_flags_finish */
+};
+
+class pass_lower_atomic_ifn : public gimple_opt_pass
+{
+public:
+ pass_lower_atomic_ifn (gcc::context *ctxt)
+ : gimple_opt_pass (pass_data_lower_atomic_ifn, ctxt)
+ {}
+
+ /* opt_pass methods: */
+ bool gate (function *fn) final override { return gate_lower_atomic_ifn (fn);
}
+ unsigned int execute (function *) final override
+ {
+ return lower_atomic_ifn ();
+ }
+}; // class pass_lower_atomic_ifn
+
+} // namespace
+
+gimple_opt_pass *
+make_pass_lower_atomic_ifn (gcc::context *ctxt)
+{
+ return new pass_lower_atomic_ifn (ctxt);
+}
diff --git a/gcc/c-family/c-common.cc b/gcc/c-family/c-common.cc
index a16288f4441..d2baa8251ef 100644
--- a/gcc/c-family/c-common.cc
+++ b/gcc/c-family/c-common.cc
@@ -46,6 +46,7 @@ along with GCC; see the file COPYING3. If not see
#include "tree-iterator.h"
#include "opts.h"
#include "gimplify.h"
+#include "internal-fn.h"
#include "substring-locations.h"
#include "spellcheck.h"
#include "c-spellcheck.h"
@@ -8793,6 +8794,8 @@ resolve_overloaded_builtin (location_t loc, tree function,
case BUILT_IN_ATOMIC_FETCH_NAND_N:
case BUILT_IN_ATOMIC_FETCH_XOR_N:
case BUILT_IN_ATOMIC_FETCH_OR_N:
+ case BUILT_IN_ATOMIC_FETCH_MIN:
+ case BUILT_IN_ATOMIC_FETCH_MAX:
orig_format = false;
/* FALLTHRU */
case BUILT_IN_SYNC_FETCH_AND_ADD_N:
@@ -8822,12 +8825,14 @@ resolve_overloaded_builtin (location_t loc, tree
function,
int n = sync_resolve_size (function, params, fetch_op, orig_format,
complain);
- tree new_function, first_param, result;
+ tree new_function, result;
enum built_in_function fncode;
if (n == 0)
return error_mark_node;
+ tree first_param = (*params)[0];
+
if (n == -1)
{
/* complain is related to SFINAE context.
@@ -8843,13 +8848,71 @@ resolve_overloaded_builtin (location_t loc, tree
function,
params);
}
+ /* Atomic fetch min/max builtins are implemented through an IFN
+ that will be either lowered to a CAS loop or expanded via a
+ direct optab, depending on the target. */
+ if (orig_code == BUILT_IN_ATOMIC_FETCH_MIN
+ || orig_code == BUILT_IN_ATOMIC_FETCH_MAX)
+ {
+ /* Similar to sync_resolve_params, we should call TYPE_MAIN_VARIANT
+ to drop volatile. */
+ tree datatype
+ = TYPE_MAIN_VARIANT (TREE_TYPE (TREE_TYPE (first_param)));
+
+ /* sync_resolve_params expects a sized builtin decl; we only
+ register the generic form and lower via IFN, therefore, validate
+ params manually. */
+ const char *err = NULL;
+ if (params->length () < 3)
+ err = "too few arguments to function %qE";
+ else if (params->length () > 3)
+ err = "too many arguments to function %qE";
+ else if (!INTEGRAL_TYPE_P (datatype))
+ err = "argument 1 of %qE must point to an integer type";
+ else if (!INTEGRAL_TYPE_P (TREE_TYPE ((*params)[1])))
+ err = "argument 2 of %qE must be an integer type";
+ else if (!INTEGRAL_TYPE_P (TREE_TYPE ((*params)[2])))
+ err = "non-integer memory model argument 3 of %qE";
+ if (err)
+ {
+ if (complain)
+ error_at (loc, err, function);
+ return error_mark_node;
+ }
+
+ tree_code op_code
+ = (orig_code == BUILT_IN_ATOMIC_FETCH_MIN) ? MIN_EXPR : MAX_EXPR;
+
+ /* INTEGRAL_TYPE_P above guarantees VALUE is integer-typed, so this
+ is int-to-int and convert () should not emit an error. */
+ tree value = convert (datatype, (*params)[1]);
+ tree memorder = (*params)[2];
+ tree op_code_tree = build_int_cst (integer_type_node, op_code);
+ /* Dummy argument to preserve the datatype of the first parameter.
+ Casts on the first parameter might get stripped away later. */
+ tree type_arg = build_zero_cst (datatype);
+
+ tree ret
+ = build_call_expr_internal_loc (loc, IFN_ATOMIC_FETCH_MINMAX,
+ datatype, 5, first_param, value,
+ memorder, op_code_tree, type_arg);
+ /* Atomics can throw under -fnon-call-exceptions; mark the call
+ NOTHROW otherwise. */
+ if (!flag_non_call_exceptions)
+ TREE_NOTHROW (ret) = 1;
+
+ /* Mark the function so pass_lower_atomic_ifn knows to run. */
+ if (cfun)
+ cfun->calls_atomic_ifn = 1;
+
+ return ret;
+ }
fncode = (enum built_in_function)((int)orig_code + exact_log2 (n) + 1);
new_function = builtin_decl_explicit (fncode);
if (!sync_resolve_params (loc, function, new_function, params,
orig_format, complain))
return error_mark_node;
- first_param = (*params)[0];
result = build_function_call_vec (loc, vNULL, new_function, params,
NULL);
if (result == error_mark_node)
diff --git a/gcc/function.h b/gcc/function.h
index 765c71309fa..dc030dbd04c 100644
--- a/gcc/function.h
+++ b/gcc/function.h
@@ -375,6 +375,10 @@ struct GTY(()) function {
/* Nonzero if function being compiled can call __builtin_eh_return. */
unsigned int calls_eh_return : 1;
+ /* Nonzero if function being compiled contains an atomic-fetch IFN that
+ pass_lower_atomic_ifn needs to consider. */
+ unsigned int calls_atomic_ifn : 1;
+
/* Nonzero if function being compiled receives nonlocal gotos
from nested functions. */
unsigned int has_nonlocal_label : 1;
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 8a4622da794..692391f67dc 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -3762,6 +3762,14 @@ expand_ATOMIC_XOR_FETCH_CMP_0 (internal_fn, gcall *call)
expand_ifn_atomic_op_fetch_cmp_0 (call);
}
+/* Expand atomic fetch minmax. */
+
+static void
+expand_ATOMIC_FETCH_MINMAX (internal_fn, gcall *)
+{
+ /* Implement this. */
+}
+
/* Expand LAUNDER to assignment, lhs = arg0. */
static void
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index a9fb933021f..4f90c65ca5f 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -601,6 +601,7 @@ DEF_INTERNAL_FN (ATOMIC_SUB_FETCH_CMP_0, ECF_LEAF, NULL)
DEF_INTERNAL_FN (ATOMIC_AND_FETCH_CMP_0, ECF_LEAF, NULL)
DEF_INTERNAL_FN (ATOMIC_OR_FETCH_CMP_0, ECF_LEAF, NULL)
DEF_INTERNAL_FN (ATOMIC_XOR_FETCH_CMP_0, ECF_LEAF, NULL)
+DEF_INTERNAL_FN (ATOMIC_FETCH_MINMAX, ECF_LEAF, NULL)
/* To implement [[fallthrough]]. If the TREE_NOTHROW or GF_CALL_NOTHROW flag
is set on the call (normally redundant with ECF_NOTHROW), it marks
diff --git a/gcc/lto-streamer-in.cc b/gcc/lto-streamer-in.cc
index ee2fa2affbc..46e5feb852f 100644
--- a/gcc/lto-streamer-in.cc
+++ b/gcc/lto-streamer-in.cc
@@ -1328,6 +1328,7 @@ input_struct_function_base (struct function *fn, class
data_in *data_in,
fn->has_musttail = bp_unpack_value (&bp, 1);
fn->has_unroll = bp_unpack_value (&bp, 1);
fn->assume_function = bp_unpack_value (&bp, 1);
+ fn->calls_atomic_ifn = bp_unpack_value (&bp, 1);
fn->va_list_fpr_size = bp_unpack_value (&bp, 8);
fn->va_list_gpr_size = bp_unpack_value (&bp, 8);
fn->last_clique = bp_unpack_value (&bp, sizeof (short) * 8);
diff --git a/gcc/lto-streamer-out.cc b/gcc/lto-streamer-out.cc
index f2feefc1532..379861f02d6 100644
--- a/gcc/lto-streamer-out.cc
+++ b/gcc/lto-streamer-out.cc
@@ -2318,6 +2318,7 @@ output_struct_function_base (struct output_block *ob,
struct function *fn)
bp_pack_value (&bp, fn->has_musttail, 1);
bp_pack_value (&bp, fn->has_unroll, 1);
bp_pack_value (&bp, fn->assume_function, 1);
+ bp_pack_value (&bp, fn->calls_atomic_ifn, 1);
bp_pack_value (&bp, fn->va_list_fpr_size, 8);
bp_pack_value (&bp, fn->va_list_gpr_size, 8);
bp_pack_value (&bp, fn->last_clique, sizeof (short) * 8);
diff --git a/gcc/passes.def b/gcc/passes.def
index 1fc867fae51..54f2eeb159d 100644
--- a/gcc/passes.def
+++ b/gcc/passes.def
@@ -206,6 +206,7 @@ along with GCC; see the file COPYING3. If not see
NEXT_PASS (pass_omp_target_link);
NEXT_PASS (pass_adjust_alignment);
NEXT_PASS (pass_harden_control_flow_redundancy);
+ NEXT_PASS (pass_lower_atomic_ifn);
NEXT_PASS (pass_all_optimizations);
PUSH_INSERT_PASSES_WITHIN (pass_all_optimizations)
NEXT_PASS (pass_remove_cgraph_callee_edges);
diff --git a/gcc/sync-builtins.def b/gcc/sync-builtins.def
index 1a5d84f5db5..704b43a683d 100644
--- a/gcc/sync-builtins.def
+++ b/gcc/sync-builtins.def
@@ -597,6 +597,13 @@ DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_IS_LOCK_FREE,
"__atomic_is_lock_free",
BT_FN_BOOL_SIZE_CONST_VPTR, ATTR_CONST_NOTHROW_LEAF_LIST)
+DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_FETCH_MIN,
+ "__atomic_fetch_min",
+ BT_FN_VOID_VAR, ATTR_NOTHROWCALL_LEAF_LIST)
+
+DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_FETCH_MAX,
+ "__atomic_fetch_max",
+ BT_FN_VOID_VAR, ATTR_NOTHROWCALL_LEAF_LIST)
DEF_SYNC_BUILTIN (BUILT_IN_ATOMIC_THREAD_FENCE,
"__atomic_thread_fence",
diff --git a/gcc/testsuite/g++.dg/template/builtin-atomic-overloads6.C
b/gcc/testsuite/g++.dg/template/builtin-atomic-overloads6.C
index 15d01c10b72..34aa4496d79 100644
--- a/gcc/testsuite/g++.dg/template/builtin-atomic-overloads6.C
+++ b/gcc/testsuite/g++.dg/template/builtin-atomic-overloads6.C
@@ -132,11 +132,32 @@ typedef __UINT32_TYPE__ uint32_t;
*/
#define BITINT_FETCHCAS_ERRS(X)
+#define FETCHMIN_TOOFEW(X) \
+ X(fetch_min, (std::declval<int*>(), int()), 0)
+#define FETCHMIN_TOOMANY(X) \
+ X(fetch_min, (std::declval<int*>(), int(), int(), int()), 1)
+#define FETCHMAX_TOOFEW(X) \
+ X(fetch_max, (std::declval<int*>(), int()), 0)
+#define FETCHMAX_TOOMANY(X) \
+ X(fetch_max, (std::declval<int*>(), int(), int(), int()), 1)
+#define FETCHMIN_BADVALUE(X) \
+ X(fetch_min, (std::declval<int*>(), Large(), int()), 2)
+#define FETCHMAX_BADVALUE(X) \
+ X(fetch_max, (std::declval<int*>(), Large(), int()), 2)
+#define MINMAX_FETCH_ERRS(X) \
+ FETCHMIN_TOOFEW(X) \
+ FETCHMIN_TOOMANY(X) \
+ FETCHMAX_TOOFEW(X) \
+ FETCHMAX_TOOMANY(X) \
+ FETCHMIN_BADVALUE(X) \
+ FETCHMAX_BADVALUE(X)
+
#define ALL_ERRS(X) \
GET_ATOMIC_GENERIC_ERRS(X) \
SYNC_SIZE_ERRS(X) \
SYNC_PARM_ERRS(X) \
- BITINT_FETCHCAS_ERRS(X)
+ BITINT_FETCHCAS_ERRS(X) \
+ MINMAX_FETCH_ERRS(X)
#define SFINAE_TYPE_CHECK(NAME, PARAMS, COUNTER) \
template <typename T, typename = void> \
@@ -152,8 +173,8 @@ ALL_ERRS(SFINAE_TYPE_CHECK)
/* { dg-error "operand type 'int' is incompatible with argument 1 of
'__atomic_load_n'" "" { target *-*-* } 110 } */
/* { dg-error "too few arguments to function '__atomic_load_n'"
"" { target *-*-* } 116 } */
/* { dg-error "too many arguments to function '__atomic_load_n'"
"" { target *-*-* } 118 } */
-/* { dg-error "template argument 1 is invalid"
"" { target *-*-* } 146 } */
-/* { dg-error "template argument 2 is invalid"
"" { target *-*-* } 146 } */
+/* { dg-error "template argument 1 is invalid"
"" { target *-*-* } 167 } */
+/* { dg-error "template argument 2 is invalid"
"" { target *-*-* } 167 } */
/* { dg-error "incorrect number of arguments to function '__atomic_load'"
"" { target *-*-* } 48 } */
/* { dg-error "argument 1 of '__atomic_load' must be a non-void pointer type"
"" { target *-*-* } 51 } */
/* { dg-error "argument 1 of '__atomic_load' must be a non-void pointer type"
"" { target *-*-* } 53 } */
@@ -167,5 +188,11 @@ ALL_ERRS(SFINAE_TYPE_CHECK)
/* { dg-error "argument 2 of '__atomic_load' must not be a pointer to a
'const' type" "" { target *-*-* } 80 } */
/* { dg-error "argument 2 of '__atomic_load' must not be a pointer to a
'volatile' type" "" { target *-*-* } 82 } */
/* { dg-error "non-integer memory model argument 3 of '__atomic_load'"
"" { target *-*-* } 93 } */
+/* { dg-error "too few arguments to function '__atomic_fetch_min'"
"" { target *-*-* } 136 } */
+/* { dg-error "too many arguments to function '__atomic_fetch_min'"
"" { target *-*-* } 138 } */
+/* { dg-error "too few arguments to function '__atomic_fetch_max'"
"" { target *-*-* } 140 } */
+/* { dg-error "too many arguments to function '__atomic_fetch_max'"
"" { target *-*-* } 142 } */
+/* { dg-error "argument 2 of '__atomic_fetch_min' must be an integer type"
"" { target *-*-* } 144 } */
+/* { dg-error "argument 2 of '__atomic_fetch_max' must be an integer type"
"" { target *-*-* } 146 } */
/* { dg-warning {invalid memory model argument 3 of '__atomic_load'
\[-Winvalid-memory-model\]} "" { target *-*-* } 95 } */
diff --git a/gcc/testsuite/g++.dg/template/builtin-atomic-overloads7.C
b/gcc/testsuite/g++.dg/template/builtin-atomic-overloads7.C
index c0996d15c9f..96bcc06a7ed 100644
--- a/gcc/testsuite/g++.dg/template/builtin-atomic-overloads7.C
+++ b/gcc/testsuite/g++.dg/template/builtin-atomic-overloads7.C
@@ -136,12 +136,33 @@ typedef __UINT32_TYPE__ uint32_t;
*/
#define BITINT_FETCHCAS_ERRS(X)
+#define FETCHMIN_TOOFEW(X) \
+ X(fetch_min, (std::declval<T>(), int()), 0)
+#define FETCHMIN_TOOMANY(X) \
+ X(fetch_min, (std::declval<T>(), int(), int(), int()), 1)
+#define FETCHMAX_TOOFEW(X) \
+ X(fetch_max, (std::declval<T>(), int()), 0)
+#define FETCHMAX_TOOMANY(X) \
+ X(fetch_max, (std::declval<T>(), int(), int(), int()), 1)
+#define FETCHMIN_BADVALUE(X) \
+ X(fetch_min, (std::declval<int*>(), std::declval<T>(), int()), 2)
+#define FETCHMAX_BADVALUE(X) \
+ X(fetch_max, (std::declval<int*>(), std::declval<T>(), int()), 2)
+#define MINMAX_FETCH_ERRS(X) \
+ FETCHMIN_TOOFEW(X) \
+ FETCHMIN_TOOMANY(X) \
+ FETCHMAX_TOOFEW(X) \
+ FETCHMAX_TOOMANY(X) \
+ FETCHMIN_BADVALUE(X) \
+ FETCHMAX_BADVALUE(X)
+
#define ALL_ERRS(X) \
GET_ATOMIC_GENERIC_ERRS(X) \
SYNC_SIZE_ERRS(X) \
SYNC_PARM_ERRS(X) \
MEMMODEL_TOOLARGE(X) \
- BITINT_FETCHCAS_ERRS(X)
+ BITINT_FETCHCAS_ERRS(X) \
+ MINMAX_FETCH_ERRS(X)
#define SFINAE_TYPE_CHECK(NAME, PARAMS, COUNTER) \
template <typename T, typename = void> \
diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c
b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c
new file mode 100644
index 00000000000..d4c5e607430
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-bad-types.c
@@ -0,0 +1,17 @@
+
+/* { dg-do compile } */
+/* { dg-options "-O2" } */
+
+int ival;
+
+int test_bad_value (float x)
+{
+ /* arg 2 must be an integer type. */
+ return __atomic_fetch_min (&ival, x, __ATOMIC_SEQ_CST); /* { dg-error "must
be an integer type" } */
+}
+
+int test_bad_memorder (int x, float m)
+{
+ /* arg 3 (memory model) must be an integer. */
+ return __atomic_fetch_max (&ival, x, m); /* { dg-error "non-integer memory
model" } */
+}
diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c
b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c
new file mode 100644
index 00000000000..00e5dcfaacc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-ifn.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-gimple" } */
+
+int global_val;
+
+int test_min (int x)
+{
+ return __atomic_fetch_min (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+int test_max (int x)
+{
+ return __atomic_fetch_max (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+/* { dg-final { scan-tree-dump-times "ATOMIC_FETCH_MINMAX" 2 "gimple" } } */
diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c
b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c
new file mode 100644
index 00000000000..764bf70762d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-lower.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fno-inline-atomics -fdump-tree-lower_atomic_ifn" } */
+
+int global_val;
+
+int test_min (int x)
+{
+ return __atomic_fetch_min (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+int test_max (int x)
+{
+ return __atomic_fetch_max (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+/* { dg-final { scan-tree-dump-times "Lowering atomic-fetch IFN to CAS loop" 2
"lower_atomic_ifn" } } */
+/* { dg-final { scan-tree-dump-times "\\.ATOMIC_COMPARE_EXCHANGE" 2
"lower_atomic_ifn" } } */
diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c
b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c
new file mode 100644
index 00000000000..e5f9ca14691
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-sign-cast.c
@@ -0,0 +1,38 @@
+/* Verify that __atomic_fetch_min/max correctly preserves signedness when the
+underlying storage is unsigned but accessed via a signed pointer cast. */
+
+/* { dg-do run } */
+/* { dg-require-effective-target sync_int_long } */
+/* { dg-options "-O2" } */
+
+extern void abort (void);
+
+unsigned val;
+
+int
+main ()
+{
+ int old;
+
+ val = (unsigned)-1;
+ old = __atomic_fetch_min ((int *)&val, 5, __ATOMIC_SEQ_CST);
+ if (old != -1 || (int)val != -1)
+ abort ();
+
+ val = (unsigned)-1;
+ old = __atomic_fetch_max ((int *)&val, 5, __ATOMIC_SEQ_CST);
+ if (old != -1 || (int)val != 5)
+ abort ();
+
+ val = (unsigned)-3;
+ old = __atomic_fetch_min ((int *)&val, -10, __ATOMIC_SEQ_CST);
+ if (old != -3 || (int)val != -10)
+ abort ();
+
+ val = (unsigned)-10;
+ old = __atomic_fetch_max ((int *)&val, -3, __ATOMIC_SEQ_CST);
+ if (old != -10 || (int)val != -3)
+ abort ();
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c
b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c
new file mode 100644
index 00000000000..5ea9ccfe403
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/atomic-fetch-minmax-type-mismatch.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fdump-tree-gimple" } */
+
+int global_val = 10;
+
+int test_min_long_long (long long x)
+{
+ return __atomic_fetch_min (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+int test_max_long_long (long long x)
+{
+ return __atomic_fetch_max (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+int test_min_short (short x)
+{
+ return __atomic_fetch_min (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+int test_min_char (char x)
+{
+ return __atomic_fetch_min (&global_val, x, __ATOMIC_SEQ_CST);
+}
+
+/* Each function must explicitly convert its argument to (int) (type of
+ global_val) before passing it to ATOMIC_FETCH_MINMAX. The four functions
+ each produce one such cast, so we expect exactly 4 in total. */
+/* { dg-final { scan-tree-dump-times "= \\(int\\) x" 4 "gimple" } } */
\ No newline at end of file
diff --git a/gcc/testsuite/gcc.dg/atomic-op-1.c
b/gcc/testsuite/gcc.dg/atomic-op-1.c
index a8a97c401b7..3b638090e5c 100644
--- a/gcc/testsuite/gcc.dg/atomic-op-1.c
+++ b/gcc/testsuite/gcc.dg/atomic-op-1.c
@@ -5,6 +5,8 @@
/* Test the execution of the __atomic_*OP builtin routines for a char. */
+#include "limits.h"
+
extern void abort(void);
char v, count, res;
@@ -167,6 +169,114 @@ test_fetch_or ()
abort ();
}
+void
+test_fetch_smin ()
+{
+ signed char sv = 10;
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE) != -10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SCHAR_MIN, __ATOMIC_ACQ_REL) != -20)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SCHAR_MAX, __ATOMIC_SEQ_CST) != SCHAR_MIN)
+ abort ();
+
+ if (sv != SCHAR_MIN)
+ abort ();
+}
+
+void
+test_fetch_umin ()
+{
+ unsigned char uv = 100;
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED) != 100)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, UCHAR_MAX, __ATOMIC_SEQ_CST) != 0)
+ abort ();
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_fetch_smax ()
+{
+ signed char sv = -10;
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED) != -10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, SCHAR_MAX, __ATOMIC_SEQ_CST) != 50)
+ abort ();
+
+ if (sv != SCHAR_MAX)
+ abort ();
+}
+
+void
+test_fetch_umax ()
+{
+ unsigned char uv = 10;
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL) != 100)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, UCHAR_MAX, __ATOMIC_SEQ_CST) != 200)
+ abort ();
+
+ if (uv != UCHAR_MAX)
+ abort ();
+}
+
/* The OP_fetch routines return the new value after the operation. */
void
@@ -328,7 +438,6 @@ test_or_fetch ()
abort ();
}
-
/* Test the OP routines with a result which isn't used. Use both variations
within each function. */
@@ -527,6 +636,130 @@ test_or ()
abort ();
}
+void
+test_smin ()
+{
+ signed char sv = 10;
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE);
+ if (sv != -10)
+ abort ();
+
+ __atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE);
+ if (sv != -20)
+ abort ();
+
+ __atomic_fetch_min (&sv, SCHAR_MIN, __ATOMIC_ACQ_REL);
+ if (sv != SCHAR_MIN)
+ abort ();
+
+ __atomic_fetch_min (&sv, SCHAR_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SCHAR_MIN)
+ abort ();
+}
+
+void
+test_umin ()
+{
+ unsigned char uv = 100;
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE);
+ if (uv != 10)
+ abort ();
+
+ __atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL);
+ if (uv != 0)
+ abort ();
+
+ __atomic_fetch_min (&uv, UCHAR_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_smax ()
+{
+ signed char sv = -10;
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE);
+ if (sv != 10)
+ abort ();
+
+ __atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL);
+ if (sv != 50)
+ abort ();
+
+ __atomic_fetch_max (&sv, SCHAR_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SCHAR_MAX)
+ abort ();
+}
+
+void
+test_umax ()
+{
+ unsigned char uv = 10;
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE);
+ if (uv != 100)
+ abort ();
+
+ __atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL);
+ if (uv != 200)
+ abort ();
+
+ __atomic_fetch_max (&uv, UCHAR_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != UCHAR_MAX)
+ abort ();
+}
+
int
main ()
{
@@ -536,6 +769,10 @@ main ()
test_fetch_nand ();
test_fetch_xor ();
test_fetch_or ();
+ test_fetch_smin ();
+ test_fetch_umin ();
+ test_fetch_smax ();
+ test_fetch_umax ();
test_add_fetch ();
test_sub_fetch ();
@@ -550,6 +787,10 @@ main ()
test_nand ();
test_xor ();
test_or ();
+ test_smin ();
+ test_umin ();
+ test_smax ();
+ test_umax ();
return 0;
}
diff --git a/gcc/testsuite/gcc.dg/atomic-op-2.c
b/gcc/testsuite/gcc.dg/atomic-op-2.c
index 949850345b5..2ec47eeb9dd 100644
--- a/gcc/testsuite/gcc.dg/atomic-op-2.c
+++ b/gcc/testsuite/gcc.dg/atomic-op-2.c
@@ -6,6 +6,8 @@
/* Test the execution of the __atomic_*OP builtin routines for a short. */
+#include "limits.h"
+
extern void abort(void);
short v, count, res;
@@ -168,6 +170,114 @@ test_fetch_or ()
abort ();
}
+void
+test_fetch_smin ()
+{
+ short sv = 10;
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE) != -10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SHRT_MIN, __ATOMIC_ACQ_REL) != -20)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SHRT_MAX, __ATOMIC_SEQ_CST) != SHRT_MIN)
+ abort ();
+
+ if (sv != SHRT_MIN)
+ abort ();
+}
+
+void
+test_fetch_umin ()
+{
+ unsigned short uv = 100;
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED) != 100)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, USHRT_MAX, __ATOMIC_SEQ_CST) != 0)
+ abort ();
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_fetch_smax ()
+{
+ short sv = -10;
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED) != -10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, SHRT_MAX, __ATOMIC_SEQ_CST) != 50)
+ abort ();
+
+ if (sv != SHRT_MAX)
+ abort ();
+}
+
+void
+test_fetch_umax ()
+{
+ unsigned short uv = 10;
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL) != 100)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, USHRT_MAX, __ATOMIC_SEQ_CST) != 200)
+ abort ();
+
+ if (uv != USHRT_MAX)
+ abort ();
+}
+
/* The OP_fetch routines return the new value after the operation. */
void
@@ -329,7 +439,6 @@ test_or_fetch ()
abort ();
}
-
/* Test the OP routines with a result which isn't used. Use both variations
within each function. */
@@ -528,6 +637,130 @@ test_or ()
abort ();
}
+void
+test_smin ()
+{
+ short sv = 10;
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE);
+ if (sv != -10)
+ abort ();
+
+ __atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE);
+ if (sv != -20)
+ abort ();
+
+ __atomic_fetch_min (&sv, SHRT_MIN, __ATOMIC_ACQ_REL);
+ if (sv != SHRT_MIN)
+ abort ();
+
+ __atomic_fetch_min (&sv, SHRT_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SHRT_MIN)
+ abort ();
+}
+
+void
+test_umin ()
+{
+ unsigned short uv = 100;
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE);
+ if (uv != 10)
+ abort ();
+
+ __atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL);
+ if (uv != 0)
+ abort ();
+
+ __atomic_fetch_min (&uv, USHRT_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_smax ()
+{
+ short sv = -10;
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE);
+ if (sv != 10)
+ abort ();
+
+ __atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL);
+ if (sv != 50)
+ abort ();
+
+ __atomic_fetch_max (&sv, SHRT_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SHRT_MAX)
+ abort ();
+}
+
+void
+test_umax ()
+{
+ unsigned short uv = 10;
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE);
+ if (uv != 100)
+ abort ();
+
+ __atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL);
+ if (uv != 200)
+ abort ();
+
+ __atomic_fetch_max (&uv, USHRT_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != USHRT_MAX)
+ abort ();
+}
+
int
main ()
{
@@ -537,6 +770,10 @@ main ()
test_fetch_nand ();
test_fetch_xor ();
test_fetch_or ();
+ test_fetch_smin ();
+ test_fetch_umin ();
+ test_fetch_smax ();
+ test_fetch_umax ();
test_add_fetch ();
test_sub_fetch ();
@@ -551,6 +788,10 @@ main ()
test_nand ();
test_xor ();
test_or ();
+ test_smin ();
+ test_umin ();
+ test_smax ();
+ test_umax ();
return 0;
}
diff --git a/gcc/testsuite/gcc.dg/atomic-op-3.c
b/gcc/testsuite/gcc.dg/atomic-op-3.c
index 9a54a2a4178..a25a6695762 100644
--- a/gcc/testsuite/gcc.dg/atomic-op-3.c
+++ b/gcc/testsuite/gcc.dg/atomic-op-3.c
@@ -5,6 +5,8 @@
/* Test the execution of the __atomic_*OP builtin routines for an int. */
+#include "limits.h"
+
extern void abort(void);
int v, count, res;
@@ -167,6 +169,114 @@ test_fetch_or ()
abort ();
}
+void
+test_fetch_smin ()
+{
+ int sv = 10;
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE) != -10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, INT_MIN, __ATOMIC_ACQ_REL) != -20)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, INT_MAX, __ATOMIC_SEQ_CST) != INT_MIN)
+ abort ();
+
+ if (sv != INT_MIN)
+ abort ();
+}
+
+void
+test_fetch_umin ()
+{
+ unsigned int uv = 100;
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED) != 100)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, UINT_MAX, __ATOMIC_SEQ_CST) != 0)
+ abort ();
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_fetch_smax ()
+{
+ int sv = -10;
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED) != -10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, INT_MAX, __ATOMIC_SEQ_CST) != 50)
+ abort ();
+
+ if (sv != INT_MAX)
+ abort ();
+}
+
+void
+test_fetch_umax ()
+{
+ unsigned int uv = 10;
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL) != 100)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, UINT_MAX, __ATOMIC_SEQ_CST) != 200)
+ abort ();
+
+ if (uv != UINT_MAX)
+ abort ();
+}
+
/* The OP_fetch routines return the new value after the operation. */
void
@@ -328,7 +438,6 @@ test_or_fetch ()
abort ();
}
-
/* Test the OP routines with a result which isn't used. Use both variations
within each function. */
@@ -527,6 +636,130 @@ test_or ()
abort ();
}
+void
+test_smin ()
+{
+ int sv = 10;
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE);
+ if (sv != -10)
+ abort ();
+
+ __atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE);
+ if (sv != -20)
+ abort ();
+
+ __atomic_fetch_min (&sv, INT_MIN, __ATOMIC_ACQ_REL);
+ if (sv != INT_MIN)
+ abort ();
+
+ __atomic_fetch_min (&sv, INT_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != INT_MIN)
+ abort ();
+}
+
+void
+test_umin ()
+{
+ unsigned int uv = 100;
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE);
+ if (uv != 10)
+ abort ();
+
+ __atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL);
+ if (uv != 0)
+ abort ();
+
+ __atomic_fetch_min (&uv, UINT_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_smax ()
+{
+ int sv = -10;
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE);
+ if (sv != 10)
+ abort ();
+
+ __atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL);
+ if (sv != 50)
+ abort ();
+
+ __atomic_fetch_max (&sv, INT_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != INT_MAX)
+ abort ();
+}
+
+void
+test_umax ()
+{
+ unsigned int uv = 10;
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE);
+ if (uv != 100)
+ abort ();
+
+ __atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL);
+ if (uv != 200)
+ abort ();
+
+ __atomic_fetch_max (&uv, UINT_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != UINT_MAX)
+ abort ();
+}
+
int
main ()
{
@@ -536,6 +769,10 @@ main ()
test_fetch_nand ();
test_fetch_xor ();
test_fetch_or ();
+ test_fetch_smin ();
+ test_fetch_umin ();
+ test_fetch_smax ();
+ test_fetch_umax ();
test_add_fetch ();
test_sub_fetch ();
@@ -550,6 +787,10 @@ main ()
test_nand ();
test_xor ();
test_or ();
+ test_smin ();
+ test_umin ();
+ test_smax ();
+ test_umax ();
return 0;
}
diff --git a/gcc/testsuite/gcc.dg/atomic-op-4.c
b/gcc/testsuite/gcc.dg/atomic-op-4.c
index 6990b0e2d75..e518615df46 100644
--- a/gcc/testsuite/gcc.dg/atomic-op-4.c
+++ b/gcc/testsuite/gcc.dg/atomic-op-4.c
@@ -7,6 +7,8 @@
/* Test the execution of the __atomic_*OP builtin routines for long long. */
+#include "limits.h"
+
extern void abort(void);
long long v, count, res;
@@ -169,6 +171,114 @@ test_fetch_or ()
abort ();
}
+void
+test_fetch_smin ()
+{
+ long long sv = 10;
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE) != -10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, LLONG_MIN, __ATOMIC_ACQ_REL) != -20)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, LLONG_MAX, __ATOMIC_SEQ_CST) != LLONG_MIN)
+ abort ();
+
+ if (sv != LLONG_MIN)
+ abort ();
+}
+
+void
+test_fetch_umin ()
+{
+ unsigned long long uv = 100;
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED) != 100)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, ULLONG_MAX, __ATOMIC_SEQ_CST) != 0)
+ abort ();
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_fetch_smax ()
+{
+ long long sv = -10;
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED) != -10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, LLONG_MAX, __ATOMIC_SEQ_CST) != 50)
+ abort ();
+
+ if (sv != LLONG_MAX)
+ abort ();
+}
+
+void
+test_fetch_umax ()
+{
+ unsigned long long uv = 10;
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL) != 100)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, ULLONG_MAX, __ATOMIC_SEQ_CST) != 200)
+ abort ();
+
+ if (uv != ULLONG_MAX)
+ abort ();
+}
+
/* The OP_fetch routines return the new value after the operation. */
void
@@ -330,7 +440,6 @@ test_or_fetch ()
abort ();
}
-
/* Test the OP routines with a result which isn't used. Use both variations
within each function. */
@@ -529,6 +638,130 @@ test_or ()
abort ();
}
+void
+test_smin ()
+{
+ long long sv = 10;
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE);
+ if (sv != -10)
+ abort ();
+
+ __atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE);
+ if (sv != -20)
+ abort ();
+
+ __atomic_fetch_min (&sv, LLONG_MIN, __ATOMIC_ACQ_REL);
+ if (sv != LLONG_MIN)
+ abort ();
+
+ __atomic_fetch_min (&sv, LLONG_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != LLONG_MIN)
+ abort ();
+}
+
+void
+test_umin ()
+{
+ unsigned long long uv = 100;
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE);
+ if (uv != 10)
+ abort ();
+
+ __atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL);
+ if (uv != 0)
+ abort ();
+
+ __atomic_fetch_min (&uv, ULLONG_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_smax ()
+{
+ long long sv = -10;
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE);
+ if (sv != 10)
+ abort ();
+
+ __atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL);
+ if (sv != 50)
+ abort ();
+
+ __atomic_fetch_max (&sv, LLONG_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != LLONG_MAX)
+ abort ();
+}
+
+void
+test_umax ()
+{
+ unsigned long long uv = 10;
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE);
+ if (uv != 100)
+ abort ();
+
+ __atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL);
+ if (uv != 200)
+ abort ();
+
+ __atomic_fetch_max (&uv, ULLONG_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != ULLONG_MAX)
+ abort ();
+}
+
int
main ()
{
@@ -538,6 +771,10 @@ main ()
test_fetch_nand ();
test_fetch_xor ();
test_fetch_or ();
+ test_fetch_smin ();
+ test_fetch_umin ();
+ test_fetch_smax ();
+ test_fetch_umax ();
test_add_fetch ();
test_sub_fetch ();
@@ -552,6 +789,10 @@ main ()
test_nand ();
test_xor ();
test_or ();
+ test_smin ();
+ test_umin ();
+ test_smax ();
+ test_umax ();
return 0;
}
diff --git a/gcc/testsuite/gcc.dg/atomic-op-5.c
b/gcc/testsuite/gcc.dg/atomic-op-5.c
index 4c6dcef8bf2..32bcc1e9580 100644
--- a/gcc/testsuite/gcc.dg/atomic-op-5.c
+++ b/gcc/testsuite/gcc.dg/atomic-op-5.c
@@ -9,6 +9,10 @@
extern void abort(void);
+#define SINT128_MIN ((__int128_t)((__uint128_t)1 << (sizeof(__int128_t) * 8 -
1)))
+#define SINT128_MAX ((__int128_t)~SINT128_MIN)
+#define UINT128_MAX ((__uint128_t)~0)
+
__int128_t v, count, res;
const __int128_t init = ~0;
@@ -169,6 +173,114 @@ test_fetch_or ()
abort ();
}
+void
+test_fetch_smin ()
+{
+ __int128_t sv = 10;
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE) != 5)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE) != -10)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SINT128_MIN, __ATOMIC_ACQ_REL) != -20)
+ abort ();
+
+ if (__atomic_fetch_min (&sv, SINT128_MAX, __ATOMIC_SEQ_CST) != SINT128_MIN)
+ abort ();
+
+ if (sv != SINT128_MIN)
+ abort ();
+}
+
+void
+test_fetch_umin ()
+{
+ __uint128_t uv = 100;
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED) != 100)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_min (&uv, UINT128_MAX, __ATOMIC_SEQ_CST) != 0)
+ abort ();
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_fetch_smax ()
+{
+ __int128_t sv = -10;
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED) != -10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE) != -5)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&sv, SINT128_MAX, __ATOMIC_SEQ_CST) != 50)
+ abort ();
+
+ if (sv != SINT128_MAX)
+ abort ();
+}
+
+void
+test_fetch_umax ()
+{
+ __uint128_t uv = 10;
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED) != 10)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE) != 50)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL) != 100)
+ abort ();
+
+ if (__atomic_fetch_max (&uv, UINT128_MAX, __ATOMIC_SEQ_CST) != 200)
+ abort ();
+
+ if (uv != UINT128_MAX)
+ abort ();
+}
+
/* The OP_fetch routines return the new value after the operation. */
void
@@ -330,7 +442,6 @@ test_or_fetch ()
abort ();
}
-
/* Test the OP routines with a result which isn't used. Use both variations
within each function. */
@@ -529,6 +640,130 @@ test_or ()
abort ();
}
+void
+test_smin ()
+{
+ __int128_t sv = 10;
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_RELAXED);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, 5, __ATOMIC_CONSUME);
+ if (sv != 5)
+ abort ();
+
+ __atomic_fetch_min (&sv, -10, __ATOMIC_ACQUIRE);
+ if (sv != -10)
+ abort ();
+
+ __atomic_fetch_min (&sv, -20, __ATOMIC_RELEASE);
+ if (sv != -20)
+ abort ();
+
+ __atomic_fetch_min (&sv, SINT128_MIN, __ATOMIC_ACQ_REL);
+ if (sv != SINT128_MIN)
+ abort ();
+
+ __atomic_fetch_min (&sv, SINT128_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SINT128_MIN)
+ abort ();
+}
+
+void
+test_umin ()
+{
+ __uint128_t uv = 100;
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 75, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_min (&uv, 10, __ATOMIC_RELEASE);
+ if (uv != 10)
+ abort ();
+
+ __atomic_fetch_min (&uv, 0, __ATOMIC_ACQ_REL);
+ if (uv != 0)
+ abort ();
+
+ __atomic_fetch_min (&uv, UINT128_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != 0)
+ abort ();
+}
+
+void
+test_smax ()
+{
+ __int128_t sv = -10;
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_RELAXED);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -5, __ATOMIC_CONSUME);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, -20, __ATOMIC_ACQUIRE);
+ if (sv != -5)
+ abort ();
+
+ __atomic_fetch_max (&sv, 10, __ATOMIC_RELEASE);
+ if (sv != 10)
+ abort ();
+
+ __atomic_fetch_max (&sv, 50, __ATOMIC_ACQ_REL);
+ if (sv != 50)
+ abort ();
+
+ __atomic_fetch_max (&sv, SINT128_MAX, __ATOMIC_SEQ_CST);
+
+ if (sv != SINT128_MAX)
+ abort ();
+}
+
+void
+test_umax ()
+{
+ __uint128_t uv = 10;
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_RELAXED);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 50, __ATOMIC_CONSUME);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 30, __ATOMIC_ACQUIRE);
+ if (uv != 50)
+ abort ();
+
+ __atomic_fetch_max (&uv, 100, __ATOMIC_RELEASE);
+ if (uv != 100)
+ abort ();
+
+ __atomic_fetch_max (&uv, 200, __ATOMIC_ACQ_REL);
+ if (uv != 200)
+ abort ();
+
+ __atomic_fetch_max (&uv, UINT128_MAX, __ATOMIC_SEQ_CST);
+
+ if (uv != UINT128_MAX)
+ abort ();
+}
+
int
main ()
{
@@ -538,6 +773,10 @@ main ()
test_fetch_nand ();
test_fetch_xor ();
test_fetch_or ();
+ test_fetch_smin ();
+ test_fetch_umin ();
+ test_fetch_smax ();
+ test_fetch_umax ();
test_add_fetch ();
test_sub_fetch ();
@@ -552,6 +791,10 @@ main ()
test_nand ();
test_xor ();
test_or ();
+ test_smin ();
+ test_umin ();
+ test_smax ();
+ test_umax ();
return 0;
}
diff --git a/gcc/tree-cfg.cc b/gcc/tree-cfg.cc
index 8fb80b8121e..e1c848c8920 100644
--- a/gcc/tree-cfg.cc
+++ b/gcc/tree-cfg.cc
@@ -8036,6 +8036,9 @@ move_sese_region_to_fn (struct function *dest_cfun,
basic_block entry_bb,
/* Adjust the maximum clique used. */
dest_cfun->last_clique = saved_cfun->last_clique;
+ /* Mark if this function contains an atomic IFN. */
+ dest_cfun->calls_atomic_ifn |= saved_cfun->calls_atomic_ifn;
+
loop->aux = NULL;
loop0->aux = NULL;
/* Loop sizes are no longer correct, fix them up. */
diff --git a/gcc/tree-inline.cc b/gcc/tree-inline.cc
index 302ab8d6b7c..27da06d405d 100644
--- a/gcc/tree-inline.cc
+++ b/gcc/tree-inline.cc
@@ -5095,6 +5095,7 @@ expand_call_inline (basic_block bb, gimple *stmt,
copy_body_data *id,
if (src_properties != prop_mask)
dst_cfun->curr_properties &= src_properties | ~prop_mask;
dst_cfun->calls_eh_return |= id->src_cfun->calls_eh_return;
+ dst_cfun->calls_atomic_ifn |= id->src_cfun->calls_atomic_ifn;
id->dst_node->has_omp_variant_constructs
|= id->src_node->has_omp_variant_constructs;
diff --git a/gcc/tree-pass.h b/gcc/tree-pass.h
index b3c97658a8f..be51227e6d3 100644
--- a/gcc/tree-pass.h
+++ b/gcc/tree-pass.h
@@ -422,6 +422,7 @@ extern gimple_opt_pass *make_pass_pre (gcc::context *ctxt);
extern unsigned int tail_merge_optimize (bool);
extern gimple_opt_pass *make_pass_profile (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_strip_predict_hints (gcc::context *ctxt);
+extern gimple_opt_pass *make_pass_lower_atomic_ifn (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_rebuild_frequencies (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lower_complex_O0 (gcc::context *ctxt);
extern gimple_opt_pass *make_pass_lower_complex (gcc::context *ctxt);
--
2.43.0
