From: Kyrylo Tkachov <[email protected]>
noce_convert_multiple_sets rejects an arm that sets the condition codes
and then uses them in a conditional select. A branch therefore stays
around updates such as this one from the hot loop of 707.ntest_r in
SPEC CPU 2026:
if (child_score > score)
{
score = child_score;
if (score > alpha)
alpha = score;
}
If an arm sets a hard condition-code register or contains a conditional
select, evaluate the arms into temporaries before the outer conditional
moves, keeping the instructions of each arm in order. If this changes
the condition codes that the jump tests, emit the outer comparison again
after the arms when it cannot trap and its inputs do not change, and
otherwise let each conditional move compute its own comparison, as
before. noce_emit_multiple_sets emits an arm this way and replaces the
code that evaluated the else arm first. When it copies an arm set by
set, it uses a source pseudo that neither arm sets instead of a copy.
Once the then arm has been evaluated this way, only the last set of each
register that is live after the block needs a conditional move.
A condition-code set counts towards the sets an arm must have, because
the other lowerings reject such an arm, so a single conditional select
with its comparison is now converted too. An arm that sets a
condition-code pseudo, leaves a condition-code register live on exit or
has no other set is rejected. Converting these arms keeps every
instruction of the arm and adds a conditional move for each output, so
they are only accepted in blocks that are optimized for speed. Arms
without condition-code sets or conditional selects are handled as
before, and the target profitability checks do not change.
For the loop above on aarch64 at -O2, ce1 used to keep
24: cc:CC=cmp(r101:SI,r111:SI)
25: pc={(cc:CC<=0)?L33:pc}
31: cc:CC=cmp(r112:SI,r101:SI)
32: r112:SI={(cc:CC>=0)?r112:SI:r101:SI}
8: r111:SI=r101:SI
33: L33:
and now emits
78: cc:CC=cmp(r112:SI,r101:SI)
79: r125:SI={(cc:CC>=0)?r112:SI:r101:SI}
80: cc:CC=cmp(r101:SI,r111:SI)
83: r112:SI={(cc:CC<=0)?r112:SI:r125:SI}
86: r111:SI={(cc:CC<=0)?r111:SI:r101:SI}
where r101 is child_score, r111 is score and r112 is alpha.
On an NVIDIA Vera system, with 707.ntest_r built at -O3 -ffast-math
-mcpu=olympus -flto=auto, the benchmark improves by 3.2% and branch
mispredictions fall by 15.7%.
Bootstrapped and tested on aarch64-linux-gnu and x86_64-linux-gnu.
Ok for trunk?
Thanks,
Kyrill
gcc/ChangeLog:
* ifcvt.cc (noce_convert_multiple_sets): Update comment.
(noce_multiple_sets_precompute_p): New function.
(noce_emit_multiple_sets): New function, split out of
noce_convert_multiple_sets_1. Copy blocks with condition-code sets
or conditional selects set by set. Fail if the instructions emitted
before the block change a hard register that is live into it.
(noce_convert_multiple_sets_1): Use it for the else arm, and for the
then arm when an arm sets a condition code or contains a conditional
select. Use the precomputed then-arm values, and emit a conditional
move only for the last set of each register that is live after the
block. When those instructions clobber the comparison at
cond_earliest, emit it again with its location and do not report a
use of cond_earliest.
(init_noce_multiple_sets_info): Skip condition-code definitions.
(bb_ok_for_noce_convert_multiple_sets): Accept hard condition-code
sets that are dead on exit, and conditional selects, outside a
PARALLEL and only when optimizing for speed. Count condition-code
sets towards the number of sets. Require a set that is not a
condition-code set.
gcc/testsuite/ChangeLog:
* gcc.dg/ifcvt-multiple-sets-cc-1.c: New test.
* gcc.dg/rtl/aarch64/ifcvt-multiple-sets-else-chain.c: New test.
* gcc.dg/rtl/x86_64/ifcvt-multiple-sets-else-chain.c: New test.
* gcc.target/aarch64/ifcvt_multiple_sets_cc_1.c: New test.
* gcc.target/i386/ifcvt-multiple-sets-cc-1.c: New test.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
gcc/ifcvt.cc | 344 ++++++++++++++----
.../gcc.dg/ifcvt-multiple-sets-cc-1.c | 219 +++++++++++
.../aarch64/ifcvt-multiple-sets-else-chain.c | 86 +++++
.../x86_64/ifcvt-multiple-sets-else-chain.c | 103 ++++++
.../aarch64/ifcvt_multiple_sets_cc_1.c | 46 +++
.../i386/ifcvt-multiple-sets-cc-1.c | 22 ++
6 files changed, 740 insertions(+), 80 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/ifcvt-multiple-sets-cc-1.c
create mode 100644
gcc/testsuite/gcc.dg/rtl/aarch64/ifcvt-multiple-sets-else-chain.c
create mode 100644
gcc/testsuite/gcc.dg/rtl/x86_64/ifcvt-multiple-sets-else-chain.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_cc_1.c
create mode 100644 gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-cc-1.c
diff --git a/gcc/ifcvt.cc b/gcc/ifcvt.cc
index fad341ead6c..92828583651 100644
--- a/gcc/ifcvt.cc
+++ b/gcc/ifcvt.cc
@@ -4078,8 +4078,10 @@ noce_finish_if_conversion (noce_if_info *if_info)
IF_INFO contains the useful information about the block structure and
jump instructions. For an IF-THEN-ELSE-JOIN, first evaluate the else
arm's sets into temporaries and retain their final values for the
- conditional moves. Return true if the replacement is valid and profitable
- and its CFG changes have been committed, otherwise return false. */
+ conditional moves. If either arm sets a condition code or contains a
+ conditional select, also evaluate the then arm's sets into temporaries
+ first. Return true if the replacement is valid and profitable and its
+ CFG changes have been committed, otherwise return false. */
static bool
noce_convert_multiple_sets (noce_if_info *if_info)
@@ -4257,11 +4259,132 @@ noce_seq_clobbers_cc_cmp_p (rtx_insn *seq, rtx cc_cmp,
rtx rev_cc_cmp)
return false;
}
+/* Return true if BB needs its computations emitted before the outer
+ conditional moves, to preserve comparisons used within the block. */
+
+static bool
+noce_multiple_sets_precompute_p (basic_block bb)
+{
+ rtx_insn *insn;
+ FOR_BB_INSNS (bb, insn)
+ if (active_insn_p (insn))
+ {
+ /* bb_ok_for_noce_convert_multiple_sets only accepts single sets. */
+ rtx set = single_set (insn);
+ gcc_checking_assert (set);
+
+ if (GET_MODE_CLASS (GET_MODE (SET_DEST (set))) == MODE_CC
+ || GET_CODE (SET_SRC (set)) == IF_THEN_ELSE)
+ return true;
+ }
+ return false;
+}
+
+/* Emit the sets of BB ahead of the conditional moves and record in
+ INSN_INFO the register that holds each value. DESTS holds the pseudo
+ registers that either arm sets, which include all registers that the
+ conditional moves assign. If COPY_SETS, copy each set in order, so that
+ condition-code definitions stay ahead of their users, and use a source
+ pseudo outside DESTS in place of a copy. Otherwise expand each value
+ into a new pseudo. Return false if the instructions emitted so far
+ change a hard register that is live on entry to BB, or if COPY_SETS and
+ a set in BB is part of a PARALLEL. */
+
+static bool
+noce_emit_multiple_sets (basic_block bb,
+ auto_delete_vec<noce_multiple_sets_info> &insn_info,
+ const_bitmap dests, bool copy_sets)
+{
+ int count = 0;
+ HARD_REG_SET modified;
+ CLEAR_HARD_REG_SET (modified);
+ for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
+ note_stores (insn, record_hard_reg_sets, &modified);
+
+ hard_reg_set_iterator hrsi;
+ unsigned regno;
+ EXECUTE_IF_SET_IN_HARD_REG_SET (modified, 0, regno, hrsi)
+ if (bitmap_bit_p (df_get_live_in (bb), regno))
+ return false;
+
+ rtx_insn *insn;
+ FOR_BB_INSNS (bb, insn)
+ {
+ if (!active_insn_p (insn))
+ continue;
+
+ /* bb_ok_for_noce_convert_multiple_sets only accepts single sets. */
+ rtx set = single_set (insn);
+ gcc_checking_assert (set);
+
+ rtx target = SET_DEST (set);
+ rtx value = copy_rtx (SET_SRC (set));
+ bool cc_p = GET_MODE_CLASS (GET_MODE (target)) == MODE_CC;
+ /* The rewiring below replaces the destinations of BB one at a time,
+ so no temporary may be one of them. */
+ gcc_checking_assert (cc_p
+ ? HARD_REGISTER_P (target)
+ : bitmap_bit_p (dests, REGNO (target)));
+
+ if (cc_p)
+ {
+ auto_bitmap seen;
+ for (int i = count - 1; i >= 0; --i)
+ if (bitmap_set_bit (seen, REGNO (insn_info[i]->target)))
+ value = simplify_replace_rtx (value, insn_info[i]->target,
+ insn_info[i]->temporary);
+ }
+ else
+ {
+ int i, ii;
+ FOR_EACH_VEC_ELT (insn_info[count]->rewired_src, i, ii)
+ value = simplify_replace_rtx (value, insn_info[ii]->target,
+ insn_info[ii]->temporary);
+ }
+
+ /* Copying only the SET would drop the rest of a PARALLEL. */
+ if (copy_sets && GET_CODE (PATTERN (insn)) != SET)
+ return false;
+
+ /* A value that no arm and no conditional move changes needs no
+ copy. */
+ if (copy_sets && !cc_p && REG_P (value) && !HARD_REGISTER_P (value)
+ && !bitmap_bit_p (dests, REGNO (value)))
+ {
+ insn_info[count++]->temporary = value;
+ continue;
+ }
+
+ rtx temporary;
+ location_t saved_location = curr_insn_location ();
+ set_curr_insn_location (INSN_LOCATION (insn));
+ if (copy_sets)
+ {
+ temporary = cc_p ? target : gen_reg_rtx (GET_MODE (target));
+ emit_insn (gen_rtx_SET (temporary, value));
+ }
+ else
+ temporary = copy_to_mode_reg (GET_MODE (target), value);
+ set_curr_insn_location (saved_location);
+ if (!cc_p)
+ {
+ gcc_checking_assert (rtx_equal_p (target, insn_info[count]->target));
+ insn_info[count++]->temporary = temporary;
+ }
+ }
+ return true;
+}
+
/* Try to emit the multiple-set conversion described by IF_INFO, selecting on
the already decoded jump condition COND. INSN_INFO holds THEN_COUNT
then-arm entries followed by entries for the else-only definitions recorded
in ELSE_ONLY_INDICES. For a diamond, evaluate ELSE_INSN_INFO first and
- retain its final values for the conditional moves.
+ retain its final values for the conditional moves. If either arm sets a
+ condition code or contains a conditional select, also evaluate the then
+ arm first, and if that changes the condition code that the jump tests,
+ emit the comparison at IF_INFO->cond_earliest again when it cannot trap
+ and its inputs are unchanged. Whenever the condition code changes and
+ is not restored this way, compare COND for each move.
LAST_NEEDS_COMPARISON is -1 on the first attempt. Record in it the last set
that needs a temporary to preserve the comparison, then use that boundary
@@ -4292,62 +4415,67 @@ noce_convert_multiple_sets_1 (noce_if_info *if_info,
rtx cond,
bool second_try = *last_needs_comparison != -1;
*use_cond_earliest = false;
- /* For an IF-THEN-ELSE-JOIN, emit the else block's computations first into
- fresh temporaries. This leaves the incoming register values available to
- the then arm. The conditional moves below select the then or else
- values. */
- if (if_info->else_bb)
- {
- int else_count = 0;
- rtx_insn *before_else = get_last_insn ();
- location_t saved_location = curr_insn_location ();
- rtx_insn *else_insn;
- FOR_BB_INSNS (if_info->else_bb, else_insn)
+ bool then_copy = noce_multiple_sets_precompute_p (if_info->then_bb);
+ bool else_copy = (if_info->else_bb
+ && noce_multiple_sets_precompute_p (if_info->else_bb));
+ bool precompute = then_copy || else_copy;
+ bool recreated_comparison = false;
+ if (if_info->else_bb || precompute)
+ {
+ auto_bitmap dests;
+ for (const noce_multiple_sets_info *info : insn_info)
+ bitmap_set_bit (dests, REGNO (info->target));
+ for (const noce_multiple_sets_info *info : else_insn_info)
+ bitmap_set_bit (dests, REGNO (info->target));
+
+ if ((if_info->else_bb
+ && !noce_emit_multiple_sets (if_info->else_bb, else_insn_info,
+ dests, else_copy))
+ || (precompute
+ && !noce_emit_multiple_sets (if_info->then_bb, insn_info,
+ dests, then_copy)))
{
- if (!active_insn_p (else_insn))
- continue;
-
- noce_multiple_sets_info *info = else_insn_info[else_count];
- rtx set = single_set (else_insn);
- gcc_checking_assert (set && REG_P (SET_DEST (set))
- && !HARD_REGISTER_P (SET_DEST (set)));
-
- rtx target = SET_DEST (set);
- rtx value = copy_rtx (SET_SRC (set));
- int i, ii;
- FOR_EACH_VEC_ELT (info->rewired_src, i, ii)
- value = simplify_replace_rtx (value,
- else_insn_info[ii]->target,
- else_insn_info[ii]->temporary);
-
- set_curr_insn_location (INSN_LOCATION (else_insn));
- rtx temporary = copy_to_mode_reg (GET_MODE (target), value);
-
- info->temporary = temporary;
- else_count++;
+ end_sequence ();
+ return false;
}
- set_curr_insn_location (saved_location);
-
- gcc_checking_assert (else_count == (int) else_insn_info.length ());
-
- /* These insns run ahead of the conditional moves. If they change a
- register the comparison reads we cannot reuse it, so bail. If they
- only clobber the condition code, drop the shared compare so that every
- move re-materializes its own. */
-
- rtx_insn *first_else
- = before_else ? NEXT_INSN (before_else) : get_insns ();
- for (rtx_insn *ei = first_else; ei; ei = NEXT_INSN (ei))
+ /* If these instructions change CC, emit the outer comparison again
+ after them, provided that it is a single comparison that cannot trap
+ and that its inputs are unchanged. Only do this when they include
+ the then arm. Otherwise the conditional moves compute the then
+ values and can change CC again. */
+ bool clobbered_cc = false;
+ rtx compare = PATTERN (if_info->cond_earliest);
+ bool can_recreate = (precompute
+ && cc_cmp
+ && GET_CODE (compare) == SET
+ && GET_CODE (SET_SRC (compare)) == COMPARE
+ && noce_operand_ok (SET_SRC (compare))
+ && reg_overlap_mentioned_p (SET_DEST (compare),
+ cc_cmp));
+ for (rtx_insn *insn = get_insns (); insn; insn = NEXT_INSN (insn))
{
- if (modified_in_p (cond, ei))
+ if (modified_in_p (cond, insn))
{
end_sequence ();
return false;
}
+ if (can_recreate && modified_in_p (SET_SRC (compare), insn))
+ can_recreate = false;
if (cc_cmp
- && (modified_in_p (cc_cmp, ei)
- || (rev_cc_cmp && modified_in_p (rev_cc_cmp, ei))))
+ && (modified_in_p (cc_cmp, insn)
+ || (rev_cc_cmp && modified_in_p (rev_cc_cmp, insn))))
+ clobbered_cc = true;
+ }
+ if (clobbered_cc)
+ {
+ if (can_recreate)
+ {
+ rtx_insn *copy = emit_insn (copy_rtx (compare));
+ INSN_LOCATION (copy) = INSN_LOCATION (if_info->cond_earliest);
+ recreated_comparison = true;
+ }
+ else
{
cc_cmp = NULL_RTX;
rev_cc_cmp = NULL_RTX;
@@ -4355,6 +4483,17 @@ noce_convert_multiple_sets_1 (noce_if_info *if_info, rtx
cond,
}
}
+ /* Record the last set of each register in the then arm. */
+ auto_sbitmap last_set (then_count);
+ bitmap_clear (last_set);
+ if (precompute)
+ {
+ auto_bitmap seen;
+ for (int i = then_count - 1; i >= 0; --i)
+ if (bitmap_set_bit (seen, REGNO (insn_info[i]->target)))
+ bitmap_set_bit (last_set, i);
+ }
+
for (count = 0; count < (int) insn_info.length (); ++count)
{
noce_multiple_sets_info *info = insn_info[count];
@@ -4367,14 +4506,31 @@ noce_convert_multiple_sets_1 (noce_if_info *if_info,
rtx cond,
rtx new_val;
rtx old_val;
- if ((unsigned) count < then_count)
+ /* With the then arm evaluated above, a set needs a conditional move
+ only if it is the last set of a register that is live after the
+ block. */
+ if (precompute
+ && (unsigned) count < then_count
+ && (!info->need_cmov || !bitmap_bit_p (last_set, count)))
{
- new_val = SET_SRC (set);
+ info->temporary = target;
+ continue;
+ }
- int i, ii;
- FOR_EACH_VEC_ELT (info->rewired_src, i, ii)
- new_val = simplify_replace_rtx (new_val, insn_info[ii]->target,
- insn_info[ii]->temporary);
+ if ((unsigned) count < then_count)
+ {
+ /* INFO still holds the value of the then arm computed above. Only
+ the end of this iteration overwrites it. */
+ if (precompute)
+ new_val = info->temporary;
+ else
+ {
+ new_val = SET_SRC (set);
+ int i, ii;
+ FOR_EACH_VEC_ELT (info->rewired_src, i, ii)
+ new_val = simplify_replace_rtx (new_val, insn_info[ii]->target,
+ insn_info[ii]->temporary);
+ }
old_val = target;
@@ -4547,7 +4703,8 @@ noce_convert_multiple_sets_1 (noce_if_info *if_info, rtx
cond,
temp_dest = temp_dest2;
if (!second_try && read_comparison)
*last_needs_comparison = count;
- *use_cond_earliest = true;
+ if (!recreated_comparison)
+ *use_cond_earliest = true;
}
else
{
@@ -4594,9 +4751,10 @@ noce_convert_multiple_sets_1 (noce_if_info *if_info, rtx
cond,
return true;
}
-/* Fill INSN_INFO with one entry per active insn in BB. Find local swap-style
- idioms and mark the first insn (1) that is only a temporary as not needing
- a conditional move, as it is going to be dead afterwards anyway.
+/* Fill INSN_INFO with one entry per pseudo-register definition in BB.
+ Find local swap-style idioms and mark the first insn (1) that is only a
+ temporary as not needing a conditional move, since its value is dead
+ after the block.
(1) int tmp = a;
a = b;
@@ -4641,15 +4799,17 @@ init_noce_multiple_sets_info (basic_block bb,
if (!active_insn_p (insn))
continue;
- noce_multiple_sets_info *info = new noce_multiple_sets_info;
- info->temporary = NULL_RTX;
- insn_info.safe_push (info);
-
rtx set = single_set (insn);
gcc_checking_assert (set);
rtx src = SET_SRC (set);
rtx dest = SET_DEST (set);
+ if (GET_MODE_CLASS (GET_MODE (dest)) == MODE_CC)
+ continue;
+
+ noce_multiple_sets_info *info = new noce_multiple_sets_info;
+ info->temporary = NULL_RTX;
+ insn_info.safe_push (info);
gcc_checking_assert (REG_P (dest) && !HARD_REGISTER_P (dest));
info->target = dest;
@@ -4673,9 +4833,12 @@ init_noce_multiple_sets_info (basic_block bb,
/* Return true iff basic block TEST_BB is suitable for conversion to a
series of conditional moves. Unless REQUIRE_MULTIPLE is false, also check
- that we have more than one set (other routines can handle a single set
- better than we would). A diamond arm may have a single set.
- Require no more than PARAM_MAX_RTL_IF_CONVERSION_INSNS sets. While going
+ that we have more than one set, counting condition-code sets (other
+ routines can handle a single set better than we would, but they reject a
+ block that sets the condition codes). A diamond arm may have a single set.
+ Require a set that is not a condition-code set and no more than
+ PARAM_MAX_RTL_IF_CONVERSION_INSNS sets, and accept condition-code sets and
+ conditional selects only if TEST_BB is optimized for speed. While going
through the insns, store the sum of their potential costs in COST. On
success, if LIVE_OUT_DESTS is nonnull, record the distinct pseudo
destinations that are live out of
@@ -4699,9 +4862,8 @@ bb_ok_for_noce_convert_multiple_sets (basic_block
test_bb, unsigned *cost,
bitmap_clear (live_out_dests);
if (has_non_simple_src)
*has_non_simple_src = false;
- bitmap bb_live_out = NULL;
- if (live_out_dests)
- bb_live_out = df_get_live_out (test_bb);
+ bitmap bb_live_out = df_get_live_out (test_bb);
+ unsigned outputs = 0;
FOR_BB_INSNS (test_bb, insn)
{
@@ -4720,26 +4882,47 @@ bb_ok_for_noce_convert_multiple_sets (basic_block
test_bb, unsigned *cost,
if (has_non_simple_src && !noce_simple_cmove_operand_p (src))
*has_non_simple_src = true;
+ bool cc_p = REG_P (dest) && GET_MODE_CLASS (GET_MODE (dest)) == MODE_CC;
+ bool select_p = GET_CODE (src) == IF_THEN_ELSE;
+
+ /* An arm with either of these is evaluated in full before the
+ conditional moves. That keeps every instruction of the arm and adds
+ a conditional move for each output, so it cannot make the code
+ smaller. Only do it when optimizing for speed. */
+ if ((cc_p || select_p) && !speed_p)
+ return false;
+
/* Dependency rewiring is keyed by register number, so restrict
- destinations to pseudos. Hard-register definitions can overlap
- without having the same mode. Do not handle anything involving
+ destinations to pseudos, apart from hard condition-code registers,
+ which are copied without renaming. Do not handle anything involving
memory loads/stores since it might violate data-race-freedom
- guarantees. Make sure we can force SRC to a register as that may
+ guarantees. Copy condition-code sets and conditional selects as they
+ are. Make sure we can force any other SRC to a register as that may
be needed in try_emit_cmove_seq. */
- if (!REG_P (dest) || HARD_REGISTER_P (dest)
+ if (!REG_P (dest) || (HARD_REGISTER_P (dest) && !cc_p)
+ || (cc_p && !HARD_REGISTER_P (dest))
|| contains_mem_rtx_p (src)
- || !noce_can_force_operand (src))
+ || ((cc_p || select_p) && GET_CODE (PATTERN (insn)) != SET)
+ || (!cc_p && !select_p && !noce_can_force_operand (src)))
return false;
+ if (cc_p)
+ for (unsigned regno = REGNO (dest); regno < END_REGNO (dest); ++regno)
+ if (bitmap_bit_p (bb_live_out, regno))
+ return false;
+
/* Destination and source must be appropriate. */
if (!noce_operand_ok (dest) || !noce_operand_ok (src))
return false;
/* We must be able to conditionally move in this mode. */
- if (!can_conditionally_move_p (GET_MODE (dest)))
+ if (!cc_p && !can_conditionally_move_p (GET_MODE (dest)))
return false;
- if (bb_live_out && bitmap_bit_p (bb_live_out, REGNO (dest)))
+ if (!cc_p)
+ outputs++;
+ if (live_out_dests && !cc_p
+ && bitmap_bit_p (bb_live_out, REGNO (dest)))
bitmap_set_bit (live_out_dests, REGNO (dest));
potential_cost += insn_cost (insn, speed_p);
@@ -4751,12 +4934,13 @@ bb_ok_for_noce_convert_multiple_sets (basic_block
test_bb, unsigned *cost,
/* If we would only put out one conditional move, the other strategies
this pass tries are better optimized and will be more appropriate, so
- require more than one set unless REQUIRE_MULTIPLE is false. A diamond
- arm may have one set.
+ require more than one set unless REQUIRE_MULTIPLE is false. Those
+ strategies reject an arm that sets a condition code, so such sets count
+ here. A diamond arm may have one set.
Some targets want to strictly limit the number of conditional moves
that are emitted, they set this through PARAM, we need to respect
that. */
- bool ok = count >= (require_multiple ? 2u : 1u) && count <= param;
+ bool ok = outputs && count >= (require_multiple ? 2u : 1u) && count <= param;
if (ok && insn_count)
*insn_count = count;
return ok;
diff --git a/gcc/testsuite/gcc.dg/ifcvt-multiple-sets-cc-1.c
b/gcc/testsuite/gcc.dg/ifcvt-multiple-sets-cc-1.c
new file mode 100644
index 00000000000..71a236b7b04
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/ifcvt-multiple-sets-cc-1.c
@@ -0,0 +1,219 @@
+/* Check arms that hold a comparison and a dependent conditional select. */
+/* { dg-do run } */
+/* { dg-options "-O3 --param=max-rtl-if-conversion-unpredictable-cost=100" } */
+/* { dg-additional-options "-fdump-rtl-ce1" { target aarch64*-*-* } } */
+/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through
noce_convert_multiple_sets" 5 "ce1" { target aarch64*-*-* } } } */
+
+#ifdef __aarch64__
+/* Keep MAX and MIN as a comparison and a conditional select. */
+#pragma GCC target "+nocssc"
+#endif
+
+struct result { int score, alpha; };
+
+__attribute__ ((noipa)) struct result
+update_scores (const int *children, unsigned int count, int beta,
+ int score, int alpha)
+{
+ for (unsigned int i = 0; i < count; ++i)
+ {
+ int child_score = children[i];
+ if (child_score >= beta)
+ return (struct result) { child_score, alpha };
+ if (child_score > score)
+ {
+ score = child_score;
+ if (score > alpha)
+ alpha = score;
+ }
+ }
+ return (struct result) { score, alpha };
+}
+
+volatile int out_x, out_y;
+
+__attribute__ ((noipa)) void
+dependent (int c, int a, int b, int x, int y)
+{
+ if (x > c)
+ {
+ x = a > b ? a : b;
+ y = x + y;
+ x = x > y ? x : y;
+ }
+ else
+ {
+ y = a < b ? a : b;
+ x = x - y;
+ }
+ out_x = x;
+ out_y = y;
+}
+
+/* Keep both inputs available while the arm results are selected. */
+__attribute__ ((noipa)) void
+snapshots (unsigned int n, int c, int a, int b, int x, int y)
+{
+ while (n--)
+ if (x > c)
+ {
+ int old = x;
+ x = y;
+ y = a > b ? a : b;
+ y += old;
+ }
+ else
+ {
+ y = x;
+ x = a < b ? a : b;
+ }
+ out_x = x;
+ out_y = y;
+}
+
+/* The outer comparison can trap, so each select tests the condition. */
+__attribute__ ((noipa)) void
+fp_outer (double d, double e, int a, int b, int x, int y)
+{
+ if (!(d > e))
+ {
+ x = a > b ? a : b;
+ y = x + y;
+ }
+ out_x = x;
+ out_y = y;
+}
+
+/* Only the then arm has a comparison. */
+__attribute__ ((noipa)) void
+mixed (int c, int a, int b, int x, int y)
+{
+ if (x > c)
+ {
+ x = a > b ? a : b;
+ y = x + y;
+ }
+ else
+ {
+ y = x + 1;
+ x = a - b;
+ }
+ out_x = x;
+ out_y = y;
+}
+
+__attribute__ ((optimize ("O0"))) int
+main (void)
+{
+ static const int children[] = { -9, 4, 2, 8, -3, 12, 6 };
+ for (unsigned int n = 0; n <= 7; ++n)
+ for (int beta = -10; beta <= 14; ++beta)
+ for (int score = -10; score <= 14; score += 3)
+ for (int alpha = -10; alpha <= 14; alpha += 3)
+ {
+ int s = score, a = alpha;
+ for (unsigned int i = 0; i < n; ++i)
+ {
+ int v = children[i];
+ if (v >= beta)
+ {
+ s = v;
+ break;
+ }
+ if (v <= s)
+ continue;
+ s = v;
+ if (a < v)
+ a = v;
+ }
+ struct result r = update_scores (children, n, beta, score, alpha);
+ if (r.score != s || r.alpha != a)
+ __builtin_abort ();
+ }
+
+ for (int c = -1; c <= 1; ++c)
+ for (int a = -3; a <= 3; ++a)
+ for (int b = -3; b <= 3; ++b)
+ for (int x = -3; x <= 3; ++x)
+ for (int y = -3; y <= 3; ++y)
+ {
+ int p, q;
+ if (x > c)
+ {
+ p = a;
+ if (b > p)
+ p = b;
+ q = p + y;
+ if (q > p)
+ p = q;
+ }
+ else
+ {
+ q = a;
+ if (b < q)
+ q = b;
+ p = x - q;
+ }
+ dependent (c, a, b, x, y);
+ if (out_x != p || out_y != q)
+ __builtin_abort ();
+
+ p = x;
+ q = y;
+ for (unsigned int i = 0; i < 3; ++i)
+ if (p > c)
+ {
+ int old = p;
+ p = q;
+ q = a > b ? a : b;
+ q += old;
+ }
+ else
+ {
+ q = p;
+ p = a < b ? a : b;
+ }
+ snapshots (3, c, a, b, x, y);
+ if (out_x != p || out_y != q)
+ __builtin_abort ();
+ }
+
+ static const double v[] = { -1.0, 0.0, 1.0, __builtin_nan ("") };
+ for (int i = 0; i < 4; ++i)
+ for (int j = 0; j < 4; ++j)
+ for (int a = -2; a <= 2; ++a)
+ for (int b = -2; b <= 2; ++b)
+ {
+ int p = 5, q = 7;
+ if (!(v[i] > v[j]))
+ {
+ p = a > b ? a : b;
+ q = p + q;
+ }
+ fp_outer (v[i], v[j], a, b, 5, 7);
+ if (out_x != p || out_y != q)
+ __builtin_abort ();
+ }
+
+ for (int c = -1; c <= 1; ++c)
+ for (int a = -2; a <= 2; ++a)
+ for (int b = -2; b <= 2; ++b)
+ for (int x = -2; x <= 2; ++x)
+ {
+ int p, q;
+ if (x > c)
+ {
+ p = a > b ? a : b;
+ q = p + 3;
+ }
+ else
+ {
+ q = x + 1;
+ p = a - b;
+ }
+ mixed (c, a, b, x, 3);
+ if (out_x != p || out_y != q)
+ __builtin_abort ();
+ }
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/ifcvt-multiple-sets-else-chain.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/ifcvt-multiple-sets-else-chain.c
new file mode 100644
index 00000000000..b00aa6771ce
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/ifcvt-multiple-sets-else-chain.c
@@ -0,0 +1,86 @@
+/* { dg-do run { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fdump-rtl-ce1" } */
+
+/* The else arm copies r to x, redefines r and then reads x through y
+ next to the new r. r is dead after the else arm. It runs when c == 0,
+ so GE holds for the flags of the outer comparison and z is the incoming
+ r. noce_convert_multiple_sets must not rewire the use of the incoming r
+ to the new r. */
+
+int __RTL (startwith ("ce1")) f (int c, int r, int p)
+{
+(function "f"
+ (param "c"
+ (DECL_RTL (reg/v:SI <0> [ c ]))
+ (DECL_RTL_INCOMING (reg:SI x0 [ c ])))
+ (param "r"
+ (DECL_RTL (reg/v:SI <1> [ r ]))
+ (DECL_RTL_INCOMING (reg:SI x1 [ r ])))
+ (param "p"
+ (DECL_RTL (reg/v:SI <2> [ p ]))
+ (DECL_RTL_INCOMING (reg:SI x2 [ p ])))
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 3 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 4 (set (reg/v:SI <0> [ c ]) (reg:SI x0 [ c ])))
+ (cinsn 5 (set (reg/v:SI <1> [ r ]) (reg:SI x1 [ r ])))
+ (cinsn 6 (set (reg/v:SI <2> [ p ]) (reg:SI x2 [ p ])))
+ (cnote 7 NOTE_INSN_FUNCTION_BEG)
+ (cinsn 8 (set (reg:SI <6>) (const_int 7)))
+ (cinsn 9 (set (reg:CC cc)
+ (compare:CC (reg/v:SI <0> [ c ]) (const_int 0))))
+ (cjump_insn 10 (set (pc)
+ (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 20)
+ (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ ) ;; block 2
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 11 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 12 (set (reg:SI <5>) (const_int 100)))
+ (cinsn 13 (set (reg:SI <6>) (reg/v:SI <2> [ p ])))
+ (edge-to 5 (flags "FALLTHRU"))
+ ) ;; block 3
+ (block 4
+ (edge-from 2)
+ (clabel 20 2)
+ (cnote 21 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:SI <3>) (reg/v:SI <1> [ r ])))
+ (cinsn 23 (set (reg/v:SI <1> [ r ])
+ (plus:SI (reg/v:SI <2> [ p ]) (const_int 1))))
+ (cinsn 24 (set (reg:SI <4>) (reg:SI <3>)))
+ (cinsn 25 (set (reg:SI <5>)
+ (if_then_else:SI (ge (reg:CC cc) (const_int 0))
+ (reg:SI <4>)
+ (reg/v:SI <1> [ r ]))))
+ (edge-to 5 (flags "FALLTHRU"))
+ ) ;; block 4
+ (block 5
+ (edge-from 3 (flags "FALLTHRU"))
+ (edge-from 4 (flags "FALLTHRU"))
+ (cnote 31 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (set (reg:SI <7>) (ashift:SI (reg:SI <5>) (const_int 8))))
+ (cinsn 33 (set (reg:SI <8>) (plus:SI (reg:SI <7>) (reg:SI <6>))))
+ (cinsn 34 (set (reg/i:SI x0) (reg:SI <8>)))
+ (cinsn 35 (use (reg/i:SI x0)))
+ (edge-to exit (flags "FALLTHRU"))
+ ) ;; block 5
+ ) ;; insn-chain
+) ;; function
+}
+
+int
+main (void)
+{
+ for (int c = 0; c < 2; c++)
+ for (int r = -3; r <= 3; r++)
+ for (int p = -3; p <= 3; p++)
+ if (f (c, r, p) != (c ? 100 * 256 + p : r * 256 + 7))
+ __builtin_abort ();
+ return 0;
+}
+
+/* { dg-final { scan-rtl-dump "if-conversion succeeded through
noce_convert_multiple_sets" "ce1" } } */
diff --git a/gcc/testsuite/gcc.dg/rtl/x86_64/ifcvt-multiple-sets-else-chain.c
b/gcc/testsuite/gcc.dg/rtl/x86_64/ifcvt-multiple-sets-else-chain.c
new file mode 100644
index 00000000000..c4777dede29
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/x86_64/ifcvt-multiple-sets-else-chain.c
@@ -0,0 +1,103 @@
+/* { dg-do run { target { { i?86-*-* x86_64-*-* } && lp64 } } } */
+/* { dg-options "-O2 -mbranch-cost=5
--param=max-rtl-if-conversion-unpredictable-cost=100 -fdump-rtl-ce1" } */
+
+/* The else arm copies r to x, redefines r and then compares y, a copy
+ of x, with the new r. r is dead after the else arm. The comparison
+ must see the incoming r, so z is (r > p ? a : b). */
+
+int __RTL (startwith ("ce1")) g (int c, int r, int p, int a, int b)
+{
+(function "g"
+ (param "c"
+ (DECL_RTL (reg/v:SI <0> [ c ]))
+ (DECL_RTL_INCOMING (reg:SI di [ c ])))
+ (param "r"
+ (DECL_RTL (reg/v:SI <1> [ r ]))
+ (DECL_RTL_INCOMING (reg:SI si [ r ])))
+ (param "p"
+ (DECL_RTL (reg/v:SI <2> [ p ]))
+ (DECL_RTL_INCOMING (reg:SI dx [ p ])))
+ (param "a"
+ (DECL_RTL (reg/v:SI <9> [ a ]))
+ (DECL_RTL_INCOMING (reg:SI cx [ a ])))
+ (param "b"
+ (DECL_RTL (reg/v:SI <10> [ b ]))
+ (DECL_RTL_INCOMING (reg:SI r8 [ b ])))
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 3 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 4 (set (reg/v:SI <0> [ c ]) (reg:SI di [ c ])))
+ (cinsn 5 (set (reg/v:SI <1> [ r ]) (reg:SI si [ r ])))
+ (cinsn 6 (set (reg/v:SI <2> [ p ]) (reg:SI dx [ p ])))
+ (cinsn 36 (set (reg/v:SI <9> [ a ]) (reg:SI cx [ a ])))
+ (cinsn 37 (set (reg/v:SI <10> [ b ]) (reg:SI r8 [ b ])))
+ (cnote 7 NOTE_INSN_FUNCTION_BEG)
+ (cinsn 8 (set (reg:SI <6>) (const_int 7)))
+ (cinsn 9 (set (reg:CCZ flags)
+ (compare:CCZ (reg/v:SI <0> [ c ]) (const_int 0))))
+ (cjump_insn 10 (set (pc)
+ (if_then_else (eq (reg:CCZ flags) (const_int 0))
+ (label_ref 20)
+ (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ ) ;; block 2
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 11 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 12 (set (reg:SI <5>) (reg/v:SI <9> [ a ])))
+ (cinsn 13 (set (reg:SI <6>) (reg/v:SI <2> [ p ])))
+ (edge-to 5 (flags "FALLTHRU"))
+ ) ;; block 3
+ (block 4
+ (edge-from 2)
+ (clabel 20 2)
+ (cnote 21 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:SI <3>) (reg/v:SI <1> [ r ])))
+ (cinsn 23 (set (reg/v:SI <1> [ r ]) (reg/v:SI <2> [ p ])))
+ (cinsn 24 (set (reg:SI <4>) (reg:SI <3>)))
+ (cinsn 25 (set (reg:CCGC flags)
+ (compare:CCGC (reg:SI <4>) (reg/v:SI <1> [ r ]))))
+ (cinsn 26 (set (reg:SI <5>)
+ (if_then_else:SI (gt (reg:CCGC flags) (const_int 0))
+ (reg/v:SI <9> [ a ])
+ (reg/v:SI <10> [ b ]))))
+ (edge-to 5 (flags "FALLTHRU"))
+ ) ;; block 4
+ (block 5
+ (edge-from 3 (flags "FALLTHRU"))
+ (edge-from 4 (flags "FALLTHRU"))
+ (cnote 31 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (parallel [(set (reg:SI <7>)
+ (ashift:SI (reg:SI <5>) (const_int 8)))
+ (clobber (reg:CC flags))]))
+ (cinsn 33 (parallel [(set (reg:SI <8>)
+ (plus:SI (reg:SI <7>) (reg:SI <6>)))
+ (clobber (reg:CC flags))]))
+ (cinsn 34 (set (reg/i:SI ax) (reg:SI <8>)))
+ (cinsn 35 (use (reg/i:SI ax)))
+ (edge-to exit (flags "FALLTHRU"))
+ ) ;; block 5
+ ) ;; insn-chain
+ (crtl
+ (return_rtx
+ (reg/i:SI ax)
+ ) ;; return_rtx
+ ) ;; crtl
+) ;; function
+}
+
+int
+main (void)
+{
+ for (int c = 0; c < 2; c++)
+ for (int r = -3; r <= 3; r++)
+ for (int p = -3; p <= 3; p++)
+ if (g (c, r, p, 11, 22)
+ != (c ? (11 << 8) + p : ((r > p ? 11 : 22) << 8) + 7))
+ __builtin_abort ();
+ return 0;
+}
+
+/* { dg-final { scan-rtl-dump "if-conversion succeeded through
noce_convert_multiple_sets" "ce1" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_cc_1.c
b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_cc_1.c
new file mode 100644
index 00000000000..ad35734d5fa
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/ifcvt_multiple_sets_cc_1.c
@@ -0,0 +1,46 @@
+/* Convert arms that hold a comparison and a conditional select. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -fdump-rtl-ce1" } */
+
+/* Keep MAX as a comparison and a conditional select. */
+#pragma GCC target "+nocssc"
+
+struct result { int score, alpha; };
+
+struct result
+update_scores (const int *children, unsigned int count, int beta,
+ int score, int alpha)
+{
+ for (unsigned int i = 0; i < count; ++i)
+ {
+ int child_score = children[i];
+ if (child_score >= beta)
+ return (struct result) { child_score, alpha };
+ if (child_score > score)
+ {
+ score = child_score;
+ if (score > alpha)
+ alpha = score;
+ }
+ }
+ return (struct result) { score, alpha };
+}
+
+/* An arm with one output and the comparison it needs. No other lowering
+ accepts an arm that sets the condition codes. */
+
+int
+single_output (const int *pc, const int *pd, int y, int n)
+{
+ int x = 0;
+ for (int i = 0; i < n; i++)
+ {
+ int c = pc[i], d = pd[i];
+ if (c > d)
+ x = x > y ? x : y;
+ x += c;
+ }
+ return x;
+}
+
+/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through
noce_convert_multiple_sets" 2 "ce1" } } */
diff --git a/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-cc-1.c
b/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-cc-1.c
new file mode 100644
index 00000000000..f9fc83c428a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/ifcvt-multiple-sets-cc-1.c
@@ -0,0 +1,22 @@
+/* Convert an arm that holds a comparison and a conditional move. */
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -mtune=generic -fdump-rtl-ce1" } */
+
+struct result { int score, best; };
+
+struct result
+update_best (const int *children, unsigned int count, int score, int best)
+{
+ for (unsigned int i = 0; i < count; ++i)
+ {
+ int child = children[i];
+ if (child > score)
+ {
+ score = child;
+ best = child > best ? (int) i : best;
+ }
+ }
+ return (struct result) { score, best };
+}
+
+/* { dg-final { scan-rtl-dump-times "if-conversion succeeded through
noce_convert_multiple_sets" 2 "ce1" } } */
--
2.54.0 (Apple Git-157)
