This patch adds vectorization support to early-break loops with gswitch
statements. Such gswitches may come from original switch-case constructs
in the source or the iftoswitch pass which rewrites if conditions with a
chain of comparisons, like below, to gswitch statements.
if (a[i] == c1 || a[i] == c2 || a[i] == c3)
In this patch, the loop exit condition type is generalized from gcond to
gimple, so that gswitch statements can be treated as valid loop exits. A
new vectorizer pattern recognization function is introduced to check if
all non-default labels of the gswitch have the same destination. If yes,
it lowers the gswitch to an equivalent gcond. To avoid excessive compile
time, a new parameter vect-max-compare-in-switch-lowering is introduced
to limit the maximum number of comparisons in the gswitch lowering. In
addition, the CRC loop optimization is adjusted a bit to make sure CRC
loops still have gcond exits.
This patch is bootstrapped and regression-tested on x86_64-linux-gnu,
arm-linux-gnueabihf and aarch64-linux-gnu.
gcc/ChangeLog:
* doc/invoke.texi: Document vect-max-compare-in-switch-lowering.
* gimple-crc-optimization.cc (crc_optimization::loop_may_calculate_crc):
Ensure CRC loop exits are still gconds.
(crc_optimization::optimize_crc_loop): Cast CRC loop exits.
* params.opt: Add option vect-max-compare-in-switch-lowering.
* tree-scalar-evolution.cc (scev_dfs::follow_ssa_edge_expr):
Generalize loop exit conditions type from gcond to gimple.
(get_loop_exit_condition): Likewise.
* tree-scalar-evolution.h (get_loop_exit_condition): Likewise.
* tree-vect-loop-manip.cc (vect_set_loop_condition_normal):
Likewise.
(vect_set_loop_condition): Likewise.
(slpeel_can_duplicate_loop_p): Likewise.
* tree-vect-loop.cc (vect_get_loop_niters): Likewise.
(vect_analyze_loop_form): Likewise.
(vect_create_loop_vinfo): Likewise.
* tree-vect-patterns.cc (vect_recog_gswitch_pattern): New
pattern recognition function for gswitch early exits.
* tree-vect-slp.cc (vect_analyze_slp): Bail out if a gswitch is
not lowered into a gcond.
* tree-vect-stmts.cc (vect_mark_stmts_to_be_vectorized): Allow
gswitch statements as loop exits.
(vectorizable_early_exit): Transform gswitch early exits.
* tree-vectorizer.h (struct vect_loop_form_info): Allow gswitch
statements as loop exits.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/vect-switch-search-line-fast.c: Update the test as
the loop is now vectorizable on multiple targets.
* gcc.dg/vect/vect-early-break_139-switch1.c: New test.
* gcc.dg/vect/vect-early-break_139-switch2.c: New test.
* gcc.dg/vect/vect-early-break_139-switch3.c: New test.
* gcc.dg/vect/vect-early-break_139-switch4.c: New test.
---
gcc/doc/invoke.texi | 4 +
gcc/gimple-crc-optimization.cc | 9 +-
gcc/params.opt | 4 +
.../vect/vect-early-break_139-switch1.c | 19 +++
.../vect/vect-early-break_139-switch2.c | 21 ++++
.../vect/vect-early-break_139-switch3.c | 24 ++++
.../vect/vect-early-break_139-switch4.c | 26 ++++
.../vect/vect-switch-search-line-fast.c | 3 +-
gcc/tree-scalar-evolution.cc | 12 +-
gcc/tree-scalar-evolution.h | 4 +-
gcc/tree-vect-loop-manip.cc | 6 +-
gcc/tree-vect-loop.cc | 10 +-
gcc/tree-vect-patterns.cc | 111 ++++++++++++++++++
gcc/tree-vect-slp.cc | 11 +-
gcc/tree-vect-stmts.cc | 74 ++++++++----
gcc/tree-vectorizer.h | 8 +-
16 files changed, 295 insertions(+), 51 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c
create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 3cf326cc22c..def84a6675a 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -16652,6 +16652,10 @@ Complex expressions slow the analyzer.
Maximum number of arguments in a PHI supported by TREE if conversion
unless the loop is marked with simd pragma.
+@item vect-max-compare-in-switch-lowering
+The maximum number of comparisons that can be performed when lowering switch
+statements in the vectorizer.
+
@item vect-max-layout-candidates
The maximum number of possible vector layouts (such as permutations)
to consider when optimizing to-be-vectorized code.
diff --git a/gcc/gimple-crc-optimization.cc b/gcc/gimple-crc-optimization.cc
index 1751be9bc97..986aa193893 100644
--- a/gcc/gimple-crc-optimization.cc
+++ b/gcc/gimple-crc-optimization.cc
@@ -937,6 +937,12 @@ crc_optimization::loop_may_calculate_crc (class loop *loop)
return false;
m_crc_loop = loop;
+
+ /* Filter out the loops, which don't have a gcond as single exit. */
+ gimple *loop_exit_cond = get_loop_exit_condition (m_crc_loop);
+ if (loop_exit_cond == NULL || !is_a <gcond *> (loop_exit_cond))
+ return false;
+
basic_block *loop_bbs = get_loop_body_in_dom_order (m_crc_loop);
/* Filter out the cases, which don't have exactly two conditions in the loop.
@@ -1274,7 +1280,8 @@ crc_optimization::optimize_crc_loop (gphi *output_crc)
remove_phi_node (&tmp_gsi, false);
/* Alter the exit condition of the loop to always exit. */
- gcond* loop_exit_cond = get_loop_exit_condition (m_crc_loop);
+ gcond *loop_exit_cond
+ = safe_dyn_cast <gcond *> (get_loop_exit_condition (m_crc_loop));
gimple_cond_make_false (loop_exit_cond);
update_stmt (loop_exit_cond);
return true;
diff --git a/gcc/params.opt b/gcc/params.opt
index dd53d830895..115400ee8d2 100644
--- a/gcc/params.opt
+++ b/gcc/params.opt
@@ -1229,6 +1229,10 @@ Whether to use canonical types.
Common Joined UInteger Var(param_vect_epilogues_nomask) Init(1)
IntegerRange(0, 1) Param Optimization NoOffload
Enable loop epilogue vectorization using smaller vector size.
+-param=vect-max-compare-in-switch-lowering=
+Common Joined UInteger Var(param_vect_max_compare_in_switch_lowering) Init(16)
Param Optimization
+Maximum number of comparisons that can be performed when lowering switch
statements in the vectorizer.
+
-param=vect-max-layout-candidates=
Common Joined UInteger Var(param_vect_max_layout_candidates) Init(32) Param
Optimization
Maximum number of possible vector layouts (such as permutations) to consider
when optimizing to-be-vectorized code.
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c
new file mode 100644
index 00000000000..bbe75d20116
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c
@@ -0,0 +1,19 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast" } */
+
+int find (int *a, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ /* This is converted to a switch statement in the iftoswitch pass. */
+ if (a[i] == 1 || a[i] == 2 || a[i] == 5 || a[i] == 7)
+ return i;
+ }
+ return -1;
+}
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c
new file mode 100644
index 00000000000..d200bea4594
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c
@@ -0,0 +1,21 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast" } */
+
+int find (int *a, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ /* This is converted to a switch statement in the iftoswitch pass. */
+ if (a[i] == 1 || a[i] == 3 || a[i] == 4 || a[i] == 5)
+ continue;
+ else
+ return i;
+ }
+ return -1;
+}
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c
new file mode 100644
index 00000000000..f696267a855
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c
@@ -0,0 +1,24 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast" } */
+
+int find (int *a, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ /* An early break in case labels of a switch statement. */
+ switch (a[i])
+ {
+ case -1 ... 1:
+ case 10:
+ case 20:
+ return i;
+ }
+ }
+ return -1;
+}
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c
new file mode 100644
index 00000000000..f8e3187843d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c
@@ -0,0 +1,26 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+/* { dg-additional-options "-Ofast" } */
+
+int find (int *a, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ /* An early break in the default label of a switch statement. */
+ switch (a[i])
+ {
+ case 1 ... 3:
+ case 5:
+ case 10:
+ continue;
+ default:
+ return i;
+ }
+ }
+ return -1;
+}
+
+/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c
b/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c
index 678512db319..17e80330daa 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c
@@ -16,5 +16,4 @@ const unsigned char *search_line_fast2 (const unsigned char
*s,
return s;
}
-/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target {
ilp32 || { amdgcn*-*-* } } } } } */
-/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 0 "vect" { target {
! { ilp32 || { amdgcn*-*-* } } } } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" } } */
diff --git a/gcc/tree-scalar-evolution.cc b/gcc/tree-scalar-evolution.cc
index ecdef7529a6..0a5172c3be4 100644
--- a/gcc/tree-scalar-evolution.cc
+++ b/gcc/tree-scalar-evolution.cc
@@ -1302,7 +1302,7 @@ scev_dfs::follow_ssa_edge_expr (gimple *at_stmt, tree
expr,
guards the exit edge. If the expression is too difficult to
analyze, then give up. */
-gcond *
+gimple *
get_loop_exit_condition (const class loop *loop)
{
return get_loop_exit_condition (single_exit (loop));
@@ -1311,16 +1311,20 @@ get_loop_exit_condition (const class loop *loop)
/* If the statement just before the EXIT_EDGE contains a condition then
return the condition, otherwise NULL. */
-gcond *
+gimple *
get_loop_exit_condition (const_edge exit_edge)
{
- gcond *res = NULL;
+ gimple *res = NULL;
if (dump_file && (dump_flags & TDF_SCEV))
fprintf (dump_file, "(get_loop_exit_condition \n ");
if (exit_edge)
- res = safe_dyn_cast <gcond *> (*gsi_last_bb (exit_edge->src));
+ {
+ gimple *g = *gsi_last_bb (exit_edge->src);
+ if (is_a <gcond *> (g) || is_a <gswitch *> (g))
+ res = g;
+ }
if (dump_file && (dump_flags & TDF_SCEV))
{
diff --git a/gcc/tree-scalar-evolution.h b/gcc/tree-scalar-evolution.h
index c7feeacf1db..54f0bf82c7b 100644
--- a/gcc/tree-scalar-evolution.h
+++ b/gcc/tree-scalar-evolution.h
@@ -22,8 +22,8 @@ along with GCC; see the file COPYING3. If not see
#define GCC_TREE_SCALAR_EVOLUTION_H
extern tree number_of_latch_executions (class loop *);
-extern gcond *get_loop_exit_condition (const class loop *);
-extern gcond *get_loop_exit_condition (const_edge);
+extern gimple *get_loop_exit_condition (const class loop *);
+extern gimple *get_loop_exit_condition (const_edge);
extern void scev_initialize (void);
extern bool scev_initialized_p (void);
diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
index 20141dbc2e5..bd04252da2d 100644
--- a/gcc/tree-vect-loop-manip.cc
+++ b/gcc/tree-vect-loop-manip.cc
@@ -1233,7 +1233,7 @@ vect_set_loop_condition_normal (loop_vec_info /*
loop_vinfo */, edge exit_edge,
{
tree indx_before_incr, indx_after_incr;
gcond *cond_stmt;
- gcond *orig_cond;
+ gimple *orig_cond;
edge pe = loop_preheader_edge (loop);
gimple_stmt_iterator incr_gsi;
bool insert_after;
@@ -1395,7 +1395,7 @@ vect_set_loop_condition (class loop *loop, edge loop_e,
loop_vec_info loop_vinfo
bool niters_maybe_zero)
{
gcond *cond_stmt;
- gcond *orig_cond = get_loop_exit_condition (loop_e);
+ gimple *orig_cond = get_loop_exit_condition (loop_e);
gimple_stmt_iterator loop_cond_gsi = gsi_for_stmt (orig_cond);
if (loop_vinfo && LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo))
@@ -2039,7 +2039,7 @@ slpeel_can_duplicate_loop_p (const class loop *loop,
const_edge exit_e,
const_edge e)
{
edge entry_e = loop_preheader_edge (loop);
- gcond *orig_cond = get_loop_exit_condition (exit_e);
+ gimple *orig_cond = get_loop_exit_condition (exit_e);
gimple_stmt_iterator loop_exit_gsi = gsi_last_bb (exit_e->src);
/* All loops have an outer scope; the only case loop->outer is NULL is for
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index b58e4355e58..02db9f5a2eb 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -627,12 +627,12 @@ vect_fixup_scalar_cycles_with_patterns (loop_vec_info
loop_vinfo)
Return the loop exit conditions. */
-static vec<gcond *>
+static vec<gimple *>
vect_get_loop_niters (class loop *loop, const_edge main_exit, tree
*assumptions,
tree *number_of_iterations, tree *number_of_iterationsm1)
{
auto_vec<edge> exits = get_loop_exit_edges (loop);
- vec<gcond *> conds;
+ vec<gimple *> conds;
conds.create (exits.length ());
class tree_niter_desc niter_desc;
tree niter_assumptions, niter, may_be_zero;
@@ -654,7 +654,7 @@ vect_get_loop_niters (class loop *loop, const_edge
main_exit, tree *assumptions,
unsigned int i;
FOR_EACH_VEC_ELT (exits, i, exit)
{
- gcond *cond = get_loop_exit_condition (exit);
+ gimple *cond = get_loop_exit_condition (exit);
if (cond)
conds.safe_push (cond);
@@ -1680,7 +1680,7 @@ vect_analyze_loop_form (class loop *loop, gimple
*loop_vectorized_call,
/* Determine what the primary and alternate exit conds are. */
for (unsigned i = 0; i < info->conds.length (); i++)
{
- gcond *cond = info->conds[i];
+ gimple *cond = info->conds[i];
if (exit_e->src == gimple_bb (cond))
std::swap (info->conds[0], info->conds[i]);
}
@@ -1745,7 +1745,7 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared
*shared,
if (!integer_onep (info->assumptions) && !orig_loop_info)
LOOP_VINFO_NITERS_ASSUMPTIONS (loop_vinfo) = info->assumptions;
- for (gcond *cond : info->conds)
+ for (gimple *cond : info->conds)
{
stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (cond);
/* Mark the statement as a condition. */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 70bf768d339..cd39bdfbd0d 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -5529,6 +5529,116 @@ vect_recog_gcond_pattern (vec_info *vinfo,
return pattern_stmt;
}
+/* Function vect_recog_gswitch_pattern
+
+ Try to find pattern like following:
+
+ switch (x) <default: <La>, case b1: <Lb>, ... case bN: <Lb>>
+
+ where all labels except the default one have the same destination and
+ convert it to a boolean pattern
+
+ mask = (x == b1) || (x == b2) || ... || (x == bN)
+ if (mask != 0)
+
+ The type of output TYPE_OUT is set in this function.
+
+ Input:
+
+ * STMT_VINFO: The stmt at the end from which the pattern
+ search begins, i.e., cast of a bool to
+ an integer type.
+
+ Output:
+
+ * TYPE_OUT: The type of the output of this pattern.
+
+ * Return value: A new stmt that will be used to replace the pattern. */
+
+static gimple *
+vect_recog_gswitch_pattern (vec_info *vinfo,
+ stmt_vec_info stmt_vinfo, tree *type_out)
+{
+ /* Currently we only support this for loop vectorization and when multiple
+ exits. */
+ loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
+ if (!loop_vinfo || !LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ return NULL;
+
+ gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo);
+ gswitch *gs = NULL;
+ if (!(gs = dyn_cast <gswitch *> (last_stmt)))
+ return NULL;
+
+ tree index = gimple_switch_index (gs);
+ tree stype = TREE_TYPE (index);
+
+ /* For now, there should be no gswitch with boolean type index. */
+ gcc_assert (!VECT_SCALAR_BOOLEAN_TYPE_P (stype));
+ tree vectype = get_mask_type_for_scalar_type (vinfo, stype);
+ if (vectype == NULL_TREE)
+ return NULL;
+
+ tree case_label_dest = NULL_TREE;
+ tree cond = NULL_TREE;
+ unsigned int num_cmps = 0;
+ for (unsigned li = 1; li < gimple_switch_num_labels (gs); li++)
+ {
+ tree label = gimple_switch_label (gs, li);
+
+ /* Check if all case labels have the same destination. */
+ tree dest = CASE_LABEL (label);
+ if (case_label_dest == NULL)
+ case_label_dest = dest;
+ else if (dest != case_label_dest)
+ return NULL;
+
+ tree low = CASE_LOW (label);
+ gcc_assert (low != NULL_TREE);
+ tree high = CASE_HIGH (label);
+ gcc_assert (high == NULL_TREE || tree_int_cst_lt (low, high));
+
+ wide_int val = wi::to_wide (low);
+ wide_int one = wi::one (TYPE_PRECISION (stype));
+ while (true)
+ {
+ /* A large number of comparisons lead to long compile time and make
+ vectorization non-profitable (most cost models will reject the
+ vectorized code). So bail out in this case. */
+ if (++num_cmps > (unsigned) param_vect_max_compare_in_switch_lowering)
+ return NULL;
+
+ tree eq_cond = vect_recog_temp_ssa_var (boolean_type_node, NULL);
+ gimple *eq_stmt = gimple_build_assign (eq_cond, EQ_EXPR, index,
+ wide_int_to_tree (stype, val));
+ append_pattern_def_seq (vinfo, stmt_vinfo, eq_stmt, vectype, stype);
+ if (cond == NULL_TREE)
+ cond = eq_cond;
+ else
+ {
+ tree new_cond = vect_recog_temp_ssa_var (boolean_type_node, NULL);
+ gimple *or_stmt = gimple_build_assign (new_cond, BIT_IOR_EXPR,
+ cond, eq_cond);
+ append_pattern_def_seq (vinfo, stmt_vinfo, or_stmt,
+ vectype, stype);
+ cond = new_cond;
+ }
+
+ /* Break if the case label has a single value or we've reached the
+ high value. */
+ if (high == NULL_TREE || wi::eq_p (val, wi::to_wide (high)))
+ break;
+
+ val = wi::add (val, one);
+ }
+ }
+
+ vect_pattern_detected ("vect_recog_gswitch_pattern", last_stmt);
+ *type_out = vectype;
+ return gimple_build_cond (NE_EXPR, cond, build_zero_cst (TREE_TYPE (cond)),
+ NULL_TREE, NULL_TREE);
+}
+
/* Function vect_recog_bool_pattern
Try to find pattern like following:
@@ -6987,6 +7097,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = {
{ vect_recog_sat_sub_pattern, "sat_sub" },
{ vect_recog_sat_trunc_pattern, "sat_trunc" },
{ vect_recog_gcond_pattern, "gcond" },
+ { vect_recog_gswitch_pattern, "gswitch" },
{ vect_recog_bool_pattern, "bool" },
/* This must come before mask conversion, and includes the parts
of mask conversion that are needed for gather and scatter
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 895fb88ab7f..217d1299656 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -5409,14 +5409,13 @@ vect_analyze_slp (vec_info *vinfo, unsigned
max_tree_size,
vec<stmt_vec_info> roots = vNULL;
roots.safe_push (cond_info);
gimple *stmt = STMT_VINFO_STMT (cond_info);
- tree args0 = gimple_cond_lhs (stmt);
- tree args1 = gimple_cond_rhs (stmt);
/* These should be enforced by cond lowering, but if it failed
bail. */
- if (gimple_cond_code (stmt) != NE_EXPR
- || TREE_TYPE (args0) != boolean_type_node
- || !integer_zerop (args1))
+ if (!is_a <gcond *> (stmt)
+ || gimple_cond_code (stmt) != NE_EXPR
+ || TREE_TYPE (gimple_cond_lhs (stmt)) != boolean_type_node
+ || !integer_zerop (gimple_cond_rhs (stmt)))
{
roots.release ();
return opt_result::failure_at (vect_location,
@@ -5428,7 +5427,7 @@ vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size,
from them. It's highly likely that the resulting SLP tree here if
both
arguments have a def will be incompatible, but we rely on it being
split
later on. */
- auto varg = loop_vinfo->lookup_def (args0);
+ auto varg = loop_vinfo->lookup_def (gimple_cond_lhs (stmt));
vec<stmt_vec_info> stmts;
vec<tree> remain = vNULL;
stmts.create (1);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 6274956e2a5..436a4f4fc44 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -743,6 +743,7 @@ vect_mark_stmts_to_be_vectorized (loop_vec_info loop_vinfo,
bool *fatal)
if (gimple_get_lhs (stmt) == NULL_TREE
&& !is_a <gcond *> (stmt)
+ && !is_a <gswitch *> (stmt)
&& !is_a <gcall *> (stmt))
return opt_result::failure_at
(stmt, "not vectorized: irregular stmt: %G", stmt);
@@ -12339,28 +12340,6 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
bool len_loop_p = LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
- /* Now build the new conditional. Pattern gimple_conds get dropped during
- codegen so we must replace the original insn. */
- gimple *orig_stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info));
- gcond *cond_stmt = as_a <gcond *>(orig_stmt);
-
- tree vectype_out = vectype;
- auto bb = gimple_bb (cond_stmt);
- edge exit_true_edge = EDGE_SUCC (bb, 0);
- if (exit_true_edge->flags & EDGE_FALSE_VALUE)
- exit_true_edge = EDGE_SUCC (bb, 1);
- gcc_assert (exit_true_edge->flags & EDGE_TRUE_VALUE);
-
- /* When vectorizing we assume that if the branch edge is taken that we're
- exiting the loop. This is not however always the case as the compiler
will
- rewrite conditions to always be a comparison against 0. To do this it
- sometimes flips the edges. This is fine for scalar, but for vector we
- then have to negate the result of the test, as we're still assuming that
if
- you take the branch edge that we found the exit condition. i.e. we need
to
- know whether we are generating a `forall` or an `exist` condition. */
- bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
- exit_true_edge->dest);
-
/* See if we support ADDHN and use that for the reduction. */
internal_fn ifn = IFN_VEC_TRUNC_ADD_HIGH;
bool addhn_supported_p
@@ -12439,6 +12418,53 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
auto_vec<tree> stmts;
stmts.safe_splice (SLP_TREE_VEC_DEFS (slp_node));
+ /* Now build the new conditional. Pattern gimple_conds get dropped during
+ codegen so we must replace the original insn. */
+ gimple *orig_stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info));
+ tree vectype_out = vectype;
+ auto bb = gimple_bb (orig_stmt);
+
+ /* Find the edge which indicates the branch is taken. If original statement
+ is a gcond, we find the edge with flag EDGE_TRUE_VALUE. For gswitch, we
+ find the edge from the non-default label, and set flags EDGE_TRUE_VALUE,
+ EDGE_FALSE_VALUE for the two edges. */
+ edge exit_true_edge = EDGE_SUCC (bb, 0);
+ if (is_a <gcond *> (orig_stmt))
+ {
+ if (exit_true_edge->flags & EDGE_FALSE_VALUE)
+ exit_true_edge = EDGE_SUCC (bb, 1);
+ gcc_assert (exit_true_edge->flags & EDGE_TRUE_VALUE);
+ }
+ else
+ {
+ gcc_assert (is_a <gswitch *> (orig_stmt));
+ gcc_assert (bb->succs->length () == 2);
+ tree dft_lab = gimple_switch_default_label (as_a <gswitch *>
(orig_stmt));
+ basic_block dft_bb = label_to_block (cfun, CASE_LABEL (dft_lab));
+ if (exit_true_edge->dest != dft_bb)
+ {
+ EDGE_SUCC (bb, 0)->flags |= EDGE_TRUE_VALUE;
+ EDGE_SUCC (bb, 1)->flags |= EDGE_FALSE_VALUE;
+ }
+ else
+ {
+ exit_true_edge = EDGE_SUCC (bb, 1);
+ EDGE_SUCC (bb, 0)->flags |= EDGE_FALSE_VALUE;
+ EDGE_SUCC (bb, 1)->flags |= EDGE_TRUE_VALUE;
+ }
+ gcc_assert (exit_true_edge->dest != dft_bb);
+ }
+
+ /* When vectorizing we assume that if the branch edge is taken that we're
+ exiting the loop. This is not however always the case as the compiler
will
+ rewrite conditions to always be a comparison against 0. To do this it
+ sometimes flips the edges. This is fine for scalar, but for vector we
+ then have to negate the result of the test, as we're still assuming that
if
+ you take the branch edge that we found the exit condition. i.e., we need
+ to know whether we are generating a `forall` or an `exist` condition. */
+ bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
+ exit_true_edge->dest);
+
/* If we're comparing against a previous forall we need to negate the
resullts
before we do the final comparison or reduction. */
if (flipped)
@@ -12536,8 +12562,8 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
gcc_assert (new_temp);
tree cst = build_zero_cst (vectype_out);
- gimple_cond_set_condition (cond_stmt, NE_EXPR, new_temp, cst);
- update_stmt (orig_stmt);
+ new_stmt = gimple_build_cond (NE_EXPR, new_temp, cst, NULL_TREE, NULL_TREE);
+ gsi_replace (&cond_gsi, new_stmt, true);
/* ??? */
SLP_TREE_VEC_DEFS (slp_node).truncate (0);
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index b7c2188ab3d..cb715459c00 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -1155,10 +1155,10 @@ public:
bool early_breaks;
/* List of loop additional IV conditionals found in the loop. */
- auto_vec<gcond *> conds;
+ auto_vec<gimple *> conds;
/* Main loop IV cond. */
- gcond* loop_iv_cond;
+ gimple *loop_iv_cond;
/* True if we have an unroll factor requested by the user through pragma GCC
unroll. */
@@ -2702,8 +2702,8 @@ struct vect_loop_form_info
tree number_of_iterations;
tree number_of_iterationsm1;
tree assumptions;
- auto_vec<gcond *> conds;
- gcond *inner_loop_cond;
+ auto_vec<gimple *> conds;
+ gimple *inner_loop_cond;
edge loop_exit;
};
extern opt_result vect_analyze_loop_form (class loop *, gimple *,
--
2.43.0
