On Thu, Sep 18, 2025 at 5:34 AM Pengfei Li <[email protected]> wrote: > > This patch adds vectorization support to early-break loops with gswitch > statements. Such gswitches may come from original switch-case constructs > in the source or the iftoswitch pass which rewrites if conditions with a > chain of comparisons, like below, to gswitch statements. > > if (a[i] == c1 || a[i] == c2 || a[i] == c3) > > In this patch, the loop exit condition type is generalized from gcond to > gimple, so that gswitch statements can be treated as valid loop exits. A > new vectorizer pattern recognization function is introduced to check if > all non-default labels of the gswitch have the same destination. If yes, > it lowers the gswitch to an equivalent gcond. To avoid excessive compile > time, a new parameter vect-max-compare-in-switch-lowering is introduced > to limit the maximum number of comparisons in the gswitch lowering. In > addition, the CRC loop optimization is adjusted a bit to make sure CRC > loops still have gcond exits. > > This patch is bootstrapped and regression-tested on x86_64-linux-gnu, > arm-linux-gnueabihf and aarch64-linux-gnu.
Does this fix https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115866 ? Thanks, Andrew > > gcc/ChangeLog: > > * doc/invoke.texi: Document vect-max-compare-in-switch-lowering. > * gimple-crc-optimization.cc > (crc_optimization::loop_may_calculate_crc): > Ensure CRC loop exits are still gconds. > (crc_optimization::optimize_crc_loop): Cast CRC loop exits. > * params.opt: Add option vect-max-compare-in-switch-lowering. > * tree-scalar-evolution.cc (scev_dfs::follow_ssa_edge_expr): > Generalize loop exit conditions type from gcond to gimple. > (get_loop_exit_condition): Likewise. > * tree-scalar-evolution.h (get_loop_exit_condition): Likewise. > * tree-vect-loop-manip.cc (vect_set_loop_condition_normal): > Likewise. > (vect_set_loop_condition): Likewise. > (slpeel_can_duplicate_loop_p): Likewise. > * tree-vect-loop.cc (vect_get_loop_niters): Likewise. > (vect_analyze_loop_form): Likewise. > (vect_create_loop_vinfo): Likewise. > * tree-vect-patterns.cc (vect_recog_gswitch_pattern): New > pattern recognition function for gswitch early exits. > * tree-vect-slp.cc (vect_analyze_slp): Bail out if a gswitch is > not lowered into a gcond. > * tree-vect-stmts.cc (vect_mark_stmts_to_be_vectorized): Allow > gswitch statements as loop exits. > (vectorizable_early_exit): Transform gswitch early exits. > * tree-vectorizer.h (struct vect_loop_form_info): Allow gswitch > statements as loop exits. > > gcc/testsuite/ChangeLog: > > * gcc.dg/vect/vect-switch-search-line-fast.c: Update the test as > the loop is now vectorizable on multiple targets. > * gcc.dg/vect/vect-early-break_139-switch1.c: New test. > * gcc.dg/vect/vect-early-break_139-switch2.c: New test. > * gcc.dg/vect/vect-early-break_139-switch3.c: New test. > * gcc.dg/vect/vect-early-break_139-switch4.c: New test. > --- > gcc/doc/invoke.texi | 4 + > gcc/gimple-crc-optimization.cc | 9 +- > gcc/params.opt | 4 + > .../vect/vect-early-break_139-switch1.c | 19 +++ > .../vect/vect-early-break_139-switch2.c | 21 ++++ > .../vect/vect-early-break_139-switch3.c | 24 ++++ > .../vect/vect-early-break_139-switch4.c | 26 ++++ > .../vect/vect-switch-search-line-fast.c | 3 +- > gcc/tree-scalar-evolution.cc | 12 +- > gcc/tree-scalar-evolution.h | 4 +- > gcc/tree-vect-loop-manip.cc | 6 +- > gcc/tree-vect-loop.cc | 10 +- > gcc/tree-vect-patterns.cc | 111 ++++++++++++++++++ > gcc/tree-vect-slp.cc | 11 +- > gcc/tree-vect-stmts.cc | 74 ++++++++---- > gcc/tree-vectorizer.h | 8 +- > 16 files changed, 295 insertions(+), 51 deletions(-) > create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c > create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c > create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c > create mode 100644 gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi > index 3cf326cc22c..def84a6675a 100644 > --- a/gcc/doc/invoke.texi > +++ b/gcc/doc/invoke.texi > @@ -16652,6 +16652,10 @@ Complex expressions slow the analyzer. > Maximum number of arguments in a PHI supported by TREE if conversion > unless the loop is marked with simd pragma. > > +@item vect-max-compare-in-switch-lowering > +The maximum number of comparisons that can be performed when lowering switch > +statements in the vectorizer. > + > @item vect-max-layout-candidates > The maximum number of possible vector layouts (such as permutations) > to consider when optimizing to-be-vectorized code. > diff --git a/gcc/gimple-crc-optimization.cc b/gcc/gimple-crc-optimization.cc > index 1751be9bc97..986aa193893 100644 > --- a/gcc/gimple-crc-optimization.cc > +++ b/gcc/gimple-crc-optimization.cc > @@ -937,6 +937,12 @@ crc_optimization::loop_may_calculate_crc (class loop > *loop) > return false; > > m_crc_loop = loop; > + > + /* Filter out the loops, which don't have a gcond as single exit. */ > + gimple *loop_exit_cond = get_loop_exit_condition (m_crc_loop); > + if (loop_exit_cond == NULL || !is_a <gcond *> (loop_exit_cond)) > + return false; > + > basic_block *loop_bbs = get_loop_body_in_dom_order (m_crc_loop); > > /* Filter out the cases, which don't have exactly two conditions in the > loop. > @@ -1274,7 +1280,8 @@ crc_optimization::optimize_crc_loop (gphi *output_crc) > remove_phi_node (&tmp_gsi, false); > > /* Alter the exit condition of the loop to always exit. */ > - gcond* loop_exit_cond = get_loop_exit_condition (m_crc_loop); > + gcond *loop_exit_cond > + = safe_dyn_cast <gcond *> (get_loop_exit_condition (m_crc_loop)); > gimple_cond_make_false (loop_exit_cond); > update_stmt (loop_exit_cond); > return true; > diff --git a/gcc/params.opt b/gcc/params.opt > index dd53d830895..115400ee8d2 100644 > --- a/gcc/params.opt > +++ b/gcc/params.opt > @@ -1229,6 +1229,10 @@ Whether to use canonical types. > Common Joined UInteger Var(param_vect_epilogues_nomask) Init(1) > IntegerRange(0, 1) Param Optimization NoOffload > Enable loop epilogue vectorization using smaller vector size. > > +-param=vect-max-compare-in-switch-lowering= > +Common Joined UInteger Var(param_vect_max_compare_in_switch_lowering) > Init(16) Param Optimization > +Maximum number of comparisons that can be performed when lowering switch > statements in the vectorizer. > + > -param=vect-max-layout-candidates= > Common Joined UInteger Var(param_vect_max_layout_candidates) Init(32) Param > Optimization > Maximum number of possible vector layouts (such as permutations) to consider > when optimizing to-be-vectorized code. > diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c > b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c > new file mode 100644 > index 00000000000..bbe75d20116 > --- /dev/null > +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch1.c > @@ -0,0 +1,19 @@ > +/* { dg-add-options vect_early_break } */ > +/* { dg-do compile } */ > +/* { dg-require-effective-target vect_early_break } */ > +/* { dg-require-effective-target vect_int } */ > + > +/* { dg-additional-options "-Ofast" } */ > + > +int find (int *a, int n) > +{ > + for (int i = 0; i < n; i++) > + { > + /* This is converted to a switch statement in the iftoswitch pass. */ > + if (a[i] == 1 || a[i] == 2 || a[i] == 5 || a[i] == 7) > + return i; > + } > + return -1; > +} > + > +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ > diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c > b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c > new file mode 100644 > index 00000000000..d200bea4594 > --- /dev/null > +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch2.c > @@ -0,0 +1,21 @@ > +/* { dg-add-options vect_early_break } */ > +/* { dg-do compile } */ > +/* { dg-require-effective-target vect_early_break } */ > +/* { dg-require-effective-target vect_int } */ > + > +/* { dg-additional-options "-Ofast" } */ > + > +int find (int *a, int n) > +{ > + for (int i = 0; i < n; i++) > + { > + /* This is converted to a switch statement in the iftoswitch pass. */ > + if (a[i] == 1 || a[i] == 3 || a[i] == 4 || a[i] == 5) > + continue; > + else > + return i; > + } > + return -1; > +} > + > +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ > diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c > b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c > new file mode 100644 > index 00000000000..f696267a855 > --- /dev/null > +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch3.c > @@ -0,0 +1,24 @@ > +/* { dg-add-options vect_early_break } */ > +/* { dg-do compile } */ > +/* { dg-require-effective-target vect_early_break } */ > +/* { dg-require-effective-target vect_int } */ > + > +/* { dg-additional-options "-Ofast" } */ > + > +int find (int *a, int n) > +{ > + for (int i = 0; i < n; i++) > + { > + /* An early break in case labels of a switch statement. */ > + switch (a[i]) > + { > + case -1 ... 1: > + case 10: > + case 20: > + return i; > + } > + } > + return -1; > +} > + > +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ > diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c > b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c > new file mode 100644 > index 00000000000..f8e3187843d > --- /dev/null > +++ b/gcc/testsuite/gcc.dg/vect/vect-early-break_139-switch4.c > @@ -0,0 +1,26 @@ > +/* { dg-add-options vect_early_break } */ > +/* { dg-do compile } */ > +/* { dg-require-effective-target vect_early_break } */ > +/* { dg-require-effective-target vect_int } */ > + > +/* { dg-additional-options "-Ofast" } */ > + > +int find (int *a, int n) > +{ > + for (int i = 0; i < n; i++) > + { > + /* An early break in the default label of a switch statement. */ > + switch (a[i]) > + { > + case 1 ... 3: > + case 5: > + case 10: > + continue; > + default: > + return i; > + } > + } > + return -1; > +} > + > +/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */ > diff --git a/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c > b/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c > index 678512db319..17e80330daa 100644 > --- a/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c > +++ b/gcc/testsuite/gcc.dg/vect/vect-switch-search-line-fast.c > @@ -16,5 +16,4 @@ const unsigned char *search_line_fast2 (const unsigned char > *s, > return s; > } > > -/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" { target > { ilp32 || { amdgcn*-*-* } } } } } */ > -/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 0 "vect" { target > { ! { ilp32 || { amdgcn*-*-* } } } } } } */ > +/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 1 "vect" } } */ > diff --git a/gcc/tree-scalar-evolution.cc b/gcc/tree-scalar-evolution.cc > index ecdef7529a6..0a5172c3be4 100644 > --- a/gcc/tree-scalar-evolution.cc > +++ b/gcc/tree-scalar-evolution.cc > @@ -1302,7 +1302,7 @@ scev_dfs::follow_ssa_edge_expr (gimple *at_stmt, tree > expr, > guards the exit edge. If the expression is too difficult to > analyze, then give up. */ > > -gcond * > +gimple * > get_loop_exit_condition (const class loop *loop) > { > return get_loop_exit_condition (single_exit (loop)); > @@ -1311,16 +1311,20 @@ get_loop_exit_condition (const class loop *loop) > /* If the statement just before the EXIT_EDGE contains a condition then > return the condition, otherwise NULL. */ > > -gcond * > +gimple * > get_loop_exit_condition (const_edge exit_edge) > { > - gcond *res = NULL; > + gimple *res = NULL; > > if (dump_file && (dump_flags & TDF_SCEV)) > fprintf (dump_file, "(get_loop_exit_condition \n "); > > if (exit_edge) > - res = safe_dyn_cast <gcond *> (*gsi_last_bb (exit_edge->src)); > + { > + gimple *g = *gsi_last_bb (exit_edge->src); > + if (is_a <gcond *> (g) || is_a <gswitch *> (g)) > + res = g; > + } > > if (dump_file && (dump_flags & TDF_SCEV)) > { > diff --git a/gcc/tree-scalar-evolution.h b/gcc/tree-scalar-evolution.h > index c7feeacf1db..54f0bf82c7b 100644 > --- a/gcc/tree-scalar-evolution.h > +++ b/gcc/tree-scalar-evolution.h > @@ -22,8 +22,8 @@ along with GCC; see the file COPYING3. If not see > #define GCC_TREE_SCALAR_EVOLUTION_H > > extern tree number_of_latch_executions (class loop *); > -extern gcond *get_loop_exit_condition (const class loop *); > -extern gcond *get_loop_exit_condition (const_edge); > +extern gimple *get_loop_exit_condition (const class loop *); > +extern gimple *get_loop_exit_condition (const_edge); > > extern void scev_initialize (void); > extern bool scev_initialized_p (void); > diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc > index 20141dbc2e5..bd04252da2d 100644 > --- a/gcc/tree-vect-loop-manip.cc > +++ b/gcc/tree-vect-loop-manip.cc > @@ -1233,7 +1233,7 @@ vect_set_loop_condition_normal (loop_vec_info /* > loop_vinfo */, edge exit_edge, > { > tree indx_before_incr, indx_after_incr; > gcond *cond_stmt; > - gcond *orig_cond; > + gimple *orig_cond; > edge pe = loop_preheader_edge (loop); > gimple_stmt_iterator incr_gsi; > bool insert_after; > @@ -1395,7 +1395,7 @@ vect_set_loop_condition (class loop *loop, edge loop_e, > loop_vec_info loop_vinfo > bool niters_maybe_zero) > { > gcond *cond_stmt; > - gcond *orig_cond = get_loop_exit_condition (loop_e); > + gimple *orig_cond = get_loop_exit_condition (loop_e); > gimple_stmt_iterator loop_cond_gsi = gsi_for_stmt (orig_cond); > > if (loop_vinfo && LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)) > @@ -2039,7 +2039,7 @@ slpeel_can_duplicate_loop_p (const class loop *loop, > const_edge exit_e, > const_edge e) > { > edge entry_e = loop_preheader_edge (loop); > - gcond *orig_cond = get_loop_exit_condition (exit_e); > + gimple *orig_cond = get_loop_exit_condition (exit_e); > gimple_stmt_iterator loop_exit_gsi = gsi_last_bb (exit_e->src); > > /* All loops have an outer scope; the only case loop->outer is NULL is for > diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc > index b58e4355e58..02db9f5a2eb 100644 > --- a/gcc/tree-vect-loop.cc > +++ b/gcc/tree-vect-loop.cc > @@ -627,12 +627,12 @@ vect_fixup_scalar_cycles_with_patterns (loop_vec_info > loop_vinfo) > Return the loop exit conditions. */ > > > -static vec<gcond *> > +static vec<gimple *> > vect_get_loop_niters (class loop *loop, const_edge main_exit, tree > *assumptions, > tree *number_of_iterations, tree > *number_of_iterationsm1) > { > auto_vec<edge> exits = get_loop_exit_edges (loop); > - vec<gcond *> conds; > + vec<gimple *> conds; > conds.create (exits.length ()); > class tree_niter_desc niter_desc; > tree niter_assumptions, niter, may_be_zero; > @@ -654,7 +654,7 @@ vect_get_loop_niters (class loop *loop, const_edge > main_exit, tree *assumptions, > unsigned int i; > FOR_EACH_VEC_ELT (exits, i, exit) > { > - gcond *cond = get_loop_exit_condition (exit); > + gimple *cond = get_loop_exit_condition (exit); > if (cond) > conds.safe_push (cond); > > @@ -1680,7 +1680,7 @@ vect_analyze_loop_form (class loop *loop, gimple > *loop_vectorized_call, > /* Determine what the primary and alternate exit conds are. */ > for (unsigned i = 0; i < info->conds.length (); i++) > { > - gcond *cond = info->conds[i]; > + gimple *cond = info->conds[i]; > if (exit_e->src == gimple_bb (cond)) > std::swap (info->conds[0], info->conds[i]); > } > @@ -1745,7 +1745,7 @@ vect_create_loop_vinfo (class loop *loop, > vec_info_shared *shared, > if (!integer_onep (info->assumptions) && !orig_loop_info) > LOOP_VINFO_NITERS_ASSUMPTIONS (loop_vinfo) = info->assumptions; > > - for (gcond *cond : info->conds) > + for (gimple *cond : info->conds) > { > stmt_vec_info loop_cond_info = loop_vinfo->lookup_stmt (cond); > /* Mark the statement as a condition. */ > diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc > index 70bf768d339..cd39bdfbd0d 100644 > --- a/gcc/tree-vect-patterns.cc > +++ b/gcc/tree-vect-patterns.cc > @@ -5529,6 +5529,116 @@ vect_recog_gcond_pattern (vec_info *vinfo, > return pattern_stmt; > } > > +/* Function vect_recog_gswitch_pattern > + > + Try to find pattern like following: > + > + switch (x) <default: <La>, case b1: <Lb>, ... case bN: <Lb>> > + > + where all labels except the default one have the same destination and > + convert it to a boolean pattern > + > + mask = (x == b1) || (x == b2) || ... || (x == bN) > + if (mask != 0) > + > + The type of output TYPE_OUT is set in this function. > + > + Input: > + > + * STMT_VINFO: The stmt at the end from which the pattern > + search begins, i.e., cast of a bool to > + an integer type. > + > + Output: > + > + * TYPE_OUT: The type of the output of this pattern. > + > + * Return value: A new stmt that will be used to replace the pattern. */ > + > +static gimple * > +vect_recog_gswitch_pattern (vec_info *vinfo, > + stmt_vec_info stmt_vinfo, tree *type_out) > +{ > + /* Currently we only support this for loop vectorization and when multiple > + exits. */ > + loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo); > + if (!loop_vinfo || !LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) > + return NULL; > + > + gimple *last_stmt = STMT_VINFO_STMT (stmt_vinfo); > + gswitch *gs = NULL; > + if (!(gs = dyn_cast <gswitch *> (last_stmt))) > + return NULL; > + > + tree index = gimple_switch_index (gs); > + tree stype = TREE_TYPE (index); > + > + /* For now, there should be no gswitch with boolean type index. */ > + gcc_assert (!VECT_SCALAR_BOOLEAN_TYPE_P (stype)); > + tree vectype = get_mask_type_for_scalar_type (vinfo, stype); > + if (vectype == NULL_TREE) > + return NULL; > + > + tree case_label_dest = NULL_TREE; > + tree cond = NULL_TREE; > + unsigned int num_cmps = 0; > + for (unsigned li = 1; li < gimple_switch_num_labels (gs); li++) > + { > + tree label = gimple_switch_label (gs, li); > + > + /* Check if all case labels have the same destination. */ > + tree dest = CASE_LABEL (label); > + if (case_label_dest == NULL) > + case_label_dest = dest; > + else if (dest != case_label_dest) > + return NULL; > + > + tree low = CASE_LOW (label); > + gcc_assert (low != NULL_TREE); > + tree high = CASE_HIGH (label); > + gcc_assert (high == NULL_TREE || tree_int_cst_lt (low, high)); > + > + wide_int val = wi::to_wide (low); > + wide_int one = wi::one (TYPE_PRECISION (stype)); > + while (true) > + { > + /* A large number of comparisons lead to long compile time and make > + vectorization non-profitable (most cost models will reject the > + vectorized code). So bail out in this case. */ > + if (++num_cmps > (unsigned) > param_vect_max_compare_in_switch_lowering) > + return NULL; > + > + tree eq_cond = vect_recog_temp_ssa_var (boolean_type_node, NULL); > + gimple *eq_stmt = gimple_build_assign (eq_cond, EQ_EXPR, index, > + wide_int_to_tree (stype, > val)); > + append_pattern_def_seq (vinfo, stmt_vinfo, eq_stmt, vectype, stype); > + if (cond == NULL_TREE) > + cond = eq_cond; > + else > + { > + tree new_cond = vect_recog_temp_ssa_var (boolean_type_node, > NULL); > + gimple *or_stmt = gimple_build_assign (new_cond, BIT_IOR_EXPR, > + cond, eq_cond); > + append_pattern_def_seq (vinfo, stmt_vinfo, or_stmt, > + vectype, stype); > + cond = new_cond; > + } > + > + /* Break if the case label has a single value or we've reached the > + high value. */ > + if (high == NULL_TREE || wi::eq_p (val, wi::to_wide (high))) > + break; > + > + val = wi::add (val, one); > + } > + } > + > + vect_pattern_detected ("vect_recog_gswitch_pattern", last_stmt); > + *type_out = vectype; > + return gimple_build_cond (NE_EXPR, cond, build_zero_cst (TREE_TYPE (cond)), > + NULL_TREE, NULL_TREE); > +} > + > /* Function vect_recog_bool_pattern > > Try to find pattern like following: > @@ -6987,6 +7097,7 @@ static vect_recog_func vect_vect_recog_func_ptrs[] = { > { vect_recog_sat_sub_pattern, "sat_sub" }, > { vect_recog_sat_trunc_pattern, "sat_trunc" }, > { vect_recog_gcond_pattern, "gcond" }, > + { vect_recog_gswitch_pattern, "gswitch" }, > { vect_recog_bool_pattern, "bool" }, > /* This must come before mask conversion, and includes the parts > of mask conversion that are needed for gather and scatter > diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc > index 895fb88ab7f..217d1299656 100644 > --- a/gcc/tree-vect-slp.cc > +++ b/gcc/tree-vect-slp.cc > @@ -5409,14 +5409,13 @@ vect_analyze_slp (vec_info *vinfo, unsigned > max_tree_size, > vec<stmt_vec_info> roots = vNULL; > roots.safe_push (cond_info); > gimple *stmt = STMT_VINFO_STMT (cond_info); > - tree args0 = gimple_cond_lhs (stmt); > - tree args1 = gimple_cond_rhs (stmt); > > /* These should be enforced by cond lowering, but if it failed > bail. */ > - if (gimple_cond_code (stmt) != NE_EXPR > - || TREE_TYPE (args0) != boolean_type_node > - || !integer_zerop (args1)) > + if (!is_a <gcond *> (stmt) > + || gimple_cond_code (stmt) != NE_EXPR > + || TREE_TYPE (gimple_cond_lhs (stmt)) != boolean_type_node > + || !integer_zerop (gimple_cond_rhs (stmt))) > { > roots.release (); > return opt_result::failure_at (vect_location, > @@ -5428,7 +5427,7 @@ vect_analyze_slp (vec_info *vinfo, unsigned > max_tree_size, > from them. It's highly likely that the resulting SLP tree here > if both > arguments have a def will be incompatible, but we rely on it > being split > later on. */ > - auto varg = loop_vinfo->lookup_def (args0); > + auto varg = loop_vinfo->lookup_def (gimple_cond_lhs (stmt)); > vec<stmt_vec_info> stmts; > vec<tree> remain = vNULL; > stmts.create (1); > diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc > index 6274956e2a5..436a4f4fc44 100644 > --- a/gcc/tree-vect-stmts.cc > +++ b/gcc/tree-vect-stmts.cc > @@ -743,6 +743,7 @@ vect_mark_stmts_to_be_vectorized (loop_vec_info > loop_vinfo, bool *fatal) > > if (gimple_get_lhs (stmt) == NULL_TREE > && !is_a <gcond *> (stmt) > + && !is_a <gswitch *> (stmt) > && !is_a <gcall *> (stmt)) > return opt_result::failure_at > (stmt, "not vectorized: irregular stmt: %G", stmt); > @@ -12339,28 +12340,6 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, > stmt_vec_info stmt_info, > bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo); > bool len_loop_p = LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo); > > - /* Now build the new conditional. Pattern gimple_conds get dropped during > - codegen so we must replace the original insn. */ > - gimple *orig_stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info)); > - gcond *cond_stmt = as_a <gcond *>(orig_stmt); > - > - tree vectype_out = vectype; > - auto bb = gimple_bb (cond_stmt); > - edge exit_true_edge = EDGE_SUCC (bb, 0); > - if (exit_true_edge->flags & EDGE_FALSE_VALUE) > - exit_true_edge = EDGE_SUCC (bb, 1); > - gcc_assert (exit_true_edge->flags & EDGE_TRUE_VALUE); > - > - /* When vectorizing we assume that if the branch edge is taken that we're > - exiting the loop. This is not however always the case as the compiler > will > - rewrite conditions to always be a comparison against 0. To do this it > - sometimes flips the edges. This is fine for scalar, but for vector we > - then have to negate the result of the test, as we're still assuming > that if > - you take the branch edge that we found the exit condition. i.e. we > need to > - know whether we are generating a `forall` or an `exist` condition. */ > - bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo), > - exit_true_edge->dest); > - > /* See if we support ADDHN and use that for the reduction. */ > internal_fn ifn = IFN_VEC_TRUNC_ADD_HIGH; > bool addhn_supported_p > @@ -12439,6 +12418,53 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, > stmt_vec_info stmt_info, > auto_vec<tree> stmts; > stmts.safe_splice (SLP_TREE_VEC_DEFS (slp_node)); > > + /* Now build the new conditional. Pattern gimple_conds get dropped during > + codegen so we must replace the original insn. */ > + gimple *orig_stmt = STMT_VINFO_STMT (vect_orig_stmt (stmt_info)); > + tree vectype_out = vectype; > + auto bb = gimple_bb (orig_stmt); > + > + /* Find the edge which indicates the branch is taken. If original > statement > + is a gcond, we find the edge with flag EDGE_TRUE_VALUE. For gswitch, we > + find the edge from the non-default label, and set flags EDGE_TRUE_VALUE, > + EDGE_FALSE_VALUE for the two edges. */ > + edge exit_true_edge = EDGE_SUCC (bb, 0); > + if (is_a <gcond *> (orig_stmt)) > + { > + if (exit_true_edge->flags & EDGE_FALSE_VALUE) > + exit_true_edge = EDGE_SUCC (bb, 1); > + gcc_assert (exit_true_edge->flags & EDGE_TRUE_VALUE); > + } > + else > + { > + gcc_assert (is_a <gswitch *> (orig_stmt)); > + gcc_assert (bb->succs->length () == 2); > + tree dft_lab = gimple_switch_default_label (as_a <gswitch *> > (orig_stmt)); > + basic_block dft_bb = label_to_block (cfun, CASE_LABEL (dft_lab)); > + if (exit_true_edge->dest != dft_bb) > + { > + EDGE_SUCC (bb, 0)->flags |= EDGE_TRUE_VALUE; > + EDGE_SUCC (bb, 1)->flags |= EDGE_FALSE_VALUE; > + } > + else > + { > + exit_true_edge = EDGE_SUCC (bb, 1); > + EDGE_SUCC (bb, 0)->flags |= EDGE_FALSE_VALUE; > + EDGE_SUCC (bb, 1)->flags |= EDGE_TRUE_VALUE; > + } > + gcc_assert (exit_true_edge->dest != dft_bb); > + } > + > + /* When vectorizing we assume that if the branch edge is taken that we're > + exiting the loop. This is not however always the case as the compiler > will > + rewrite conditions to always be a comparison against 0. To do this it > + sometimes flips the edges. This is fine for scalar, but for vector we > + then have to negate the result of the test, as we're still assuming > that if > + you take the branch edge that we found the exit condition. i.e., we > need > + to know whether we are generating a `forall` or an `exist` condition. > */ > + bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo), > + exit_true_edge->dest); > + > /* If we're comparing against a previous forall we need to negate the > resullts > before we do the final comparison or reduction. */ > if (flipped) > @@ -12536,8 +12562,8 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, > stmt_vec_info stmt_info, > gcc_assert (new_temp); > > tree cst = build_zero_cst (vectype_out); > - gimple_cond_set_condition (cond_stmt, NE_EXPR, new_temp, cst); > - update_stmt (orig_stmt); > + new_stmt = gimple_build_cond (NE_EXPR, new_temp, cst, NULL_TREE, > NULL_TREE); > + gsi_replace (&cond_gsi, new_stmt, true); > > /* ??? */ > SLP_TREE_VEC_DEFS (slp_node).truncate (0); > diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h > index b7c2188ab3d..cb715459c00 100644 > --- a/gcc/tree-vectorizer.h > +++ b/gcc/tree-vectorizer.h > @@ -1155,10 +1155,10 @@ public: > bool early_breaks; > > /* List of loop additional IV conditionals found in the loop. */ > - auto_vec<gcond *> conds; > + auto_vec<gimple *> conds; > > /* Main loop IV cond. */ > - gcond* loop_iv_cond; > + gimple *loop_iv_cond; > > /* True if we have an unroll factor requested by the user through pragma > GCC > unroll. */ > @@ -2702,8 +2702,8 @@ struct vect_loop_form_info > tree number_of_iterations; > tree number_of_iterationsm1; > tree assumptions; > - auto_vec<gcond *> conds; > - gcond *inner_loop_cond; > + auto_vec<gimple *> conds; > + gimple *inner_loop_cond; > edge loop_exit; > }; > extern opt_result vect_analyze_loop_form (class loop *, gimple *, > -- > 2.43.0 >
