The then/else order of the VEC_COND_EXPRs created by
vect_create_epilog_for_reduction meeds to line up with the
main VEC_COND_EXPR.
Tested on aarch64-linux-gnu, x86_64-linux-gnu and armeb-eabi.
Approved by Richi in the PR, applied as r276175.
Richard
2019-09-27 Richard Sandiford <richard.sandif...@arm.com>
gcc/
PR tree-optimization/91909
* tree-vect-loop.c (vect_create_epilog_for_reduction): Take a
reduc_index parameter. When handling COND_REDUCTION, make sure
that the reduction phi operand is in the correct arm of the
VEC_COND_EXPR.
(vectorizable_reduction): Pass reduc_index to the above.
Index: gcc/tree-vect-loop.c
===================================================================
--- gcc/tree-vect-loop.c 2019-09-27 09:09:26.747845171 +0100
+++ gcc/tree-vect-loop.c 2019-09-27 09:20:30.946948486 +0100
@@ -4235,6 +4235,8 @@ get_initial_defs_for_reduction (slp_tree
INDUC_CODE is the code for epilog reduction if INTEGER_INDUC_COND_REDUCTION.
NEUTRAL_OP is the value given by neutral_op_for_slp_reduction; it is
null if this is not an SLP reduction
+ REDUC_INDEX says which rhs operand of the STMT_INFO is the reduction phi
+ (counting from 0)
This function:
1. Creates the reduction def-use cycles: sets the arguments for
@@ -4285,7 +4287,7 @@ vect_create_epilog_for_reduction (vec<tr
bool double_reduc,
slp_tree slp_node,
slp_instance slp_node_instance,
- tree neutral_op)
+ tree neutral_op, int reduc_index)
{
tree induc_val = NULL_TREE;
stmt_vec_info prev_phi_info;
@@ -4469,11 +4471,17 @@ vect_create_epilog_for_reduction (vec<tr
tree ccompare = unshare_expr (gimple_assign_rhs1 (vec_stmt));
/* Create a conditional, where the condition is taken from vec_stmt
- (CCOMPARE), then is the induction index (INDEX_BEFORE_INCR) and
- else is the phi (NEW_PHI_TREE). */
- tree index_cond_expr = build3 (VEC_COND_EXPR, cr_index_vector_type,
- ccompare, indx_before_incr,
- new_phi_tree);
+ (CCOMPARE). The then and else values mirror the main VEC_COND_EXPR:
+ the reduction phi corresponds to NEW_PHI_TREE and the new values
+ correspond to INDEX_BEFORE_INCR. */
+ gcc_assert (reduc_index >= 1);
+ tree index_cond_expr;
+ if (reduc_index == 2)
+ index_cond_expr = build3 (VEC_COND_EXPR, cr_index_vector_type,
+ ccompare, indx_before_incr, new_phi_tree);
+ else
+ index_cond_expr = build3 (VEC_COND_EXPR, cr_index_vector_type,
+ ccompare, new_phi_tree, indx_before_incr);
induction_index = make_ssa_name (cr_index_vector_type);
gimple *index_condition = gimple_build_assign (induction_index,
index_cond_expr);
@@ -7159,7 +7167,7 @@ vectorizable_reduction (stmt_vec_info st
vect_create_epilog_for_reduction (vect_defs, stmt_info, reduc_def_phi,
orig_code, epilog_copies, reduc_fn, phis,
double_reduc, slp_node, slp_node_instance,
- neutral_op);
+ neutral_op, reduc_index);
return true;
}
