> -----Original Message----- > From: Alfie Richards <[email protected]> > Sent: 30 June 2026 09:01 > To: Richard Biener <[email protected]>; Tamar Christina > <[email protected]> > Cc: [email protected]; [email protected]; [email protected]; > [email protected] > Subject: Re: [PATCH v3 2/2] vect: Introduce LOOP_VINFO_IV_INCREMENT > > On 29/06/2026 07:58, Richard Biener wrote: > > On Fri, 26 Jun 2026, Tamar Christina wrote: > > > >>> -----Original Message----- > >>> From: Alfie Richards <[email protected]> > >>> Sent: 24 June 2026 14:46 > >>> To: [email protected] > >>> Cc: Alfie Richards <[email protected]>; [email protected]; Tamar > >>> Christina <[email protected]>; [email protected]; > [email protected]; > >>> [email protected] > >>> Subject: [PATCH v3 2/2] vect: Introduce LOOP_VINFO_IV_INCREMENT > >>> > >>> Introduce LOOP_VINFO_IV_INCREMENT which stores the number of > scalar > >>> iterations an iteration of the vectorized loop has processed. Then > >>> updates IV updates and reductions to use this value instead of VF. > >>> > >>> Update IV update logic so the IV_INCREMENT can be loop-variant. > >>> > >>> Simplify/remove SELECT_VL IV increment special cases by instead using > >>> IV_INCREMENT. > >>> > >>> As part of this change, the grouped load/store data pointer logic changes > >>> from: > >>> > >>> ADDRESS_1 = PHI <... , ADDRESS_3> > >>> V1 = LOAD ADDRESS_1 > >>> ADDRESS_2 = ADDRESS_1 + BUMP > >>> V2 = LOAD ADDRESS_2 > >>> ADDRESS_3 = ADDRESS_2 + BUMP > >>> > >>> to: > >>> > >>> ADDRESS_1 = PHI <... , ADDRESS_3> > >>> V1 = LOAD ADDRESS_1 > >>> ADDRESS_2 = ADDRESS_1 + BUMP > >>> V2 = LOAD ADDRESS_2 > >>> > >>> ADDRESS_3 = ADDRESS_1 + STEP * IV_INCREMENT > >>> > >>> gcc/ChangeLog: > >>> > >>> * tree-ssa-loop-manip.h: Add argument to create_iv. > >>> * tree-ssa-loop-manip.cc (create_iv): Change step to not > >>> necessarily be loop invariant. > >>> * tree-vect-data-refs.cc (bump_vector_ptr): Remove logic for > >>> updating use-def chain. > >>> * tree-vect-loop-manip.cc (vect_set_loop_controls_directly): > >>> Change to use LOOP_VINFO_IV_INCREMENT instead of VF. > >>> (vect_set_loop_condition_partial_vectors_avx512): > >>> Change to use LOOP_VINFO_IV_INCREMENT instead of VF. > >>> (vect_gen_vector_loop_niters): Update to handle > >>> LOOP_VINFO_IV_INCREMENT. > >>> (vect_get_loop_iv_increment): New function. > >>> * tree-vect-loop.cc (vectorizable_induction): Update to use > >>> LOOP_VINFO_IV_INCREMENT instead of VF, and remove SELECT_VL > >>> logic. > >>> (vect_update_ivs_after_vectorizer_for_early_breaks): > >>> Update to use LOOP_VINFO_IV_INCREMENT over VF and move IV > >>> update to vect_iv_increment_position. > >>> (vect_transform_loop): Add initialization of > >>> LOOP_VINFO_IV_INCREMENT. > >>> * tree-vect-stmts.cc (vect_get_strided_load_store_ops): > >>> Update to use dr_increment and dr_bump, remove SELECT_VL logic. > >>> (vect_get_data_ptr_increment): Change to return the increment > >>> needed to advance the pointer to the next iteration. > >>> (vect_get_data_ptr_bump): New function. > >>> (vectorizable_scan_store): Update to use vect_get_data_ptr_bump > >>> and to remove SELECT_VL logic. > >>> (vectorizable_store): Update to use vect_get_data_ptr_bump and > >>> vect_get_data_ptr_increment. > >>> (vectorizable_load): Update to use vect_get_data_ptr_bump and > >>> vect_get_data_ptr_increment. > >>> * tree-vectorizer.h (loop_vec_info): Add iv_increment field. > >>> (LOOP_VINFO_IV_INCREMENT): New macro. > >>> (LOOP_VINFO_IV_INCREMENT_INVARIANT_P): New macro. > >>> (bump_vector_ptr): Remove incr_ptr argument. > >>> (vect_get_loop_iv_increment): New function. > >>> > >>> gcc/testsuite/ChangeLog: > >>> > >>> * gcc.target/riscv/rvv/autovec/pr112399.c: Update for different > >>> scheduling. > >>> --- > >>> .../gcc.target/riscv/rvv/autovec/pr112399.c | 2 +- > >>> gcc/tree-ssa-loop-manip.cc | 21 +- > >>> gcc/tree-ssa-loop-manip.h | 3 +- > >>> gcc/tree-vect-data-refs.cc | 43 +-- > >>> gcc/tree-vect-loop-manip.cc | 88 ++++-- > >>> gcc/tree-vect-loop.cc | 113 ++++---- > >>> gcc/tree-vect-stmts.cc | 262 +++++++++--------- > >>> gcc/tree-vectorizer.h | 10 +- > >>> 8 files changed, 283 insertions(+), 259 deletions(-) > >>> > >>> diff --git a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112399.c > >>> b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112399.c > >>> index 948e12b8474..3e5b9457d68 100644 > >>> --- a/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112399.c > >>> +++ b/gcc/testsuite/gcc.target/riscv/rvv/autovec/pr112399.c > >>> @@ -7,12 +7,12 @@ > >>> ** foo: > >>> ** ... > >>> ** vsetvli\s*[a-x0-9]+,\s*[a-x0-9]+,\s*e64,\s*m1,\s*tu,\s*m[au] > >>> -** slli\s*[a-x0-9]+,\s*[a-x0-9]+,\s*3 > >>> ** vle64\.v\s*v[0-9]+,\s*0\([a-x0-9]+\) > >>> ** vle64\.v\s*v[0-9]+,\s*0\([a-x0-9]+\) > >>> ** vfmul\.vv\s*v[0-9]+,\s*v[0-9]+,\s*v[0-9]+ > >>> ** vle64\.v\s*\s*v[0-9]+,\s*0\([a-x0-9]+\) > >>> ** vfmacc\.vv\s*\s*v[0-9]+,\s*v[0-9]+,\s*v[0-9]+ > >>> +** slli\s*[a-x0-9]+,\s*[a-x0-9]+,\s*3 > >>> ** add\s*[a-x0-9]+,\s*[a-x0-9]+,\s*[a-x0-9]+ > >>> ** add\s*[a-x0-9]+,\s*[a-x0-9]+,\s*[a-x0-9]+ > >>> ** add\s*[a-x0-9]+,\s*[a-x0-9]+,\s*[a-x0-9]+ > >>> diff --git a/gcc/tree-ssa-loop-manip.cc b/gcc/tree-ssa-loop-manip.cc > >>> index df6cd8629bc..486667bd937 100644 > >>> --- a/gcc/tree-ssa-loop-manip.cc > >>> +++ b/gcc/tree-ssa-loop-manip.cc > >>> @@ -89,12 +89,14 @@ insert_iv_increment (gimple_stmt_iterator > *incr_pos, > >>> bool after, > >>> INCR_POS (after it if AFTER is true, before it otherwise). INCR_POS > >>> and > >>> AFTER can be computed using standard_iv_increment_position. The > ssa > >>> versions > >>> of the variable before and after increment will be stored in > >>> VAR_BEFORE > and > >>> - VAR_AFTER (unless they are NULL). */ > >>> + VAR_AFTER (unless they are NULL). If LOOP_INVARIANT is true then > the > >>> + calculation step will be added to the pre-header. Otherwise, the step > >>> + calculation is added to incr_pos. */ > >>> > >>> void > >>> create_iv (tree base, tree_code incr_op, tree step, tree var, class loop > *loop, > >>> gimple_stmt_iterator *incr_pos, bool after, tree *var_before, > >>> - tree *var_after) > >>> + tree *var_after, bool loop_invariant) > >>> { > >>> gphi *phi; > >>> tree initial, step1; > >>> @@ -150,13 +152,18 @@ create_iv (tree base, tree_code incr_op, tree > step, > >>> tree var, class loop *loop, > >>> step = fold_build1 (NEGATE_EXPR, TREE_TYPE (step), step); > >>> incr_op = POINTER_PLUS_EXPR; > >>> } > >>> - /* Gimplify the step if necessary. We put the computations in front of > the > >>> - loop (i.e. the step should be loop invariant). */ > >>> - step = force_gimple_operand (step, &stmts, true, NULL_TREE); > >>> - if (stmts) > >>> - gsi_insert_seq_on_edge_immediate (pe, stmts); > >>> > >>> gimple_seq incr_stmts = nullptr; > >>> + /* Gimplify the step if necessary. */ > >>> + if (loop_invariant) > >>> + { > >>> + step = force_gimple_operand (step, &stmts, true, NULL_TREE); > >>> + if (stmts) > >>> + gsi_insert_seq_on_edge_immediate (pe, stmts); > >>> + } > >>> + else > >>> + step = force_gimple_operand (step, &incr_stmts, true, NULL_TREE); > >>> + > >>> gimple_seq_add_stmt (&incr_stmts, > >>> gimple_build_assign (va, incr_op, vb, step)); > >>> insert_iv_increment (incr_pos, after, incr_stmts); > >>> diff --git a/gcc/tree-ssa-loop-manip.h b/gcc/tree-ssa-loop-manip.h > >>> index e5289ea0cd7..521aac2dbac 100644 > >>> --- a/gcc/tree-ssa-loop-manip.h > >>> +++ b/gcc/tree-ssa-loop-manip.h > >>> @@ -24,7 +24,8 @@ typedef void (*transform_callback)(class loop *, > void *); > >>> > >>> extern void insert_iv_increment (gimple_stmt_iterator *, bool, > gimple_seq); > >>> extern void create_iv (tree, tree_code, tree, tree, class loop *, > >>> - gimple_stmt_iterator *, bool, tree *, tree *); > >>> + gimple_stmt_iterator *, bool, tree *, tree *, > >>> + bool = true); > >>> extern void rewrite_into_loop_closed_ssa (bitmap, unsigned); > >>> extern void verify_loop_closed_ssa (bool, class loop * = NULL); > >>> > >> > >> So I am going to assume that Richi's comments last time amount to an OK > for this part > >> since I can't approve this part. > >> > >>> diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc > >>> index 6311e0d03df..4de1a15ae2c 100644 > >>> --- a/gcc/tree-vect-data-refs.cc > >>> +++ b/gcc/tree-vect-data-refs.cc > >>> @@ -6015,7 +6015,8 @@ vect_create_data_ref_ptr (vec_info *vinfo, > >>> stmt_vec_info stmt_info, > >>> > >>> create_iv (aggr_ptr_init, PLUS_EXPR, > >>> iv_step, aggr_ptr, loop, &incr_gsi, insert_after, > >>> - &indx_before_incr, &indx_after_incr); > >>> + &indx_before_incr, &indx_after_incr, > >>> + LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)); > >>> incr = gsi_stmt (incr_gsi); > >>> > >>> /* Copy the points-to information if it exists. */ > >>> @@ -6059,30 +6060,12 @@ vect_create_data_ref_ptr (vec_info *vinfo, > >>> stmt_vec_info stmt_info, > >>> > >>> /* Function bump_vector_ptr > >>> > >>> - Increment a pointer (to a vector type) by vector-size. If requested, > >>> - i.e. if PTR-INCR is given, then also connect the new increment stmt > >>> - to the existing def-use update-chain of the pointer, by modifying > >>> - the PTR_INCR as illustrated below: > >>> - > >>> - The pointer def-use update-chain before this function: > >>> - DATAREF_PTR = phi (p_0, p_2) > >>> - .... > >>> - PTR_INCR: p_2 = DATAREF_PTR + step > >>> - > >>> - The pointer def-use update-chain after this function: > >>> - DATAREF_PTR = phi (p_0, p_2) > >>> - .... > >>> - NEW_DATAREF_PTR = DATAREF_PTR + BUMP > >>> - .... > >>> - PTR_INCR: p_2 = NEW_DATAREF_PTR + step > >>> + Increment DATAREF_PTR by UPDATE. > >>> > >>> Input: > >>> DATAREF_PTR - ssa_name of a pointer (to vector type) that is being > updated > >>> in the loop. > >>> - PTR_INCR - optional. The stmt that updates the pointer in each > >>> iteration > of > >>> - the loop. The increment amount across iterations is expected > >>> - to be vector_size. > >>> - BSI - location where the new update stmt is to be placed. > >>> + GSI - location where the new update stmt is to be placed. > >>> STMT_INFO - the original scalar memory-access stmt that is being > >>> vectorized. > >>> UPDATE - The offset by which to bump the pointer. > >>> > >>> @@ -6092,13 +6075,11 @@ vect_create_data_ref_ptr (vec_info *vinfo, > >>> stmt_vec_info stmt_info, > >>> > >>> tree > >>> bump_vector_ptr (vec_info *vinfo, > >>> - tree dataref_ptr, gimple *ptr_incr, gimple_stmt_iterator *gsi, > >>> + tree dataref_ptr, gimple_stmt_iterator *gsi, > >>> stmt_vec_info stmt_info, tree update) > >>> { > >>> struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info); > >>> gimple *incr_stmt; > >>> - ssa_op_iter iter; > >>> - use_operand_p use_p; > >>> tree new_dataref_ptr; > >>> > >>> if (TREE_CODE (dataref_ptr) == SSA_NAME) > >>> @@ -6129,20 +6110,6 @@ bump_vector_ptr (vec_info *vinfo, > >>> /* Copy the points-to information if it exists. */ > >>> duplicate_ssa_name_ptr_info (new_dataref_ptr, DR_PTR_INFO (dr)); > >>> > >>> - if (!ptr_incr) > >>> - return new_dataref_ptr; > >>> - > >>> - /* Update the vector-pointer's cross-iteration increment. */ > >>> - FOR_EACH_SSA_USE_OPERAND (use_p, ptr_incr, iter, SSA_OP_USE) > >>> - { > >>> - tree use = USE_FROM_PTR (use_p); > >>> - > >>> - if (use == dataref_ptr) > >>> - SET_USE (use_p, new_dataref_ptr); > >>> - else > >>> - gcc_assert (operand_equal_p (use, update, 0)); > >>> - } > >>> - > >>> return new_dataref_ptr; > >>> } > >>> > >>> diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc > >>> index a64a79d3299..dd5b20f5a0e 100644 > >>> --- a/gcc/tree-vect-loop-manip.cc > >>> +++ b/gcc/tree-vect-loop-manip.cc > >>> @@ -514,7 +514,6 @@ vect_set_loop_controls_directly (class loop *loop, > >>> loop_vec_info loop_vinfo, > >>> tree ctrl_type = rgc->type; > >>> unsigned int nitems_per_iter = rgc->max_nscalars_per_iter * > >>> rgc->factor; > >>> poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type) * > rgc- > >>>> factor; > >>> - poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); > >>> tree length_limit = NULL_TREE; > >>> /* For length, we need length_limit to ensure length in range. */ > >>> if (!use_masks_p) > >>> @@ -525,7 +524,15 @@ vect_set_loop_controls_directly (class loop > *loop, > >>> loop_vec_info loop_vinfo, > >>> of the vector loop, and the number that it should skip during the > >>> first iteration of the vector loop. */ > >>> tree nitems_total = niters; > >>> - tree nitems_step = build_int_cst (iv_type, vf); > >>> + tree nitems_vf > >>> + = build_int_cst (iv_type, LOOP_VINFO_VECT_FACTOR (loop_vinfo)); > >>> + tree nitems_step > >>> + = LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo) > >>> + ? gimple_convert (preheader_seq, iv_type, > >>> + LOOP_VINFO_IV_INCREMENT (loop_vinfo)) > >>> + : gimple_convert (&loop_cond_gsi, true, GSI_SAME_STMT, > >>> UNKNOWN_LOCATION, > >>> + iv_type, LOOP_VINFO_IV_INCREMENT (loop_vinfo)); > >>> + > >>> tree nitems_skip = niters_skip; > >>> if (nitems_per_iter != 1) > >>> { > >>> @@ -535,8 +542,14 @@ vect_set_loop_controls_directly (class loop > *loop, > >>> loop_vec_info loop_vinfo, > >>> tree iv_factor = build_int_cst (iv_type, nitems_per_iter); > >>> nitems_total = gimple_build (preheader_seq, MULT_EXPR, > compare_type, > >>> nitems_total, compare_factor); > >>> - nitems_step = gimple_build (preheader_seq, MULT_EXPR, iv_type, > >>> - nitems_step, iv_factor); > >>> + nitems_vf = gimple_build (preheader_seq, MULT_EXPR, iv_type, > >>> + nitems_vf, iv_factor); > >>> + nitems_step = LOOP_VINFO_IV_INCREMENT_INVARIANT_P > (loop_vinfo) > >>> + ? gimple_build (preheader_seq, MULT_EXPR, iv_type, > >>> + nitems_step, iv_factor) > >>> + : gimple_build (&loop_cond_gsi, true, GSI_SAME_STMT, > >>> + UNKNOWN_LOCATION, MULT_EXPR, > >>> iv_type, > >>> + nitems_step, iv_factor); > >>> if (nitems_skip) > >>> nitems_skip = gimple_build (preheader_seq, MULT_EXPR, > >>> compare_type, > >>> nitems_skip, compare_factor); > >>> @@ -578,9 +591,21 @@ vect_set_loop_controls_directly (class loop > *loop, > >>> loop_vec_info loop_vinfo, > >>> insert_after, &index_before_incr, > >>> &index_after_incr); > >>> tree vectype = build_zero_cst (rgc->type); > >>> tree len = gimple_build (header_seq, IFN_SELECT_VL, iv_type, > >>> - index_before_incr, nitems_step, > >>> + index_before_incr, nitems_vf, > >>> vectype); > >>> gimple_seq_add_stmt (header_seq, gimple_build_assign (step, > >>> len)); > >>> + len = gimple_convert (header_seq, sizetype, len); > >>> + > >>> + /* Remove the previous initialiation of IV_INCREMENT to VARYING. > >>> */ > >>> + gimple *varying_def > >>> + = SSA_NAME_DEF_STMT (LOOP_VINFO_IV_INCREMENT > >>> (loop_vinfo)); > >>> + auto def_gsi = gsi_for_stmt (varying_def); > >>> + gsi_remove (&def_gsi, true); > >>> + > >>> + /* Set the LOOP_VINFO_IV_INCREMENT to be len. */ > >>> + gassign* assign_iv_increment > >>> + = gimple_build_assign (LOOP_VINFO_IV_INCREMENT (loop_vinfo), > >>> len); > >>> + gimple_seq_add_stmt (header_seq, assign_iv_increment); > >>> } > >>> else > >>> { > >>> @@ -593,7 +618,7 @@ vect_set_loop_controls_directly (class loop *loop, > >>> loop_vec_info loop_vinfo, > >>> nitems_step)); > >>> } > >>> *iv_step = step; > >>> - *compare_step = nitems_step; > >>> + *compare_step = nitems_vf; > >>> return LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo) ? > index_after_incr > >>> : > >>> index_before_incr; > >>> } > >>> @@ -601,7 +626,8 @@ vect_set_loop_controls_directly (class loop *loop, > >>> loop_vec_info loop_vinfo, > >>> /* Create increment IV. */ > >>> create_iv (build_int_cst (iv_type, 0), PLUS_EXPR, nitems_step, > NULL_TREE, > >>> loop, &incr_gsi, insert_after, &index_before_incr, > >>> - &index_after_incr); > >>> + &index_after_incr, > >>> + LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)); > >>> > >>> tree zero_index = build_int_cst (compare_type, 0); > >>> tree test_index, test_limit, first_limit; > >>> @@ -636,7 +662,7 @@ vect_set_loop_controls_directly (class loop *loop, > >>> loop_vec_info loop_vinfo, > >>> COMPARE_TYPE. */ > >>> test_index = index_before_incr; > >>> tree adjust = gimple_convert (preheader_seq, compare_type, > >>> - nitems_step); > >>> + nitems_vf); > >>> if (nitems_skip) > >>> adjust = gimple_build (preheader_seq, MINUS_EXPR, compare_type, > >>> adjust, nitems_skip); > >>> @@ -1040,14 +1066,16 @@ > >>> vect_set_loop_condition_partial_vectors_avx512 (class loop *loop, > >>> iv_type, niters, skip); > >>> } > >>> > >>> - /* The iteration step is the vectorization factor. */ > >>> - tree iv_step = build_int_cst (iv_type, vf); > >>> - > >>> - /* Create the decrement IV. */ > >>> - tree index_before_incr, index_after_incr; > >>> gimple_stmt_iterator incr_gsi; > >>> + tree index_before_incr, index_after_incr; > >>> bool insert_after; > >>> vect_iv_increment_position (exit_edge, &incr_gsi, &insert_after); > >>> + > >>> + /* The iteration step is the vectorization factor. */ > >>> + tree iv_step = gimple_convert (&preheader_seq, iv_type, > >>> + LOOP_VINFO_IV_INCREMENT (loop_vinfo)); > >>> + > >>> + /* Create the decrement IV. */ > >>> create_iv (niters_adj, MINUS_EXPR, iv_step, NULL_TREE, loop, > >>> &incr_gsi, insert_after, &index_before_incr, > >>> &index_after_incr); > >>> @@ -1230,7 +1258,7 @@ > vect_set_loop_condition_partial_vectors_avx512 > >>> (class loop *loop, > >>> loop handles exactly VF scalars per iteration. */ > >>> > >>> static gcond * > >>> -vect_set_loop_condition_normal (loop_vec_info /* loop_vinfo */, edge > >>> exit_edge, > >>> +vect_set_loop_condition_normal (loop_vec_info loop_vinfo, edge > exit_edge, > >>> class loop *loop, tree niters, tree > >>> step, > >>> tree final_iv, bool niters_maybe_zero, > >>> gimple_stmt_iterator loop_cond_gsi) > >>> @@ -1325,7 +1353,10 @@ vect_set_loop_condition_normal > (loop_vec_info > >>> /* loop_vinfo */, edge exit_edge, > >>> > >>> vect_iv_increment_position (exit_edge, &incr_gsi, &insert_after); > >>> create_iv (init, PLUS_EXPR, step, NULL_TREE, loop, > >>> - &incr_gsi, insert_after, &indx_before_incr, > >>> &indx_after_incr); > >>> + &incr_gsi, insert_after, > >>> + &indx_before_incr, &indx_after_incr, > >>> + LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)); > >>> + > >>> indx_after_incr = force_gimple_operand_gsi (&loop_cond_gsi, > >>> indx_after_incr, > >>> true, NULL_TREE, true, > >>> GSI_SAME_STMT); > >>> @@ -2979,7 +3010,8 @@ vect_gen_vector_loop_niters (loop_vec_info > >>> loop_vinfo, tree niters, > >>> /* To silence some unexpected warnings, simply initialize to 0. */ > >>> unsigned HOST_WIDE_INT const_vf = 0; > >>> if (vf.is_constant (&const_vf) > >>> - && !LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)) > >>> + && !LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> /* Create: niters / vf, which is equivalent to niters >> log2(vf) > >>> when > >>> vf is a power of two, and when not we approximate > >>> using a > >>> @@ -3054,6 +3086,30 @@ vect_gen_vector_loop_niters (loop_vec_info > >>> loop_vinfo, tree niters, > >>> return; > >>> } > >>> > >>> +/* Finds the amount IV's should be incremented by each iteration. > >>> + Stored in LOOP_VINFO_IV_INCREMENT. */ > >>> + > >>> +tree > >>> +vect_get_loop_iv_increment (loop_vec_info loop_vinfo) > >>> +{ > >>> + if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) > >>> + { > >>> + /* For now, set this as varying. > >>> + Fill this in later when building the loop controls. */ > >>> + tree iv_increment = make_temp_ssa_name (sizetype, NULL, > >>> "iv_increment"); > >>> + > >>> + gcall *varying_call = gimple_build_call_internal (IFN_VARYING, 0); > >>> + gimple_call_set_lhs (varying_call, iv_increment); > >>> + > >>> + gimple_stmt_iterator gsi = gsi_after_labels (loop_vinfo->loop- > >header); > >>> + gsi_insert_before (&gsi, varying_call, GSI_NEW_STMT); > >>> + > >>> + return iv_increment; > >>> + } > >>> + else > >>> + return build_int_cst (sizetype, LOOP_VINFO_VECT_FACTOR > (loop_vinfo)); > >>> +} > >>> + > >>> /* Given NITERS_VECTOR which is the number of iterations for vectorized > >>> loop specified by LOOP_VINFO after vectorization, compute the > number > >>> of iterations before vectorization (niters_vector * vf) and store it > >>> diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc > >>> index 0167d52b28b..931b942d7cb 100644 > >>> --- a/gcc/tree-vect-loop.cc > >>> +++ b/gcc/tree-vect-loop.cc > >>> @@ -9524,9 +9524,7 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> tree new_name; > >>> gphi *induction_phi; > >>> tree induc_def, vec_dest; > >>> - poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); > >>> unsigned i; > >>> - tree expr; > >>> tree index_vectype = NULL_TREE; > >>> gimple_stmt_iterator si; > >>> enum vect_induction_op_type induction_type > >>> @@ -9633,7 +9631,8 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> if (!target_supports_op_p (step_vectype, PLUS_EXPR, optab_default) > >>> || !target_supports_op_p (step_vectype, MINUS_EXPR, > optab_default)) > >>> return false; > >>> - if (!nunits.is_constant ()) > >>> + if (!nunits.is_constant () > >>> + || !LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> if (!target_supports_op_p (step_vectype, MULT_EXPR, > optab_default)) > >>> return false; > >>> @@ -9734,7 +9733,8 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> unsigned HOST_WIDE_INT const_nunits; > >>> if (nested_in_vect_loop) > >>> nivs = nvects; > >>> - else if (nunits.is_constant (&const_nunits)) > >>> + else if (nunits.is_constant (&const_nunits) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> /* Compute the number of distinct IVs we need. First reduce > >>> group_size if it is a multiple of const_nunits so we get > >>> @@ -9750,10 +9750,12 @@ vectorizable_induction (loop_vec_info > >>> loop_vinfo, > >>> nivs = 1; > >>> } > >>> gimple_seq init_stmts = NULL; > >>> + gimple_seq lupdate_mul_stmts = NULL; > >>> tree lupdate_mul = NULL_TREE; > >>> if (!nested_in_vect_loop) > >>> { > >>> - if (nunits.is_constant (&const_nunits)) > >>> + if (nunits.is_constant (&const_nunits) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> /* The number of iterations covered in one vector iteration. > >>> */ > >>> unsigned lup_mul = (nvects * const_nunits) / group_size; > >>> @@ -9766,14 +9768,23 @@ vectorizable_induction (loop_vec_info > >>> loop_vinfo, > >>> } > >>> else > >>> { > >>> + gimple_seq *update_stmts > >>> + = LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo) > >>> + ? &init_stmts > >>> + : &lupdate_mul_stmts; > >>> if (SCALAR_FLOAT_TYPE_P (stept)) > >>> { > >>> - tree tem = build_int_cst (integer_type_node, vf); > >>> - lupdate_mul = gimple_build (&init_stmts, FLOAT_EXPR, stept, > >>> tem); > >>> + tree increment > >>> + = gimple_convert (update_stmts, integer_type_node, > >>> + LOOP_VINFO_IV_INCREMENT (loop_vinfo)); > >>> + lupdate_mul = gimple_build (update_stmts, FLOAT_EXPR, stept, > >>> + increment); > >>> } > >>> else > >>> - lupdate_mul = build_int_cst (stept, vf); > >>> - lupdate_mul = gimple_build_vector_from_val (&init_stmts, > >>> step_vectype, > >>> + lupdate_mul = gimple_convert (update_stmts, stept, > >>> + LOOP_VINFO_IV_INCREMENT > >>> (loop_vinfo)); > >>> + lupdate_mul = gimple_build_vector_from_val (update_stmts, > >>> + step_vectype, > >>> lupdate_mul); > >>> } > >>> } > >>> @@ -9796,7 +9807,8 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> { > >>> gimple_seq stmts = NULL; > >>> bool invariant = true; > >>> - if (nunits.is_constant (&const_nunits)) > >>> + if (nunits.is_constant (&const_nunits) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> tree_vector_builder step_elts (step_vectype, const_nunits, 1); > >>> tree_vector_builder init_elts (vectype, const_nunits, 1); > >>> @@ -9904,36 +9916,12 @@ vectorizable_induction (loop_vec_info > >>> loop_vinfo, > >>> /* Create the iv update inside the loop */ > >>> tree up = vec_step; > >>> if (lupdate_mul) > >>> - { > >>> - if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) > >>> - { > >>> - /* When we're using loop_len produced by SELEC_VL, the > >>> - non-final iterations are not always processing VF > >>> - elements. So vectorize induction variable instead of > >>> - > >>> - _21 = vect_vec_iv_.6_22 + { VF, ... }; > >>> - > >>> - We should generate: > >>> - > >>> - _35 = .SELECT_VL (ivtmp_33, VF); > >>> - vect_cst__22 = [vec_duplicate_expr] _35; > >>> - _21 = vect_vec_iv_.6_22 + vect_cst__22; */ > >>> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo); > >>> - tree len = vect_get_loop_len (loop_vinfo, NULL, lens, 1, > >>> - vectype, 0, 0, false); > >>> - if (SCALAR_FLOAT_TYPE_P (stept)) > >>> - expr = gimple_build (&stmts, FLOAT_EXPR, stept, len); > >>> - else > >>> - expr = gimple_convert (&stmts, stept, len); > >>> - lupdate_mul = gimple_build_vector_from_val (&stmts, > >>> step_vectype, > >>> - expr); > >>> - up = gimple_build (&stmts, MULT_EXPR, > >>> - step_vectype, vec_step, lupdate_mul); > >>> - } > >>> - else > >>> - up = gimple_build (&init_stmts, MULT_EXPR, step_vectype, > >>> - vec_step, lupdate_mul); > >>> - } > >>> + { > >>> + if (lupdate_mul_stmts) > >>> + gimple_seq_add_seq (&stmts, lupdate_mul_stmts); > >>> + up = gimple_build (&stmts, MULT_EXPR, step_vectype, vec_step, > >>> + lupdate_mul); > >>> + } > >>> vec_def = gimple_convert (&stmts, step_vectype, induc_def); > >>> vec_def = gimple_build (&stmts, PLUS_EXPR, step_vectype, vec_def, > up); > >>> vec_def = gimple_convert (&stmts, vectype, vec_def); > >>> @@ -9964,7 +9952,8 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> if (!nested_in_vect_loop) > >>> { > >>> /* Fill up to the number of vectors we need for the whole group. > >>> */ > >>> - if (nunits.is_constant (&const_nunits)) > >>> + if (nunits.is_constant (&const_nunits) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> nivs = least_common_multiple (group_size, const_nunits) / > >>> const_nunits; > >>> else > >>> nivs = 1; > >>> @@ -9980,7 +9969,8 @@ vectorizable_induction (loop_vec_info > loop_vinfo, > >>> stmts by adding VF' * stride to the IVs generated above. */ > >>> if (ivn < nvects) > >>> { > >>> - if (nunits.is_constant (&const_nunits)) > >>> + if (nunits.is_constant (&const_nunits) > >>> + && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> { > >>> unsigned vfp = (least_common_multiple (group_size, > >>> const_nunits) > >>> / group_size); > >>> @@ -11073,7 +11063,6 @@ > >>> vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info > loop_vinfo) > >>> > >>> tree phi_var = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo); > >>> tree niters_skip = LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo); > >>> - poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo); > >>> tree ty_var = TREE_TYPE (phi_var); > >>> auto loop = LOOP_VINFO_LOOP (loop_vinfo); > >>> tree induc_var = niters_skip ? copy_ssa_name (phi_var) : phi_var; > >>> @@ -11090,27 +11079,15 @@ > >>> vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info > loop_vinfo) > >>> gimple_seq init_stmts = NULL; > >>> gimple_seq stmts = NULL; > >>> gimple_seq iv_stmts = NULL; > >>> - tree tree_vf = build_int_cst (ty_var, vf); > >>> - > >>> - /* For loop len targets we have to use .SELECT_VL (ivtmp_33, VF); > instead of > >>> - just += VF as the VF can change in between two loop iterations. */ > >>> - if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) > >>> - { > >>> - vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo); > >>> - tree_vf = vect_get_loop_len (loop_vinfo, NULL, lens, 1, > >>> - NULL_TREE, 0, 0, true); > >>> - } > >>> + tree tree_iv_incr = LOOP_VINFO_IV_INCREMENT (loop_vinfo); > >>> > >>> tree iter_var; > >>> if (POINTER_TYPE_P (ty_var)) > >>> - { > >>> - tree offset = gimple_convert (&stmts, sizetype, tree_vf); > >>> - iter_var = gimple_build (&stmts, POINTER_PLUS_EXPR, ty_var, > induc_def, > >>> - gimple_convert (&stmts, sizetype, offset)); > >>> - } > >>> + iter_var = gimple_build (&stmts, POINTER_PLUS_EXPR, ty_var, > induc_def, > >>> + tree_iv_incr); > >>> else > >>> { > >>> - tree offset = gimple_convert (&stmts, ty_var, tree_vf); > >>> + tree offset = gimple_convert (&stmts, ty_var, tree_iv_incr); > >>> iter_var = gimple_build (&stmts, PLUS_EXPR, ty_var, induc_def, > offset); > >>> } > >>> > >>> @@ -11157,10 +11134,17 @@ > >>> vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info > loop_vinfo) > >>> /* Write the init_stmts in the loop-preheader block. */ > >>> auto psi = gsi_last_nondebug_bb (pe->src); > >>> gsi_insert_seq_after (&psi, init_stmts, GSI_LAST_NEW_STMT); > >>> - /* Write the adjustments in the header block. */ > >>> - basic_block bb = loop->header; > >>> - auto si = gsi_after_labels (bb); > >>> - gsi_insert_seq_before (&si, stmts, GSI_SAME_STMT); > >>> + > >>> + /* Write the adjustments at the end of the iv increment. */ > >>> + bool insert_after; > >>> + gimple_stmt_iterator incr_gsi; > >>> + vect_iv_increment_position (LOOP_VINFO_MAIN_EXIT (loop_vinfo), > >>> &incr_gsi, > >>> + &insert_after); > >>> + > >>> + if (insert_after) > >>> + gsi_insert_seq_after (&incr_gsi, stmts, GSI_NEW_STMT); > >>> + else > >>> + gsi_insert_seq_before (&incr_gsi, stmts, GSI_NEW_STMT); > >>> } > >>> > >>> /* Function vect_transform_loop. > >>> @@ -11259,6 +11243,9 @@ vect_transform_loop (loop_vec_info > loop_vinfo, > >>> gimple *loop_vectorized_call) > >>> check_profitability, niters_no_overflow, > >>> &advance); > >>> > >>> + LOOP_VINFO_IV_INCREMENT (loop_vinfo) > >>> + = vect_get_loop_iv_increment (loop_vinfo); > >>> + > >>> /* Assign hierarchical discriminators to the vectorized loop. */ > >>> poly_uint64 vf_val = LOOP_VINFO_VECT_FACTOR (loop_vinfo); > >>> unsigned int vf_int = constant_lower_bound (vf_val); > >>> diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc > >>> index b69f9de5de8..bc1a99bf571 100644 > >>> --- a/gcc/tree-vect-stmts.cc > >>> +++ b/gcc/tree-vect-stmts.cc > >>> @@ -3170,44 +3170,35 @@ vect_get_gather_scatter_ops (class loop > *loop, > >>> slp_tree slp_node, > >>> the gather load or scatter store operation described by GS_INFO. > >>> STMT_INFO is the load or store statement. > >>> > >>> - Set *DATAREF_BUMP to the amount that should be added to the base > >>> - address after each copy of the vectorized statement. Set *VEC_OFFSET > >>> - to an invariant offset vector in which element I has the value > >>> - I * DR_STEP / SCALE. */ > >>> + Set *DR_INCREMENT to the amount that should be added to pointer > base > >>> address > >>> + to get to the next iteration's base address. > >>> + Set *DR_BUMP to the amount that should be added to the base > >>> + address after each copy of the vectorized statement in a grouped read. > >>> + Set *VEC_OFFSET to an invariant offset vector in which element I has > the > >>> + value I * DR_STEP / SCALE. */ > >>> > >>> static void > >>> vect_get_strided_load_store_ops (stmt_vec_info stmt_info, slp_tree > node, > >>> tree vectype, tree offset_vectype, > >>> loop_vec_info loop_vinfo, > >>> gimple_stmt_iterator *gsi, > >>> - tree *dataref_bump, tree *vec_offset, > >>> - vec_loop_lens *loop_lens) > >>> + tree *dr_step, tree *dr_bump, > >>> + tree *vec_offset) > >>> { > >>> struct data_reference *dr = STMT_VINFO_DATA_REF (stmt_info); > >>> > >>> - if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) > >>> - { > >>> - /* _31 = .SELECT_VL (ivtmp_29, POLY_INT_CST [4, 4]); > >>> - ivtmp_8 = _31 * 16 (step in bytes); > >>> - .MASK_LEN_SCATTER_STORE (vectp_a.9_7, ... ); > >>> - vectp_a.9_26 = vectp_a.9_7 + ivtmp_8; */ > >>> - tree loop_len > >>> - = vect_get_loop_len (loop_vinfo, gsi, loop_lens, 1, vectype, 0, 0, > >>> true); > >>> - tree tmp > >>> - = fold_build2 (MULT_EXPR, sizetype, > >>> - fold_convert (sizetype, unshare_expr (DR_STEP (dr))), > >>> - loop_len); > >>> - *dataref_bump = force_gimple_operand_gsi (gsi, tmp, true, > NULL_TREE, > >>> true, > >>> - GSI_SAME_STMT); > >>> - } > >>> - else > >>> - { > >>> - tree bump > >>> - = size_binop (MULT_EXPR, > >>> - fold_convert (sizetype, unshare_expr (DR_STEP (dr))), > >>> - size_int (TYPE_VECTOR_SUBPARTS (vectype))); > >>> - *dataref_bump = cse_and_gimplify_to_preheader (loop_vinfo, > bump); > >>> - } > >>> + tree dr_step_temp > >>> + = size_binop (MULT_EXPR, > >>> + fold_convert (sizetype, unshare_expr (DR_STEP (dr))), > >>> + LOOP_VINFO_IV_INCREMENT (loop_vinfo)); > >>> + *dr_step = LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo) > >>> + ? cse_and_gimplify_to_preheader (loop_vinfo, dr_step_temp) > >>> + : force_gimple_operand_gsi (gsi, dr_step_temp, false, NULL_TREE, > >>> + true, GSI_SAME_STMT); > >>> + tree bump = size_binop (MULT_EXPR, > >>> + fold_convert (sizetype, unshare_expr (DR_STEP > >>> (dr))), > >>> + size_int (TYPE_VECTOR_SUBPARTS (vectype))); > >>> + *dr_bump = cse_and_gimplify_to_preheader (loop_vinfo, bump); > >>> > >>> internal_fn ifn > >>> = DR_IS_READ (dr) ? IFN_MASK_LEN_STRIDED_LOAD : > >>> IFN_MASK_LEN_STRIDED_STORE; > >>> @@ -3233,47 +3224,31 @@ vect_get_strided_load_store_ops > >>> (stmt_vec_info stmt_info, slp_tree node, > >>> *vec_offset = cse_and_gimplify_to_preheader (loop_vinfo, offset); > >>> } > >>> > >>> -/* Prepare the pointer IVs which needs to be updated by a variable > amount. > >>> - Such variable amount is the outcome of .SELECT_VL. In this case, we > >>> can > >>> - allow each iteration process the flexible number of elements as long > >>> as > >>> - the number <= vf elements. > >>> - > >>> - Return data reference according to SELECT_VL. > >>> - If new statements are needed, insert them before GSI. */ > >>> +/* Return the amount that should be added to a vector pointer, > represented > >>> by > >>> + DR_INFO, to increment to the next vectorized iteration. */ > >>> > >>> static tree > >>> -vect_get_loop_variant_data_ptr_increment ( > >>> - vec_info *vinfo, tree aggr_type, gimple_stmt_iterator *gsi, > >>> - vec_loop_lens *loop_lens, dr_vec_info *dr_info, > >>> - vect_memory_access_type memory_access_type) > >>> +vect_get_data_ptr_step (vec_info *vinfo, dr_vec_info *dr_info, > >>> + vect_memory_access_type memory_access_type) > >>> { > >>> + if (memory_access_type == VMAT_INVARIANT) > >>> + return size_zero_node; > >>> + > >>> loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo); > >>> - tree step = vect_dr_behavior (vinfo, dr_info)->step; > >>> + > >>> + /* For BB SLP there is no next iteration. */ > >>> + if (!loop_vinfo) > >>> + return build_zero_cst (sizetype); > >>> + > >>> + tree step = vect_dr_behavior (loop_vinfo, dr_info)->step; > >>> > >>> /* gather/scatter never reach here. */ > >>> gcc_assert (!mat_gather_scatter_p (memory_access_type)); > >>> > >>> - /* When we support SELECT_VL pattern, we dynamic adjust > >>> - the memory address by .SELECT_VL result. > >>> + tree iv_increment = LOOP_VINFO_IV_INCREMENT (loop_vinfo); > >>> > >>> - The result of .SELECT_VL is the number of elements to > >>> - be processed of each iteration. So the memory address > >>> - adjustment operation should be: > >>> - > >>> - addr = addr + .SELECT_VL (ARG..) * step; > >>> - */ > >>> - tree loop_len > >>> - = vect_get_loop_len (loop_vinfo, gsi, loop_lens, 1, aggr_type, 0, 0, > true); > >>> - tree len_type = TREE_TYPE (loop_len); > >>> - /* Since the outcome of .SELECT_VL is element size, we should adjust > >>> - it into bytesize so that it can be used in address pointer variable > >>> - amount IVs adjustment. */ > >>> - tree tmp = fold_build2 (MULT_EXPR, len_type, loop_len, > >>> - wide_int_to_tree (len_type, wi::to_widest (step))); > >>> - tree bump = make_temp_ssa_name (len_type, NULL, "ivtmp"); > >>> - gassign *assign = gimple_build_assign (bump, tmp); > >>> - gsi_insert_before (gsi, assign, GSI_SAME_STMT); > >>> - return bump; > >>> + return fold_build2 (MULT_EXPR, sizetype, iv_increment, > >>> + fold_convert (sizetype, step)); > >>> } > >>> > >>> /* Return the amount that should be added to a vector pointer to move > >>> @@ -3282,19 +3257,18 @@ vect_get_loop_variant_data_ptr_increment > ( > >>> vectorization. */ > >>> > >>> static tree > >>> -vect_get_data_ptr_increment (vec_info *vinfo, gimple_stmt_iterator *gsi, > >>> - dr_vec_info *dr_info, tree aggr_type, > >>> - vect_memory_access_type memory_access_type, > >>> - vec_loop_lens *loop_lens) > >>> +vect_get_data_ptr_bump (vec_info *vinfo, > >>> + dr_vec_info *dr_info, tree aggr_type, > >>> + vect_memory_access_type memory_access_type) > >>> { > >>> if (memory_access_type == VMAT_INVARIANT) > >>> return size_zero_node; > >>> > >>> loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo); > >>> + /* We do not support SLP loads where num_vec != 1 with SELECT_VL so > this > >>> value > >>> + should never be needed. */ > >>> if (loop_vinfo && LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)) > >>> - return vect_get_loop_variant_data_ptr_increment (vinfo, aggr_type, > gsi, > >>> - loop_lens, dr_info, > >>> - memory_access_type); > >>> + return NULL_TREE; > >>> > >>> tree iv_step = TYPE_SIZE_UNIT (aggr_type); > >>> tree step = vect_dr_behavior (vinfo, dr_info)->step; > >>> @@ -7945,19 +7919,12 @@ vectorizable_scan_store (vec_info *vinfo, > >>> stmt_vec_info stmt_info, > >>> perms[i] = vect_gen_perm_mask_checked (vectype, indices); > >>> } > >>> > >>> - vec_loop_lens *loop_lens > >>> - = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo) > >>> - ? &LOOP_VINFO_LENS (loop_vinfo) > >>> - : NULL); > >>> - > >>> tree vec_oprnd1 = NULL_TREE; > >>> tree vec_oprnd2 = NULL_TREE; > >>> tree vec_oprnd3 = NULL_TREE; > >>> tree dataref_ptr = DR_BASE_ADDRESS (dr_info->dr); > >>> tree dataref_offset = build_int_cst (ref_type, 0); > >>> - tree bump = vect_get_data_ptr_increment (vinfo, gsi, dr_info, > >>> - vectype, VMAT_CONTIGUOUS, > >>> - loop_lens); > >>> + tree bump = vect_get_data_ptr_bump (vinfo, dr_info, vectype, > >>> VMAT_CONTIGUOUS); > >>> tree ldataref_ptr = NULL_TREE; > >>> tree orig = NULL_TREE; > >>> if (STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) == 4 && > >>> !inscan_var_store) > >>> @@ -8131,7 +8098,6 @@ vectorizable_store (vec_info *vinfo, > >>> enum vect_def_type mask_dt = vect_unknown_def_type; > >>> tree dataref_ptr = NULL_TREE; > >>> tree dataref_offset = NULL_TREE; > >>> - gimple *ptr_incr = NULL; > >>> int j; > >>> stmt_vec_info first_stmt_info; > >>> bool grouped_store; > >>> @@ -8558,15 +8524,26 @@ vectorizable_store (vec_info *vinfo, > >>> if (!costing_p) > >>> { > >>> ivstep = stride_step; > >>> + > >>> + tree increment = fold_convert (TREE_TYPE (ivstep), > >>> + LOOP_VINFO_IV_INCREMENT > >>> (loop_vinfo)); > >>> + > >>> ivstep = fold_build2 (MULT_EXPR, TREE_TYPE (ivstep), ivstep, > >>> - build_int_cst (TREE_TYPE (ivstep), vf)); > >>> + increment); > >>> > >>> standard_iv_increment_position (loop, &incr_gsi, > >>> &insert_after); > >>> > >>> stride_base = cse_and_gimplify_to_preheader (loop_vinfo, > >>> stride_base); > >>> - ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); > >>> + if (LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> + ivstep = force_gimple_operand_gsi (&incr_gsi, unshare_expr > >>> (ivstep), > >>> + true, NULL_TREE, true, > >>> + GSI_SAME_STMT); > >>> + else > >>> + ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); > >>> + > >>> create_iv (stride_base, PLUS_EXPR, ivstep, NULL, loop, > >>> &incr_gsi, > >>> - insert_after, &offvar, NULL); > >>> + insert_after, &offvar, NULL, > >>> + LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)); > >>> > >> > >> This is flipped is it not? The invariant case now puts it inside the header > rather than > >> the pre-header? It looks like it's correct in vectorizable_load but > >> inverted > here. > >> > >>> stride_step = cse_and_gimplify_to_preheader (loop_vinfo, > >>> stride_step); > >>> } > >>> @@ -8705,12 +8682,15 @@ vectorizable_store (vec_info *vinfo, > >>> if (!known_eq (poffset, 0)) > >>> offset = size_int (poffset); > >>> > >>> - tree bump; > >>> + tree dr_increment; > >>> + tree dr_bump; > >>> + > >>> tree vec_offset = NULL_TREE; > >>> if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) > >>> { > >>> aggr_type = NULL_TREE; > >>> - bump = NULL_TREE; > >>> + dr_increment = NULL_TREE; > >>> + dr_bump = NULL_TREE; > >>> } > >>> else if (mat_gather_scatter_p (memory_access_type)) > >>> { > >>> @@ -8721,7 +8701,8 @@ vectorizable_store (vec_info *vinfo, > >>> vect_get_strided_load_store_ops (stmt_info, slp_node, vtype, > >>> ls.strided_offset_vectype, > >>> loop_vinfo, gsi, > >>> - &bump, &vec_offset, loop_lens); > >>> + &dr_increment, &dr_bump, > >>> + &vec_offset); > >>> } > >>> } > >>> else > >>> @@ -8731,8 +8712,12 @@ vectorizable_store (vec_info *vinfo, > >>> else > >>> aggr_type = vectype; > >>> if (!costing_p) > >>> - bump = vect_get_data_ptr_increment (vinfo, gsi, dr_info, aggr_type, > >>> - memory_access_type, loop_lens); > >>> + { > >>> + dr_increment = vect_get_data_ptr_step (vinfo, dr_info, > >>> + memory_access_type); > >>> + dr_bump = vect_get_data_ptr_bump (vinfo, dr_info, aggr_type, > >>> + memory_access_type); > >>> + } > >>> } > >>> > >>> if (loop_vinfo && mask_node && !costing_p) > >>> @@ -8785,7 +8770,7 @@ vectorizable_store (vec_info *vinfo, > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, first_stmt_info, > >>> aggr_type, NULL, offset, > >>> &dummy, > >>> - gsi, &ptr_incr, false, bump); > >>> + gsi, NULL, false, > >>> dr_increment); > >>> } > >>> } > >>> else if (!costing_p) > >>> @@ -8793,8 +8778,8 @@ vectorizable_store (vec_info *vinfo, > >>> gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); > >>> if (mask_node) > >>> vec_mask = vec_masks[j]; > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, > >>> + stmt_info, dr_bump); > >>> } > >>> > >>> if (costing_p) > >>> @@ -8941,16 +8926,16 @@ vectorizable_store (vec_info *vinfo, > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, > >>> first_stmt_info, > >>> aggr_type, NULL, > >>> offset, > >>> - &dummy, gsi, &ptr_incr, > >>> false, > >>> - bump); > >>> + &dummy, gsi, NULL, false, > >>> + dr_increment); > >>> } > >>> } > >>> else if (!costing_p) > >>> { > >>> gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); > >>> if (!STMT_VINFO_GATHER_SCATTER_P (stmt_info)) > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, > >>> - gsi, stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, > >>> + gsi, stmt_info, dr_bump); > >>> } > >>> > >>> new_stmt = NULL; > >>> @@ -9347,8 +9332,8 @@ vectorizable_store (vec_info *vinfo, > >>> else if (!costing_p) > >>> dataref_ptr = vect_create_data_ref_ptr (vinfo, first_stmt_info, > aggr_type, > >>> simd_lane_access_p ? loop : > >>> NULL, > >>> - offset, &dummy, gsi, &ptr_incr, > >>> - simd_lane_access_p, bump); > >>> + offset, &dummy, gsi, NULL, > >>> + simd_lane_access_p, dr_increment); > >>> > >>> new_stmt = NULL; > >>> gcc_assert (!grouped_store); > >>> @@ -9398,8 +9383,8 @@ vectorizable_store (vec_info *vinfo, > >>> > >>> if (i > 0) > >>> /* Bump the vector pointer. */ > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, > >>> + dr_bump); > >>> > >>> unsigned misalign; > >>> unsigned HOST_WIDE_INT align; > >>> @@ -9722,7 +9707,6 @@ vectorizable_load (vec_info *vinfo, > >>> tree dummy; > >>> tree dataref_ptr = NULL_TREE; > >>> tree dataref_offset = NULL_TREE; > >>> - gimple *ptr_incr = NULL; > >>> int i, j; > >>> unsigned int group_size; > >>> poly_uint64 group_gap_adj; > >>> @@ -10231,16 +10215,22 @@ vectorizable_load (vec_info *vinfo, > >>> vectemp = {tmp1, tmp2, ...} > >>> */ > >>> > >>> - ivstep = fold_build2 (MULT_EXPR, TREE_TYPE (stride_step), > >>> stride_step, > >>> - build_int_cst (TREE_TYPE (stride_step), vf)); > >>> + tree increment = fold_convert (TREE_TYPE (stride_step), > >>> + LOOP_VINFO_IV_INCREMENT > >>> (loop_vinfo)); > >>> + ivstep = fold_build2 (MULT_EXPR, TREE_TYPE (stride_step), > >>> + stride_step, increment); > >>> > >>> standard_iv_increment_position (loop, &incr_gsi, > >>> &insert_after); > >>> > >>> stride_base = cse_and_gimplify_to_preheader (loop_vinfo, > >>> stride_base); > >>> - ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); > >>> - create_iv (stride_base, PLUS_EXPR, ivstep, NULL, > >>> - loop, &incr_gsi, insert_after, > >>> - &offvar, NULL); > >>> + if (LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)) > >>> + ivstep = cse_and_gimplify_to_preheader (loop_vinfo, ivstep); > >>> + else > >>> + ivstep = force_gimple_operand_gsi (&incr_gsi, unshare_expr > >>> (ivstep), > >>> + true, NULL_TREE, true, > >>> + GSI_SAME_STMT); > >>> + create_iv (stride_base, PLUS_EXPR, ivstep, NULL, loop, &incr_gsi, > >>> + insert_after, &offvar, NULL, true); > >>> > >>> stride_step = cse_and_gimplify_to_preheader (loop_vinfo, > >>> stride_step); > >>> } > >>> @@ -10710,7 +10700,8 @@ vectorizable_load (vec_info *vinfo, > >>> ? size_binop (PLUS_EXPR, offset, size_int (poffset)) > >>> : size_int (poffset)); > >>> > >>> - tree bump; > >>> + tree dr_increment; > >>> + tree dr_bump; > >>> tree vec_offset = NULL_TREE; > >>> > >>> auto_vec<tree> vec_offsets; > >>> @@ -10730,8 +10721,12 @@ vectorizable_load (vec_info *vinfo, > >>> > >>> aggr_type = build_array_type_nelts (elem_type, group_size * > >>> nunits); > >>> if (!costing_p) > >>> - bump = vect_get_data_ptr_increment (vinfo, gsi, dr_info, aggr_type, > >>> - memory_access_type, loop_lens); > >>> + { > >>> + dr_increment = vect_get_data_ptr_step (vinfo, dr_info, > >>> + memory_access_type); > >>> + dr_bump = vect_get_data_ptr_bump (vinfo, dr_info, aggr_type, > >>> + memory_access_type); > >>> + } > >>> > >>> unsigned int inside_cost = 0, prologue_cost = 0; > >>> /* For costing some adjacent vector loads, we'd like to cost with > >>> @@ -10778,12 +10773,12 @@ vectorizable_load (vec_info *vinfo, > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, first_stmt_info, > >>> aggr_type, > >>> at_loop, offset, &dummy, gsi, > >>> - &ptr_incr, false, bump); > >>> + NULL, false, dr_increment); > >>> else > >>> { > >>> gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, > >>> + stmt_info, dr_bump); > >>> } > >>> if (mask_node) > >>> vec_mask = vec_masks[j]; > >>> @@ -10909,7 +10904,7 @@ vectorizable_load (vec_info *vinfo, > >>> if (STMT_VINFO_GATHER_SCATTER_P (stmt_info)) > >>> { > >>> aggr_type = NULL_TREE; > >>> - bump = NULL_TREE; > >>> + dr_increment = NULL_TREE; > >>> if (!costing_p) > >>> vect_get_gather_scatter_ops (loop, slp_node, &dataref_ptr, > >>> &vec_offsets); > >>> @@ -10922,11 +10917,12 @@ vectorizable_load (vec_info *vinfo, > >>> vect_get_strided_load_store_ops (stmt_info, slp_node, > >>> vectype, > >>> > >>> ls.strided_offset_vectype, > >>> loop_vinfo, gsi, > >>> - &bump, &vec_offset, loop_lens); > >>> + &dr_increment, &dr_bump, > >>> + &vec_offset); > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, first_stmt_info, > >>> aggr_type, > >>> at_loop, offset, &dummy, > >>> gsi, > >>> - &ptr_incr, false, bump); > >>> + NULL, false, dr_increment); > >>> } > >>> } > >>> > >>> @@ -10950,8 +10946,8 @@ vectorizable_load (vec_info *vinfo, > >>> final_mask, vec_mask, > >>> gsi); > >>> > >>> if (i > 0 && !STMT_VINFO_GATHER_SCATTER_P (stmt_info)) > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, > >>> - gsi, stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, > >>> + stmt_info, dr_bump); > >>> } > >>> > >>> /* 2. Create the vector-load in the loop. */ > >>> @@ -11336,8 +11332,12 @@ vectorizable_load (vec_info *vinfo, > >>> > >>> aggr_type = vectype; > >>> if (!costing_p) > >>> - bump = vect_get_data_ptr_increment (vinfo, gsi, dr_info, aggr_type, > >>> - memory_access_type, loop_lens); > >>> + { > >>> + dr_increment = vect_get_data_ptr_step (vinfo, dr_info, > >>> + memory_access_type); > >>> + dr_bump = vect_get_data_ptr_bump (vinfo, dr_info, aggr_type, > >>> + memory_access_type); > >>> + } > >>> > >>> poly_uint64 group_elt = 0; > >>> unsigned int inside_cost = 0, prologue_cost = 0; > >>> @@ -11368,8 +11368,8 @@ vectorizable_load (vec_info *vinfo, > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, > >>> first_stmt_info_for_drptr, > >>> aggr_type, at_loop, offset, > >>> &dummy, > >>> - gsi, &ptr_incr, simd_lane_access_p, > >>> - bump); > >>> + gsi, NULL, simd_lane_access_p, > >>> + dr_increment); > >>> /* Adjust the pointer by the difference to first_stmt. */ > >>> data_reference_p ptrdr > >>> = STMT_VINFO_DATA_REF (first_stmt_info_for_drptr); > >>> @@ -11377,7 +11377,7 @@ vectorizable_load (vec_info *vinfo, > >>> size_binop (MINUS_EXPR, > >>> DR_INIT > >>> (first_dr_info->dr), > >>> DR_INIT (ptrdr))); > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, > >>> stmt_info, diff); > >>> if (alignment_support_scheme == dr_explicit_realign) > >>> { > >>> @@ -11393,17 +11393,17 @@ vectorizable_load (vec_info *vinfo, > >>> dataref_ptr > >>> = vect_create_data_ref_ptr (vinfo, first_stmt_info, aggr_type, > >>> at_loop, > >>> - offset, &dummy, gsi, &ptr_incr, > >>> - simd_lane_access_p, bump); > >>> + offset, &dummy, gsi, NULL, > >>> + simd_lane_access_p, dr_increment); > >>> } > >>> else if (!costing_p) > >>> { > >>> gcc_assert (!LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo)); > >>> if (dataref_offset) > >>> - dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, bump); > >>> + dataref_offset = int_const_binop (PLUS_EXPR, dataref_offset, > >>> dr_bump); > >>> else > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, > >>> + dr_bump); > >>> } > >>> > >>> auto_vec<tree> dr_chain; > >>> @@ -11429,8 +11429,8 @@ vectorizable_load (vec_info *vinfo, > >>> final_mask, vec_mask, gsi); > >>> > >>> if (i > 0) > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, > >>> - gsi, stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, > >>> + dr_bump); > >>> } > >>> > >>> /* 2. Create the vector-load in the loop. */ > >>> @@ -11776,8 +11776,7 @@ vectorizable_load (vec_info *vinfo, > >>> > >>> bump = size_binop (MULT_EXPR, vs, TYPE_SIZE_UNIT > >>> (elem_type)); > >>> bump = size_binop (MINUS_EXPR, bump, size_one_node); > >>> - ptr = bump_vector_ptr (vinfo, dataref_ptr, NULL, gsi, stmt_info, > >>> - bump); > >>> + ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, bump); > >>> new_stmt = gimple_build_assign (NULL_TREE, BIT_AND_EXPR, > >>> ptr, > >>> build_int_cst (TREE_TYPE > >>> (ptr), > >>> > >>> -(HOST_WIDE_INT) > >>> align)); > >>> @@ -11934,8 +11933,8 @@ vectorizable_load (vec_info *vinfo, > >>> if (tree_int_cst_sgn (vect_dr_behavior (vinfo, > >>> dr_info)->step) == -1) > >>> bump_val = -bump_val; > >>> tree bump = wide_int_to_tree (sizetype, bump_val); > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, > >>> + bump); > >>> group_elt = 0; > >>> } > >>> } > >>> @@ -11950,8 +11949,7 @@ vectorizable_load (vec_info *vinfo, > >>> if (tree_int_cst_sgn (vect_dr_behavior (vinfo, dr_info)->step) == > >>> -1) > >>> bump_val = -bump_val; > >>> tree bump = wide_int_to_tree (sizetype, bump_val); > >>> - dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, ptr_incr, gsi, > >>> - stmt_info, bump); > >>> + dataref_ptr = bump_vector_ptr (vinfo, dataref_ptr, gsi, stmt_info, > bump); > >>> } > >>> > >>> if (ls.slp_perm) > >>> diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h > >>> index 0b646d6b29e..92444ad46b4 100644 > >>> --- a/gcc/tree-vectorizer.h > >>> +++ b/gcc/tree-vectorizer.h > >>> @@ -978,6 +978,10 @@ public: > >>> it can be that no unrolling is needed, and thus this is 1. */ > >>> poly_uint64 vectorization_factor; > >>> > >>> + /* Gimple operand for the number of scalar iteration handed per loop > >>> + iteration, and therefore how much to increment each IV by. */ > >>> + tree iv_increment; > >>> + > >>> /* If this loop is an epilogue loop whose main loop can be skipped, > >>> MAIN_LOOP_EDGE is the edge from the main loop to this loop's > >>> preheader. SKIP_MAIN_LOOP_EDGE is then the edge that skips the > >>> @@ -1302,6 +1306,9 @@ public: > >>> #define LOOP_VINFO_ALLOW_MUTUAL_ALIGNMENT(L) (L)- > >>>> allow_mutual_alignment > >>> #define LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS(L) (L)- > >>>> partial_load_store_bias > >>> #define LOOP_VINFO_VECT_FACTOR(L) (L)->vectorization_factor > >>> +#define LOOP_VINFO_IV_INCREMENT(L) (L)->iv_increment > >>> +#define LOOP_VINFO_IV_INCREMENT_INVARIANT_P(L)\ > >>> + (!(L) || !LOOP_VINFO_USING_SELECT_VL_P (L)) > >> > >> I considered making a comment here since it's weird that loop_vec_info > >> can be null when calling a LOOP_VINFO_ macro. It looks like that's because > >> it's also called in areas where we're doing BB vectorization. > >> > >> But we don't seem to have a VINFO_ macro group. So I was considering > >> asking for you to drop the LOOP_ prefix and check is_a <..> () instead of > >> NULL check to distinguish between a NULL value being passed and it not > >> being a loop. (and so when vinfo is available pass it instead of > >> loop_vinfo). > > > > Yeah, sorry - when introducing 'vinfo' I simply directly accessed members > > rather than adding a VINFO () wrapping :/ But then for BB vect there > > shouldn't be any "IV" so I suggest to wrap the caller. > Hi Richi, > > Sorry I dont follow what youre suggesting, what do you mean wrap the caller? >
In the callers where loop_vinfo can be NULL, change from LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo) into loop_vinfo && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo) so it's explicit at the call site. Thanks, Tamar > KR, > Alfie > > > >> But I don't think it makes much of a difference here. An actual NULL would > >> fail elsewhere and so the vect changes are OK with the condition above > corrected. > >> > >> But I can't approve the RISCV test update (even though it looks like > >> trivial > scheduling changes). > >> > >> Thanks, > >> Tamar > >> > >>> #define LOOP_VINFO_MAX_VECT_FACTOR(L) (L)- > >>>> max_vectorization_factor > >>> #define LOOP_VINFO_MASKS(L) (L)->masks > >>> #define LOOP_VINFO_LENS(L) (L)->lens > >>> @@ -2650,7 +2657,7 @@ extern tree vect_create_data_ref_ptr (vec_info > *, > >>> tree *, gimple_stmt_iterator *, > >>> gimple **, bool, > >>> tree = NULL_TREE); > >>> -extern tree bump_vector_ptr (vec_info *, tree, gimple *, > gimple_stmt_iterator > >>> *, > >>> +extern tree bump_vector_ptr (vec_info *, tree, gimple_stmt_iterator *, > >>> stmt_vec_info, tree); > >>> extern void vect_copy_ref_info (tree, tree); > >>> extern tree vect_create_destination_var (tree, tree); > >>> @@ -2684,6 +2691,7 @@ extern opt_loop_vec_info vect_analyze_loop > (class > >>> loop *, gimple *, > >>> extern tree vect_build_loop_niters (loop_vec_info, bool * = NULL); > >>> extern void vect_gen_vector_loop_niters (loop_vec_info, tree, tree *, > >>> tree *, bool); > >>> +extern tree vect_get_loop_iv_increment (loop_vec_info); > >>> extern tree vect_halve_mask_nunits (tree, machine_mode); > >>> extern tree vect_double_mask_nunits (tree, machine_mode); > >>> extern void vect_record_loop_mask (loop_vec_info, vec_loop_masks *, > >>> -- > >>> 2.34.1 > >> > >> > >
