On Wed, 10 Jun 2026, Tamar Christina wrote: > > -----Original Message----- > > From: kangmengbo <[email protected]> > > Sent: 09 June 2026 09:39 > > To: [email protected] > > Cc: [email protected]; Tamar Christina <[email protected]>; > > [email protected]; [email protected]; [email protected]; > > kangmengbo <[email protected]> > > Subject: [PATCH] [PATCH] RISC-V: Replace vector IVs with scalar IVs > > > > This patch builds upon the framework introduced by commit > > 65a3849eb46df2fbac6b41ff78dae13c85387f9e ("Synthesize early-break > > live-out from scalar IV") to further improve code generation for > > RISC-V targets. > > > > The aforementioned commit established a general mechanism to synthesize > > early-break live-out values from scalar IVs, enabling DCE to remove > > vector dependencies. This patch refines that logic for a specific > > corner case encountered on RVV: when a vect_internal_def represents > > a latch update (post-increment). > > > > In this scenario, the original logic could compute a next-iteration > > value instead of the current one, leading to an off-by-one issue that > > prevented the full removal of vector-to-scalar extraction sequences > > (vadd.vi, vmv.x.s). By correcting this, our patch ensures the > > performance benefits of the original framework are fully realized on > > RISC-V, producing tighter and more efficient code for early-break loops. > > > > For a loop like: > > > > int64_t* find_val(int64_t* begin, int64_t* end, int64_t val) { > > while (begin != end && *begin != val) { > > ++begin; > > } > > return begin; > > } > > > > This patch eliminates redundant extraction sequences (e.g., `vadd.vi` > > followed by `vmv.x.s`) from the loop's exit path, resulting in smaller > > and faster code. > > > > > > gcc/ChangeLog: > > > > * tree-vect-loop.cc (vectorizable_early_break_live_operation): > > Replace constructing a vector-based IV finish value by adding a fixed > > VF/op_offset with sourcing the scalar PHI argument from the loop > > latch edge when handling main exits. This ensures the scalar IV > > reflects the actual number of processed elements (including partial > > vectors) instead of using an imprecise `phi + VF` adjustment. > > The change synthesizes early-break live-outs from the macro scalar IV > > where appropriate, avoiding a vector dependence on the original > > vector IV and enabling DCE to remove redundant vector extraction. > > > > * tree-vect-loop.cc (vectorizable_live_operation): > > Call `vectorizable_early_break_live_operation` to attempt producing a > > scalar-derived live-out for early-break cases; fall back to the > > previous `vectorizable_live_operation_1` path when not applicable. > > This integrates the new early-break scalar-IV synthesis into the > > existing live-value handling. > > > > * tree-vect-loop.cc (vect_update_ivs_after_vectorizer_for_early_breaks): > > Adjust initialization of the induction `init_var` when `niters_skip` > > is non-zero to correctly handle pointer-typed iteration counters by > > building pointer arithmetic (negated skip) instead of assuming a > > plain integer subtraction. Preserve existing behavior for integer > > iteration variables. > > > > gcc/testsuite/ChangeLog: > > > > * g++.target/riscv/rvv/autovec/early-break-iv-extract.C: New test. > > Add a compile-only testcase that exercises `find_if` style loops under > > RVV autovectorization and verifies that the generated assembler does > > not contain undesired scalar-from-vector extraction sequences such as > > `vmv.x.s`. This test guards against regressions that would reintroduce > > redundant `vadd.vi`/`vadd.vx`/`vmv` patterns or force unnecessary > > scalar extraction from vectors in early-break scenarios. > > > > Hi Kangmengbo, > > Thanks for the patch! > > I don't think this is a RISCV specific issue nor an early break one, it's just > often hidden by other optimization passes when using non-early break. > > So for instance the issue happens with just > > #include <stdint.h> > > #define N 64 > > int64_t live_iv (int64_t start, int n, int64_t *x) > { > int64_t iv = start; > > for (int j = 0; j < n; ++j) > { > iv += 5; > x[j] = iv; > } > > return iv; > } > > Compiled with -O3 -fno-tree-scev-cprop > > Which guides where the actual fix should go I think. > > The issue is that during the splitting of the guard block between the > vector body and the scalar epilogue when skip_epilog we add a skip > edge between the guard block and the scalar loop. > > vect_update_ivs_after_vectorizer takes care of updating the main > exit, since we don't really need a scalar IV here, if we've taken the > main latch exit the final value is just init_expr + niters_vector * step * VF. > > But when the edge for the epilog is split, the values are copied from > the original exit which has the live value instead of the calculated > final value. > > Unfortunately we can't call vect_update_ivs_after_vectorizer on guard_e > because the LOOP_VINFO_MAIN_EXIT contains PHI nodes we've forced > live. So the order of the nodes between the skip edge and the exit edge > may not be the same. > > g:65a3849eb46df2fbac6b41ff78dae13c85387f9e worked around this by > in vect_update_ivs_after_vectorizer rewrite all out of loop usages which > fixes the missed optimization but is a bit blunt. > > In the fix for PR123983 this was reverted, but re-introducing the missed > optimization. > > It looks like your patch tries to work around it by re-computing the value > If the live value is used and early break. But the recomputing of the PHI > like this is a bit messy and we're duplicating code here a bit, and doesn't > fix the general case. > > So I think this is a fixup that needs to happen when skip_epilog, > > And so I wonder whether the fixup code for > > /* If we have a peeled vector iteration we will never skip the epilog > loop > and we can simplify the cfg a lot by not doing the edge split. */ > if (skip_epilog > || (LOOP_VINFO_EARLY_BREAKS (loop_vinfo) > && !LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))) > > Can't be adjusted so that we can call vect_update_ivs_after_vectorizer > > So for instance > > diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc > index d2c854e16d4..86c9f88770c 100644 > --- a/gcc/tree-vect-loop-manip.cc > +++ b/gcc/tree-vect-loop-manip.cc > @@ -3877,6 +3877,13 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree > niters, tree nitersm1, > update_e, > recalculate_peel_niters_init); > > + if (skip_epilog > + || (LOOP_VINFO_EARLY_BREAKS (loop_vinfo) > + && !LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo)) > + vect_update_ivs_after_vectorizer (loop_vinfo, vector_iters_vf, > + guard_e, > + recalculate_peel_niters_init); > + > /* Recalculate the dominators after adding the guard edge. */ > if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) > iterate_fix_dominators (CDI_DOMINATORS, doms, false); > > fixes the attached testcase, but isn't safe due to the PHI ordering. > > So now that we have 2 testcases Richi, do you agree that this is best fixed > in the iv updating > code after peeling rather than during vectorization of the live operations?
I agree that we should handle this better when we setup things during peeling. We should ideally be able to remove that requirement of lock-step PHIs -- only the loop header PHIs should need to appear in the same order and we should be able to correlate others by backtracking to the respective loop header PHI. Iff needed at all. But I know it's always a pain to change the peeling code because it's so fragile (fix that! somehow!). Thanks and sorry for the delay, Richard. > > Thanks, > Tamar > > > Signed-off-by: kangmengbo <[email protected]> > > --- > > > > Hi all, > > > > This patch is a follow-up and architecture-specific tuning based on the > > excellent vectorization improvements introduced in commit 65a3849eb46d. > > > > While the baseline commit improved the handling of vectorization and loop > > induction variables, we observed that for early-exit loops or loops > > returning > > a pointer/iterator, the RISC-V backend still generates redundant vector > > instructions to track and extract the final induction variable. > > > > Consider this common search loop: > > > > int64_t* find_val(int64_t* begin, int64_t* end, int64_t val) { > > while (begin != end && *begin != val) { > > ++begin; > > } > > return begin; > > } > > > > When built with `-O3 -march=rv64gcv`, the exit path previously contained > > costly sequences like `vadd.vi` and `vslidedown.vi` followed by `vmv.x.s` > > just to extract the final scalar value from a vector register. Furthermore, > > unnecessary vector additions (`vadd.vv`) were kept inside the loop body. > > > > This patch eliminates these sequences, replacing the vector-to-scalar > > extraction overhead with pure scalar arithmetic. > > > > Assembly Delta Example: > > > > Before: > > .L14: > > addi a5,a5,2 > > beq a5,a6,.L23 > > vmv1r.v v2,v3 > > .L5: > > vle64.v v1,0(a0) > > vadd.vv v3,v2,v5 // Overhead inside loop > > addi a0,a0,16 > > vmseq.vv v1,v1,v4 > > vcpop.m a4,v1 > > beq a4,zero,.L14 > > slli a5,a5,3 > > add a0,a7,a5 > > // ... some other codes > > .L23: > > slli t1,t1,4 > > add a0,a7,t1 > > bne a5,a3,.L6 > > vadd.vi v1,v2,8 // \ > > vslidedown.vi v1,v1,1 // > Vector-to-scalar extraction overhead > > vmv.x.s a0,v1 // / > > ret > > > > After: > > .L14: > > addi a5,a5,2 > > beq a6,a5,.L23 > > .L5: > > vle64.v v1,0(a0) > > addi a0,a0,16 > > vmseq.vv v1,v1,v2 > > vcpop.m a4,v1 > > beq a4,zero,.L5 > > slli a5,a5,3 > > add a0,t1,a5 > > // some other codes > > .L23: > > slli a7,a7,4 > > add a0,t1,a7 > > bne a6,a3,.L6 > > slli a6,a6,3 // \ > > add a0,t1,a6 // > Replaced with clean scalar operations > > ret > > > > Tested on riscv64-unknown-linux-gnu. > > Regression tested with `make check-gcc RUNTESTFLAGS="riscv.exp"`, no new > > failures introduced. > > > > Looking forward to your feedback. > > > > Thanks, > > Kangmengbo > > > > .../rvv/autovec/early-break-iv-extract.C | 24 +++ > > gcc/tree-vect-loop.cc | 170 +++++++++++++++++- > > 2 files changed, 185 insertions(+), 9 deletions(-) > > create mode 100644 gcc/testsuite/g++.target/riscv/rvv/autovec/early-break- > > iv-extract.C > > > > diff --git > > a/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C > > b/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C > > new file mode 100644 > > index 00000000000..5dbe85ef33b > > --- /dev/null > > +++ b/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C > > @@ -0,0 +1,24 @@ > > +/* { dg-do compile } */ > > +/* { dg-require-effective-target riscv_v } */ > > +/* { dg-options "-O3 -march=rv64gcv -mabi=lp64d" } */ > > + > > +#include <stdint.h> > > + > > +template<typename _Iterator, typename _Predicate> > > +inline _Iterator my_find_if (_Iterator __first, _Iterator __last, > > _Predicate > > __pred) > > +{ > > + while (__first != __last && !__pred (*__first)) > > + ++__first; > > + return __first; > > +} > > + > > +struct Pred { > > + int64_t val; > > + bool operator()(int64_t x) const { return x == val; } > > +}; > > + > > +int64_t* find_val (int64_t* begin, int64_t* end, int64_t val) { > > + return my_find_if (begin, end, Pred{val}); > > +} > > + > > +/* { dg-final { scan-assembler-not {vmv.x.s} } } */ > > diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc > > index 51e7f05100d..0c590c90a73 100644 > > --- a/gcc/tree-vect-loop.cc > > +++ b/gcc/tree-vect-loop.cc > > @@ -10135,6 +10135,143 @@ vectorizable_live_operation_1 (loop_vec_info > > loop_vinfo, basic_block exit_bb, > > return new_tree; > > } > > > > +static tree > > +vectorizable_early_break_live_operation (loop_vec_info loop_vinfo, > > + stmt_vec_info stmt_info, edge e, > > + tree lhs, > > + gimple_stmt_iterator *exit_gsi, > > + bool early_break_first_element_p) > > +{ > > + if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo)) > > + return NULL_TREE; > > + > > + class loop *loop = LOOP_VINFO_LOOP (loop_vinfo); > > + stmt_vec_info ind_stmt_info = NULL; > > + tree op_offset = NULL_TREE; > > + bool op_offset_is_postinc = false; > > + > > + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def) > > + { > > + ind_stmt_info = stmt_info; > > + } > > + else if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_internal_def) > > + { > > + /* Elegant fix: Just check if this internal def is the latch > > + update of an induction variable. If so, it's the post-increment! */ > > + gphi_iterator si; > > + for (si = gsi_start_phis (loop->header); !gsi_end_p (si); gsi_next > > (&si)) > > + { > > + gphi *phi = si.phi (); > > + stmt_vec_info phi_info = loop_vinfo->lookup_stmt (phi); > > + if (phi_info && STMT_VINFO_DEF_TYPE (phi_info) == vect_induction_def) > > + { > > + tree latch_arg = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge > > (loop)); > > + if (latch_arg == lhs) > > + { > > + ind_stmt_info = phi_info; > > + op_offset = build_one_cst ( > > + TREE_TYPE (LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo))); > > + op_offset_is_postinc = true; > > + break; > > + } > > + } > > + } > > + } > > + > > + if (ind_stmt_info) > > + { > > + /* Synthesize the induction variable from the scalar completion > > counter. > > + This completely severs the use-def chain with the vector IV, > > + allowing DCE to completely optimize away the vector updates! */ > > + tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART > > (ind_stmt_info); > > + tree static_base = PHI_ARG_DEF_FROM_EDGE (as_a<gphi > > *>(ind_stmt_info->stmt), > > + loop_preheader_edge (loop)); > > + tree phi_var = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo); > > + tree lhs_type = TREE_TYPE (lhs); > > + gimple_seq stmts = NULL; > > + > > + tree mult_type = POINTER_TYPE_P (lhs_type) ? sizetype : lhs_type; > > + tree conv_step; > > + if (SCALAR_FLOAT_TYPE_P (mult_type) > > + && !SCALAR_FLOAT_TYPE_P (TREE_TYPE (step_expr))) > > + conv_step = gimple_build (&stmts, FLOAT_EXPR, mult_type, step_expr); > > + else > > + conv_step = gimple_convert (&stmts, mult_type, step_expr); > > + > > + /* Ultimate fix: Precisely distinguish between early breaks from the > > first > > + element versus full vector iterations, ensuring strict type > > isolation > > + during AST building. GIMPLE requires pointers and integers to be > > + strictly isolated. */ > > + tree phi_type = TREE_TYPE (phi_var); > > + if (!early_break_first_element_p) > > + { > > + /* Main exit: The correct scalar iterations counter is exactly the > > + post-increment latch definition of the macro scalar IV, which > > + inherently accounts for exactly how many elements were processed > > + (including partial vectors!) */ > > + gphi *macro_phi = as_a<gphi *>(SSA_NAME_DEF_STMT (phi_var)); > > + phi_var = PHI_ARG_DEF_FROM_EDGE (macro_phi, loop_latch_edge > > (loop)); > > + } > > + else if (op_offset) > > + { > > + if (POINTER_TYPE_P (phi_type)) > > + { > > + tree offset = gimple_convert (&stmts, sizetype, op_offset); > > + if (op_offset_is_postinc && early_break_first_element_p) > > + { > > + tree neg_offset > > + = gimple_build (&stmts, NEGATE_EXPR, sizetype, offset); > > + phi_var = gimple_build (&stmts, POINTER_PLUS_EXPR, phi_type, > > + phi_var, neg_offset); > > + } > > + else > > + phi_var = gimple_build (&stmts, POINTER_PLUS_EXPR, phi_type, > > + phi_var, offset); > > + } > > + else > > + { > > + if (op_offset_is_postinc && early_break_first_element_p) > > + phi_var = gimple_build (&stmts, MINUS_EXPR, phi_type, phi_var, > > + op_offset); > > + else > > + phi_var > > + = gimple_build (&stmts, PLUS_EXPR, phi_type, phi_var, > > + op_offset); > > + } > > + } > > + > > + tree conv_phi_var; > > + if (SCALAR_FLOAT_TYPE_P (mult_type)) > > + conv_phi_var = gimple_build (&stmts, FLOAT_EXPR, mult_type, phi_var); > > + else > > + conv_phi_var = gimple_convert (&stmts, mult_type, phi_var); > > + tree offset > > + = gimple_build (&stmts, MULT_EXPR, mult_type, conv_step, > > conv_phi_var); > > + > > + tree new_tree; > > + if (POINTER_TYPE_P (lhs_type)) > > + new_tree = gimple_build (&stmts, POINTER_PLUS_EXPR, lhs_type, > > static_base, > > + offset); > > + else > > + new_tree > > + = gimple_build (&stmts, PLUS_EXPR, lhs_type, static_base, offset); > > + > > + if (dump_enabled_p ()) > > + { > > + dump_printf_loc (MSG_NOTE, vect_location, > > + "Synthesizing early-break live out from macro scalar " > > + "IV, avoiding vector dependence!\n"); > > + } > > + > > + *exit_gsi = gsi_after_labels (e->dest); > > + gsi_insert_seq_before (exit_gsi, stmts, GSI_SAME_STMT); > > + > > + return new_tree; > > + } > > + > > + return NULL_TREE; > > +} > > + > > /* Function vectorizable_live_operation. > > > > STMT_INFO computes a value that is used outside the loop. Check if > > @@ -10370,12 +10507,13 @@ vectorizable_live_operation (vec_info *vinfo, > > stmt_vec_info stmt_info, > > } > > > > gimple_stmt_iterator exit_gsi; > > - tree new_tree > > - = vectorizable_live_operation_1 (loop_vinfo, > > - e->dest, vectype, > > - slp_node, bitsize, > > - tmp_bitstart, tmp_vec_lhs, > > - lhs_type, &exit_gsi); > > + tree new_tree = vectorizable_early_break_live_operation ( > > + loop_vinfo, stmt_info, e, lhs, &exit_gsi, > > + early_break_first_element_p); > > + if (!new_tree) > > + new_tree = vectorizable_live_operation_1 ( > > + loop_vinfo, e->dest, vectype, slp_node, bitsize, > > + tmp_bitstart, tmp_vec_lhs, lhs_type, &exit_gsi); > > > > auto gsi = gsi_for_stmt (use_stmt); > > tree lhs_phi = gimple_phi_result (use_stmt); > > @@ -11111,10 +11249,24 @@ > > vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info loop_vinfo) > > } > > > > tree init_var = build_zero_cst (ty_var); > > - if (niters_skip) > > - init_var = gimple_build (&init_stmts, MINUS_EXPR, ty_var, init_var, > > - gimple_convert (&init_stmts, ty_var, niters_skip)); > > > > + if (niters_skip) > > + { > > + if (POINTER_TYPE_P (ty_var)) > > + { > > + tree skip_sz = gimple_convert (&init_stmts, sizetype, niters_skip); > > + tree neg_skip > > + = gimple_build (&init_stmts, NEGATE_EXPR, sizetype, skip_sz); > > + init_var = gimple_build (&init_stmts, POINTER_PLUS_EXPR, ty_var, > > init_var, > > + neg_skip); > > + } > > + else > > + { > > + init_var > > + = gimple_build (&init_stmts, MINUS_EXPR, ty_var, init_var, > > + gimple_convert (&init_stmts, ty_var, niters_skip)); > > + } > > + } > > add_phi_arg (induction_phi, iter_var, > > loop_latch_edge (loop), UNKNOWN_LOCATION); > > add_phi_arg (induction_phi, init_var, > > -- > > 2.25.1 > > -- Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
