This patch builds upon the framework introduced by commit
65a3849eb46df2fbac6b41ff78dae13c85387f9e ("Synthesize early-break
live-out from scalar IV") to further improve code generation for
RISC-V targets.
The aforementioned commit established a general mechanism to synthesize
early-break live-out values from scalar IVs, enabling DCE to remove
vector dependencies. This patch refines that logic for a specific
corner case encountered on RVV: when a vect_internal_def represents
a latch update (post-increment).
In this scenario, the original logic could compute a next-iteration
value instead of the current one, leading to an off-by-one issue that
prevented the full removal of vector-to-scalar extraction sequences
(vadd.vi, vmv.x.s). By correcting this, our patch ensures the
performance benefits of the original framework are fully realized on
RISC-V, producing tighter and more efficient code for early-break loops.
For a loop like:
int64_t* find_val(int64_t* begin, int64_t* end, int64_t val) {
while (begin != end && *begin != val) {
++begin;
}
return begin;
}
This patch eliminates redundant extraction sequences (e.g., `vadd.vi`
followed by `vmv.x.s`) from the loop's exit path, resulting in smaller
and faster code.
gcc/ChangeLog:
* tree-vect-loop.cc (vectorizable_early_break_live_operation):
Replace constructing a vector-based IV finish value by adding a fixed
VF/op_offset with sourcing the scalar PHI argument from the loop
latch edge when handling main exits. This ensures the scalar IV
reflects the actual number of processed elements (including partial
vectors) instead of using an imprecise `phi + VF` adjustment.
The change synthesizes early-break live-outs from the macro scalar IV
where appropriate, avoiding a vector dependence on the original
vector IV and enabling DCE to remove redundant vector extraction.
* tree-vect-loop.cc (vectorizable_live_operation):
Call `vectorizable_early_break_live_operation` to attempt producing a
scalar-derived live-out for early-break cases; fall back to the
previous `vectorizable_live_operation_1` path when not applicable.
This integrates the new early-break scalar-IV synthesis into the
existing live-value handling.
* tree-vect-loop.cc (vect_update_ivs_after_vectorizer_for_early_breaks):
Adjust initialization of the induction `init_var` when `niters_skip`
is non-zero to correctly handle pointer-typed iteration counters by
building pointer arithmetic (negated skip) instead of assuming a
plain integer subtraction. Preserve existing behavior for integer
iteration variables.
gcc/testsuite/ChangeLog:
* g++.target/riscv/rvv/autovec/early-break-iv-extract.C: New test.
Add a compile-only testcase that exercises `find_if` style loops under
RVV autovectorization and verifies that the generated assembler does
not contain undesired scalar-from-vector extraction sequences such as
`vmv.x.s`. This test guards against regressions that would reintroduce
redundant `vadd.vi`/`vadd.vx`/`vmv` patterns or force unnecessary
scalar extraction from vectors in early-break scenarios.
Signed-off-by: kangmengbo <[email protected]>
---
Hi all,
This patch is a follow-up and architecture-specific tuning based on the
excellent vectorization improvements introduced in commit 65a3849eb46d.
While the baseline commit improved the handling of vectorization and loop
induction variables, we observed that for early-exit loops or loops returning
a pointer/iterator, the RISC-V backend still generates redundant vector
instructions to track and extract the final induction variable.
Consider this common search loop:
int64_t* find_val(int64_t* begin, int64_t* end, int64_t val) {
while (begin != end && *begin != val) {
++begin;
}
return begin;
}
When built with `-O3 -march=rv64gcv`, the exit path previously contained
costly sequences like `vadd.vi` and `vslidedown.vi` followed by `vmv.x.s`
just to extract the final scalar value from a vector register. Furthermore,
unnecessary vector additions (`vadd.vv`) were kept inside the loop body.
This patch eliminates these sequences, replacing the vector-to-scalar
extraction overhead with pure scalar arithmetic.
Assembly Delta Example:
Before:
.L14:
addi a5,a5,2
beq a5,a6,.L23
vmv1r.v v2,v3
.L5:
vle64.v v1,0(a0)
vadd.vv v3,v2,v5 // Overhead inside loop
addi a0,a0,16
vmseq.vv v1,v1,v4
vcpop.m a4,v1
beq a4,zero,.L14
slli a5,a5,3
add a0,a7,a5
// ... some other codes
.L23:
slli t1,t1,4
add a0,a7,t1
bne a5,a3,.L6
vadd.vi v1,v2,8 // \
vslidedown.vi v1,v1,1 // > Vector-to-scalar extraction overhead
vmv.x.s a0,v1 // /
ret
After:
.L14:
addi a5,a5,2
beq a6,a5,.L23
.L5:
vle64.v v1,0(a0)
addi a0,a0,16
vmseq.vv v1,v1,v2
vcpop.m a4,v1
beq a4,zero,.L5
slli a5,a5,3
add a0,t1,a5
// some other codes
.L23:
slli a7,a7,4
add a0,t1,a7
bne a6,a3,.L6
slli a6,a6,3 // \
add a0,t1,a6 // > Replaced with clean scalar operations
ret
Tested on riscv64-unknown-linux-gnu.
Regression tested with `make check-gcc RUNTESTFLAGS="riscv.exp"`, no new
failures introduced.
Looking forward to your feedback.
Thanks,
Kangmengbo
.../rvv/autovec/early-break-iv-extract.C | 24 +++
gcc/tree-vect-loop.cc | 170 +++++++++++++++++-
2 files changed, 185 insertions(+), 9 deletions(-)
create mode 100644
gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C
diff --git
a/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C
b/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C
new file mode 100644
index 00000000000..5dbe85ef33b
--- /dev/null
+++ b/gcc/testsuite/g++.target/riscv/rvv/autovec/early-break-iv-extract.C
@@ -0,0 +1,24 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target riscv_v } */
+/* { dg-options "-O3 -march=rv64gcv -mabi=lp64d" } */
+
+#include <stdint.h>
+
+template<typename _Iterator, typename _Predicate>
+inline _Iterator my_find_if (_Iterator __first, _Iterator __last, _Predicate
__pred)
+{
+ while (__first != __last && !__pred (*__first))
+ ++__first;
+ return __first;
+}
+
+struct Pred {
+ int64_t val;
+ bool operator()(int64_t x) const { return x == val; }
+};
+
+int64_t* find_val (int64_t* begin, int64_t* end, int64_t val) {
+ return my_find_if (begin, end, Pred{val});
+}
+
+/* { dg-final { scan-assembler-not {vmv.x.s} } } */
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 51e7f05100d..0c590c90a73 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -10135,6 +10135,143 @@ vectorizable_live_operation_1 (loop_vec_info
loop_vinfo, basic_block exit_bb,
return new_tree;
}
+static tree
+vectorizable_early_break_live_operation (loop_vec_info loop_vinfo,
+ stmt_vec_info stmt_info, edge e,
+ tree lhs,
+ gimple_stmt_iterator *exit_gsi,
+ bool early_break_first_element_p)
+{
+ if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ return NULL_TREE;
+
+ class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
+ stmt_vec_info ind_stmt_info = NULL;
+ tree op_offset = NULL_TREE;
+ bool op_offset_is_postinc = false;
+
+ if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
+ {
+ ind_stmt_info = stmt_info;
+ }
+ else if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_internal_def)
+ {
+ /* Elegant fix: Just check if this internal def is the latch
+ update of an induction variable. If so, it's the post-increment! */
+ gphi_iterator si;
+ for (si = gsi_start_phis (loop->header); !gsi_end_p (si); gsi_next (&si))
+ {
+ gphi *phi = si.phi ();
+ stmt_vec_info phi_info = loop_vinfo->lookup_stmt (phi);
+ if (phi_info && STMT_VINFO_DEF_TYPE (phi_info) == vect_induction_def)
+ {
+ tree latch_arg = PHI_ARG_DEF_FROM_EDGE (phi, loop_latch_edge (loop));
+ if (latch_arg == lhs)
+ {
+ ind_stmt_info = phi_info;
+ op_offset = build_one_cst (
+ TREE_TYPE (LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo)));
+ op_offset_is_postinc = true;
+ break;
+ }
+ }
+ }
+ }
+
+ if (ind_stmt_info)
+ {
+ /* Synthesize the induction variable from the scalar completion counter.
+ This completely severs the use-def chain with the vector IV,
+ allowing DCE to completely optimize away the vector updates! */
+ tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (ind_stmt_info);
+ tree static_base = PHI_ARG_DEF_FROM_EDGE (as_a<gphi
*>(ind_stmt_info->stmt),
+ loop_preheader_edge (loop));
+ tree phi_var = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo);
+ tree lhs_type = TREE_TYPE (lhs);
+ gimple_seq stmts = NULL;
+
+ tree mult_type = POINTER_TYPE_P (lhs_type) ? sizetype : lhs_type;
+ tree conv_step;
+ if (SCALAR_FLOAT_TYPE_P (mult_type)
+ && !SCALAR_FLOAT_TYPE_P (TREE_TYPE (step_expr)))
+ conv_step = gimple_build (&stmts, FLOAT_EXPR, mult_type, step_expr);
+ else
+ conv_step = gimple_convert (&stmts, mult_type, step_expr);
+
+ /* Ultimate fix: Precisely distinguish between early breaks from the first
+ element versus full vector iterations, ensuring strict type isolation
+ during AST building. GIMPLE requires pointers and integers to be
+ strictly isolated. */
+ tree phi_type = TREE_TYPE (phi_var);
+ if (!early_break_first_element_p)
+ {
+ /* Main exit: The correct scalar iterations counter is exactly the
+ post-increment latch definition of the macro scalar IV, which
+ inherently accounts for exactly how many elements were processed
+ (including partial vectors!) */
+ gphi *macro_phi = as_a<gphi *>(SSA_NAME_DEF_STMT (phi_var));
+ phi_var = PHI_ARG_DEF_FROM_EDGE (macro_phi, loop_latch_edge (loop));
+ }
+ else if (op_offset)
+ {
+ if (POINTER_TYPE_P (phi_type))
+ {
+ tree offset = gimple_convert (&stmts, sizetype, op_offset);
+ if (op_offset_is_postinc && early_break_first_element_p)
+ {
+ tree neg_offset
+ = gimple_build (&stmts, NEGATE_EXPR, sizetype, offset);
+ phi_var = gimple_build (&stmts, POINTER_PLUS_EXPR, phi_type,
+ phi_var, neg_offset);
+ }
+ else
+ phi_var = gimple_build (&stmts, POINTER_PLUS_EXPR, phi_type,
+ phi_var, offset);
+ }
+ else
+ {
+ if (op_offset_is_postinc && early_break_first_element_p)
+ phi_var = gimple_build (&stmts, MINUS_EXPR, phi_type, phi_var,
+ op_offset);
+ else
+ phi_var
+ = gimple_build (&stmts, PLUS_EXPR, phi_type, phi_var,
+ op_offset);
+ }
+ }
+
+ tree conv_phi_var;
+ if (SCALAR_FLOAT_TYPE_P (mult_type))
+ conv_phi_var = gimple_build (&stmts, FLOAT_EXPR, mult_type, phi_var);
+ else
+ conv_phi_var = gimple_convert (&stmts, mult_type, phi_var);
+ tree offset
+ = gimple_build (&stmts, MULT_EXPR, mult_type, conv_step, conv_phi_var);
+
+ tree new_tree;
+ if (POINTER_TYPE_P (lhs_type))
+ new_tree = gimple_build (&stmts, POINTER_PLUS_EXPR, lhs_type,
static_base,
+ offset);
+ else
+ new_tree
+ = gimple_build (&stmts, PLUS_EXPR, lhs_type, static_base, offset);
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "Synthesizing early-break live out from macro scalar "
+ "IV, avoiding vector dependence!\n");
+ }
+
+ *exit_gsi = gsi_after_labels (e->dest);
+ gsi_insert_seq_before (exit_gsi, stmts, GSI_SAME_STMT);
+
+ return new_tree;
+ }
+
+ return NULL_TREE;
+}
+
/* Function vectorizable_live_operation.
STMT_INFO computes a value that is used outside the loop. Check if
@@ -10370,12 +10507,13 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
}
gimple_stmt_iterator exit_gsi;
- tree new_tree
- = vectorizable_live_operation_1 (loop_vinfo,
- e->dest, vectype,
- slp_node, bitsize,
- tmp_bitstart, tmp_vec_lhs,
- lhs_type, &exit_gsi);
+ tree new_tree = vectorizable_early_break_live_operation (
+ loop_vinfo, stmt_info, e, lhs, &exit_gsi,
+ early_break_first_element_p);
+ if (!new_tree)
+ new_tree = vectorizable_live_operation_1 (
+ loop_vinfo, e->dest, vectype, slp_node, bitsize,
+ tmp_bitstart, tmp_vec_lhs, lhs_type, &exit_gsi);
auto gsi = gsi_for_stmt (use_stmt);
tree lhs_phi = gimple_phi_result (use_stmt);
@@ -11111,10 +11249,24 @@ vect_update_ivs_after_vectorizer_for_early_breaks
(loop_vec_info loop_vinfo)
}
tree init_var = build_zero_cst (ty_var);
- if (niters_skip)
- init_var = gimple_build (&init_stmts, MINUS_EXPR, ty_var, init_var,
- gimple_convert (&init_stmts, ty_var, niters_skip));
+ if (niters_skip)
+ {
+ if (POINTER_TYPE_P (ty_var))
+ {
+ tree skip_sz = gimple_convert (&init_stmts, sizetype, niters_skip);
+ tree neg_skip
+ = gimple_build (&init_stmts, NEGATE_EXPR, sizetype, skip_sz);
+ init_var = gimple_build (&init_stmts, POINTER_PLUS_EXPR, ty_var,
init_var,
+ neg_skip);
+ }
+ else
+ {
+ init_var
+ = gimple_build (&init_stmts, MINUS_EXPR, ty_var, init_var,
+ gimple_convert (&init_stmts, ty_var, niters_skip));
+ }
+ }
add_phi_arg (induction_phi, iter_var,
loop_latch_edge (loop), UNKNOWN_LOCATION);
add_phi_arg (induction_phi, init_var,
--
2.25.1
