Replaces several bits of incorrect, repeated, or misleading logic with
logic based on iv increment reasoning functions.
Additionally, changes vect_get_loop_iv_increment to return a GENERIC
tree statement, so that it can be used in vect_do_peeling, without
adding statements that get duplicated.
Then add vect_force_iv_increment_to_gimple to create the potentially
required gimple statements and SSA nodes for IV increment.
Also updates some tests which change due to added extra knowledge about
loop bounds.
gcc/ChangeLog:
* config/aarch64/aarch64.cc (record_potential_advsimd_unrolling):
Change to use vect_iv_increment_for_cost.
(aarch64_vector_costs::analyze_loop_vinfo): Likewise.
(aarch64_vect_vf_for_cost): Likewise.
(determine_suggested_unroll_factor): Likewise.
* config/i386/i386.cc (ix86_vector_costs::finish_cost): Likewise.
* config/riscv/riscv-vector-costs.cc (estimate_lmul): Likewise.
(costs::record_potential_vls_unrolling): Likewise.
(estimated_loop_iters): Likewise.
(costs::compare_lmul_to): Likewise.
(costs::better_main_loop_than_p): Likewise.
* config/rs6000/rs6000.cc
(rs6000_cost_data::determine_suggested_unroll_factor):
Likewise.
* tree-vect-data-refs.cc (vect_enhance_data_refs_alignment):
Likewise.
* tree-vect-loop-manip.cc (vect_set_loop_controls_directly):
Add assert for invalid case.
(vect_set_loop_condition_normal): Change to remove uses of loop
vf.
(vect_gen_vector_loop_niters): Remove incorrect uses of loop VF
and change to use new iv reasoning functions.
(vect_get_loop_iv_increment): Change to not add new functions.
(vect_force_iv_increment_to_gimple): New function.
(vect_gen_vector_loop_niters_mult_vf): Change to use
iv_increment reasoning functions.
(vect_do_peeling): Likewise.
* tree-vect-loop.cc (_loop_vec_info::_loop_vec_info):
Initialize iv_increment to null.
(vect_need_peeling_or_partial_vectors_p): Change to use
iv_increment reasoning functions.
(vect_known_niters_smaller_than_vf): Change name to...
(vect_known_niters_smaller_than_iv_increment): ...this and
update to use new iv increment reasoning functions.
(vect_analyze_loop_1): Set IV_INCREMENT to null when restarting
analysis.
(vect_transform_loop): Change iv_increment intialization then
use vect_force_iv_increment_to_gimple to emit the value, and
use new iv reasoning functions.
(vect_analyze_loop_costing): Change to use new iv increment
reasoning functions.
(vect_determine_partial_vectors_and_peeling): Likewise.
(vect_get_peel_iters_epilogue): Likewise.
(vect_estimate_min_profitable_iters): Likewise.
(vect_update_early_break_profiles): Likewise.
(vect_iv_limit_for_partial_vectors): Likewise.
* tree-vectorizer.h (vect_vf_for_cost): Change to...
(vect_iv_increment_for_cost): ...this
(vect_max_vf): Change to use maybe_upper_bound.
(vect_max_iv_increment): New iv increment reasoning function.
(vect_min_iv_increment): Likewise.
(vect_known_const_iv_increment): Likewise.
(vect_known_iv_increment): Likewise.
(vect_likely_min_iv_increment): Likewise.
(vect_apply_runtime_profitability_check_p): Change to use new iv
increment reasoning functions.
(vect_force_iv_increment_to_gimple): New function.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/sve/noeffect11.c: Update test.
* gcc.target/aarch64/sve/roundeven_1.c: Update test.
* gcc.target/aarch64/sve/strided_load_5.c: Update test.
* gcc.target/aarch64/sve/strided_load_6.c: Update test.
* gcc.target/aarch64/sve/strided_store_6.c: Update test.
---
gcc/config/aarch64/aarch64.cc | 9 +-
gcc/config/i386/i386.cc | 2 +-
gcc/config/riscv/riscv-vector-costs.cc | 14 +-
gcc/config/rs6000/rs6000.cc | 2 +-
.../gcc.target/aarch64/sve/noeffect11.c | 4 +-
.../gcc.target/aarch64/sve/roundeven_1.c | 2 +-
.../gcc.target/aarch64/sve/strided_load_5.c | 5 +-
.../gcc.target/aarch64/sve/strided_load_6.c | 3 +-
.../gcc.target/aarch64/sve/strided_store_6.c | 6 +-
gcc/tree-vect-data-refs.cc | 5 +-
gcc/tree-vect-loop-manip.cc | 90 +++++++++----
gcc/tree-vect-loop.cc | 125 +++++++++++-------
gcc/tree-vectorizer.h | 116 +++++++++++++---
13 files changed, 270 insertions(+), 113 deletions(-)
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index b961833de02..3bdbe307b62 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -18003,7 +18003,8 @@ record_potential_advsimd_unrolling (loop_vec_info
loop_vinfo)
number of iterations is small enough, the number of statements might
not be, which is why we need to estimate the number of statements too. */
unsigned int estimated_vq = aarch64_estimated_sve_vq ();
- unsigned int advsimd_vf = CEIL (vect_vf_for_cost (loop_vinfo), estimated_vq);
+ unsigned int advsimd_vf
+ = CEIL (vect_iv_increment_for_cost (loop_vinfo), estimated_vq);
unsigned HOST_WIDE_INT unrolled_advsimd_niters
= LOOP_VINFO_INT_NITERS (loop_vinfo) / advsimd_vf;
if (unrolled_advsimd_niters > (unsigned int) param_max_completely_peel_times)
@@ -18025,7 +18026,7 @@ aarch64_vector_costs::analyze_loop_vinfo (loop_vec_info
loop_vinfo)
auto scalar_niters = max_stmt_executions_int (loop);
if (scalar_niters >= 0)
{
- unsigned int vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vf = vect_iv_increment_for_cost (loop_vinfo);
if (LOOP_VINFO_MASKS (loop_vinfo).is_empty ())
m_num_vector_iterations = scalar_niters / vf;
else
@@ -19201,7 +19202,7 @@ aarch64_external_adjust_stmt_cost (vect_cost_for_stmt
kind, slp_tree node,
static inline unsigned int
aarch64_vect_vf_for_cost (loop_vec_info loop_vinfo)
{
- unsigned int estimated_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int estimated_vf = vect_iv_increment_for_cost (loop_vinfo);
/* If we know we have a single partial vector iteration, cap the VF
to the number of scalar iterations for costing purposes. */
@@ -19563,7 +19564,7 @@ determine_suggested_unroll_factor (loop_vec_info
loop_vinfo)
if (LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
{
unsigned int niters = LOOP_VINFO_INT_NITERS (loop_vinfo);
- unsigned int estimated_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int estimated_vf = vect_iv_increment_for_cost (loop_vinfo);
max_unroll_factor = MIN (max_unroll_factor, CEIL (niters, estimated_vf));
}
diff --git a/gcc/config/i386/i386.cc b/gcc/config/i386/i386.cc
index ad34cfeda02..42a6ae3b57f 100644
--- a/gcc/config/i386/i386.cc
+++ b/gcc/config/i386/i386.cc
@@ -27676,7 +27676,7 @@ ix86_vector_costs::finish_cost (const vector_costs
*scalar_costs)
loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (m_vinfo);
if (loop_vinfo && !m_costing_for_scalar)
{
- unsigned int vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vf = vect_iv_increment_for_cost (loop_vinfo);
for (auto inst : LOOP_VINFO_SLP_INSTANCES (loop_vinfo))
if ((SLP_INSTANCE_KIND (inst) == slp_inst_kind_reduc_group
|| SLP_INSTANCE_KIND (inst) == slp_inst_kind_reduc_chain)
diff --git a/gcc/config/riscv/riscv-vector-costs.cc
b/gcc/config/riscv/riscv-vector-costs.cc
index d26a390793c..d9945324a7d 100644
--- a/gcc/config/riscv/riscv-vector-costs.cc
+++ b/gcc/config/riscv/riscv-vector-costs.cc
@@ -666,7 +666,7 @@ static int
estimate_lmul (loop_vec_info loop_vinfo, machine_mode mode)
{
gcc_assert (GET_MODE_BITSIZE (mode).is_constant ());
- int estimated_vf = vect_vf_for_cost (loop_vinfo);
+ int estimated_vf = vect_iv_increment_for_cost (loop_vinfo);
int estimated_lmul = estimated_vf * GET_MODE_BITSIZE (mode).to_constant ()
/ TARGET_MIN_VLEN;
estimated_lmul = std::min (estimated_lmul, (int) RVV_M8);
@@ -1022,7 +1022,7 @@ costs::record_potential_vls_unrolling (loop_vec_info
loop_vinfo)
sure that it is within the complete unrolling limit. Even if the
number of iterations is small enough, the number of statements might
not be, which is why we need to estimate the number of statements too. */
- unsigned int vls_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vls_vf = vect_iv_increment_for_cost (loop_vinfo);
unsigned HOST_WIDE_INT unrolled_vls_niters
= LOOP_VINFO_INT_NITERS (loop_vinfo) / vls_vf;
if (unrolled_vls_niters > (unsigned int) param_max_completely_peel_times)
@@ -1068,7 +1068,7 @@ estimated_loop_iters (loop_vec_info loop_vinfo)
return HOST_WIDE_INT_M1U;
unsigned HOST_WIDE_INT scalar_niters = LOOP_VINFO_INT_NITERS (loop_vinfo);
- unsigned int vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vf = vect_iv_increment_for_cost (loop_vinfo);
return (LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)
? CEIL (scalar_niters, vf)
: scalar_niters / vf);
@@ -1145,8 +1145,8 @@ costs::compare_lmul_to (const vector_costs *uncast_other)
const
/* First, filter out LMULs that exceed the maximum if we have one. */
if (rvv_max_lmul == RVV_CONV_DYNAMIC)
{
- int this_vf = vect_vf_for_cost (this_loop_vinfo);
- int other_vf = vect_vf_for_cost (other_loop_vinfo);
+ int this_vf = vect_iv_increment_for_cost (this_loop_vinfo);
+ int other_vf = vect_iv_increment_for_cost (other_loop_vinfo);
unsigned int largest_elsz
= tree_to_uhwi (TYPE_SIZE (this->m_largest_type));
@@ -1193,9 +1193,9 @@ costs::better_main_loop_than_p (const vector_costs
*uncast_other) const
dump_printf_loc (MSG_NOTE, vect_location,
"Comparing two main loops (%s at VF %d vs %s at VF %d)\n",
GET_MODE_NAME (this_loop_vinfo->vector_mode),
- vect_vf_for_cost (this_loop_vinfo),
+ vect_iv_increment_for_cost (this_loop_vinfo),
GET_MODE_NAME (other_loop_vinfo->vector_mode),
- vect_vf_for_cost (other_loop_vinfo));
+ vect_iv_increment_for_cost (other_loop_vinfo));
int diff = compare_lmul_to (uncast_other);
if (diff)
diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc
index 9581d122a9d..b34b31e6712 100644
--- a/gcc/config/rs6000/rs6000.cc
+++ b/gcc/config/rs6000/rs6000.cc
@@ -5604,7 +5604,7 @@ rs6000_cost_data::determine_suggested_unroll_factor
(loop_vec_info loop_vinfo)
/* Check if we can conclude it's good to unroll from the estimated
iteration count. */
HOST_WIDE_INT est_niter = get_estimated_loop_iterations_int (loop);
- unsigned int vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vf = vect_iv_increment_for_cost (loop_vinfo);
unsigned int unrolled_vf = vf * uf;
if (est_niter == -1 || est_niter < unrolled_vf)
/* When the estimated iteration of this loop is unknown, it's possible
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/noeffect11.c
b/gcc/testsuite/gcc.target/aarch64/sve/noeffect11.c
index 5c8c3565da6..0f30f11196c 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/noeffect11.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/noeffect11.c
@@ -6,7 +6,7 @@
/*
** f1:
** ...
-** whilelo p[0-9]+\.d, x[0-9]+, x[0-9]+
+** whilelo p[0-9]+\.d, [xw][0-9]+, [xw][0-9]+
** ...
** ld1d z[0-9]+\.d, p[0-9]+/z, \[x[0-9]+, x[0-9]+, lsl 3\]
** ld1d z[0-9]+\.d, p[0-9]+/z, \[x[0-9]+, x[0-9]+, lsl 3\]
@@ -36,7 +36,7 @@ int f1 (const unsigned long *restrict a, const unsigned long
*b, int n)
/*
** f2:
** ...
-** whilelo p[0-9]+\.d, x[0-9]+, x[0-9]+
+** whilelo p[0-9]+\.d, [xw][0-9]+, [xw][0-9]+
** ...
** ld1d z[0-9]+\.d, p[0-9]+/z, \[x[0-9]+, x[0-9]+, lsl 3\]
** ld1d z[0-9]+\.d, p[0-9]+/z, \[x[0-9]+, x[0-9]+, lsl 3\]
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/roundeven_1.c
b/gcc/testsuite/gcc.target/aarch64/sve/roundeven_1.c
index 1348ae8fb13..d0b7441c771 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/roundeven_1.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/roundeven_1.c
@@ -24,5 +24,5 @@ revh (_Float16 *__restrict d, _Float16 *__restrict a, int n)
/* { dg-final { scan-assembler-times {\tfrintn\tz[0-9]+\.d, p[0-9]+/m,
z[0-9]+\.d} 1 } } */
/* { dg-final { scan-assembler-times {\tfrintn\tz[0-9]+\.s, p[0-9]+/m,
z[0-9]+\.s} 1 } } */
-/* { dg-final { scan-assembler-times {\tfrintn\tz[0-9]+\.h, p[0-9]+/m,
z[0-9]+\.h} 3 } } */
+/* { dg-final { scan-assembler-times {\tfrintn\tz[0-9]+\.h, p[0-9]+/m,
z[0-9]+\.h} 2 } } */
/* { dg-final { scan-assembler-not {\tfrintn\tv[0-9]+\.} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/strided_load_5.c
b/gcc/testsuite/gcc.target/aarch64/sve/strided_load_5.c
index 415b466d698..e39f3f45567 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/strided_load_5.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/strided_load_5.c
@@ -29,6 +29,9 @@
TEST_ALL (TEST_LOOP)
+/* { dg-final { scan-assembler-times {\tld1d\tz[0-9]+\.d, p[0-7]/z, \[x[0-9]+,
z[0-9]+.d\]\n} 15 } } */
+
+/* We can use signed values for float gather statters as we know VL < 2048 and
+ so the ranges will be expressible as int32 offsets. */
/* { dg-final { scan-assembler-times {\tld1w\tz[0-9]+\.s, p[0-7]/z, \[x[0-9]+,
z[0-9]+.s, uxtw\]\n} 12 } } */
/* { dg-final { scan-assembler-times {\tld1w\tz[0-9]+\.s, p[0-7]/z, \[x[0-9]+,
z[0-9]+.s, sxtw\]\n} 3 } } */
-/* { dg-final { scan-assembler-times {\tld1d\tz[0-9]+\.d, p[0-7]/z, \[x[0-9]+,
z[0-9]+.d\]\n} 15 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/strided_load_6.c
b/gcc/testsuite/gcc.target/aarch64/sve/strided_load_6.c
index 9e0001551c1..4f87d9e5103 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/strided_load_6.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/strided_load_6.c
@@ -3,5 +3,6 @@
#include "strided_load_5.c"
-/* { dg-final { scan-assembler-not {\[x[0-9]+, z[0-9]+\.s} } } */
+/* { dg-final { scan-assembler-times {\tld1w\tz[0-9]+\.s, p[0-7]/z, \[x[0-9]+,
z[0-9]+.s, uxtw\]\n} 12 } } */
+/* { dg-final { scan-assembler-times {\tld1w\tz[0-9]+\.s, p[0-7]/z, \[x[0-9]+,
z[0-9]+.s, sxtw\]\n} 3 } } */
/* { dg-final { scan-assembler-times {\tld1d\tz[0-9]+\.d, p[0-7]/z, \[x[0-9]+,
z[0-9]+.d\]\n} 15 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/strided_store_6.c
b/gcc/testsuite/gcc.target/aarch64/sve/strided_store_6.c
index 111d52584d9..3e18f7a9054 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/strided_store_6.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/strided_store_6.c
@@ -3,5 +3,9 @@
#include "strided_store_5.c"
-/* { dg-final { scan-assembler-not {\[x[0-9]+, z[0-9]+\.s} } } */
/* { dg-final { scan-assembler-times {\tst1d\tz[0-9]+\.d, p[0-7], \[x[0-9]+,
z[0-9]+.d\]\n} 15 } } */
+
+/* We can use signed values for float gather statters as we know VL < 2048 and
+ so the ranges will be expressible as int32 offsets. */
+/* { dg-final { scan-assembler-times {\tst1w\tz[0-9]+\.s, p[0-7], \[x[0-9]+,
z[0-9]+.s, uxtw\]\n} 12 } } */
+/* { dg-final { scan-assembler-times {\tst1w\tz[0-9]+\.s, p[0-7], \[x[0-9]+,
z[0-9]+.s, sxtw\]\n} 3 } } */
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index e213869af57..81f01aa2803 100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -2556,7 +2556,8 @@ vect_enhance_data_refs_alignment (loop_vec_info
loop_vinfo)
unsigned int load_outside_cost = 0;
unsigned int store_inside_cost = 0;
unsigned int store_outside_cost = 0;
- unsigned int estimated_npeels = vect_vf_for_cost (loop_vinfo) / 2;
+ unsigned int estimated_npeels
+ = vect_iv_increment_for_cost (loop_vinfo) / 2;
stmt_vector_for_cost dummy;
dummy.create (2);
@@ -2783,7 +2784,7 @@ vect_enhance_data_refs_alignment (loop_vec_info
loop_vinfo)
if (do_peeling
&& LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
{
- unsigned int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
unsigned int max_peel = npeel == 0 ? assumed_vf - 1 : npeel;
if ((unsigned HOST_WIDE_INT) LOOP_VINFO_INT_NITERS (loop_vinfo)
< assumed_vf + max_peel)
diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
index 51c8b42deff..979486eca79 100644
--- a/gcc/tree-vect-loop-manip.cc
+++ b/gcc/tree-vect-loop-manip.cc
@@ -681,6 +681,12 @@ vect_set_loop_controls_directly (class loop *loop,
loop_vec_info loop_vinfo,
tree test_index, test_limit, first_limit;
if (might_wrap_p)
{
+ /* This should never happen as either, this is a DECREMENTING_IV loop
+ which is handled above, or it is a HSSR loop, where all but the
+ base mask are handled by permutes, and the base mask overflow is
+ avoided separately. */
+ gcc_assert (LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo));
+
/* In principle the loop should stop iterating once the incremented
IV reaches a value greater than or equal to:
@@ -709,7 +715,7 @@ vect_set_loop_controls_directly (class loop *loop,
loop_vec_info loop_vinfo,
COMPARE_TYPE. */
test_index = index_before_incr;
tree adjust = gimple_convert (preheader_seq, compare_type,
- nitems_vf);
+ nitems_step);
if (nitems_skip)
adjust = gimple_build (preheader_seq, MINUS_EXPR, compare_type,
adjust, nitems_skip);
@@ -724,9 +730,8 @@ vect_set_loop_controls_directly (class loop *loop,
loop_vec_info loop_vinfo,
/* The first vector iteration can handle at most NITEMS_STEP
items. NITEMS_STEP <= CONST_LIMIT, and adding
NITEMS_SKIP to that cannot overflow. */
- tree const_limit = build_int_cst (compare_type,
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)
- * nitems_per_iter);
+ tree const_limit
+ = gimple_convert (preheader_seq, compare_type, nitems_step);
first_limit = gimple_build (preheader_seq, MIN_EXPR, compare_type,
nitems_total, const_limit);
first_limit = gimple_build (preheader_seq, PLUS_EXPR, compare_type,
@@ -1326,6 +1331,8 @@ vect_set_loop_condition_normal (loop_vec_info loop_vinfo,
edge exit_edge,
In both cases the loop limit is NITERS - STEP. */
gimple_seq seq = NULL;
limit = force_gimple_operand (niters, &seq, true, NULL_TREE);
+ step = force_gimple_operand (fold_convert (TREE_TYPE (limit), step),
&seq,
+ true, NULL_TREE);
limit = gimple_build (&seq, MINUS_EXPR, TREE_TYPE (limit), limit, step);
if (seq)
{
@@ -3004,7 +3011,6 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
tree ni_minus_gap, var;
tree niters_vector, step_vector;
tree type = niters ? TREE_TYPE (niters) : sizetype;
- poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
edge pe = loop_preheader_edge (LOOP_VINFO_LOOP (loop_vinfo));
/* With a counting exit before the loop body, the final scalar iteration
@@ -3034,8 +3040,8 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
ni_minus_gap = niters;
/* To silence some unexpected warnings, simply initialize to 0. */
- unsigned HOST_WIDE_INT const_vf = 0;
- if (vf.is_constant (&const_vf)
+ unsigned int const_iv_increment = 0;
+ if (vect_known_const_iv_increment (loop_vinfo, &const_iv_increment)
&& !LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)
&& LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo))
{
@@ -3046,7 +3052,7 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
overflow, we can generate niters / vf; otherwise we generate
(niters - vf) / vf + 1 by using the fact that we know ratio
will be at least one. */
- tree var_vf = build_int_cst (type, const_vf);
+ tree var_vf = build_int_cst (type, const_iv_increment);
if (niters_no_overflow)
niters_vector = fold_build2 (TRUNC_DIV_EXPR, type, ni_minus_gap,
var_vf);
@@ -3064,7 +3070,7 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
else
{
niters_vector = ni_minus_gap;
- step_vector = build_int_cst (type, vf);
+ step_vector = LOOP_VINFO_IV_INCREMENT (loop_vinfo);
}
if (!is_gimple_val (niters_vector))
@@ -3087,20 +3093,21 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
wi::div_trunc (wi::max_value
(TYPE_PRECISION (type),
TYPE_SIGN (type)),
- const_vf,
+ const_iv_increment,
TYPE_SIGN (type)));
set_range_info (niters_vector, vr);
}
/* For VF == 1 the vector IV might also overflow so we cannot
assert a minimum value of 1. */
- else if (const_vf > 1)
+ else if (const_iv_increment > 1)
{
int_range<1> vr (type,
wi::one (TYPE_PRECISION (type)),
wi::rshift (wi::max_value (TYPE_PRECISION (type),
TYPE_SIGN (type))
- - (const_vf - 1),
- exact_log2 (const_vf), TYPE_SIGN
(type))
+ - (const_iv_increment - 1),
+ exact_log2 (const_iv_increment),
+ TYPE_SIGN (type))
+ 1);
set_range_info (niters_vector, vr);
}
@@ -3118,22 +3125,47 @@ vect_gen_vector_loop_niters (loop_vec_info loop_vinfo,
tree niters,
tree
vect_get_loop_iv_increment (loop_vec_info loop_vinfo)
{
- if (LOOP_VINFO_USING_SELECT_VL_P (loop_vinfo))
- {
- /* For now, set this as varying.
- Fill this in later when building the loop controls. */
- tree iv_increment = make_temp_ssa_name (sizetype, NULL, "iv_increment");
+ if (!LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo))
+ /* For now, set this as varying. */
+ return build_call_expr_internal_loc (UNKNOWN_LOCATION, IFN_VARYING,
+ sizetype, 0);
+ else
+ return build_int_cst (sizetype, LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+}
- gcall *varying_call = gimple_build_call_internal (IFN_VARYING, 0);
- gimple_call_set_lhs (varying_call, iv_increment);
+/* Emits any required statements for emitting the loop IV increment
+ for LOOP_VINFO and changes it to be an SSA node.
- gimple_stmt_iterator gsi = gsi_after_labels (loop_vinfo->loop->header);
- gsi_insert_before (&gsi, varying_call, GSI_NEW_STMT);
+ Additionally, if the *STEP_VECTOR uses the IV increment value, changes it
+ to use the new SSA node. */
- return iv_increment;
+void
+vect_force_iv_increment_to_gimple (loop_vec_info loop_vinfo, tree *step_vector)
+{
+ /* Emit the IV calculation. */
+ gimple_seq force_iv_increment_seq = {};
+ tree iv_increment = create_tmp_var (sizetype, "iv_increment");
+ iv_increment
+ = force_gimple_operand (LOOP_VINFO_IV_INCREMENT (loop_vinfo),
+ &force_iv_increment_seq, true, iv_increment);
+ if (step_vector && *step_vector)
+ {
+ if (*step_vector == LOOP_VINFO_IV_INCREMENT (loop_vinfo))
+ *step_vector = iv_increment;
+ else
+ *step_vector
+ = force_gimple_operand (*step_vector, &force_iv_increment_seq, true,
+ NULL_TREE);
}
+ LOOP_VINFO_IV_INCREMENT (loop_vinfo) = iv_increment;
+
+ if (!force_iv_increment_seq)
+ return;
+
+ if (LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo))
+ add_preheader_seq (LOOP_VINFO_LOOP (loop_vinfo), force_iv_increment_seq);
else
- return build_int_cst (sizetype, LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+ add_header_seq (LOOP_VINFO_LOOP (loop_vinfo), force_iv_increment_seq);
}
/* Given NITERS_VECTOR which is the number of iterations for vectorized
@@ -3147,14 +3179,16 @@ vect_gen_vector_loop_niters_mult_vf (loop_vec_info
loop_vinfo,
tree *niters_vector_mult_vf_ptr)
{
/* We should be using a step_vector of VF if VF is variable. */
- int vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo).to_constant ();
+ unsigned int iv_increment;
+ gcc_assert (vect_known_const_iv_increment (loop_vinfo, &iv_increment));
tree type = TREE_TYPE (niters_vector);
- tree tree_vf = build_int_cst (type, vf);
+ tree tree_iv_inc = build_int_cst (type, iv_increment);
basic_block exit_bb = LOOP_VINFO_MAIN_EXIT (loop_vinfo)->dest;
+
gcc_assert (niters_vector_mult_vf_ptr != NULL);
tree niters_vector_mult_vf = fold_build2 (MULT_EXPR, type,
- niters_vector, tree_vf);
+ niters_vector, tree_iv_inc);
if (!is_gimple_val (niters_vector_mult_vf))
{
@@ -3417,7 +3451,7 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters,
tree nitersm1,
loop->num_nodes);
prob_vector = profile_probability::guessed_always ().apply_scale (9, 10);
- estimated_vf = vect_vf_for_cost (loop_vinfo);
+ estimated_vf = vect_iv_increment_for_cost (loop_vinfo);
if (estimated_vf == 2)
estimated_vf = 3;
prob_prolog = prob_epilog = profile_probability::guessed_always ()
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 78bfd607c5d..749aae69f11 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -751,6 +751,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared,
th (0),
versioning_threshold (0),
vectorization_factor (0),
+ iv_increment (NULL),
main_loop_edge (nullptr),
skip_main_loop_edge (nullptr),
skip_this_loop_edge (nullptr),
@@ -972,11 +973,15 @@ vect_min_prec_for_max_niters (loop_vec_info loop_vinfo,
unsigned int factor)
static bool
vect_need_peeling_or_partial_vectors_p (loop_vec_info loop_vinfo)
{
- unsigned HOST_WIDE_INT const_vf;
-
if (LOOP_VINFO_PEELING_FOR_GAPS (loop_vinfo))
return true;
+ /* If we do not statically know the IV increment, then we always need peeling
+ or partial vectors. */
+ poly_uint64 known_iv_increment;
+ if (!vect_known_iv_increment (loop_vinfo, &known_iv_increment))
+ return true;
+
loop_vec_info main_loop_vinfo
= (LOOP_VINFO_EPILOGUE_P (loop_vinfo)
? LOOP_VINFO_MAIN_LOOP_INFO (loop_vinfo) : loop_vinfo);
@@ -988,17 +993,19 @@ vect_need_peeling_or_partial_vectors_p (loop_vec_info
loop_vinfo)
unsigned int peel_niter
= LOOP_VINFO_PEELING_FOR_ALIGNMENT (main_loop_vinfo);
return !multiple_p (LOOP_VINFO_INT_NITERS (loop_vinfo) - peel_niter,
- LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+ known_iv_increment);
}
+ unsigned HOST_WIDE_INT const_iv_increment;
if (!LOOP_VINFO_PEELING_FOR_ALIGNMENT (main_loop_vinfo)
- && LOOP_VINFO_VECT_FACTOR (loop_vinfo).is_constant (&const_vf))
+ && known_iv_increment.is_constant (&const_iv_increment)
+ && pow2p_hwi (const_iv_increment))
{
/* When the number of iterations is a multiple of the vectorization
factor and we are not doing prologue or forced epilogue peeling
the epilogue isn't necessary. */
if (tree_ctz (LOOP_VINFO_NITERS (loop_vinfo))
- >= (unsigned) exact_log2 (const_vf))
+ >= (unsigned) exact_log2 (const_iv_increment))
return false;
}
@@ -1728,9 +1735,9 @@ vect_create_loop_vinfo (class loop *loop, vec_info_shared
*shared,
either way. */
static bool
-vect_known_niters_smaller_than_vf (loop_vec_info loop_vinfo)
+vect_known_niters_smaller_than_iv_increment (loop_vec_info loop_vinfo)
{
- unsigned int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int minimum_inc = vect_min_iv_increment (loop_vinfo);
HOST_WIDE_INT max_niter;
if (LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
@@ -1738,7 +1745,7 @@ vect_known_niters_smaller_than_vf (loop_vec_info
loop_vinfo)
else
max_niter = max_stmt_executions_int (LOOP_VINFO_LOOP (loop_vinfo));
- if (max_niter != -1 && (unsigned HOST_WIDE_INT) max_niter < assumed_vf)
+ if (max_niter != -1 && (unsigned HOST_WIDE_INT) max_niter < minimum_inc)
return true;
return false;
@@ -1753,12 +1760,12 @@ vect_analyze_loop_costing (loop_vec_info loop_vinfo,
unsigned *suggested_unroll_factor)
{
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
- unsigned int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
/* Only loops that can handle partially-populated vectors can have iteration
counts less than the vectorization factor. */
if (!LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)
- && vect_known_niters_smaller_than_vf (loop_vinfo))
+ && vect_known_niters_smaller_than_iv_increment (loop_vinfo))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -1781,19 +1788,18 @@ vect_analyze_loop_costing (loop_vec_info loop_vinfo,
= LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo);
loop_vec_info main_loop_vinfo
= LOOP_VINFO_MAIN_LOOP_INFO (loop_vinfo);
- unsigned lowest_vf
- = constant_lower_bound (LOOP_VINFO_VECT_FACTOR (orig_loop_vinfo));
int prolog_peeling = 0;
if (!vect_use_loop_mask_for_alignment_p (main_loop_vinfo))
prolog_peeling = LOOP_VINFO_PEELING_FOR_ALIGNMENT (main_loop_vinfo);
+ unsigned int known_iv_increment;
if (prolog_peeling >= 0
- && known_eq (LOOP_VINFO_VECT_FACTOR (orig_loop_vinfo),
- lowest_vf))
+ && vect_known_const_iv_increment (orig_loop_vinfo,
+ &known_iv_increment))
{
unsigned gap
= LOOP_VINFO_PEELING_FOR_GAPS (main_loop_vinfo) ? 1 : 0;
scalar_niters = ((scalar_niters - gap - prolog_peeling)
- % lowest_vf + gap);
+ % known_iv_increment + gap);
}
}
/* Reject vectorizing for a single scalar iteration, even if
@@ -1919,7 +1925,8 @@ vect_analyze_loop_costing (loop_vec_info loop_vinfo,
would be to stick with the scalar code. */
if (loop_cost_model (loop) == VECT_COST_MODEL_VERY_CHEAP
&& !LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo)
- && min_profitable_estimate > (int) vect_vf_for_cost (loop_vinfo))
+ && min_profitable_estimate
+ > (int) vect_iv_increment_for_cost (loop_vinfo))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
@@ -1936,7 +1943,7 @@ vect_analyze_loop_costing (loop_vec_info loop_vinfo,
vectorization factor of the main loop. */
if (LOOP_VINFO_EPILOGUE_P (loop_vinfo))
estimated_niter
- = vect_vf_for_cost (LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo)) - 1;
+ = vect_iv_increment_for_cost (LOOP_VINFO_ORIG_LOOP_INFO (loop_vinfo)) -
1;
else
{
estimated_niter = estimated_stmt_executions_int (loop);
@@ -1967,7 +1974,8 @@ vect_analyze_loop_costing (loop_vec_info loop_vinfo,
likely alignment prologue). */
if (LOOP_VINFO_NITERS_UNCOUNTED_P (loop_vinfo)
&& estimated_niter == -1
- && min_profitable_estimate > (int) vect_vf_for_cost (loop_vinfo))
+ && min_profitable_estimate
+ > (int) vect_iv_increment_for_cost (loop_vinfo))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_NOTE, vect_location,
@@ -2202,7 +2210,7 @@ vect_determine_partial_vectors_and_peeling (loop_vec_info
loop_vinfo,
if ((param_vect_partial_vector_usage == 1
|| loop_vinfo->suggested_unroll_factor > 1)
&& !LOOP_VINFO_EPILOGUE_P (loop_vinfo)
- && !vect_known_niters_smaller_than_vf (loop_vinfo))
+ && !vect_known_niters_smaller_than_iv_increment (loop_vinfo))
;
else
LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo) = true;
@@ -2761,6 +2769,7 @@ again:
/* Reset the vectorization factor. */
LOOP_VINFO_VECT_FACTOR (loop_vinfo) = 0;
+ LOOP_VINFO_IV_INCREMENT (loop_vinfo) = NULL_TREE;
/* Free the SLP instances. */
FOR_EACH_VEC_ELT (LOOP_VINFO_SLP_INSTANCES (loop_vinfo), j, instance)
vect_free_slp_instance (instance);
@@ -2908,7 +2917,7 @@ vect_analyze_loop_1 (class loop *loop, vec_info_shared
*shared,
{
if (suggested_unroll_factor == 1)
{
- int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
suggested_unroll_factor = user_unroll / assumed_vf;
if (suggested_unroll_factor > 1)
{
@@ -3986,7 +3995,7 @@ vect_is_simple_reduction (loop_vec_info loop_info,
stmt_vec_info phi_info,
static int
vect_get_peel_iters_epilogue (loop_vec_info loop_vinfo, int
peel_iters_prologue)
{
- int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo) || peel_iters_prologue == -1)
{
if (dump_enabled_p ())
@@ -4079,7 +4088,7 @@ vect_estimate_min_profitable_iters (loop_vec_info
loop_vinfo,
unsigned vec_epilogue_cost = 0;
int scalar_single_iter_cost = 0;
int scalar_outside_cost = 0;
- int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
int npeel = LOOP_VINFO_PEELING_FOR_ALIGNMENT (loop_vinfo);
vector_costs *target_cost_data = loop_vinfo->vector_costs;
@@ -4390,7 +4399,7 @@ vect_estimate_min_profitable_iters (loop_vec_info
loop_vinfo,
= LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
bool need_iterate_p
= (!LOOP_VINFO_EPILOGUE_P (loop_vinfo)
- && !vect_known_niters_smaller_than_vf (loop_vinfo));
+ && !vect_known_niters_smaller_than_iv_increment (loop_vinfo));
/* Calculate how many statements to be added. */
unsigned int prologue_stmts = 0;
@@ -11047,7 +11056,7 @@ vect_update_early_break_profiles (loop_vec_info
loop_vinfo)
return;
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
- unsigned int vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int vf = vect_iv_increment_for_cost (loop_vinfo);
auto_vec<std::pair<edge, profile_probability>, 8> updates;
for (edge e : get_loop_exit_edges (loop))
@@ -11427,8 +11436,6 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
tree niters_vector = NULL_TREE;
tree step_vector = NULL_TREE;
tree niters_vector_mult_vf = NULL_TREE;
- poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
- unsigned int lowest_vf = constant_lower_bound (vf);
gimple *stmt;
bool check_profitability = false;
unsigned int th;
@@ -11500,13 +11507,15 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
bool niters_no_overflow = uncounted_p ? false /* Not known. */
: loop_niters_no_overflow (loop_vinfo);
+ LOOP_VINFO_IV_INCREMENT (loop_vinfo)
+ = vect_get_loop_iv_increment (loop_vinfo);
+
epilogue = vect_do_peeling (loop_vinfo, niters, nitersm1, &niters_vector,
&step_vector, &niters_vector_mult_vf, th,
check_profitability, niters_no_overflow,
&advance);
- LOOP_VINFO_IV_INCREMENT (loop_vinfo)
- = vect_get_loop_iv_increment (loop_vinfo);
+ vect_force_iv_increment_to_gimple (loop_vinfo, &step_vector);
/* Assign hierarchical discriminators to the vectorized loop. */
poly_uint64 vf_val = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
@@ -11541,13 +11550,16 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
if (niters_vector == NULL_TREE && !uncounted_p)
{
+ unsigned int const_iv_increment;
if (LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo)
+ && LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo)
&& !LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo)
- && known_eq (lowest_vf, vf))
+ && vect_known_const_iv_increment (loop_vinfo, &const_iv_increment))
{
niters_vector
= build_int_cst (TREE_TYPE (LOOP_VINFO_NITERS (loop_vinfo)),
- LOOP_VINFO_INT_NITERS (loop_vinfo) / lowest_vf);
+ LOOP_VINFO_INT_NITERS (loop_vinfo)
+ / const_iv_increment);
step_vector = build_one_cst (TREE_TYPE (niters));
}
else if (vect_use_loop_mask_for_alignment_p (loop_vinfo))
@@ -11680,7 +11692,7 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
niters_vector_mult_vf, !niters_no_overflow);
}
- unsigned int assumed_vf = vect_vf_for_cost (loop_vinfo);
+ unsigned int assumed_vf = vect_iv_increment_for_cost (loop_vinfo);
/* True if the final iteration might not handle a full vector's
worth of scalar iterations. */
@@ -11702,6 +11714,7 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
int alignment_npeels = LOOP_VINFO_PEELING_FOR_ALIGNMENT (loop_vinfo);
if (alignment_npeels && LOOP_VINFO_USING_PARTIAL_VECTORS_P (loop_vinfo))
{
+ unsigned lowest_vf = vect_min_iv_increment (loop_vinfo);
/* When the amount of peeling is known at compile time, the first
iteration will have exactly alignment_npeels active elements.
In the worst case it will have at least one. */
@@ -11717,37 +11730,44 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
loop->nb_iterations_upper_bound
= (final_iter_may_be_partial
? wi::udiv_ceil (loop->nb_iterations_upper_bound + bias_for_lowest,
- lowest_vf) - 1
+ vect_min_iv_increment (loop_vinfo)) - 1
: wi::udiv_floor (loop->nb_iterations_upper_bound + bias_for_lowest,
- lowest_vf) - 1);
+ vect_min_iv_increment (loop_vinfo)) - 1);
+
+ poly_uint64 main_known_iv_increment;
+ poly_uint64 loop_known_iv_increment;
if (main_vinfo
/* Both peeling for alignment and peeling for gaps can end up
with the scalar epilogue running for more than VF-1 iterations. */
&& !main_vinfo->peeling_for_alignment
- && !main_vinfo->peeling_for_gaps)
+ && !main_vinfo->peeling_for_gaps
+ && vect_known_iv_increment (main_vinfo, &main_known_iv_increment)
+ && vect_known_iv_increment (loop_vinfo, &loop_known_iv_increment))
{
unsigned int bound;
poly_uint64 main_iters
- = upper_bound (LOOP_VINFO_VECT_FACTOR (main_vinfo),
+ = upper_bound (main_known_iv_increment,
LOOP_VINFO_COST_MODEL_THRESHOLD (main_vinfo));
main_iters
= upper_bound (main_iters,
LOOP_VINFO_VERSIONING_THRESHOLD (main_vinfo));
if (can_div_away_from_zero_p (main_iters,
- LOOP_VINFO_VECT_FACTOR (loop_vinfo),
+ loop_known_iv_increment,
&bound))
loop->nb_iterations_upper_bound
= wi::umin ((bound_wide_int) (bound - 1),
loop->nb_iterations_upper_bound);
}
- }
+ }
if (loop->any_likely_upper_bound)
loop->nb_iterations_likely_upper_bound
= (final_iter_may_be_partial
? wi::udiv_ceil (loop->nb_iterations_likely_upper_bound
- + bias_for_lowest, lowest_vf) - 1
+ + bias_for_lowest,
+ vect_likely_min_iv_increment (loop_vinfo)) - 1
: wi::udiv_floor (loop->nb_iterations_likely_upper_bound
- + bias_for_lowest, lowest_vf) - 1);
+ + bias_for_lowest,
+ vect_likely_min_iv_increment (loop_vinfo)) - 1);
if (loop->any_estimate)
loop->nb_iterations_estimate
= (final_iter_may_be_partial
@@ -11778,7 +11798,7 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
/* Loops vectorized with a variable factor won't benefit from
unrolling/peeling. */
- if (!vf.is_constant ())
+ if (!vect_known_const_iv_increment (loop_vinfo))
{
loop->unroll = 1;
if (dump_enabled_p ())
@@ -12042,7 +12062,7 @@ vect_iv_limit_for_partial_vectors (loop_vec_info
loop_vinfo)
{
tree niters_skip = LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo);
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
- unsigned HOST_WIDE_INT max_vf = vect_max_vf (loop_vinfo);
+ unsigned HOST_WIDE_INT max_iv_increment = vect_max_iv_increment (loop_vinfo);
/* Calculate the value that the induction variable must be able
to hit in order to ensure that we end the loop with an all-false mask.
@@ -12058,17 +12078,24 @@ vect_iv_limit_for_partial_vectors (loop_vec_info
loop_vinfo)
if (TREE_CODE (niters_skip) == INTEGER_CST)
iv_limit += wi::to_widest (niters_skip);
else
- iv_limit += max_vf - 1;
+ iv_limit += max_iv_increment - 1;
}
else if (LOOP_VINFO_PEELING_FOR_ALIGNMENT (loop_vinfo))
/* Make a conservatively-correct assumption. */
- iv_limit += max_vf - 1;
-
- /* IV_LIMIT is the maximum number of latch iterations, which is also
- the maximum in-range IV value. Round this value down to the previous
- vector alignment boundary and then add an extra full iteration. */
- poly_uint64 vf = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
- iv_limit = (iv_limit & -(int) known_alignment (vf)) + max_vf;
+ iv_limit += max_iv_increment - 1;
+
+ poly_uint64 known_iv_increment;
+ if (vect_known_iv_increment (loop_vinfo, &known_iv_increment))
+ /* IV_LIMIT is the maximum number of latch iterations, which is also
+ the maximum in-range IV value. Round this value down to the
+ previous vector alignment boundary and then add an extra full
+ iteration. */
+ iv_limit = (iv_limit & -(int) known_alignment (known_iv_increment))
+ + max_iv_increment;
+ else
+ /* Otherwise, if we do not know the vector alignment boundary, then
+ the limit is one max iv increment over the max iv. */
+ iv_limit = iv_limit + max_iv_increment;
}
return iv_limit;
}
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 7184f193b07..5b75483f2de 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2375,17 +2375,6 @@ vect_get_num_copies (vec_info *vinfo, slp_tree node)
return vect_get_num_vectors (vf, vectype);
}
-/* Return the vectorization factor that should be used for costing
- purposes while vectorizing the loop described by LOOP_VINFO.
- Pick a reasonable estimate if the vectorization factor isn't
- known at compile time. */
-
-inline unsigned int
-vect_vf_for_cost (loop_vec_info loop_vinfo)
-{
- return estimated_poly_value (LOOP_VINFO_VECT_FACTOR (loop_vinfo));
-}
-
/* Estimate the number of elements in VEC_TYPE for costing purposes.
Pick a reasonable estimate if the exact number isn't known at
compile time. */
@@ -2401,12 +2390,108 @@ vect_nunits_for_cost (tree vec_type)
inline unsigned HOST_WIDE_INT
vect_max_vf (loop_vec_info loop_vinfo)
{
- unsigned HOST_WIDE_INT vf;
- if (LOOP_VINFO_VECT_FACTOR (loop_vinfo).is_constant (&vf))
- return vf;
+ widest_int vf_upper_bound;
+ if (maybe_upper_bound (LOOP_VINFO_VECT_FACTOR (loop_vinfo), &vf_upper_bound,
+ UNSIGNED)
+ && wi::fits_uhwi_p (vf_upper_bound))
+ return vf_upper_bound.to_uhwi ();
return MAX_VECTORIZATION_FACTOR;
}
+/* Return the maximum possible vectorization factor for LOOP_VINFO. */
+
+inline unsigned HOST_WIDE_INT
+vect_max_iv_increment (loop_vec_info loop_vinfo)
+{
+ return vect_max_vf (loop_vinfo);
+}
+
+/* Return the minimum possible vectorization factor for LOOP_VINFO. */
+
+inline unsigned HOST_WIDE_INT
+vect_min_iv_increment (loop_vec_info loop_vinfo)
+{
+ /* Invariant iv increments are still required to always process 1 scalar
+ iteration. */
+ if (!LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo))
+ return 1;
+
+ else
+ return constant_lower_bound (LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+}
+
+/* Return the the constant iv increment for LOOP_VINFO if statically known. */
+
+inline bool
+vect_known_const_iv_increment (loop_vec_info loop_vinfo,
+ unsigned int *iv_increment = NULL)
+{
+ if (vect_max_iv_increment (loop_vinfo) != vect_min_iv_increment (loop_vinfo))
+ return false;
+
+ if (iv_increment != NULL)
+ *iv_increment = vect_min_iv_increment (loop_vinfo);
+ return true;
+}
+
+/* Return the the polyint iv increment for LOOP_VINFO if statically known. */
+
+inline bool
+vect_known_iv_increment (loop_vec_info loop_vinfo,
+ poly_uint64 *iv_increment = NULL)
+{
+ unsigned int const_iv_increment;
+ if (vect_known_const_iv_increment (loop_vinfo, &const_iv_increment))
+ {
+ if (iv_increment != NULL)
+ *iv_increment = const_iv_increment;
+ return true;
+ }
+
+ /* If IV increment is loop variant or we have capped vf and don't know it
+ is a constant then it is not statically known. */
+ if (LOOP_VINFO_IV_INCREMENT_INVARIANT_P (loop_vinfo))
+ return false;
+
+ if (iv_increment != NULL)
+ *iv_increment = LOOP_VINFO_VECT_FACTOR (loop_vinfo);
+ return true;
+}
+
+/* Return the iv increment that should be used for costing
+ purposes while vectorizing the loop described by LOOP_VINFO.
+ Pick a reasonable estimate if the vectorization factor isn't
+ known at compile time. */
+
+inline unsigned int
+vect_iv_increment_for_cost (loop_vec_info loop_vinfo)
+{
+ unsigned int known_iv_increment;
+ if (vect_known_const_iv_increment (loop_vinfo, &known_iv_increment))
+ return known_iv_increment;
+
+ /* We assume even with variant iv increment the loop will normally
+ process VF iterations. */
+ unsigned int estimate
+ = estimated_poly_value (LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+
+ return estimate;
+}
+
+
+/* Return the likely samllest normal iv increment value. */
+inline unsigned HOST_WIDE_INT
+vect_likely_min_iv_increment (loop_vec_info loop_vinfo)
+{
+ unsigned int known_const_iv_increment;
+ if (vect_known_const_iv_increment (loop_vinfo, &known_const_iv_increment))
+ return known_const_iv_increment;
+
+ /* We assume the IV increment for loops with loop variant IV is still
+ normally VF, so use it for the lower bound of IV increment regardless. */
+ return constant_lower_bound (LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+}
+
/* Return the size of the value accessed by unvectorized data reference
DR_INFO. This is only valid once STMT_VINFO_VECTYPE has been calculated
for the associated gimple statement, since that guarantees that DR_INFO
@@ -2428,7 +2513,7 @@ vect_apply_runtime_profitability_check_p (loop_vec_info
loop_vinfo)
{
unsigned int th = LOOP_VINFO_COST_MODEL_THRESHOLD (loop_vinfo);
return (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo)
- && th >= vect_vf_for_cost (loop_vinfo));
+ && th >= vect_iv_increment_for_cost (loop_vinfo));
}
/* Return true if CODE is a lane-reducing opcode. */
@@ -2691,6 +2776,7 @@ extern tree vect_build_loop_niters (loop_vec_info, bool *
= NULL);
extern void vect_gen_vector_loop_niters (loop_vec_info, tree, tree *,
tree *, bool);
extern tree vect_get_loop_iv_increment (loop_vec_info);
+extern void vect_force_iv_increment_to_gimple (loop_vec_info, tree *);
extern tree vect_halve_mask_nunits (tree, machine_mode);
extern tree vect_double_mask_nunits (tree, machine_mode);
extern void vect_record_loop_mask (loop_vec_info, vec_loop_masks *,
--
2.56.0
