This is part of enablement of support for partial vectors in basic
block SLP vectorization.
Adapt the existing vect_verify_loop_lens function used by the loop
vectorizer to verify that partial vector length requirements recorded
during the analysis phase can be satisfied. Unlike masked partial
vectors, the elements-to-subelements scaling factor is not always one,
so apply it when determining the maximum precision needed to store
lengths used to implement partial tail sizes.
A different scheme is used for recording lengths for BB SLP
vectorization but it reuses as much loop vectorizer code as possible:
instead of using the passed-in number of vectors as an index to find
the relevant rgroup controls in vect_record_len and vect_get_len,
rgroup controls are instead indexed by the number of active lanes in
the partial tail vector. Full vectors do not need recorded length
requirements. Consequently, the number of controls per
rgroup_controls instance is one for BB SLP. (One length for each
unique partial tail size.)
The max_nscalars_per_iter of an rgroup is already assumed by
vect_set_bb_slp_controls_directly to hold the number of lanes in a
mask to be generated. Carry that assumption over to LEN-style partial
vectors: in vect_record_len, record the partial tail size for a given
rgroup as max_nscalars_per_iter. Because of how rgroups are indexed,
the value of max_nscalars_per_iter will either be zero (on
initialization) or one more than the rgroup index.
As part of analyzing whether an SLP region can be vectorized, after
analyzing all statements in the region, if lengths have been recorded
then call vect_verify_loop_lens to check whether we can generate
the necessary length controls. If not, return false from
vect_slp_analyze_bb_1.
Costing in vect_bb_vectorization_profitable_p is extended to
account for partial vector setup costs for an SLP vectorization
region that uses LEN-style controls. This is similar to WHILE_ULT
and AVX512 costing except that the cost kind is scalar_stmt instead
of vector_stmt (as in the equivalent loop vectorization code).
As in the case of masks, this is imprecise because lengths can be
shared between SLP nodes and some subgraphs might not be scheduled.
After scheduling any profitable subgraphs and generating any
invariant statements, unless AVX512-style partial vectors are in
use, vect_set_bb_slp_controls_partial_vectors is called. This
function is readily adapted to iterate over the recorded lens
instead of the recorded masks. vect_set_bb_slp_controls_directly
now scales the partial tail size by the rgroup's factor (which will
be 1 for WHILE_ULT-style partial vectors). If using LEN-style
partial vectors, it assigns that scaled length to the SSA name
created by vect_get_len for the control.
Dead stmts that set up lengths for unprofitable subgraphs are
expected to be eliminated by later passes.
gcc/ChangeLog:
* tree-vect-loop.cc (vect_min_prec_for_slp):
Permit the elements-to-subelements scaling factor to be other
than 1 and use it to scale the partial tail size.
(vect_verify_loop_lens): Change parameter type from loop_vec_info
to vec_info *. Use dyn_cast to get a loop_vec_info or null from
the argument.
Access lengths directly instead of via LOOP_VINFO_LENS. Avoid
calling vect_min_prec_for_max_niters for BB SLP because the
concept of iterations is not applicable. Instead, call
vect_min_prec_for_slp to get an alternative value of
min_ni_prec. Set rgroup_compare_type, rgroup_iv_type and
partial_vector_style directly instead of via LOOP_VINFO_
macros.
(vect_record_len): Use dyn_cast to get a loop_vec_info or null
from the vec_info argument. If used for BB SLP, derive the number
of active lanes in the tail vector of the sequence from the
passed-in vector type and group size. If the tail vector is not
partial then record no length requirement; otherwise, use the
partial tail size as the index of the rgroup control to be
populated (instead of the nvectors argument). When doing BB SLP,
use the partial tail size instead of the number of scalar values
operated on by each scalar iteration as max_nscalars_per_iter.
(vect_get_len): Don't call make_temp_ssa_name with "loop_len"
as the name, if used for BB SLP. For BB SLP, return a constant
length for full vectors. Otherwise, derive the number of active
lanes in the partial tail vector from the passed-in vector type
and group size and use the partial tail size (instead of the
nvectors argument) as the index of the rgroup control. Set
nvectors to 1 and index to 0 because only the partial tail requires
a recorded control. Assert that no SSA name is created for
"adjusted_loop_len" if used for BB SLP. If the passed-in
elements-to-subelements factor differs from that recorded for the
partial tail then calculate the overall scaling factor required to
build EXACT_DIV_EXPR or MULT_EXPR using the partial tail size.
* tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
Permit the set of recorded lengths to be non-empty.
Add code to count setup costs for LEN-style partial vectors.
(vect_slp_analyze_bb_1): Return false if both mask and length
requirements for partial vectors were recorded for a BB SLP
vectorization region. Permit the set of recorded lengths to be
non-empty. If lengths have been recorded for an SLP region then
call vect_verify_loop_lens to check whether we can generate the
necessary controls. If not, return false.
(vect_set_bb_slp_controls_directly): If the partial vector
style is not WHILE_ULT then require it to be LEN and scale the
partial tail size in max_nscalars_per_iter by
subelements-per-element before using it as the righthand side of
the assignment to the SSA name made by vect_get_len.
(vect_set_bb_slp_controls_partial_vectors): Permit the set of
recorded lengths to be non-empty if the partial vector style is
LEN. If the set of recorded masks is empty then iterate over
the set of recorded lengths instead.
* tree-vectorizer.h (rgroup_controls): Update the description
of max_nscalars_per_iter.
(vect_verify_loop_lens): Add function declaration for use by
vect_slp_analyze_bb_1. This function no longer has static
linkage.
---
gcc/tree-vect-loop.cc | 193 ++++++++++++++++++++++++++++++++----------
gcc/tree-vect-slp.cc | 55 +++++++++---
gcc/tree-vectorizer.h | 5 +-
3 files changed, 195 insertions(+), 58 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index aad672631a2..50086731703 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -991,7 +991,7 @@ vect_need_peeling_or_partial_vectors_p (loop_vec_info
loop_vinfo)
/* Given some rgroup CONTROLS used for basic block SLP vectorization,
calculate the minimum precision necessary to represent the maximum
- normalized SLP group size. */
+ normalized partial tail size. */
static unsigned int
vect_min_prec_for_slp (vec<rgroup_controls> *controls)
{
@@ -1002,8 +1002,9 @@ vect_min_prec_for_slp (vec<rgroup_controls> *controls)
{
if (!rgc->type)
continue;
- gcc_assert (rgc->factor == 1);
- max_normalized_group_size = i + 1;
+ unsigned int normalized_group_size = (i + 1) * rgc->factor;
+ max_normalized_group_size
+ = MAX (max_normalized_group_size, normalized_group_size);
}
return wi::min_precision (max_normalized_group_size, UNSIGNED);
@@ -1351,20 +1352,22 @@ vect_verify_full_masking_avx512 (vec_info *vinfo)
precision of the target supported length is larger than the precision
required by loop niters. */
-static bool
-vect_verify_loop_lens (loop_vec_info loop_vinfo)
+bool
+vect_verify_loop_lens (vec_info *vinfo)
{
- if (LOOP_VINFO_LENS (loop_vinfo).is_empty ())
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+
+ if (vinfo->lens.is_empty ())
return false;
- if (!VECTOR_MODE_P (loop_vinfo->vector_mode))
+ if (!VECTOR_MODE_P (vinfo->vector_mode))
return false;
machine_mode len_load_mode, len_store_mode;
- if (!get_len_load_store_mode (loop_vinfo->vector_mode, true)
+ if (!get_len_load_store_mode (vinfo->vector_mode, true)
.exists (&len_load_mode))
return false;
- if (!get_len_load_store_mode (loop_vinfo->vector_mode, false)
+ if (!get_len_load_store_mode (vinfo->vector_mode, false)
.exists (&len_store_mode))
return false;
@@ -1382,39 +1385,47 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo)
/* If the backend requires a bias of -1 for LEN_LOAD, we must not emit
len_loads with a length of zero. In order to avoid that we prohibit
more than one loop length here. */
- if (partial_load_bias == -1
- && LOOP_VINFO_LENS (loop_vinfo).length () > 1)
+ if (partial_load_bias == -1 && vinfo->lens.length () > 1)
return false;
- LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) = partial_load_bias;
+ vinfo->partial_load_store_bias = partial_load_bias;
- unsigned int max_nitems_per_iter = 1;
+ unsigned int min_ni_prec;
unsigned int i;
rgroup_controls *rgl;
- /* Find the maximum number of items per iteration for every rgroup. */
- FOR_EACH_VEC_ELT (LOOP_VINFO_LENS (loop_vinfo), i, rgl)
+
+ if (loop_vinfo)
{
- unsigned nitems_per_iter = rgl->max_nscalars_per_iter * rgl->factor;
- max_nitems_per_iter = MAX (max_nitems_per_iter, nitems_per_iter);
- }
+ unsigned int max_nitems_per_iter = 1;
+
+ /* Find the maximum number of items per iteration for every rgroup. */
+ FOR_EACH_VEC_ELT (LOOP_VINFO_LENS (loop_vinfo), i, rgl)
+ {
+ unsigned nitems_per_iter = rgl->max_nscalars_per_iter * rgl->factor;
+ max_nitems_per_iter = MAX (max_nitems_per_iter, nitems_per_iter);
+ }
- /* Work out how many bits we need to represent the length limit. */
- unsigned int min_ni_prec
- = vect_min_prec_for_max_niters (loop_vinfo, max_nitems_per_iter);
+ /* Work out how many bits we need to represent the length limit. */
+ min_ni_prec
+ = vect_min_prec_for_max_niters (loop_vinfo, max_nitems_per_iter);
- /* Now use the maximum of below precisions for one suitable IV type:
- - the IV's natural precision
- - the precision needed to hold: the maximum number of scalar
- iterations multiplied by the scale factor (min_ni_prec above)
- - the Pmode precision
+ /* Now use the maximum of below precisions for one suitable IV type:
+ - the IV's natural precision
+ - the precision needed to hold: the maximum number of scalar
+ iterations multiplied by the scale factor (min_ni_prec above)
+ - the Pmode precision
- If min_ni_prec is less than the precision of the current niters,
- we prefer to still use the niters type. Prefer to use Pmode and
- wider IV to avoid narrow conversions. */
+ If min_ni_prec is less than the precision of the current niters,
+ we prefer to still use the niters type. Prefer to use Pmode and
+ wider IV to avoid narrow conversions. */
+
+ unsigned int ni_prec
+ = TYPE_PRECISION (TREE_TYPE (LOOP_VINFO_NITERS (loop_vinfo)));
+ min_ni_prec = MAX (min_ni_prec, ni_prec);
+ }
+ else
+ min_ni_prec = vect_min_prec_for_slp (&vinfo->lens);
- unsigned int ni_prec
- = TYPE_PRECISION (TREE_TYPE (LOOP_VINFO_NITERS (loop_vinfo)));
- min_ni_prec = MAX (min_ni_prec, ni_prec);
min_ni_prec = MAX (min_ni_prec, GET_MODE_BITSIZE (Pmode));
tree iv_type = NULL_TREE;
@@ -1446,9 +1457,9 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo)
return false;
}
- LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) = vect_partial_vectors_len;
+ vinfo->rgroup_compare_type = iv_type;
+ vinfo->rgroup_iv_type = iv_type;
+ vinfo->partial_vector_style = vect_partial_vectors_len;
return true;
}
@@ -10934,16 +10945,51 @@ vect_record_len (vec_info *vinfo, unsigned int
nvectors, tree vectype,
gcc_assert (!is_a <bb_vec_info> (vinfo) || group_size != 0);
gcc_assert (nvectors != 0);
vec_lens *lens = &vinfo->lens;
- if (lens->length () < nvectors)
- lens->safe_grow_cleared (nvectors, true);
- rgroup_controls *rgl = &(*lens)[nvectors - 1];
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ unsigned int key;
+ unsigned int tail_size = group_size;
+
+ if (loop_vinfo)
+ key = nvectors;
+ else
+ {
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
+
+ /* Vector-length agnostic types do not have a known length but that
+ does not matter because we do not use them for SLP regions that need
+ a sequence of more than one vector. */
+ if (nvectors > 1)
+ {
+ unsigned int const_nunits = nunits.to_constant ();
+ tail_size = ((group_size - 1) % const_nunits) + 1;
+ }
+ else
+ {
+ gcc_assert (known_le (poly_uint64 (tail_size), nunits));
+ }
+
+ /* Only the tail vector in the sequence potentially has a special
+ length, and only then if some of its lanes are inactive. */
+ if (known_eq (tail_size, nunits))
+ return;
+
+ key = tail_size;
+ }
+
+ gcc_assert (key != 0);
+ if (lens->length () < key)
+ lens->safe_grow_cleared (key, true);
+ rgroup_controls *rgl = &(*lens)[key - 1];
/* The number of scalars per iteration, scalar occupied bytes and
the number of vectors are both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor)
- .to_constant ();
+ unsigned int nscalars_per_iter;
+ if (loop_vinfo)
+ nscalars_per_iter = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ else
+ nscalars_per_iter = tail_size;
if (rgl->max_nscalars_per_iter < nscalars_per_iter)
{
@@ -10973,8 +11019,53 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
bool adjusted, unsigned int group_size)
{
gcc_assert (!is_a <bb_vec_info> (vinfo) || group_size != 0);
+ gcc_assert (nvectors != 0);
+ gcc_assert (index < nvectors);
+
vec_lens *lens = &vinfo->lens;
- rgroup_controls *rgl = &(*lens)[nvectors - 1];
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ unsigned int key;
+ unsigned int tail_size = group_size;
+
+ if (loop_vinfo)
+ key = nvectors;
+ else
+ {
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
+
+ /* Only the tail vector in the sequence potentially has a special
+ length, because it could be partial. */
+ if (index < nvectors - 1)
+ {
+ unsigned int const_nunits = nunits.to_constant ();
+ return build_int_cst (vinfo->rgroup_compare_type,
+ const_nunits * factor);
+ }
+
+ /* Vector-length agnostic types do not have a known length but that does
+ not matter because we do not use them for SLP regions that need a
+ sequence of more than one vector. */
+ if (nvectors > 1)
+ {
+ unsigned int const_nunits = nunits.to_constant ();
+ tail_size = ((group_size - 1) % const_nunits) + 1;
+ }
+ else
+ {
+ gcc_assert (known_le (poly_uint64 (tail_size), nunits));
+ }
+
+ /* A full tail vector does not need a recorded length. */
+ if (known_eq (tail_size, nunits))
+ return build_int_cst (vinfo->rgroup_compare_type, tail_size * factor);
+
+ key = tail_size;
+ nvectors = 1;
+ index = 0;
+ }
+
+ gcc_assert (key != 0);
+ rgroup_controls *rgl = &(*lens)[key - 1];
bool use_bias_adjusted_len =
vinfo->partial_load_store_bias != 0;
@@ -10988,7 +11079,8 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
tree len_type = vinfo->rgroup_compare_type;
gcc_assert (len_type != NULL_TREE);
- tree len = make_temp_ssa_name (len_type, NULL, "loop_len");
+ tree len = make_temp_ssa_name (len_type, NULL,
+ loop_vinfo ? "loop_len" : "slp_len");
/* Provide a dummy definition until the real one is available. */
SSA_NAME_DEF_STMT (len) = gimple_build_nop ();
@@ -10997,6 +11089,7 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
if (use_bias_adjusted_len)
{
gcc_assert (i == 0);
+ gcc_assert (loop_vinfo);
tree adjusted_len =
make_temp_ssa_name (len_type, NULL, "adjusted_loop_len");
SSA_NAME_DEF_STMT (adjusted_len) = gimple_build_nop ();
@@ -11032,9 +11125,15 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
{
/* The number of scalars per iteration, scalar occupied bytes and
the number of vectors are both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor).to_constant ();
+ unsigned int nscalars_per_iter;
+ if (loop_vinfo)
+ nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ else
+ nscalars_per_iter = tail_size;
+
unsigned int rglvecsize = rgl->factor * rgl->max_nscalars_per_iter;
unsigned int vecsize = nscalars_per_iter * factor;
if (rglvecsize > vecsize)
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index eacc8f1d79b..ed3b49a001a 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9810,8 +9810,16 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
if (rgc.type)
control_count++;
}
+ else if (!bb_vinfo->lens.is_empty ())
+ {
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_len);
- gcc_assert (bb_vinfo->lens.is_empty ());
+ for (auto rgc : bb_vinfo->lens)
+ if (rgc.type)
+ control_count++;
+
+ control_kind = scalar_stmt;
+ }
const int slp_instance_count = slp_instances.length (),
control_cost_per_inst_q = control_count / slp_instance_count;
@@ -10418,14 +10426,31 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
return false;
}
+ /* For now, we don't expect to mix both masking and length approaches for one
+ basic block. Fail if both are recorded. */
+ if (!bb_vinfo->masks.is_empty () && !bb_vinfo->lens.is_empty ())
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "not vectorized: can't vectorize a basic block with partial vectors "
+ "because we don't expect to mix different approaches with partial "
+ "vectors for the same basic block.\n");
+ return false;
+ }
+
/* Check whether we can generate the necessary controls. */
- gcc_assert (bb_vinfo->lens.is_empty ());
if (!bb_vinfo->masks.is_empty ())
{
if (!vect_verify_full_masking (bb_vinfo)
&& !vect_verify_full_masking_avx512 (bb_vinfo))
return false;
}
+ else if (!bb_vinfo->lens.is_empty ())
+ {
+ if (!vect_verify_loop_lens (bb_vinfo))
+ return false;
+ }
/* Mark all the statements that we vectorize. */
vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
@@ -10448,8 +10473,7 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
tree compare_type = bb_vinfo->rgroup_compare_type;
tree ctrl_type = rgc->type;
- gcc_assert (rgc->factor == 1);
- unsigned int nitems = rgc->max_nscalars_per_iter;
+ unsigned int nitems = rgc->max_nscalars_per_iter * rgc->factor;
tree zero_index = build_int_cst (compare_type, 0);
tree nitems_cst = build_int_cst (compare_type, nitems);
@@ -10457,9 +10481,16 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
tree ctrl = rgc->controls[0];
tree init_ctrl;
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- init_ctrl
- = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
+ if (bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult)
+ {
+ init_ctrl
+ = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
+ }
+ else
+ {
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_len);
+ init_ctrl = nitems_cst;
+ }
gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
gimple_seq_add_stmt (seq, stmt);
@@ -10471,15 +10502,19 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
static void
vect_set_bb_slp_controls_partial_vectors (bb_vec_info bb_vinfo)
{
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- gcc_assert (!bb_vinfo->masks.is_empty ());
+ gcc_assert ((bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult
+ && !bb_vinfo->masks.is_empty ())
+ || (bb_vinfo->partial_vector_style == vect_partial_vectors_len
+ && !bb_vinfo->lens.is_empty ()));
gimple_seq seq = NULL;
+ bool use_masks_p = !bb_vinfo->masks.is_empty ();
/* Iterate over all the rgroups and fill in their controls. */
rgroup_controls *rgc;
unsigned int i;
- auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
+ auto_vec<rgroup_controls> *controls
+ = use_masks_p ? &bb_vinfo->masks.rgc_vec : &bb_vinfo->lens;
FOR_EACH_VEC_ELT (*controls, i, rgc)
if (!rgc->controls.is_empty ())
{
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index b1d69966841..e84bb108d89 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -714,7 +714,9 @@ struct rgroup_controls {
for all members of the group, or 1 for BB SLP vectorization.
For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
highest minimum number of subparts of all the vector types that use this
- rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
+ rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI).
+ For BB SLP vectorization with vect_partial_vectors_len, this is the
+ partial tail vector size used to index the rgroup. */
unsigned int max_nscalars_per_iter;
/* For the largest nS recorded above, the loop controls divide each scalar
@@ -2769,6 +2771,7 @@ extern int vect_get_known_peeling_cost (loop_vec_info,
int);
extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
extern bool vect_verify_full_masking (vec_info *);
extern bool vect_verify_full_masking_avx512 (vec_info *);
+extern bool vect_verify_loop_lens (vec_info *);
/* Nonlinear induction. */
extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,
--
2.43.0
