This is part of enablement of support for partial vectors in basic
block SLP vectorization.
Adapt the existing vect_verify_loop_lens function used by the loop
vectorizer to verify that partial vector length requirements recorded
during the analysis phase can be satisfied. Unlike masked partial
vectors, the elements-to-subelements scaling factor is not always one,
so apply it when determining the maximum precision needed to store
lengths used to implement SLP group sizes.
A different scheme is used for recording lengths for BB SLP
vectorization but it reuses as much loop vectorizer code as possible:
instead of using the passed-in number of vectors as an index to find
the relevant rgroup controls in vect_record_len and vect_get_len,
rgroup controls are instead indexed by the number of lanes in an SLP
tree node. Since there is no scheme for mapping nvectors to
rgroup_controls, nvectors must always be one. Consequently, the number
of controls per rgroup_controls instance is also one for BB SLP. (One
length for each unique number of lanes.)
The max_nscalars_per_iter of an rgroup is already assumed by
vect_set_bb_slp_controls_directly to hold the number of lanes in a
mask to be generated. Carry that assumption over to LEN-style partial
vectors: in vect_record_len, record the largest SLP group size for a
given rgroup as max_nscalars_per_iter. Because of how rgroups are
indexed, the value of max_nscalars_per_iter will either be zero (on
initialization) or one more than the rgroup index.
As part of analyzing whether an SLP region can be vectorized, after
analyzing all statements in the region, if lengths have been recorded
then call vect_verify_loop_lens to check whether we can generate
the necessary length controls. If not, return false from
vect_slp_analyze_bb_1.
Costing in vect_bb_vectorization_profitable_p is extended to
account for partial vector setup costs for an SLP vectorization
region that uses LEN style controls. This is similar to WHILE_ULT
and AVX512 costing except that the cost kind is scalar_stmt instead
of vector_stmt (as in the equivalent loop vectorization code).
As in the case of masks, this is imprecise because lengths can be
shared between SLP nodes and some subgraphs might not be scheduled.
After scheduling any profitable subgraphs and generating any
invariant statements, unless AVX512-style partial vectors are in
use, vect_set_bb_slp_controls_partial_vectors is called. This
function is readily adapted to iterate over the recorded lens
instead of the recorded masks. vect_set_bb_slp_controls_directly
now scales the SLP group size by the rgroup's factor (which will
be 1 for WHILE_ULT-style partial vectors). If using LEN-style
partial vectors, it assigns that scaled length to the SSA name
created by vect_get_len for the control.
Dead stmts that set up lengths for unprofitable subgraphs are
expected to be eliminated by later passes.
gcc/ChangeLog:
* tree-vect-loop.cc (vect_min_prec_for_max_group_size):
Permit the elements-to-subelements scaling factor to be other
than 1 and use it to scale group_size.
(vect_verify_loop_lens): Change parameter type from loop_vec_info
to vec_info *. Use dyn_cast to get a loop_vec_info or null from
the argument.
Access lengths directly instead of via LOOP_VINFO_LENS. Avoid
calling vect_min_prec_for_max_niters for BB SLP because the
concept of iterations is not applicable. Instead, call
vect_min_prec_for_max_group_size to get an alternative value of
min_ni_prec. Set rgroup_compare_type, rgroup_iv_type and
partial_vector_style directly instead of via LOOP_VINFO_
macros.
(vect_record_len): Use dyn_cast to get
a loop_vec_info or null from the vec_info argument.
If used for BB SLP, get the group size from the SLP tree
node and use that as the index of the rgroup control to be
populated (instead of the nvectors argument). Assert that
nvectors is 1, if used for BB SLP. When doing BB SLP, use the
SLP group size instead of the number of scalar values operated
on by each scalar iteration as max_nscalars_per_iter.
(vect_get_len): Don't call make_temp_ssa_name with "loop_len"
as the name, if used for BB SLP. Assert that nvectors is
1 and only the first vector is queried, if used for BB SLP.
If used for BB SLP, get the group size from the SLP tree node
and use that (instead of the nvectors argument) as the index
of the rgroup control. Assert that no SSA name is created for
"adjusted_loop_len" if used for BB SLP.
If the passed-in elements-to-subelements factor differs from
that recorded for the SLP group size then calculate the
overall scaling factor required to build EXACT_DIV_EXPR or
MULT_EXPR by using the constant lower bound of the number of
subparts in the passed-in vector type, to match the
calculation of max_nscalars_per_iter for the rgroup.
* tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
Permit the set of recorded lengths to be non-empty.
Add code to count setup costs for LEN-style partial vectors.
(vect_slp_analyze_bb_1): Return false if both mask and length
requirements for partial vectors were recorded for a BB SLP
vectorization region. Permit the set of recorded lengths to be
non-empty. If lengths have been recorded for an SLP region then
call vect_verify_loop_lens to check whether we can generate the
necessary controls. If not, return false.
(vect_set_bb_slp_controls_directly): If the partial vector
style is not WHILE_ULT then require it to be LEN and scale
max_nscalars_per_iter by subelements-per-element before using
it as the righthand side of the assignment to the SSA name made
by vect_get_len.
(vect_set_bb_slp_controls_partial_vectors): Permit the set of
recorded lengths to be non-empty if the partial vector style is
LEN. If the set of recorded masks is empty then iterate over
the set of recorded lengths instead.
* tree-vectorizer.h (rgroup_controls): Update the description
of max_nscalars_per_iter.
(vect_verify_loop_lens): Add function declaration for use by
vect_slp_analyze_bb_1. This function no longer has static
linkage.
---
gcc/tree-vect-loop.cc | 138 ++++++++++++++++++++++++++++--------------
gcc/tree-vect-slp.cc | 55 ++++++++++++++---
gcc/tree-vectorizer.h | 8 ++-
3 files changed, 142 insertions(+), 59 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 4709a749181..b5443bef376 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -1002,8 +1002,7 @@ vect_min_prec_for_max_group_size (vec<rgroup_controls>
*controls)
{
if (!rgc->type)
continue;
- gcc_assert (rgc->factor == 1);
- unsigned int group_size = i + 1;
+ unsigned int group_size = (i + 1) * rgc->factor;
max_group_size = MAX (max_group_size, group_size);
}
@@ -1339,20 +1338,22 @@ vect_verify_full_masking_avx512 (vec_info *vinfo)
precision of the target supported length is larger than the precision
required by loop niters. */
-static bool
-vect_verify_loop_lens (loop_vec_info loop_vinfo)
+bool
+vect_verify_loop_lens (vec_info *vinfo)
{
- if (LOOP_VINFO_LENS (loop_vinfo).is_empty ())
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+
+ if (vinfo->lens.is_empty ())
return false;
- if (!VECTOR_MODE_P (loop_vinfo->vector_mode))
+ if (!VECTOR_MODE_P (vinfo->vector_mode))
return false;
machine_mode len_load_mode, len_store_mode;
- if (!get_len_load_store_mode (loop_vinfo->vector_mode, true)
+ if (!get_len_load_store_mode (vinfo->vector_mode, true)
.exists (&len_load_mode))
return false;
- if (!get_len_load_store_mode (loop_vinfo->vector_mode, false)
+ if (!get_len_load_store_mode (vinfo->vector_mode, false)
.exists (&len_store_mode))
return false;
@@ -1370,39 +1371,47 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo)
/* If the backend requires a bias of -1 for LEN_LOAD, we must not emit
len_loads with a length of zero. In order to avoid that we prohibit
more than one loop length here. */
- if (partial_load_bias == -1
- && LOOP_VINFO_LENS (loop_vinfo).length () > 1)
+ if (partial_load_bias == -1 && vinfo->lens.length () > 1)
return false;
- LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) = partial_load_bias;
+ vinfo->partial_load_store_bias = partial_load_bias;
- unsigned int max_nitems_per_iter = 1;
+ unsigned int min_ni_prec;
unsigned int i;
rgroup_controls *rgl;
- /* Find the maximum number of items per iteration for every rgroup. */
- FOR_EACH_VEC_ELT (LOOP_VINFO_LENS (loop_vinfo), i, rgl)
+
+ if (loop_vinfo)
{
- unsigned nitems_per_iter = rgl->max_nscalars_per_iter * rgl->factor;
- max_nitems_per_iter = MAX (max_nitems_per_iter, nitems_per_iter);
- }
+ unsigned int max_nitems_per_iter = 1;
- /* Work out how many bits we need to represent the length limit. */
- unsigned int min_ni_prec
- = vect_min_prec_for_max_niters (loop_vinfo, max_nitems_per_iter);
+ /* Find the maximum number of items per iteration for every rgroup. */
+ FOR_EACH_VEC_ELT (LOOP_VINFO_LENS (loop_vinfo), i, rgl)
+ {
+ unsigned nitems_per_iter = rgl->max_nscalars_per_iter * rgl->factor;
+ max_nitems_per_iter = MAX (max_nitems_per_iter, nitems_per_iter);
+ }
- /* Now use the maximum of below precisions for one suitable IV type:
- - the IV's natural precision
- - the precision needed to hold: the maximum number of scalar
- iterations multiplied by the scale factor (min_ni_prec above)
- - the Pmode precision
+ /* Work out how many bits we need to represent the length limit. */
+ min_ni_prec
+ = vect_min_prec_for_max_niters (loop_vinfo, max_nitems_per_iter);
- If min_ni_prec is less than the precision of the current niters,
- we prefer to still use the niters type. Prefer to use Pmode and
- wider IV to avoid narrow conversions. */
+ /* Now use the maximum of below precisions for one suitable IV type:
+ - the IV's natural precision
+ - the precision needed to hold: the maximum number of scalar
+ iterations multiplied by the scale factor (min_ni_prec above)
+ - the Pmode precision
+
+ If min_ni_prec is less than the precision of the current niters,
+ we prefer to still use the niters type. Prefer to use Pmode and
+ wider IV to avoid narrow conversions. */
+
+ unsigned int ni_prec
+ = TYPE_PRECISION (TREE_TYPE (LOOP_VINFO_NITERS (loop_vinfo)));
+ min_ni_prec = MAX (min_ni_prec, ni_prec);
+ }
+ else
+ min_ni_prec = vect_min_prec_for_max_group_size (&vinfo->lens);
- unsigned int ni_prec
- = TYPE_PRECISION (TREE_TYPE (LOOP_VINFO_NITERS (loop_vinfo)));
- min_ni_prec = MAX (min_ni_prec, ni_prec);
min_ni_prec = MAX (min_ni_prec, GET_MODE_BITSIZE (Pmode));
tree iv_type = NULL_TREE;
@@ -1434,9 +1443,9 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo)
return false;
}
- LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) = vect_partial_vectors_len;
+ vinfo->rgroup_compare_type = iv_type;
+ vinfo->rgroup_iv_type = iv_type;
+ vinfo->partial_vector_style = vect_partial_vectors_len;
return true;
}
@@ -10874,16 +10883,31 @@ vect_record_len (vec_info *vinfo, unsigned int
nvectors, tree vectype,
gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
gcc_assert (nvectors != 0);
vec_lens *lens = &vinfo->lens;
- if (lens->length () < nvectors)
- lens->safe_grow_cleared (nvectors, true);
- rgroup_controls *rgl = &(*lens)[nvectors - 1];
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ unsigned int key;
+
+ if (loop_vinfo)
+ key = nvectors;
+ else
+ {
+ gcc_assert (nvectors == 1);
+ key = SLP_TREE_LANES (slp_node);
+ }
+
+ gcc_assert (key != 0);
+ if (lens->length () < key)
+ lens->safe_grow_cleared (key, true);
+ rgroup_controls *rgl = &(*lens)[key - 1];
/* The number of scalars per iteration, scalar occupied bytes and
the number of vectors are both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor)
- .to_constant ();
+ unsigned int nscalars_per_iter;
+ if (loop_vinfo)
+ nscalars_per_iter = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ else
+ nscalars_per_iter = SLP_TREE_LANES (slp_node);
if (rgl->max_nscalars_per_iter < nscalars_per_iter)
{
@@ -10912,8 +10936,21 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
bool adjusted, slp_tree slp_node)
{
gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
+ gcc_assert (nvectors != 0);
+ gcc_assert (index < nvectors);
+
vec_lens *lens = &vinfo->lens;
- rgroup_controls *rgl = &(*lens)[nvectors - 1];
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ unsigned int key;
+ if (loop_vinfo)
+ key = nvectors;
+ else
+ {
+ gcc_assert (nvectors == 1);
+ key = SLP_TREE_LANES (slp_node);
+ }
+ gcc_assert (key != 0);
+ rgroup_controls *rgl = &(*lens)[key - 1];
bool use_bias_adjusted_len =
vinfo->partial_load_store_bias != 0;
@@ -10927,7 +10964,8 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
tree len_type = vinfo->rgroup_compare_type;
gcc_assert (len_type != NULL_TREE);
- tree len = make_temp_ssa_name (len_type, NULL, "loop_len");
+ tree len = make_temp_ssa_name (len_type, NULL,
+ loop_vinfo ? "loop_len" : "slp_len");
/* Provide a dummy definition until the real one is available. */
SSA_NAME_DEF_STMT (len) = gimple_build_nop ();
@@ -10936,6 +10974,7 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
if (use_bias_adjusted_len)
{
gcc_assert (i == 0);
+ gcc_assert (loop_vinfo);
tree adjusted_len =
make_temp_ssa_name (len_type, NULL, "adjusted_loop_len");
SSA_NAME_DEF_STMT (adjusted_len) = gimple_build_nop ();
@@ -10971,9 +11010,16 @@ vect_get_len (vec_info *vinfo, gimple_stmt_iterator
*gsi, unsigned int nvectors,
{
/* The number of scalars per iteration, scalar occupied bytes and
the number of vectors are both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor).to_constant ();
+ unsigned int nscalars_per_iter;
+ if (loop_vinfo)
+ nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ else
+ nscalars_per_iter
+ = constant_lower_bound (TYPE_VECTOR_SUBPARTS (vectype));
+
unsigned int rglvecsize = rgl->factor * rgl->max_nscalars_per_iter;
unsigned int vecsize = nscalars_per_iter * factor;
if (rglvecsize > vecsize)
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index fcd889945bd..e919282bd43 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9678,8 +9678,16 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
if (rgc.type)
control_count++;
}
+ else if (!bb_vinfo->lens.is_empty ())
+ {
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_len);
- gcc_assert (bb_vinfo->lens.is_empty ());
+ for (auto rgc : bb_vinfo->lens)
+ if (rgc.type)
+ control_count++;
+
+ control_kind = scalar_stmt;
+ }
const int slp_instance_count = slp_instances.length (),
control_cost_per_inst_q = control_count / slp_instance_count;
@@ -10281,14 +10289,31 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
return false;
}
+ /* For now, we don't expect to mix both masking and length approaches for one
+ basic block. Fail if both are recorded. */
+ if (!bb_vinfo->masks.is_empty () && !bb_vinfo->lens.is_empty ())
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "not vectorized: can't vectorize a basic block with partial vectors "
+ "because we don't expect to mix different approaches with partial "
+ "vectors for the same basic block.\n");
+ return false;
+ }
+
/* Check whether we can generate the necessary controls. */
- gcc_assert (bb_vinfo->lens.is_empty ());
if (!bb_vinfo->masks.is_empty ())
{
if (!vect_verify_full_masking (bb_vinfo)
&& !vect_verify_full_masking_avx512 (bb_vinfo))
return false;
}
+ else if (!bb_vinfo->lens.is_empty ())
+ {
+ if (!vect_verify_loop_lens (bb_vinfo))
+ return false;
+ }
/* Mark all the statements that we vectorize. */
vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
@@ -10311,8 +10336,7 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
tree compare_type = bb_vinfo->rgroup_compare_type;
tree ctrl_type = rgc->type;
- gcc_assert (rgc->factor == 1);
- unsigned int nitems = rgc->max_nscalars_per_iter;
+ unsigned int nitems = rgc->max_nscalars_per_iter * rgc->factor;
tree zero_index = build_int_cst (compare_type, 0);
tree nitems_cst = build_int_cst (compare_type, nitems);
@@ -10320,9 +10344,16 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
tree ctrl = rgc->controls[0];
tree init_ctrl;
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- init_ctrl
- = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
+ if (bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult)
+ {
+ init_ctrl
+ = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
+ }
+ else
+ {
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_len);
+ init_ctrl = nitems_cst;
+ }
gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
gimple_seq_add_stmt (seq, stmt);
@@ -10334,15 +10365,19 @@ vect_set_bb_slp_controls_directly (bb_vec_info
bb_vinfo, gimple_seq *seq,
static void
vect_set_bb_slp_controls_partial_vectors (bb_vec_info bb_vinfo)
{
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- gcc_assert (!bb_vinfo->masks.is_empty ());
+ gcc_assert ((bb_vinfo->partial_vector_style == vect_partial_vectors_while_ult
+ && !bb_vinfo->masks.is_empty ())
+ || (bb_vinfo->partial_vector_style == vect_partial_vectors_len
+ && !bb_vinfo->lens.is_empty ()));
gimple_seq seq = NULL;
+ bool use_masks_p = !bb_vinfo->masks.is_empty ();
/* Iterate over all the rgroups and fill in their controls. */
rgroup_controls *rgc;
unsigned int i;
- auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
+ auto_vec<rgroup_controls> *controls
+ = use_masks_p ? &bb_vinfo->masks.rgc_vec : &bb_vinfo->lens;
FOR_EACH_VEC_ELT (*controls, i, rgc)
if (!rgc->controls.is_empty ())
{
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 5ba9331cfcb..634c02e675f 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -706,9 +706,10 @@ struct rgroup_controls {
/* The largest nS for all rgroups that use these controls.
For vect_partial_vectors_avx512 this is the constant nscalars_per_iter
for all members of the group, or 1 for BB SLP vectorization.
- For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
- highest minimum number of subparts of all the vector types that use this
- rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
+ For BB SLP vectorization with vect_partial_vectors_while_ult or
+ vect_partial_vectors_len, this is the highest minimum number of subparts
of
+ all the vector types that use this rgroup (e.g., 16 from VNx4HI, VNx4SI,
+ VNx4HF and VNx16QI). */
unsigned int max_nscalars_per_iter;
/* For the largest nS recorded above, the loop controls divide each scalar
@@ -2779,6 +2780,7 @@ extern int vect_get_known_peeling_cost (loop_vec_info,
int);
extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
extern bool vect_verify_full_masking (vec_info *);
extern bool vect_verify_full_masking_avx512 (vec_info *);
+extern bool vect_verify_loop_lens (vec_info *);
/* Nonlinear induction. */
extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,
--
2.43.0
