This is part of enablement of support for partial vectors in basic
block SLP vectorization.
No changes to vect_record_mask are needed for AVX512-style partial
vectors because the same scheme is used for recording masks as for
WHILE_ULT.
Adapt the existing vect_verify_full_masking_avx512 function used by the
loop vectorizer to verify that partial vector mask requirements
recorded during the analysis phase can be satisfied.
As part of analyzing whether an SLP region can be vectorized, after
analyzing all statements in the region, vect_verify_full_masking is
called to check whether we can generate the necessary partial vector
masks in WHILE_ULT style. If not, call vect_verify_full_masking_avx512
to determine whether AVX512 style can be used instead before returning
false from vect_slp_analyze_bb_1.
No changes are needed to estimated costing of partial vector setup
in vect_bb_vectorization_profitable_p because the costs for AVX512 style
partial vectors can be estimated like costs for WHILE_ULT. In
particular, there is no need to account for saturation costs caused by
differing precision of IV and compare types unlike loop vectorization.
During the transform phase, vect_get_mask is called to get a temporary
SSA name for each mask. Update that function to use the SLP group
size (instead of the number of values from one scalar iteration that
the rgroup operates on) as the index into the vector of rgroup
controls.
gcc/ChangeLog:
* tree-vect-loop.cc (vect_get_mask): Use SLP_TREE_LANES as the
index into rgc_vec if doing BB SLP vectorization.
(vect_verify_full_masking_avx512): Change
parameter type from loop_vec_info to vec_info *. Use dyn_cast
to get a loop_vec_info pointer or null. Access the masks member
of vec_info directly instead of via the LOOP_VINFO_MASKS macro.
When this function is used for BB SLP, call
vect_min_prec_for_max_group_size to find the minimum precision
based on the largest SLP group size and reuse the same value as
iv_precision. Since max_nscalars_per_iter is unused for BB SLP
with AVX512-style partial vectors, set it to 1.
Instead of using nscalars_per_iter as the index into the
rgroup_controls array for BB SLP, use the second value of each pair
in the mask_set, which holds the SLP group size. Avoid calling
vect_max_vf for BB SLP because the concept of a vectorization
factor is not applicable.
Set the rgroup_compare_type, rgroup_iv_type and partial_vector_style
members of vec_info directly instead of using LOOP_VINFO_
macros.
* tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
Update assertion to permit AVX512 as the partial vectors
style. Costing is the same as for WHILE_ULT style.
(vect_slp_analyze_bb_1): If vect_verify_full_masking returns
false then fall back to trying vect_verify_full_masking_avx512
before giving up.
(vect_set_bb_slp_controls_partial_vectors_avx512): New
function. Assert that the partial vectors style is AVX512 and
that some masks were recorded. Iterate over all the rgroups.
Assert that there is only one control per rgroup (because
vect_get_mask creates only one control per recorded mask
for BB SLP, instead of one per vector for loop vectorization).
The rgroup index gives the SLP group size minus 1. If the
group size is not less than the number of subparts in the
comparison type chosen by vect_verify_full_masking_avx512
then build a constant -1 of the truth type chosen for the
rgroup; otherwise, build an ascending sequence to be compared
with another vector created by gimple_build_vector_from_val, in
which the SLP group size is broadcast. Call gimple_build to
build an LT_EXPR to create the mask by comparing the two vectors.
Call gimple_build_assign to build an assignment of the vector mask
to the temporary SSA name made by vect_get_mask. Call
gimple_seq_add_stmt to add the assignment to a gimple_seq shared
by all rgroups. Use the insert_seq_on_entry method of
the bb_vec_info to insert all stmts at entry to the SLP region.
(vect_set_bb_slp_controls): If the partial vectors style
is AVX512 then call a new function,
vect_set_bb_slp_controls_partial_vectors_avx512, instead
of vect_set_bb_slp_controls_partial_vectors.
* tree-vectorizer.h (rgroup_controls): Update the description
of max_nscalars_per_iter.
(vect_verify_full_masking_avx512): Declare
this function, which no longer has static linkage, to allow it
to be called by vect_slp_analyze_bb_1.
---
gcc/tree-vect-loop.cc | 106 +++++++++++++++++++++++++++++-------------
gcc/tree-vect-slp.cc | 76 ++++++++++++++++++++++++++++--
gcc/tree-vectorizer.h | 3 +-
3 files changed, 146 insertions(+), 39 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index e0bdc262439..4709a749181 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -1167,26 +1167,41 @@ vect_verify_full_masking (vec_info *vinfo)
whether we can actually generate AVX512 style masks. Return true if so,
storing the type of the scalar IV in LOOP_VINFO_RGROUP_IV_TYPE. */
-static bool
-vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
+bool
+vect_verify_full_masking_avx512 (vec_info *vinfo)
{
/* Produce differently organized rgc_vec and differently check
we can produce masks. */
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ bb_vec_info bb_vinfo = dyn_cast<bb_vec_info> (vinfo);
/* Use a normal loop if there are no statements that need masking.
This only happens in rare degenerate cases: it means that the loop
has no loads, no stores, and no live-out values. */
- if (LOOP_VINFO_MASKS (loop_vinfo).is_empty ())
+ if (vinfo->masks.is_empty ())
return false;
+ unsigned int min_ni_width;
+ if (loop_vinfo)
+ min_ni_width = wi::min_precision (vect_max_vf (loop_vinfo), UNSIGNED);
+ else
+ min_ni_width = vect_min_prec_for_max_group_size (&bb_vinfo->masks.rgc_vec);
+
/* For the decrementing IV we need to represent all values in
[0, niter + niter_skip] where niter_skip is the elements we
skip in the first iteration for prologue peeling. */
tree iv_type = NULL_TREE;
- widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
- unsigned int iv_precision = UINT_MAX;
- if (iv_limit != -1)
- iv_precision = wi::min_precision (iv_limit, UNSIGNED);
+ unsigned int iv_precision;
+ if (loop_vinfo)
+ {
+ widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
+ iv_precision = UINT_MAX;
+
+ if (iv_limit != -1)
+ iv_precision = wi::min_precision (iv_limit, UNSIGNED);
+ }
+ else
+ iv_precision = min_ni_width;
/* First compute the type for the IV we use to track the remaining
scalar iterations. */
@@ -1206,24 +1221,42 @@ vect_verify_full_masking_avx512 (loop_vec_info
loop_vinfo)
return false;
/* Produce the rgroup controls. */
- for (auto const &mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
+ for (auto const &mask : vinfo->masks.mask_set)
{
- vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ vec_masks *masks = &vinfo->masks;
tree vectype = mask.first;
- unsigned nvectors = mask.second;
+
+ unsigned int nvectors;
+ if (loop_vinfo)
+ nvectors = mask.second;
+ else
+ nvectors = 1;
/* The number of scalars per iteration and the number of vectors are
both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ unsigned int nscalars_per_iter;
+ if (loop_vinfo)
+ nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ else
+ nscalars_per_iter = 1;
- /* We index the rgroup_controls vector with nscalars_per_iter
- which we keep constant and instead have a varying nvectors,
- remembering the vector mask with the fewest nV. */
- if (masks->rgc_vec.length () < nscalars_per_iter)
- masks->rgc_vec.safe_grow_cleared (nscalars_per_iter, true);
- rgroup_controls *rgm = &(*masks).rgc_vec[nscalars_per_iter - 1];
+ /* For loop vectorization, we index the rgroup_controls vector with
+ nscalars_per_iter which we keep constant and instead have a varying
+ nvectors, remembering the vector mask with the fewest nV. For BB SLP
+ vectorization, we instead index rgroup_controls with the SLP group
+ size (and nV is always 1). */
+ unsigned int key;
+ if (loop_vinfo)
+ key = nscalars_per_iter;
+ else
+ key = mask.second;
+
+ if (masks->rgc_vec.length () < key)
+ masks->rgc_vec.safe_grow_cleared (key, true);
+ rgroup_controls *rgm = &(*masks).rgc_vec[key - 1];
if (!rgm->type || rgm->factor > nvectors)
{
@@ -1237,11 +1270,8 @@ vect_verify_full_masking_avx512 (loop_vec_info
loop_vinfo)
/* There is no fixed compare type we are going to use but we have to
be able to get at one for each mask group. */
- unsigned int min_ni_width
- = wi::min_precision (vect_max_vf (loop_vinfo), UNSIGNED);
-
bool ok = true;
- for (auto &rgc : LOOP_VINFO_MASKS (loop_vinfo).rgc_vec)
+ for (auto &rgc : vinfo->masks.rgc_vec)
{
tree mask_type = rgc.type;
if (!mask_type)
@@ -1294,13 +1324,13 @@ vect_verify_full_masking_avx512 (loop_vec_info
loop_vinfo)
}
if (!ok)
{
- release_vec_loop_controls (&LOOP_VINFO_MASKS (loop_vinfo).rgc_vec);
+ release_vec_loop_controls (&vinfo->masks.rgc_vec);
return false;
}
- LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = error_mark_node;
- LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) = vect_partial_vectors_avx512;
+ vinfo->rgroup_compare_type = error_mark_node;
+ vinfo->rgroup_iv_type = iv_type;
+ vinfo->partial_vector_style = vect_partial_vectors_avx512;
return true;
}
@@ -10753,12 +10783,22 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator
*gsi,
}
else if (vinfo->partial_vector_style == vect_partial_vectors_avx512)
{
- /* The number of scalars per iteration and the number of vectors are
- both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor)
- .to_constant ();
+ unsigned int nscalars_per_iter;
+
+ if (loop_vinfo)
+ {
+ /* The number of scalars per iteration and the number of vectors are
+ both compile-time constants. */
+ nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ }
+ else
+ {
+ gcc_assert (nvectors == 1);
+ nscalars_per_iter = SLP_TREE_LANES (slp_node);
+ }
rgroup_controls *rgm = &masks->rgc_vec[nscalars_per_iter - 1];
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 5348b44f64a..fcd889945bd 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9670,8 +9670,10 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
if (!bb_vinfo->masks.is_empty ())
{
- gcc_assert (bb_vinfo->partial_vector_style
- == vect_partial_vectors_while_ult);
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512
+ || bb_vinfo->partial_vector_style
+ == vect_partial_vectors_while_ult);
+
for (auto rgc : bb_vinfo->masks.rgc_vec)
if (rgc.type)
control_count++;
@@ -10283,7 +10285,8 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
gcc_assert (bb_vinfo->lens.is_empty ());
if (!bb_vinfo->masks.is_empty ())
{
- if (!vect_verify_full_masking (bb_vinfo))
+ if (!vect_verify_full_masking (bb_vinfo)
+ && !vect_verify_full_masking_avx512 (bb_vinfo))
return false;
}
@@ -10350,14 +10353,77 @@ vect_set_bb_slp_controls_partial_vectors (bb_vec_info
bb_vinfo)
bb_vinfo->insert_seq_on_entry (NULL, seq);
}
+/* Set up the rgroup controls for the region described by BB_VINFO in AVX512
+ style, given that the region uses partial vectors. */
+
+static void
+vect_set_bb_slp_controls_partial_vectors_avx512 (bb_vec_info bb_vinfo)
+{
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512);
+ gcc_assert (!bb_vinfo->masks.is_empty ());
+
+ gimple_seq seq = NULL;
+
+ /* Iterate over all the rgroups and fill in their controls. */
+ rgroup_controls *rgc;
+ unsigned int i;
+ auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
+ FOR_EACH_VEC_ELT (*controls, i, rgc)
+ {
+ if (rgc->controls.is_empty ())
+ continue;
+
+ tree ctrl_type = rgc->type;
+ poly_uint64 nitems_per_ctrl = TYPE_VECTOR_SUBPARTS (ctrl_type);
+
+ /* Provide a definition of the one vector in the control group. */
+ gcc_assert (rgc->controls.length () == 1);
+ tree ctrl = rgc->controls[0];
+ tree init_ctrl;
+
+ /* See whether it is a full control. */
+ if (known_ge (i + 1, nitems_per_ctrl))
+ init_ctrl = build_minus_one_cst (ctrl_type);
+ else
+ {
+ tree vectype = rgc->compare_type;
+ tree nitems_cst = build_int_cst (TREE_TYPE (vectype), i + 1);
+
+ /* Build a vector, CMP_SERIES, to be compared with NITEMS_VEC. For
+ example, if CMP_SERIES is { 0, 1, 2, 3, ... } and NITEMS_VEC is
+ { 2, 2, 2, 2, ... } then the result of a less-than comparison is
+ { T, T, F, F, ... }, which enables the first two lanes. */
+ unsigned n = TYPE_VECTOR_SUBPARTS (ctrl_type).to_constant ();
+ tree_vector_builder builder (vectype, n, 1);
+ gcc_assert (rgc->max_nscalars_per_iter == 1);
+ for (unsigned j = 0; j < n; ++j)
+ {
+ builder.quick_push (build_int_cst (TREE_TYPE (vectype), j));
+ }
+ tree cmp_series = builder.build ();
+
+ tree nitems_vec
+ = gimple_build_vector_from_val (&seq, vectype, nitems_cst);
+ init_ctrl
+ = gimple_build (&seq, LT_EXPR, ctrl_type, cmp_series, nitems_vec);
+ }
+ gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
+ gimple_seq_add_stmt (&seq, stmt);
+ }
+
+ bb_vinfo->insert_seq_on_entry (NULL, seq);
+}
+
/* Set up the rgroup controls for the region described by BB_VINFO, if
the region uses partial vectors. */
static void
vect_set_bb_slp_controls (bb_vec_info bb_vinfo)
{
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
+ if (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512)
+ vect_set_bb_slp_controls_partial_vectors_avx512 (bb_vinfo);
+ else
+ vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
}
/* Subroutine of vect_slp_bb. Try to vectorize the statements for all
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 6f233e9045b..5ba9331cfcb 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -705,7 +705,7 @@ public:
struct rgroup_controls {
/* The largest nS for all rgroups that use these controls.
For vect_partial_vectors_avx512 this is the constant nscalars_per_iter
- for all members of the group.
+ for all members of the group, or 1 for BB SLP vectorization.
For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
highest minimum number of subparts of all the vector types that use this
rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
@@ -2778,6 +2778,7 @@ extern bool vect_can_vectorize_without_simd_p
(code_helper);
extern int vect_get_known_peeling_cost (loop_vec_info, int);
extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
extern bool vect_verify_full_masking (vec_info *);
+extern bool vect_verify_full_masking_avx512 (vec_info *);
/* Nonlinear induction. */
extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,
--
2.43.0
