This is part of enablement of support for partial vectors in basic
block SLP vectorization.
No changes to vect_record_mask are needed for AVX512-style partial
vectors because the same scheme is used for recording masks as for
WHILE_ULT.
Adapt the existing vect_verify_full_masking_avx512 function used by the
loop vectorizer to verify that partial vector mask requirements
recorded during the analysis phase can be satisfied, and choose a mask
type for each partial vector tail length. To allow mask-splitting code
in vect_get_mask to be reused despite having been designed for loop
vectorization, a slightly artificial constraint is imposed which is that
the chosen mask type must not only be wide enough to represent the mask,
but also as wide as the widest type to be masked. This avoids the
potential need to zero-extend masks.
As part of analyzing whether an SLP region can be vectorized, after
analyzing all statements in the region, vect_verify_full_masking is
called to check whether we can generate the necessary partial vector
masks in WHILE_ULT style. If not, call vect_verify_full_masking_avx512
to determine whether AVX512 style can be used instead before returning
false from vect_slp_analyze_bb_1.
No changes are needed to estimated costing of partial vector setup
in vect_bb_vectorization_profitable_p because the costs for AVX512 style
partial vectors can be estimated like costs for WHILE_ULT. In
particular, there is no need to account for saturation costs caused by
differing precision of IV and compare types unlike loop vectorization.
During the transform phase, vect_get_mask is called to get a temporary
SSA name for each mask. Update that function to use the partial tail
size (instead of the number of values from one scalar iteration that
the rgroup operates on) as the index into the vector of rgroup
controls.
gcc/ChangeLog:
* tree-vect-loop.cc (vect_get_mask): For BB SLP, use the
partial tail vector size (calculated from the passed-in group
size and vector type) to select the rgroup_controls. Since an
instance of rgroup_controls represents one partial tail mask
instead of a sequence of masks, assert that the selected mask
type has at least as many subparts as the vector type to be
masked. Zero the mask index so that existing code shared with
the loop vectorizer always selects the lowest part of the first
mask.
(vect_verify_full_masking_avx512): Change parameter type from
loop_vec_info to vec_info *. Use dyn_cast to get a loop_vec_info
pointer or null. Access the masks member of vec_info directly
instead of via the LOOP_VINFO_MASKS macro.
Avoid calling vect_max_vf for BB SLP because the concept of a
vectorization factor is not applicable; instead, use
vect_min_prec_for_slp to find the minimum precision
based on the largest partial tail size and reuse the same value as
iv_precision. Since max_nscalars_per_iter is unused for BB SLP
with AVX512-style partial vectors, set it to 1.
Instead of using nscalars_per_iter as the index into the
rgroup_controls array for BB SLP, use the second value of each pair
in the mask_set, which holds the partial tail size. Instead of
recording the mask type that required the fewest controls (as
for loop vectorization), the BB SLP vectorizer records the mask
type that has the most subparts.
Set the rgroup_compare_type, rgroup_iv_type and partial_vector_style
members of vec_info directly instead of using LOOP_VINFO_
macros.
* tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
Update assertion to permit AVX512 as the partial vectors
style. Costing is the same as for WHILE_ULT style.
(vect_slp_analyze_bb_1): If vect_verify_full_masking returns
false then fall back to trying vect_verify_full_masking_avx512
before giving up.
(vect_set_bb_slp_controls_partial_vectors_avx512): New
function. Assert that the partial vectors style is AVX512 and
that some masks were recorded. Iterate over all the rgroups.
Assert that there is only one control per rgroup (because
vect_get_mask creates only one control per recorded mask
requirement for BB SLP, instead of one control per mask in
a sequence of NVECTORS recorded for loop vectorization).
The rgroup index gives the partial tail size minus 1. Build an
ascending sequence to be compared with another vector created by
gimple_build_vector_from_val, in which the partial tail size is
broadcast. Call gimple_build to build an LT_EXPR to create the
mask by comparing the two vectors. Call gimple_build_assign to
build an assignment of the vector mask to the temporary SSA name
made by vect_get_mask. Call gimple_seq_add_stmt to add the
assignment to a gimple_seq shared by all rgroups. Use the
insert_seq_on_entry method of the bb_vec_info to insert all stmts
at entry to the SLP region.
(vect_set_bb_slp_controls): If the partial vectors style
is AVX512 then call a new function,
vect_set_bb_slp_controls_partial_vectors_avx512, instead
of vect_set_bb_slp_controls_partial_vectors.
* tree-vectorizer.h (rgroup_controls): Update the description
of max_nscalars_per_iter.
(vect_verify_full_masking_avx512): Declare
this function, which no longer has static linkage, to allow it
to be called by vect_slp_analyze_bb_1.
---
gcc/tree-vect-loop.cc | 156 ++++++++++++++++++++++++++++++------------
gcc/tree-vect-slp.cc | 67 ++++++++++++++++--
gcc/tree-vectorizer.h | 6 +-
3 files changed, 177 insertions(+), 52 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 344a65f636a..aad672631a2 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -1166,26 +1166,41 @@ vect_verify_full_masking (vec_info *vinfo)
whether we can actually generate AVX512 style masks. Return true if so,
storing the type of the scalar IV in LOOP_VINFO_RGROUP_IV_TYPE. */
-static bool
-vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
+bool
+vect_verify_full_masking_avx512 (vec_info *vinfo)
{
/* Produce differently organized rgc_vec and differently check
we can produce masks. */
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ bb_vec_info bb_vinfo = dyn_cast<bb_vec_info> (vinfo);
/* Use a normal loop if there are no statements that need masking.
This only happens in rare degenerate cases: it means that the loop
has no loads, no stores, and no live-out values. */
- if (LOOP_VINFO_MASKS (loop_vinfo).is_empty ())
+ if (vinfo->masks.is_empty ())
return false;
+ unsigned int min_ni_width;
+ if (loop_vinfo)
+ min_ni_width = wi::min_precision (vect_max_vf (loop_vinfo), UNSIGNED);
+ else
+ min_ni_width = vect_min_prec_for_slp (&bb_vinfo->masks.rgc_vec);
+
/* For the decrementing IV we need to represent all values in
[0, niter + niter_skip] where niter_skip is the elements we
skip in the first iteration for prologue peeling. */
tree iv_type = NULL_TREE;
- widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
- unsigned int iv_precision = UINT_MAX;
- if (iv_limit != -1)
- iv_precision = wi::min_precision (iv_limit, UNSIGNED);
+ unsigned int iv_precision;
+ if (loop_vinfo)
+ {
+ widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
+ iv_precision = UINT_MAX;
+
+ if (iv_limit != -1)
+ iv_precision = wi::min_precision (iv_limit, UNSIGNED);
+ }
+ else
+ iv_precision = min_ni_width;
/* First compute the type for the IV we use to track the remaining
scalar iterations. */
@@ -1205,42 +1220,70 @@ vect_verify_full_masking_avx512 (loop_vec_info
loop_vinfo)
return false;
/* Produce the rgroup controls. */
- for (auto const &mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
+ for (auto const &mask : vinfo->masks.mask_set)
{
- vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ vec_masks *masks = &vinfo->masks;
tree vectype = mask.first;
- unsigned nvectors = mask.second;
- /* The number of scalars per iteration and the number of vectors are
- both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ if (loop_vinfo)
+ {
+ /* For loop vectorization, we index the rgroup_controls vector with
+ nscalars_per_iter, which we keep constant while the required mask
+ type and number of vectors to be masked can vary. The number of
+ scalars per iteration and the number of vectors are both
+ compile-time constants. */
+ unsigned int nvectors = mask.second;
+ unsigned int nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ unsigned int key = nscalars_per_iter;
- /* We index the rgroup_controls vector with nscalars_per_iter
- which we keep constant and instead have a varying nvectors,
- remembering the vector mask with the fewest nV. */
- if (masks->rgc_vec.length () < nscalars_per_iter)
- masks->rgc_vec.safe_grow_cleared (nscalars_per_iter, true);
- rgroup_controls *rgm = &(*masks).rgc_vec[nscalars_per_iter - 1];
+ if (masks->rgc_vec.length () < key)
+ masks->rgc_vec.safe_grow_cleared (key, true);
+ rgroup_controls *rgm = &(*masks).rgc_vec[key - 1];
- if (!rgm->type || rgm->factor > nvectors)
+ /* Use the mask type that requires the fewest controls. */
+ if (!rgm->type || rgm->factor > nvectors)
+ {
+ rgm->type = truth_type_for (vectype);
+ rgm->compare_type = NULL_TREE;
+ rgm->max_nscalars_per_iter = nscalars_per_iter;
+ rgm->factor = nvectors;
+ rgm->bias_adjusted_ctrl = NULL_TREE;
+ }
+ }
+ else
{
- rgm->type = truth_type_for (vectype);
- rgm->compare_type = NULL_TREE;
- rgm->max_nscalars_per_iter = nscalars_per_iter;
- rgm->factor = nvectors;
- rgm->bias_adjusted_ctrl = NULL_TREE;
+ /* For BB SLP vectorization, we index rgroup_controls with the partial
+ tail size, which we keep constant while the required mask type can
+ vary. Only the tail vector can require a control, so there is no
+ question of the number of vectors to be masked varying. */
+ unsigned int key = mask.second;
+ tree mask_type = truth_type_for (vectype);
+
+ if (masks->rgc_vec.length () < key)
+ masks->rgc_vec.safe_grow_cleared (key, true);
+ rgroup_controls *rgm = &(*masks).rgc_vec[key - 1];
+
+ /* Use the mask type that has the most subparts. */
+ if (!rgm->type
+ || known_gt (TYPE_VECTOR_SUBPARTS (mask_type),
+ TYPE_VECTOR_SUBPARTS (rgm->type)))
+ {
+ rgm->type = mask_type;
+ rgm->compare_type = NULL_TREE;
+ rgm->max_nscalars_per_iter = 1;
+ rgm->factor = 1;
+ rgm->bias_adjusted_ctrl = NULL_TREE;
+ }
}
}
/* There is no fixed compare type we are going to use but we have to
be able to get at one for each mask group. */
- unsigned int min_ni_width
- = wi::min_precision (vect_max_vf (loop_vinfo), UNSIGNED);
-
bool ok = true;
- for (auto &rgc : LOOP_VINFO_MASKS (loop_vinfo).rgc_vec)
+ for (auto &rgc : vinfo->masks.rgc_vec)
{
tree mask_type = rgc.type;
if (!mask_type)
@@ -1293,13 +1336,13 @@ vect_verify_full_masking_avx512 (loop_vec_info
loop_vinfo)
}
if (!ok)
{
- release_vec_loop_controls (&LOOP_VINFO_MASKS (loop_vinfo).rgc_vec);
+ release_vec_loop_controls (&vinfo->masks.rgc_vec);
return false;
}
- LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = error_mark_node;
- LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) = vect_partial_vectors_avx512;
+ vinfo->rgroup_compare_type = error_mark_node;
+ vinfo->rgroup_iv_type = iv_type;
+ vinfo->partial_vector_style = vect_partial_vectors_avx512;
return true;
}
@@ -10786,19 +10829,42 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator
*gsi,
}
else if (vinfo->partial_vector_style == vect_partial_vectors_avx512)
{
- /* The number of scalars per iteration and the number of vectors are
- both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- vinfo->vectorization_factor)
- .to_constant ();
+ unsigned int nscalars_per_iter;
+
+ if (loop_vinfo)
+ {
+ /* The number of scalars per iteration and the number of vectors are
+ both compile-time constants. */
+ nscalars_per_iter
+ = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ vinfo->vectorization_factor)
+ .to_constant ();
+ }
+ else
+ {
+ nscalars_per_iter = tail_size;
+ }
rgroup_controls *rgm = &masks->rgc_vec[nscalars_per_iter - 1];
- /* The stored nV is dependent on the mask type produced. */
- gcc_assert (exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- TYPE_VECTOR_SUBPARTS (rgm->type)).to_constant ()
- == rgm->factor);
+ if (loop_vinfo)
+ {
+ /* The stored nV is dependent on the mask type produced. */
+ gcc_assert (exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
+ TYPE_VECTOR_SUBPARTS (rgm->type))
+ .to_constant ()
+ == rgm->factor);
+ }
+ else
+ {
+ /* Only a partial tail vector requires a mask and the width of the
+ tail vector must not exceed the width of the mask. */
+ gcc_assert (known_le (TYPE_VECTOR_SUBPARTS (vectype),
+ TYPE_VECTOR_SUBPARTS (rgm->type)));
+ gcc_assert (rgm->factor == 1);
+ index = 0;
+ }
+
nvectors = rgm->factor;
/* Populate the rgroup's mask array, if this is the first time we've
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 7244b82c9fc..eacc8f1d79b 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9802,8 +9802,10 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
if (!bb_vinfo->masks.is_empty ())
{
- gcc_assert (bb_vinfo->partial_vector_style
- == vect_partial_vectors_while_ult);
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512
+ || bb_vinfo->partial_vector_style
+ == vect_partial_vectors_while_ult);
+
for (auto rgc : bb_vinfo->masks.rgc_vec)
if (rgc.type)
control_count++;
@@ -10420,7 +10422,8 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
gcc_assert (bb_vinfo->lens.is_empty ());
if (!bb_vinfo->masks.is_empty ())
{
- if (!vect_verify_full_masking (bb_vinfo))
+ if (!vect_verify_full_masking (bb_vinfo)
+ && !vect_verify_full_masking_avx512 (bb_vinfo))
return false;
}
@@ -10487,14 +10490,68 @@ vect_set_bb_slp_controls_partial_vectors (bb_vec_info
bb_vinfo)
bb_vinfo->insert_seq_on_entry (NULL, seq);
}
+/* Set up the rgroup controls for the region described by BB_VINFO in AVX512
+ style, given that the region uses partial vectors. */
+
+static void
+vect_set_bb_slp_controls_partial_vectors_avx512 (bb_vec_info bb_vinfo)
+{
+ gcc_assert (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512);
+ gcc_assert (!bb_vinfo->masks.is_empty ());
+
+ gimple_seq seq = NULL;
+
+ /* Iterate over all the rgroups and fill in their controls. */
+ rgroup_controls *rgc;
+ unsigned int i;
+ auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
+ FOR_EACH_VEC_ELT (*controls, i, rgc)
+ {
+ if (rgc->controls.is_empty ())
+ continue;
+
+ tree ctrl_type = rgc->type;
+
+ /* Provide a definition of the one vector in the control group. */
+ gcc_assert (rgc->controls.length () == 1);
+ tree ctrl = rgc->controls[0];
+ tree vectype = rgc->compare_type;
+ tree nitems_cst = build_int_cst (TREE_TYPE (vectype), i + 1);
+
+ /* Build a vector, CMP_SERIES, to be compared with NITEMS_VEC. For
+ example, if CMP_SERIES is { 0, 1, 2, 3, ... } and NITEMS_VEC is
+ { 2, 2, 2, 2, ... } then the result of a less-than comparison is
+ { T, T, F, F, ... }, which enables the first two lanes. */
+ unsigned n = TYPE_VECTOR_SUBPARTS (ctrl_type).to_constant ();
+ tree_vector_builder builder (vectype, n, 1);
+ gcc_assert (rgc->max_nscalars_per_iter == 1);
+ for (unsigned j = 0; j < n; ++j)
+ {
+ builder.quick_push (build_int_cst (TREE_TYPE (vectype), j));
+ }
+ tree cmp_series = builder.build ();
+
+ tree nitems_vec
+ = gimple_build_vector_from_val (&seq, vectype, nitems_cst);
+ tree init_ctrl
+ = gimple_build (&seq, LT_EXPR, ctrl_type, cmp_series, nitems_vec);
+ gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
+ gimple_seq_add_stmt (&seq, stmt);
+ }
+
+ bb_vinfo->insert_seq_on_entry (NULL, seq);
+}
+
/* Set up the rgroup controls for the region described by BB_VINFO, if
the region uses partial vectors. */
static void
vect_set_bb_slp_controls (bb_vec_info bb_vinfo)
{
- gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
- vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
+ if (bb_vinfo->partial_vector_style == vect_partial_vectors_avx512)
+ vect_set_bb_slp_controls_partial_vectors_avx512 (bb_vinfo);
+ else
+ vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
}
/* Subroutine of vect_slp_bb. Try to vectorize the statements for all
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index dbde6d444f7..b1d69966841 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -711,7 +711,7 @@ public:
struct rgroup_controls {
/* The largest nS for all rgroups that use these controls.
For vect_partial_vectors_avx512 this is the constant nscalars_per_iter
- for all members of the group.
+ for all members of the group, or 1 for BB SLP vectorization.
For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
highest minimum number of subparts of all the vector types that use this
rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
@@ -720,7 +720,8 @@ struct rgroup_controls {
/* For the largest nS recorded above, the loop controls divide each scalar
into FACTOR equal-sized pieces. This is useful if we need to split
element-based accesses into byte-based accesses.
- For vect_partial_vectors_avx512 this records nV instead. */
+ For vect_partial_vectors_avx512 this records nV instead, or 1 for BB SLP
+ vectorization. */
unsigned int factor;
/* This is a vector type with MAX_NSCALARS_PER_ITER * VF / nV elements.
@@ -2767,6 +2768,7 @@ extern bool vect_can_vectorize_without_simd_p
(code_helper);
extern int vect_get_known_peeling_cost (loop_vec_info, int);
extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
extern bool vect_verify_full_masking (vec_info *);
+extern bool vect_verify_full_masking_avx512 (vec_info *);
/* Nonlinear induction. */
extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,
--
2.43.0
