This is part of enablement of support for partial vectors in basic
block SLP vectorization.
Adapt existing functions used by the loop vectorizer to record partial
vector masks during the analysis phase, verify that the recorded
requirements can be satisfied, and get SSA names that refer to those
masks for use during transformation.
A different scheme is used for recording masks for BB SLP vectorization
but it reuses as much loop vectorizer code as possible: instead of
recording the number of vectors to be masked in vect_record_mask and
later using that as the index to find the relevant rgroup controls in
vect_get_mask, rgroup controls are instead indexed by the number of
lanes in an SLP tree node, scaled by the ratio of the maximum number
of subparts in any vector type to the number of subparts in the
vector type to be masked. This allows a mask of 12 lanes for VNx16QI
to be reinterpreted as a mask of 3 lanes for VNx4SI, for example.
The main purpose of this reinterpretation is to avoid having to
introduce an extra dimension to storage of rgroup controls. Sharing of
mask constants typically has no effect on compiler output because the
'fre' pass eliminates the resultant VIEW_CONVERT_EXPR expressions.
Since the number of vectors is not recorded for BB SLP, it must always
be one. Consequently, the number of controls per rgroup_controls
instance is also one for BB SLP. (One mask for each unique number of
lanes.)
As part of analyzing whether an SLP region can be vectorized, after
analyzing all statements in the region, call vect_verify_full_masking
to check whether we can generate the necessary partial vector masks.
If not, return false from vect_slp_analyze_bb_1.
If vect_slp_analyze_bb_1 returns true (because vectorization of the
SLP region is possible) then vect_bb_vectorization_profitable_p is
called to estimate whether it would be profitable. New code in
that function estimates partial vector setup costs for an SLP region
based on its recorded requirements. This is imprecise because
masks can be shared between SLP nodes and we don't yet know which
subgraphs will be scheduled.
After scheduling any profitable subgraphs and generating any
invariant statements, a new function, vect_set_bb_slp_controls, is
called to insert any new stmts required to set up partial vector
masks on entry to the SLP vectorization region.
For now, only the WHILE_ULT style of partial vectors is supported
but there is scope to extend that to support other styles in future
commits.
A drawback of using inheritance from class vec_info to record partial
vector requirements for both BB SLP and loop vectorisation is that
stmts might be inserted to set up masks that are used by fewer SLP
subgraphs than expected, or even by none. Dead stmts are expected
to be eliminated by later passes.
gcc/ChangeLog:
* tree-vect-loop.cc (can_produce_all_loop_masks_p): Renamed as
can_produce_all_masks_p.
(can_produce_all_masks_p): Change parameter of type
loop_vec_info to vec_info *. Access the recorded masks
directly instead of via LOOP_VINFO_MASKS.
(vect_get_max_nscalars_per_iter): As above.
(vect_min_prec_for_max_group_size): New helper to find the
maximum group size in an SLP vectorization region by a linear
search through the vector of rgroup_controls.
(vect_verify_full_masking): Change parameter of type
loop_vec_info to vec_info *. Access the recorded masks directly
instead of via LOOP_VINFO_MASKS. When this function is used
for loop vectorization, the index into the vector of rgroup
controls is derived from the number of vectors, which is the
second value of each pair in mask_set; when used for BB SLP
vectorization, the rgroup index is instead derived
from the SLP group size, which is still the second value,
scaled by max_nunits / nunits.
For BB SLP, use the SLP group size as nscalars_per_iter, such
that the truth type of an rgroup is only replaced by the truth
type of a vectype for which a wider mask is needed. (If
previous and new mask requirements have the same rgroup index
and group size then nunits must also be equal.)
Call vect_min_prec_for_max_group_size to get an alternative
value of min_ni_width for BB SLP. Avoid calling
vect_iv_limit_for_partial_vectors for BB SLP because the concept
of an induction variable is not applicable. Instead, set
iv_precision to the calculated min_ni_width.
Set rgroup_compare_type, rgroup_iv_type and
partial_vector_style directly instead of via LOOP_VINFO_
macros.
(vect_analyze_loop_2): Reset max_nunits to zero at the same time
as emptying the mask_set.
(vect_record_mask): If used for BB SLP, get the group size
from the SLP tree node and use that as the second value of the
pair added to mask_set (instead of the nvectors argument).
Assert that nvectors is 1, if used for BB SLP. Add code
to increase the max_nunits of vec_masks if the passed-in
vectype is known to have a greater number of subparts.
(vect_get_mask): If used for BB SLP, get the group size
from the SLP tree node, scale that by max_nunits / nunits,
and use the product as the index of the rgroup control to
be queried (instead of using the nvectors argument). Don't
call make_temp_ssa_name with "loop_mask" as the name, if
used for BB SLP. Assert that nvectors is 1 and only the first
vector is queried, if used for BB SLP.
* tree-vect-slp.cc (vect_bb_vectorization_profitable_p):
Calculate the number of rgroup controls that need to be
produced for the whole SLP region. Divide that by the
number of SLP instances to get the average number of
rgroup controls per instance. Call record_stmt_cost to
assign a fair proportion of the vector_stmt costs in
vect_prologue to each instance. Spread remainder of the
division across the SLP instances until none is left.
(vect_slp_analyze_bb_1): If masks have been recorded for
an SLP region then call vect_verify_full_masking to
check whether we can generate the necessary controls.
If not, return false.
(vect_set_bb_slp_controls_directly): New helper.
Assert that there is only one rgroup control (because
vect_get_mask creates only one control per recorded mask
for BB SLP, instead of one per vector for loop vectorization).
Call build_int_cst to build constant bounds for a WHILE_ULT
statement. Call vect_gen_while to generate and return a mask
of the truth type and number of lanes chosen by
vect_verify_full_masking. Call gimple_build_assign to build
an assignment of the vector mask to the temporary SSA name
made by vect_get_mask. Call gimple_seq_add_stmt to add the
assignment to the gimple_seq passed by the caller.
(vect_set_bb_slp_controls_partial_vectors): Assert that the
partial vectors style is WHILE_ULT and that some masks were
recorded. Iterate over all the rgroups and call
vect_set_bb_slp_controls_directly to set up each control,
passing the address of a gimple_seq to populate. Lastly,
use the insert_seq_on_entry method of the bb_vec_info to
insert the stmts contained in that gimple_seq at entry to
the SLP vectorization region.
(vect_set_bb_slp_controls): Assert that the partial vectors
style is WHILE_ULT and call
vect_set_bb_slp_controls_partial_vectors.
(vect_slp_region): If a partial vectors style has been set
by vect_verify_full_masking, call vect_set_bb_slp_controls.
* tree-vectorizer.h (rgroup_controls): Update the description
of max_nscalars_per_iter.
(vec_masks): Add max_nunits as a member of vec_masks, with a
default initial value of 0. This is used to track the
maximum number of subparts of all the vector types recorded
in mask_set.
(vect_verify_full_masking): Declare function for use by
vect_slp_analyze_bb_1. This function no longer has static
linkage.
---
gcc/tree-vect-loop.cc | 190 ++++++++++++++++++++++++++++++++----------
gcc/tree-vect-slp.cc | 110 ++++++++++++++++++++++++
gcc/tree-vectorizer.h | 11 ++-
3 files changed, 267 insertions(+), 44 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index ec733d01e77..e0bdc262439 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -891,14 +891,14 @@ cse_and_gimplify_to_preheader (loop_vec_info loop_vinfo,
tree expr)
}
/* Return true if we can use CMP_TYPE as the comparison type to produce
- all masks required to mask LOOP_VINFO. */
+ all masks required to mask VINFO. */
static bool
-can_produce_all_loop_masks_p (loop_vec_info loop_vinfo, tree cmp_type)
+can_produce_all_masks_p (vec_info *vinfo, tree cmp_type)
{
rgroup_controls *rgm;
unsigned int i;
- FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo).rgc_vec, i, rgm)
+ FOR_EACH_VEC_ELT (vinfo->masks.rgc_vec, i, rgm)
if (rgm->type != NULL_TREE
&& !direct_internal_fn_supported_p (IFN_WHILE_ULT,
cmp_type, rgm->type,
@@ -908,15 +908,15 @@ can_produce_all_loop_masks_p (loop_vec_info loop_vinfo,
tree cmp_type)
}
/* Calculate the maximum number of scalars per iteration for every
- rgroup in LOOP_VINFO. */
+ rgroup in VINFO. */
static unsigned int
-vect_get_max_nscalars_per_iter (loop_vec_info loop_vinfo)
+vect_get_max_nscalars_per_iter (vec_info *vinfo)
{
unsigned int res = 1;
unsigned int i;
rgroup_controls *rgm;
- FOR_EACH_VEC_ELT (LOOP_VINFO_MASKS (loop_vinfo).rgc_vec, i, rgm)
+ FOR_EACH_VEC_ELT (vinfo->masks.rgc_vec, i, rgm)
res = MAX (res, rgm->max_nscalars_per_iter);
return res;
}
@@ -989,37 +989,91 @@ vect_need_peeling_or_partial_vectors_p (loop_vec_info
loop_vinfo)
return true;
}
-/* Each statement in LOOP_VINFO can be masked where necessary. Check
- whether we can actually generate the masks required. Return true if so,
- storing the type of the scalar IV in LOOP_VINFO_RGROUP_COMPARE_TYPE. */
+/* Given some rgroup CONTROLS used for basic block SLP vectorization,
+ return the maximum SLP group size from the recorded partial vector
+ requirements. */
+static unsigned int
+vect_min_prec_for_max_group_size (vec<rgroup_controls> *controls)
+{
+ unsigned int i, max_group_size = 1;
+ rgroup_controls *rgc;
-static bool
-vect_verify_full_masking (loop_vec_info loop_vinfo)
+ FOR_EACH_VEC_ELT (*controls, i, rgc)
+ {
+ if (!rgc->type)
+ continue;
+ gcc_assert (rgc->factor == 1);
+ unsigned int group_size = i + 1;
+ max_group_size = MAX (max_group_size, group_size);
+ }
+
+ return wi::min_precision (max_group_size, UNSIGNED);
+}
+
+/* Each statement in a loop or basic block SLP vectorization region represented
+ by VINFO can be masked where necessary. Check whether we can actually
+ generate the masks required. Return true if so, storing the type of the
+ scalar IV and the type that IV should be converted to for comparison
purposes
+ in VINFO. For BB SLP vectorization, both types are the same and are used to
+ build constant bounds for WHILE_ULT stmts. */
+bool
+vect_verify_full_masking (vec_info *vinfo)
{
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ bb_vec_info bb_vinfo = dyn_cast<bb_vec_info> (vinfo);
unsigned int min_ni_width;
/* Use a normal loop if there are no statements that need masking.
This only happens in rare degenerate cases: it means that the loop
has no loads, no stores, and no live-out values. */
- if (LOOP_VINFO_MASKS (loop_vinfo).is_empty ())
+ if (vinfo->masks.is_empty ())
return false;
/* Produce the rgroup controls. */
- for (auto mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
+ for (auto mask : vinfo->masks.mask_set)
{
- vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ vec_masks *masks = &vinfo->masks;
tree vectype = mask.first;
- unsigned nvectors = mask.second;
+ unsigned int nvectors_or_group_size = mask.second;
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
- if (masks->rgc_vec.length () < nvectors)
- masks->rgc_vec.safe_grow_cleared (nvectors, true);
- rgroup_controls *rgm = &(*masks).rgc_vec[nvectors - 1];
/* The number of scalars per iteration and the number of vectors are
both compile-time constants. */
- unsigned int nscalars_per_iter
- = exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ unsigned int nscalars_per_iter, key;
+ if (loop_vinfo)
+ {
+ unsigned int nvectors = nvectors_or_group_size;
+
+ /* With VNx16QI, the rgroup operates on nV * 16 / VF values from
+ each scalar iteration; with VNx4SI, it instead operates on
+ nV * 4 / VF values. Calculate a value of nS suitable for both
+ cases. */
+ nscalars_per_iter
+ = exact_div (nvectors * nunits, vinfo->vectorization_factor)
+ .to_constant ();
+ key = nvectors;
+ }
+ else
+ {
+ unsigned int group_size = nvectors_or_group_size;
+
+ /* With VNx16QI, the rgroup operates on GS1 * max(4,16) / 16 values;
+ with VNx4SI, it instead operates on GS2 * max(4,16) / 4 values.
+ Use the same mask for both cases (e.g., key is 12 for VNx16QI with
+ group_size 12, but also for VNx4SI with group_size 3). */
+ key = exact_div (group_size * vinfo->masks.max_nunits, nunits)
+ .to_constant ();
+ nscalars_per_iter = group_size;
+ }
+
+ if (masks->rgc_vec.length () < key)
+ masks->rgc_vec.safe_grow_cleared (key, true);
+ rgroup_controls *rgm = &(*masks).rgc_vec[key - 1];
+ /* A mask produced for nS values with a vector type that has nL lanes can
+ be reinterpreted as a mask for nS/2 values with a vector type that has
+ nL/2 lanes, so plan to build a mask of a type suitable for the vector
+ type that is used to operate on the highest nS. */
if (rgm->max_nscalars_per_iter < nscalars_per_iter)
{
rgm->max_nscalars_per_iter = nscalars_per_iter;
@@ -1028,23 +1082,31 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
}
}
- unsigned int max_nscalars_per_iter
- = vect_get_max_nscalars_per_iter (loop_vinfo);
+ unsigned int max_nscalars_per_iter = vect_get_max_nscalars_per_iter (vinfo);
/* Work out how many bits we need to represent the limit. */
- min_ni_width
- = vect_min_prec_for_max_niters (loop_vinfo, max_nscalars_per_iter);
+ if (loop_vinfo)
+ min_ni_width
+ = vect_min_prec_for_max_niters (loop_vinfo, max_nscalars_per_iter);
+ else
+ min_ni_width = vect_min_prec_for_max_group_size (&bb_vinfo->masks.rgc_vec);
/* Find a scalar mode for which WHILE_ULT is supported. */
opt_scalar_int_mode cmp_mode_iter;
tree cmp_type = NULL_TREE;
tree iv_type = NULL_TREE;
- widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
- unsigned int iv_precision = UINT_MAX;
+ unsigned int iv_precision;
+ if (loop_vinfo)
+ {
+ widest_int iv_limit = vect_iv_limit_for_partial_vectors (loop_vinfo);
+ iv_precision = UINT_MAX;
- if (iv_limit != -1)
- iv_precision = wi::min_precision (iv_limit * max_nscalars_per_iter,
- UNSIGNED);
+ if (iv_limit != -1)
+ iv_precision
+ = wi::min_precision (iv_limit * max_nscalars_per_iter, UNSIGNED);
+ }
+ else
+ iv_precision = min_ni_width;
FOR_EACH_MODE_IN_CLASS (cmp_mode_iter, MODE_INT)
{
@@ -1054,7 +1116,7 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
{
tree this_type = build_nonstandard_integer_type (cmp_bits, true);
if (this_type
- && can_produce_all_loop_masks_p (loop_vinfo, this_type))
+ && can_produce_all_masks_p (vinfo, this_type))
{
/* Although we could stop as soon as we find a valid mode,
there are at least two reasons why that's not always the
@@ -1091,13 +1153,13 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
if (!cmp_type)
{
- LOOP_VINFO_MASKS (loop_vinfo).rgc_vec.release ();
+ vinfo->masks.rgc_vec.release ();
return false;
}
- LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo) = cmp_type;
- LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo) = iv_type;
- LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo) =
vect_partial_vectors_while_ult;
+ vinfo->rgroup_compare_type = cmp_type;
+ vinfo->rgroup_iv_type = iv_type;
+ vinfo->partial_vector_style = vect_partial_vectors_while_ult;
return true;
}
@@ -2723,6 +2785,7 @@ again:
loop_vinfo->vector_costs = nullptr;
/* Reset accumulated rgroup information. */
LOOP_VINFO_MASKS (loop_vinfo).mask_set.empty ();
+ LOOP_VINFO_MASKS (loop_vinfo).max_nunits = 0;
release_vec_loop_controls (&LOOP_VINFO_MASKS (loop_vinfo).rgc_vec);
release_vec_loop_controls (&LOOP_VINFO_LENS (loop_vinfo));
/* Reset assorted flags. */
@@ -10583,14 +10646,31 @@ vect_record_mask (vec_info *vinfo, unsigned int
nvectors,
gcc_assert (nvectors != 0);
vec_masks *masks = &vinfo->masks;
loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ unsigned int nvectors_or_group_size;
- if (loop_vinfo && scalar_mask)
+ if (loop_vinfo)
{
- scalar_cond_masked_key cond (scalar_mask, nvectors);
- loop_vinfo->scalar_cond_masked_set.add (cond);
+ nvectors_or_group_size = nvectors;
+ if (scalar_mask)
+ {
+ scalar_cond_masked_key cond (scalar_mask, nvectors);
+ loop_vinfo->scalar_cond_masked_set.add (cond);
+ }
+ }
+ else
+ {
+ gcc_assert (nvectors == 1);
+ nvectors_or_group_size = SLP_TREE_LANES (slp_node);
+
+ /* Keep track of the highest number of subparts, so we can scale the SLP
+ group size by that value to determine when mask requirements for
+ different group sizes can be satisfied by a common mask. */
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
+ if (known_gt (nunits, masks->max_nunits))
+ masks->max_nunits = nunits;
}
- masks->mask_set.add (std::make_pair (vectype, nvectors));
+ masks->mask_set.add (std::make_pair (vectype, nvectors_or_group_size));
}
/* Given a complete set of masks for the loop or SLP region represented by
@@ -10608,11 +10688,31 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator
*gsi,
slp_tree slp_node)
{
gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
- vec_masks *masks = &LOOP_VINFO_MASKS (vinfo);
+ gcc_assert (nvectors != 0);
+ gcc_assert (index < nvectors);
+
+ vec_masks *masks = &vinfo->masks;
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
if (vinfo->partial_vector_style == vect_partial_vectors_while_ult)
{
- rgroup_controls *rgm = &(masks->rgc_vec)[nvectors - 1];
+ unsigned int key;
+
+ if (loop_vinfo)
+ key = nvectors;
+ else
+ {
+ gcc_assert (nvectors == 1);
+ unsigned int group_size = SLP_TREE_LANES (slp_node);
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
+ unsigned int nscalars_per_iter
+ = exact_div (group_size * vinfo->masks.max_nunits, nunits)
+ .to_constant ();
+ key = nscalars_per_iter;
+ }
+
+ gcc_assert (key != 0);
+ rgroup_controls *rgm = &(masks->rgc_vec)[key - 1];
tree mask_type = rgm->type;
/* Populate the rgroup's mask array, if this is the first time we've
@@ -10622,7 +10722,9 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator
*gsi,
rgm->controls.safe_grow_cleared (nvectors, true);
for (unsigned int i = 0; i < nvectors; ++i)
{
- tree mask = make_temp_ssa_name (mask_type, NULL, "loop_mask");
+ tree mask
+ = make_temp_ssa_name (mask_type, NULL,
+ loop_vinfo ? "loop_mask" : "slp_mask");
/* Provide a dummy definition until the real one is available. */
SSA_NAME_DEF_STMT (mask) = gimple_build_nop ();
rgm->controls[i] = mask;
@@ -10673,7 +10775,9 @@ vect_get_mask (vec_info *vinfo, gimple_stmt_iterator
*gsi,
rgm->controls.safe_grow_cleared (nvectors, true);
for (unsigned int i = 0; i < nvectors; ++i)
{
- tree mask = make_temp_ssa_name (rgm->type, NULL, "loop_mask");
+ tree mask
+ = make_temp_ssa_name (rgm->type, NULL,
+ loop_vinfo ? "loop_mask" : "slp_mask");
/* Provide a dummy definition until the real one is available. */
SSA_NAME_DEF_STMT (mask) = gimple_build_nop ();
rgm->controls[i] = mask;
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 67cdc95433b..5348b44f64a 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9660,6 +9660,29 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
SLP_INSTANCE_TREE (instance), visited);
}
+ /* Costing each partial vector would be excessive for many SLP instances,
+ because it is common to materialise identical masks/lengths for related
+ operations. This alternative allows costs to be shared between SLP
+ subgraphs, but does not take into account any further savings that might
be
+ discovered during lowering. */
+ int control_count = 0;
+ enum vect_cost_for_stmt control_kind = vector_stmt;
+
+ if (!bb_vinfo->masks.is_empty ())
+ {
+ gcc_assert (bb_vinfo->partial_vector_style
+ == vect_partial_vectors_while_ult);
+ for (auto rgc : bb_vinfo->masks.rgc_vec)
+ if (rgc.type)
+ control_count++;
+ }
+
+ gcc_assert (bb_vinfo->lens.is_empty ());
+
+ const int slp_instance_count = slp_instances.length (),
+ control_cost_per_inst_q = control_count / slp_instance_count;
+ int control_cost_per_inst_r = control_count % slp_instance_count;
+
/* Then DFS walk scalar stmts, performing costing and handling
still live scalar stmts via the previously computed vector coverage. */
stmt_vector_for_cost scalar_costs = vNULL;
@@ -9681,6 +9704,20 @@ vect_bb_vectorization_profitable_p (bb_vec_info bb_vinfo,
worklist.safe_push (stmt);
}
vect_bb_slp_scalar_cost (bb_vinfo, worklist, &scalar_costs, svisited);
+
+ /* We cannot ascribe shared costs of partial setup to any one instance,
+ but we can try to split them fairly between instances. */
+ int stmt_cost = control_cost_per_inst_q;
+ if (control_cost_per_inst_r)
+ {
+ control_cost_per_inst_r--;
+ stmt_cost++;
+ }
+
+ if (stmt_cost)
+ record_stmt_cost (&instance->cost_vec, stmt_cost, control_kind, nullptr,
+ nullptr, NULL_TREE, 0, vect_prologue);
+
vector_costs.safe_splice (instance->cost_vec);
instance->cost_vec.release ();
}
@@ -10242,6 +10279,14 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
return false;
}
+ /* Check whether we can generate the necessary controls. */
+ gcc_assert (bb_vinfo->lens.is_empty ());
+ if (!bb_vinfo->masks.is_empty ())
+ {
+ if (!vect_verify_full_masking (bb_vinfo))
+ return false;
+ }
+
/* Mark all the statements that we vectorize. */
vect_bb_slp_mark_stmts_vectorized (bb_vinfo);
@@ -10253,6 +10298,68 @@ vect_slp_analyze_bb_1 (bb_vec_info bb_vinfo, int
n_stmts, bool &fatal,
return true;
}
+/* Helper for vect_set_bb_slp_controls_partial_vectors. Generate definitions
+ for all the rgroup controls in RGC. */
+
+static void
+vect_set_bb_slp_controls_directly (bb_vec_info bb_vinfo, gimple_seq *seq,
+ rgroup_controls *rgc)
+{
+ tree compare_type = bb_vinfo->rgroup_compare_type;
+
+ tree ctrl_type = rgc->type;
+ gcc_assert (rgc->factor == 1);
+ unsigned int nitems = rgc->max_nscalars_per_iter;
+ tree zero_index = build_int_cst (compare_type, 0);
+ tree nitems_cst = build_int_cst (compare_type, nitems);
+
+ gcc_assert (rgc->controls.length () == 1);
+ tree ctrl = rgc->controls[0];
+ tree init_ctrl;
+
+ gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
+ init_ctrl
+ = vect_gen_while (seq, ctrl_type, zero_index, nitems_cst, "max_mask");
+
+ gimple *stmt = gimple_build_assign (ctrl, init_ctrl);
+ gimple_seq_add_stmt (seq, stmt);
+}
+
+/* Set up the rgroup controls for the region described by BB_VINFO, given that
+ the region uses partial vectors. */
+
+static void
+vect_set_bb_slp_controls_partial_vectors (bb_vec_info bb_vinfo)
+{
+ gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
+ gcc_assert (!bb_vinfo->masks.is_empty ());
+
+ gimple_seq seq = NULL;
+
+ /* Iterate over all the rgroups and fill in their controls. */
+ rgroup_controls *rgc;
+ unsigned int i;
+ auto_vec<rgroup_controls> *controls = &bb_vinfo->masks.rgc_vec;
+ FOR_EACH_VEC_ELT (*controls, i, rgc)
+ if (!rgc->controls.is_empty ())
+ {
+ /* Set up the control for this group. */
+ vect_set_bb_slp_controls_directly (bb_vinfo, &seq, rgc);
+ }
+
+ bb_vinfo->insert_seq_on_entry (NULL, seq);
+}
+
+/* Set up the rgroup controls for the region described by BB_VINFO, if
+ the region uses partial vectors. */
+
+static void
+vect_set_bb_slp_controls (bb_vec_info bb_vinfo)
+{
+ gcc_assert (bb_vinfo->partial_vector_style ==
vect_partial_vectors_while_ult);
+ vect_set_bb_slp_controls_partial_vectors (bb_vinfo);
+}
+
/* Subroutine of vect_slp_bb. Try to vectorize the statements for all
basic blocks in BBS, returning true on success.
The region has N_STMTS statements and has the datarefs given by DATAREFS.
*/
@@ -10415,6 +10522,9 @@ vect_slp_region (vec<basic_block> bbs,
vec<data_reference_p> datarefs,
bb_vinfo->insert_seq_on_entry (NULL,
bb_vinfo->inv_pattern_def_seq);
}
+
+ if (bb_vinfo->partial_vector_style != vect_partial_vectors_none)
+ vect_set_bb_slp_controls (bb_vinfo);
}
else
{
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index f0adaccb03f..6f233e9045b 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -705,7 +705,10 @@ public:
struct rgroup_controls {
/* The largest nS for all rgroups that use these controls.
For vect_partial_vectors_avx512 this is the constant nscalars_per_iter
- for all members of the group. */
+ for all members of the group.
+ For BB SLP vectorization with vect_partial_vectors_while_ult, this is the
+ highest minimum number of subparts of all the vector types that use this
+ rgroup (e.g., 16 from VNx4HI, VNx4SI, VNx4HF and VNx16QI). */
unsigned int max_nscalars_per_iter;
/* For the largest nS recorded above, the loop controls divide each scalar
@@ -742,6 +745,11 @@ struct vec_masks
/* rgroup_controls used for the partial vector scheme. */
auto_vec<rgroup_controls> rgc_vec;
+
+ /* For BB SLP vectorization, the maximum number of subparts of all of the
+ vector types recorded in mask_set. Used to scale SLP group sizes to
+ allow a compatible mask to be located in rgc_vec. */
+ poly_uint64 max_nunits = 0;
};
typedef auto_vec<rgroup_controls> vec_lens;
@@ -2769,6 +2777,7 @@ extern bool vect_can_vectorize_without_simd_p (tree_code);
extern bool vect_can_vectorize_without_simd_p (code_helper);
extern int vect_get_known_peeling_cost (loop_vec_info, int);
extern tree cse_and_gimplify_to_preheader (loop_vec_info, tree);
+extern bool vect_verify_full_masking (vec_info *);
/* Nonlinear induction. */
extern tree vect_peel_nonlinear_iv_init (gimple_seq*, tree, tree,
--
2.43.0
