Modify the interface used by the vectorizer to record and get
masks, in preparation for extending use of that interface to
basic block SLP vectorization (as well as keeping its existing
use as part of loop vectorization). Add an optional parameter
of type slp_tree. At the moment, this is unused; in
future, it will be used to get the number of active lanes from
an SLP node in a more type-safe way than passing an int.
Rename the types that store sets of rgroup controls used in the
partial vector scheme, to the less loop-specific names vec_masks
and vec_lens.
Move certain members of _loop_vec_info to its superclass,
vec_info, in anticipation of those members being reused for
BB SLP vectorization too.
Notably, using_partial_vectors_p, must_use_partial_vectors_p
and can_use_partial_vectors_p are *not* expected to be reused
for BB SLP because it does not make sense to maintain that
state for a whole SLP region. Decisions about use of partial
vectors will be made immediately on a per-statement basis during
analysis of the SLP tree instead of being deferred so that
vectorization of a loop can be determined holistically. It
follows that any existing code using the function-like macros
LOOP_VINFO_USING_PARTIAL_VECTORS_P,
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P,
LOOP_VINFO_MUST_USE_PARTIAL_VECTORS_P, LOOP_VINFO_FULLY_MASKED_P,
and LOOP_VINFO_FULLY_WITH_LENGTH_P will need to be rewritten for
BB SLP.
This is intended to be a pure refactoring and to remain
minimal (e.g., vec_info and slp_tree pointers are not passed
into vect_get_len whenever such arguments are available).
gcc/ChangeLog:
* tree-vect-loop.cc (_loop_vec_info::_loop_vec_info):
Delete initializers for vectorization_factor,
rgroup_compare_type, partial_vector_style and
partial_load_store_bias.
(vect_verify_full_masking_avx512): Update comment.
(vect_estimate_min_profitable_iters): As above.
(vectorize_fold_left_reduction): Delete the parameters
that point to vec_loop_masks and vec_loop_lens and
don't pass those arguments to vect_get_mask or vect_get_len.
(vect_reduction_update_partial_vector_usage): Don't pass a
pointer to vec_loop_masks or vec_loop_lens when calling
vect_record_mask or vect_record_len.
(vect_transform_reduction): Don't pass a
pointer to vec_loop_masks or vec_loop_lens when calling
vect_get_mask, vect_get_len or vectorize_fold_left_reduction.
(vectorizable_live_operation_1): Don't pass a
pointer to vec_loop_masks or vec_loop_lens when calling
vect_get_mask or vect_get_len.
(vectorizable_live_operation): Don't pass a
pointer to vec_loop_masks or vec_loop_lens when calling
vect_record_mask or vect_record_len.
(vect_record_loop_mask): Rename as vect_record_mask.
(vect_record_mask): Change the type of the first parameter from
loop_vec_info to a pointer to vec_info.
Get vec_masks from vec_info instead of requiring an extra
argument for vec_loop_masks (old type name).
Only add a mask to the scalar_cond_masked_set if called with a
loop_vec_info.
Add a (currently unused) parameter of type slp_tree.
(vect_get_loop_mask): Rename as vect_get_mask.
(vect_get_mask): Change the type of the first parameter from
loop_vec_info to a pointer to vec_info.
Get vec_masks from vec_info instead of requiring an extra
argument for vec_loop_masks (old type name).
Access members of the vec_info directly instead of via
LOOP_VINFO_ accessor macros.
Add a (currently unused) parameter of type slp_tree.
(vect_record_loop_len): Rename as vect_record_len.
(vect_record_len): Change the type of the first parameter from
loop_vec_info to a pointer to vec_info.
Get vec_lens from vec_info instead of requiring an extra
argument for vec_loop_lens (old name).
Access members of the vec_info directly instead of via
LOOP_VINFO_ accessor macros.
Add a (currently unused) parameter of type slp_tree.
(vect_get_loop_len): Rename as vect_get_len.
(vect_get_len): Change the type of the first parameter from
loop_vec_info to a pointer to vec_info.
Get vec_lens from vec_info instead of requiring an extra
argument for vec_loop_lens (old name).
Access members of the vec_info directly instead of via
LOOP_VINFO_ accessor macros.
Add a (currently unused) parameter of type slp_tree.
(vect_gen_loop_len_mask): Delete the parameter that points to
vec_loop_lens (old name) and don't pass vec_lens to
vect_get_len.
* tree-vect-stmts.cc (check_load_store_for_partial_vectors):
Don't pass a pointer to vec_loop_masks or vec_loop_lens when
calling vect_record_mask or vect_record_len.
(prepare_vec_mask): Change the type of the first parameter from
loop_vec_info to a pointer to vec_info and use it when querying
vec_cond_masked_set.
Rename the loop_mask parameter as loop_or_tail_mask.
(vectorizable_call): Don't pass a pointer to vec_loop_masks or
vec_loop_lens when calling vect_record_mask, vect_record_len,
vect_get_mask or vect_get_len.
(vectorizable_simd_clone_call): As above.
(vectorizable_operation): As above.
Replace uses of local vec_loop_masks and vec_loop_lens pointers
for flow control with Boolean local variables that can
be true for BB SLP if using partial vectors.
(vectorizable_store): Replace uses of local vec_loop_masks and
vec_loop_lens pointers for flow control with Boolean local
variables that can be true for BB SLP if using partial vectors.
Stop passing a vec_loop_masks or vec_loop_lens pointer as an
explicit argument to vect_get_loop_mask, vect_get_loop_len
(old function names).
Replace invocation of LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS with
direct access to partial_load_store_bias member of vec_info.
Pass a pointer to vec_info to vect_get_len, vect_get_mask and
prepare_vec_mask instead of passing a loop_vec_info.
(vectorizable_load): As above.
(vectorizable_condition): Stop passing a vec_loop_masks or
vec_loop_lens pointer as an explicit argument to
vect_record_loop_len, vect_record_loop_mask, vect_get_loop_len,
or vect_get_loop_mask (old function names).
Replace uses of local vec_loop_masks and vec_loop_lens pointers
for flow control with Boolean local variables that can
be true for BB SLP if using partial vectors.
(vectorizable_early_exit): Stop passing a vec_loop_masks or
vec_loop_lens pointer as an explicit argument to
vect_record_loop_len, vect_record_loop_mask,
vect_get_loop_mask, prepare_vec_mask and
vect_gen_loop_len_mask.
* tree-vectorizer.cc (vec_info::vec_info):
Add initializers for vectorization_factor,
rgroup_compare_type, rgroup_iv_type,
partial_vector_style, partial_load_store_bias.
* tree-vectorizer.h (struct scalar_cond_masked_key):
Update reference to vect_record_loop_mask.
(class vec_info): Move members vectorization_factor,
masks, lens, vec_cond_masked_set, rgroup_compare_type,
rgroup_iv_type, partial_vector_style and
partial_load_store_bias from _loop_vec_info to its
superclass.
(class _loop_vec_info): As above.
(struct vec_loop_masks): Rename as vec_masks.
(vec_loop_lens): Rename as vec_lens.
(vect_record_loop_mask): Rename as vect_record_mask.
(vect_get_loop_mask): Rename as vect_get_mask.
(vect_record_loop_len): Rename as vect_record_len.
(vect_get_loop_len): Rename as vect_get_len.
(vect_record_mask): New name of vect_record_loop_mask.
Replace first parameter of type loop_vec_info with a
pointer to vec_info.
Don't require the caller to pass a pointer to vec_loop_masks.
Add an optional parameter of type slp_tree.
(vect_get_mask): New name of vect_get_loop_mask.
Replace first parameter of type loop_vec_info with a pointer
to vec_info.
Don't require the caller to pass a pointer to vec_loop_masks.
Add an optional parameter of type slp_tree.
(vect_record_len): New name of vect_record_loop_len.
Replace first parameter of type loop_vec_info with a pointer
to vec_info.
Don't require the caller to pass a pointer to vec_loop_lens.
Add an optional parameter of type slp_tree.
(vect_get_len): New name of vect_get_loop_len.
Replace first parameter of type loop_vec_info with a pointer
to vec_info.
Don't require the caller to pass a pointer to vec_loop_lens.
Add an optional parameter of type slp_tree.
(vect_gen_loop_len_mask): Don't require the caller to pass a
pointer to vec_loop_lens.
(prepare_vec_mask): Change the type of the first parameter from
loop_vec_info into a pointer to vec_info.
---
gcc/tree-vect-loop.cc | 175 ++++++++++++------------
gcc/tree-vect-stmts.cc | 290 ++++++++++++++++-----------------------
gcc/tree-vectorizer.cc | 7 +-
gcc/tree-vectorizer.h | 302 +++++++++++++++++++++--------------------
4 files changed, 363 insertions(+), 411 deletions(-)
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index a9335ed68bb..728c59cf25d 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -732,7 +732,6 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared)
scalar_costs (nullptr),
th (0),
versioning_threshold (0),
- vectorization_factor (0),
main_loop_edge (nullptr),
skip_main_loop_edge (nullptr),
skip_this_loop_edge (nullptr),
@@ -741,9 +740,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared)
max_vectorization_factor (0),
mask_skip_niters (NULL_TREE),
mask_skip_niters_pfa_offset (NULL_TREE),
- rgroup_compare_type (NULL_TREE),
simd_if_cond (NULL_TREE),
- partial_vector_style (vect_partial_vectors_none),
unaligned_dr (NULL),
peeling_for_alignment (0),
ptr_mask (0),
@@ -759,7 +756,6 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared)
using_decrementing_iv_p (false),
using_select_vl_p (false),
allow_mutual_alignment (false),
- partial_load_store_bias (0),
peeling_for_gaps (false),
peeling_for_niter (false),
early_breaks (false),
@@ -1011,7 +1007,7 @@ vect_verify_full_masking (loop_vec_info loop_vinfo)
/* Produce the rgroup controls. */
for (auto mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
{
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
tree vectype = mask.first;
unsigned nvectors = mask.second;
@@ -1150,7 +1146,7 @@ vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
/* Produce the rgroup controls. */
for (auto const &mask : LOOP_VINFO_MASKS (loop_vinfo).mask_set)
{
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ vec_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
tree vectype = mask.first;
unsigned nvectors = mask.second;
@@ -1189,7 +1185,7 @@ vect_verify_full_masking_avx512 (loop_vec_info loop_vinfo)
if (!mask_type)
continue;
- /* For now vect_get_loop_mask only supports integer mode masks
+ /* For now vect_get_mask only supports integer mode masks
when we need to split it. */
if (GET_MODE_CLASS (TYPE_MODE (mask_type)) != MODE_INT
|| TYPE_PRECISION (TREE_TYPE (mask_type)) != 1)
@@ -4207,7 +4203,7 @@ vect_estimate_min_profitable_iters (loop_vec_info
loop_vinfo,
out of mask registers for example. */
/* ??? We are also failing to account for smaller vector masks
- we generate by splitting larger masks in vect_get_loop_mask. */
+ we generate by splitting larger masks in vect_get_mask. */
/* In the worst case, we need to generate each mask in the prologue
and in the loop body. We need one splat per group and one
@@ -6414,12 +6410,9 @@ get_masked_reduction_fn (internal_fn reduc_fn, tree
vectype_in)
static bool
vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
- gimple_stmt_iterator *gsi,
- slp_tree slp_node,
+ gimple_stmt_iterator *gsi, slp_tree slp_node,
code_helper code, internal_fn reduc_fn,
- int num_ops, tree vectype_in,
- int reduc_index, vec_loop_masks *masks,
- vec_loop_lens *lens)
+ int num_ops, tree vectype_in, int reduc_index)
{
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
tree vectype_out = SLP_TREE_VECTYPE (slp_node);
@@ -6500,8 +6493,8 @@ vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
tree bias = NULL_TREE;
if (LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
{
- tree loop_mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_in, i);
+ tree loop_mask
+ = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in, i);
if (is_cond_op)
mask = prepare_vec_mask (loop_vinfo, TREE_TYPE (loop_mask),
loop_mask, vec_opmask[i], gsi);
@@ -6512,8 +6505,8 @@ vectorize_fold_left_reduction (loop_vec_info loop_vinfo,
mask = vec_opmask[i];
if (LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))
{
- len = vect_get_loop_len (loop_vinfo, gsi, lens, vec_num, vectype_in,
- i, 1, false);
+ len
+ = vect_get_len (loop_vinfo, gsi, vec_num, vectype_in, i, 1, false);
signed char biasval = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
bias = build_int_cst (intQI_type_node, biasval);
if (!is_cond_op)
@@ -6753,14 +6746,12 @@ vect_reduction_update_partial_vector_usage
(loop_vec_info loop_vinfo,
{
internal_fn mask_reduc_fn
= get_masked_reduction_fn (reduc_fn, vectype_in);
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
- vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
unsigned nvectors = vect_get_num_copies (loop_vinfo, slp_node);
if (mask_reduc_fn == IFN_MASK_LEN_FOLD_LEFT_PLUS)
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_in, 1);
+ vect_record_len (loop_vinfo, nvectors, vectype_in, 1);
else
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_in, NULL);
+ vect_record_mask (loop_vinfo, nvectors, vectype_in, NULL_TREE);
}
}
@@ -7978,8 +7969,6 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
&& internal_fn_mask_index ((internal_fn)code) != -1)
? (internal_fn)code : get_conditional_internal_fn (code, op.type));
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
- vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
bool mask_by_cond_expr = use_mask_by_cond_expr_p (code, cond_fn, vectype_in);
/* Transform. */
@@ -8011,10 +8000,10 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
{
internal_fn reduc_fn = VECT_REDUC_INFO_FN (reduc_info);
gcc_assert (code.is_tree_code () || cond_fn_p);
- return vectorize_fold_left_reduction
- (loop_vinfo, stmt_info, gsi, slp_node,
- code, reduc_fn, op.num_ops, vectype_in,
- reduc_index, masks, lens);
+ return vectorize_fold_left_reduction (loop_vinfo, stmt_info, gsi,
+ slp_node, code, reduc_fn,
+ op.num_ops, vectype_in,
+ reduc_index);
}
bool single_defuse_cycle = VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info);
@@ -8224,9 +8213,8 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
yet. */
gcc_assert (!lane_reducing);
- tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_in,
- mask_index++);
+ tree mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in,
+ mask_index++);
gcall *call;
if (code.is_internal_fn () && cond_fn_p)
{
@@ -8254,9 +8242,8 @@ vect_transform_reduction (loop_vec_info loop_vinfo,
if (masked_loop_p && mask_by_cond_expr)
{
- tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_in,
- mask_index++);
+ tree mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype_in,
+ mask_index++);
build_vect_cond_expr (code, vop, mask, gsi);
}
@@ -10117,9 +10104,7 @@ vectorizable_live_operation_1 (loop_vec_info
loop_vinfo, basic_block exit_bb,
gcc_assert (SLP_TREE_LANES (slp_node) == 1);
gimple_seq tem = NULL;
gimple_stmt_iterator gsi = gsi_last (tem);
- tree len = vect_get_loop_len (loop_vinfo, &gsi,
- &LOOP_VINFO_LENS (loop_vinfo),
- 1, vectype, 0, 1, false);
+ tree len = vect_get_len (loop_vinfo, &gsi, 1, vectype, 0, 1, false);
gimple_seq_add_seq (&stmts, tem);
/* LAST_INDEX = LEN - 1. */
@@ -10146,9 +10131,7 @@ vectorizable_live_operation_1 (loop_vec_info
loop_vinfo, basic_block exit_bb,
tree scalar_type = TREE_TYPE (vectype);
gimple_seq tem = NULL;
gimple_stmt_iterator gsi = gsi_last (tem);
- tree mask = vect_get_loop_mask (loop_vinfo, &gsi,
- &LOOP_VINFO_MASKS (loop_vinfo),
- 1, vectype, 0);
+ tree mask = vect_get_mask (loop_vinfo, &gsi, 1, vectype, 0);
tree scalar_res;
gimple_seq_add_seq (&stmts, tem);
@@ -10309,14 +10292,10 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
{
if (direct_internal_fn_supported_p (IFN_EXTRACT_LAST, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_mask (loop_vinfo,
- &LOOP_VINFO_MASKS (loop_vinfo),
- 1, vectype, NULL);
+ vect_record_mask (loop_vinfo, 1, vectype, NULL_TREE);
else if (can_vec_extract_var_idx_p (
TYPE_MODE (vectype), TYPE_MODE (TREE_TYPE (vectype))))
- vect_record_loop_len (loop_vinfo,
- &LOOP_VINFO_LENS (loop_vinfo),
- 1, vectype, 1);
+ vect_record_len (loop_vinfo, 1, vectype, 1);
else
{
if (dump_enabled_p ())
@@ -10577,18 +10556,23 @@ vect_double_mask_nunits (tree old_type, machine_mode
new_mode)
return build_truth_vector_type_for_mode (nunits, new_mode);
}
-/* Record that a fully-masked version of LOOP_VINFO would need MASKS to
- contain a sequence of NVECTORS masks that each control a vector of type
- VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND
- these vector masks with the vector version of SCALAR_MASK. */
+/* Record that a fully-masked version of the loop or SLP region represented by
+ VINFO would need a sequence of NVECTORS masks that each control a
+ vector of type VECTYPE. If SCALAR_MASK is nonnull and VINFO represents a
+ loop, the fully-masked loop would AND these vector masks with the vector
+ version of SCALAR_MASK. SLP_NODE can be null unless VINFO represents an
+ SLP region. */
void
-vect_record_loop_mask (loop_vec_info loop_vinfo, vec_loop_masks *masks,
- unsigned int nvectors, tree vectype, tree scalar_mask)
+vect_record_mask (vec_info *vinfo, unsigned int nvectors,
+ tree vectype, tree scalar_mask, slp_tree slp_node)
{
+ gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
gcc_assert (nvectors != 0);
+ vec_masks *masks = &vinfo->masks;
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
- if (scalar_mask)
+ if (loop_vinfo && scalar_mask)
{
scalar_cond_masked_key cond (scalar_mask, nvectors);
loop_vinfo->scalar_cond_masked_set.add (cond);
@@ -10597,20 +10581,24 @@ vect_record_loop_mask (loop_vec_info loop_vinfo,
vec_loop_masks *masks,
masks->mask_set.add (std::make_pair (vectype, nvectors));
}
-/* Given a complete set of masks MASKS, extract mask number INDEX
- for an rgroup that operates on NVECTORS vectors of type VECTYPE,
- where 0 <= INDEX < NVECTORS. Insert any set-up statements before GSI.
+/* Given a complete set of masks for the loop or SLP region represented by
+ VINFO, extract mask number INDEX for an rgroup that operates on NVECTORS
+ vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. Insert any set-up
+ statements before GSI. SLP_NODE can be null unless VINFO represents a basic
+ block SLP region.
- See the comment above vec_loop_masks for more details about the mask
+ See the comment above vec_masks for more details about the mask
arrangement. */
tree
-vect_get_loop_mask (loop_vec_info loop_vinfo,
- gimple_stmt_iterator *gsi, vec_loop_masks *masks,
- unsigned int nvectors, tree vectype, unsigned int index)
+vect_get_mask (vec_info *vinfo, gimple_stmt_iterator *gsi,
+ unsigned int nvectors, tree vectype, unsigned int index,
+ slp_tree slp_node)
{
- if (LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo)
- == vect_partial_vectors_while_ult)
+ gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
+ vec_masks *masks = &LOOP_VINFO_MASKS (vinfo);
+
+ if (vinfo->partial_vector_style == vect_partial_vectors_while_ult)
{
rgroup_controls *rgm = &(masks->rgc_vec)[nvectors - 1];
tree mask_type = rgm->type;
@@ -10649,14 +10637,14 @@ vect_get_loop_mask (loop_vec_info loop_vinfo,
}
return mask;
}
- else if (LOOP_VINFO_PARTIAL_VECTORS_STYLE (loop_vinfo)
- == vect_partial_vectors_avx512)
+ else if (vinfo->partial_vector_style == vect_partial_vectors_avx512)
{
/* The number of scalars per iteration and the number of vectors are
both compile-time constants. */
unsigned int nscalars_per_iter
= exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ vinfo->vectorization_factor)
+ .to_constant ();
rgroup_controls *rgm = &masks->rgc_vec[nscalars_per_iter - 1];
@@ -10717,16 +10705,19 @@ vect_get_loop_mask (loop_vec_info loop_vinfo,
gcc_unreachable ();
}
-/* Record that LOOP_VINFO would need LENS to contain a sequence of NVECTORS
- lengths for controlling an operation on VECTYPE. The operation splits
- each element of VECTYPE into FACTOR separate subelements, measuring the
- length as a number of these subelements. */
+/* Record that the loop or SLP region represented by VINFO would need a
sequence
+ of NVECTORS lengths for controlling an operation on VECTYPE. The operation
+ splits each element of VECTYPE into FACTOR separate subelements, measuring
+ the length as a number of these subelements. SLP_NODE can be null unless
+ VINFO represents a basic block SLP region. */
void
-vect_record_loop_len (loop_vec_info loop_vinfo, vec_loop_lens *lens,
- unsigned int nvectors, tree vectype, unsigned int factor)
+vect_record_len (vec_info *vinfo, unsigned int nvectors, tree vectype,
+ unsigned int factor, slp_tree slp_node)
{
+ gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
gcc_assert (nvectors != 0);
+ vec_lens *lens = &vinfo->lens;
if (lens->length () < nvectors)
lens->safe_grow_cleared (nvectors, true);
rgroup_controls *rgl = &(*lens)[nvectors - 1];
@@ -10735,7 +10726,8 @@ vect_record_loop_len (loop_vec_info loop_vinfo,
vec_loop_lens *lens,
the number of vectors are both compile-time constants. */
unsigned int nscalars_per_iter
= exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ vinfo->vectorization_factor)
+ .to_constant ();
if (rgl->max_nscalars_per_iter < nscalars_per_iter)
{
@@ -10751,20 +10743,23 @@ vect_record_loop_len (loop_vec_info loop_vinfo,
vec_loop_lens *lens,
}
}
-/* Given a complete set of lengths LENS, extract length number INDEX
- for an rgroup that operates on NVECTORS vectors of type VECTYPE,
- where 0 <= INDEX < NVECTORS. Return a value that contains FACTOR
- multiplied by the number of elements that should be processed.
- Insert any set-up statements before GSI. */
+/* Given a complete set of lengths for the loop or SLP region represented by
+ VINFO, extract length number INDEX for an rgroup that operates on NVECTORS
+ vectors of type VECTYPE, where 0 <= INDEX < NVECTORS. Return a value that
+ contains FACTOR multiplied by the number of elements that should be
+ processed. Insert any set-up statements before GSI. SLP_NODE can be null
+ unless VINFO represents a basic block SLP region. */
tree
-vect_get_loop_len (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
- vec_loop_lens *lens, unsigned int nvectors, tree vectype,
- unsigned int index, unsigned int factor, bool adjusted)
+vect_get_len (vec_info *vinfo, gimple_stmt_iterator *gsi, unsigned int
nvectors,
+ tree vectype, unsigned int index, unsigned int factor,
+ bool adjusted, slp_tree slp_node)
{
+ gcc_assert (!is_a<bb_vec_info> (vinfo) || slp_node != nullptr);
+ vec_lens *lens = &vinfo->lens;
rgroup_controls *rgl = &(*lens)[nvectors - 1];
bool use_bias_adjusted_len =
- LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo) != 0;
+ vinfo->partial_load_store_bias != 0;
/* Populate the rgroup's len array, if this is the first time we've
used it. */
@@ -10773,7 +10768,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
gimple_stmt_iterator *gsi,
rgl->controls.safe_grow_cleared (nvectors, true);
for (unsigned int i = 0; i < nvectors; ++i)
{
- tree len_type = LOOP_VINFO_RGROUP_COMPARE_TYPE (loop_vinfo);
+ tree len_type = vinfo->rgroup_compare_type;
gcc_assert (len_type != NULL_TREE);
tree len = make_temp_ssa_name (len_type, NULL, "loop_len");
@@ -10808,7 +10803,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
gimple_stmt_iterator *gsi,
are N times bigger than X's. */
gcc_assert (multiple_p (nunits1, nunits2));
factor = exact_div (nunits1, nunits2).to_constant ();
- tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
+ tree iv_type = vinfo->rgroup_iv_type;
gimple_seq seq = NULL;
loop_len = gimple_build (&seq, EXACT_DIV_EXPR, iv_type, loop_len,
build_int_cst (iv_type, factor));
@@ -10822,13 +10817,13 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
gimple_stmt_iterator *gsi,
the number of vectors are both compile-time constants. */
unsigned int nscalars_per_iter
= exact_div (nvectors * TYPE_VECTOR_SUBPARTS (vectype),
- LOOP_VINFO_VECT_FACTOR (loop_vinfo)).to_constant ();
+ vinfo->vectorization_factor).to_constant ();
unsigned int rglvecsize = rgl->factor * rgl->max_nscalars_per_iter;
unsigned int vecsize = nscalars_per_iter * factor;
if (rglvecsize > vecsize)
{
unsigned int fac = rglvecsize / vecsize;
- tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
+ tree iv_type = vinfo->rgroup_iv_type;
gimple_seq seq = NULL;
loop_len = gimple_build (&seq, EXACT_DIV_EXPR, iv_type, loop_len,
build_int_cst (iv_type, fac));
@@ -10838,7 +10833,7 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
gimple_stmt_iterator *gsi,
else if (rglvecsize < vecsize)
{
unsigned int fac = vecsize / rglvecsize;
- tree iv_type = LOOP_VINFO_RGROUP_IV_TYPE (loop_vinfo);
+ tree iv_type = vinfo->rgroup_iv_type;
gimple_seq seq = NULL;
loop_len = gimple_build (&seq, MULT_EXPR, iv_type, loop_len,
build_int_cst (iv_type, fac));
@@ -10856,14 +10851,14 @@ vect_get_loop_len (loop_vec_info loop_vinfo,
gimple_stmt_iterator *gsi,
*/
tree
vect_gen_loop_len_mask (loop_vec_info loop_vinfo, gimple_stmt_iterator *gsi,
- gimple_stmt_iterator *cond_gsi, vec_loop_lens *lens,
- unsigned int nvectors, tree vectype, tree stmt,
- unsigned int index, unsigned int factor)
+ gimple_stmt_iterator *cond_gsi, unsigned int nvectors,
+ tree vectype, tree stmt, unsigned int index,
+ unsigned int factor)
{
tree all_one_mask = build_all_ones_cst (vectype);
tree all_zero_mask = build_zero_cst (vectype);
- tree len = vect_get_loop_len (loop_vinfo, gsi, lens, nvectors, vectype,
index,
- factor, true);
+ tree len
+ = vect_get_len (loop_vinfo, gsi, nvectors, vectype, index, factor, true);
tree bias = build_int_cst (intQI_type_node,
LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo));
tree len_mask = make_temp_ssa_name (TREE_TYPE (stmt), NULL, "vec_len_mask");
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 5ce9db48e28..83adb17735f 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -1440,8 +1440,6 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
}
unsigned int nvectors = vect_get_num_copies (loop_vinfo, slp_node);
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
- vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
machine_mode vecmode = TYPE_MODE (vectype);
bool is_load = (vls_type == VLS_LOAD);
if (memory_access_type == VMAT_LOAD_STORE_LANES)
@@ -1452,10 +1450,9 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
elsvals)
: vect_store_lanes_supported (vectype, group_size, true));
if (ifn == IFN_MASK_LEN_LOAD_LANES || ifn == IFN_MASK_LEN_STORE_LANES)
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
+ vect_record_len (loop_vinfo, nvectors, vectype, 1);
else if (ifn == IFN_MASK_LOAD_LANES || ifn == IFN_MASK_STORE_LANES)
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
- scalar_mask);
+ vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
else
{
if (dump_enabled_p ())
@@ -1497,14 +1494,13 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
memory_type,
off_vectype, scale,
elsvals))
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
+ vect_record_len (loop_vinfo, nvectors, vectype, 1);
else if (internal_gather_scatter_fn_supported_p (ifn, vectype,
memory_type,
off_vectype, scale,
elsvals)
|| memory_access_type == VMAT_GATHER_SCATTER_LEGACY)
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
- scalar_mask);
+ vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
else
{
if (dump_enabled_p ())
@@ -1560,7 +1556,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
{
nvectors = group_memory_nvectors (group_size * vf, nunits);
unsigned factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode);
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, factor);
+ vect_record_len (loop_vinfo, nvectors, vectype, factor);
using_partial_vectors_p = true;
}
else if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode)
@@ -1568,7 +1564,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
elsvals))
{
nvectors = group_memory_nvectors (group_size * vf, nunits);
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
scalar_mask);
+ vect_record_mask (loop_vinfo, nvectors, vectype, scalar_mask);
using_partial_vectors_p = true;
}
@@ -1584,30 +1580,31 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
}
/* Return the mask input to a masked load or store. VEC_MASK is the vectorized
- form of the scalar mask condition and LOOP_MASK, if nonnull, is the mask
- that needs to be applied to all loads and stores in a vectorized loop.
- Return VEC_MASK if LOOP_MASK is null or if VEC_MASK is already masked,
- otherwise return VEC_MASK & LOOP_MASK.
+ form of the scalar mask condition and LOOP_OR_TAIL_MASK, if nonnull, is the
+ mask that needs to be applied because partial vectors are used in the loop
or
+ SLP region represented by VINFO. Return VEC_MASK if LOOP_OR_TAIL_MASK is
+ null or if VEC_MASK is already masked, otherwise return VEC_MASK &
+ LOOP_OR_TAIL_MASK.
MASK_TYPE is the type of both masks. If new statements are needed,
insert them before GSI. */
tree
-prepare_vec_mask (loop_vec_info loop_vinfo, tree mask_type, tree loop_mask,
+prepare_vec_mask (vec_info *vinfo, tree mask_type, tree loop_or_tail_mask,
tree vec_mask, gimple_stmt_iterator *gsi)
{
gcc_assert (useless_type_conversion_p (mask_type, TREE_TYPE (vec_mask)));
- if (!loop_mask)
+ if (!loop_or_tail_mask)
return vec_mask;
- gcc_assert (TREE_TYPE (loop_mask) == mask_type);
+ gcc_assert (TREE_TYPE (loop_or_tail_mask) == mask_type);
- if (loop_vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_mask }))
+ if (vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_or_tail_mask }))
return vec_mask;
tree and_res = make_temp_ssa_name (mask_type, NULL, "vec_mask_and");
- gimple *and_stmt = gimple_build_assign (and_res, BIT_AND_EXPR,
- vec_mask, loop_mask);
+ gimple *and_stmt
+ = gimple_build_assign (and_res, BIT_AND_EXPR, vec_mask, loop_or_tail_mask);
gsi_insert_before (gsi, and_stmt, GSI_SAME_STMT);
return and_res;
@@ -3682,8 +3679,6 @@ vectorizable_call (vec_info *vinfo,
internal_fn cond_fn = (internal_fn_mask_index (ifn) != -1
? ifn : get_conditional_internal_fn (ifn));
internal_fn cond_len_fn = get_len_internal_fn (cond_fn);
- vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
- vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
unsigned int nvectors = vect_get_num_copies (vinfo, slp_node);
if (cost_vec) /* transformation not required. */
{
@@ -3727,11 +3722,10 @@ vectorizable_call (vec_info *vinfo,
if (cond_len_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_len_fn, vectype_out,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_out,
- 1);
+ vect_record_len (loop_vinfo, nvectors, vectype_out, 1);
else
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_out,
- scalar_mask);
+ vect_record_mask (loop_vinfo, nvectors, vectype_out,
+ scalar_mask);
}
}
return true;
@@ -3811,9 +3805,8 @@ vectorizable_call (vec_info *vinfo,
if (masked_loop_p)
{
unsigned int vec_num = vec_oprnds0.length ();
- vargs[varg++] = vect_get_loop_mask (loop_vinfo, gsi,
- masks, vec_num,
- vectype_out, i);
+ vargs[varg++] = vect_get_mask (loop_vinfo, gsi, vec_num,
+ vectype_out, i);
}
else
{
@@ -3872,8 +3865,8 @@ vectorizable_call (vec_info *vinfo,
if (len_opno >= 0 && len_loop_p)
{
unsigned int vec_num = vec_oprnds0.length ();
- tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
- vec_num, vectype_out, i, 1,
true);
+ tree len = vect_get_len (loop_vinfo, gsi, vec_num,
+ vectype_out, i, 1, true);
signed char biasval
= LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
tree bias = build_int_cst (intQI_type_node, biasval);
@@ -3883,8 +3876,8 @@ vectorizable_call (vec_info *vinfo,
else if (mask_opno >= 0 && masked_loop_p)
{
unsigned int vec_num = vec_oprnds0.length ();
- tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_out, i);
+ tree mask = vect_get_mask (loop_vinfo, gsi, vec_num,
+ vectype_out, i);
vargs[mask_opno]
= prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
vargs[mask_opno], gsi);
@@ -4455,9 +4448,7 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
if (masked_call_offset)
/* When there is an explicit mask we require the
number of elements to match up. */
- vect_record_loop_mask (loop_vinfo,
- &LOOP_VINFO_MASKS (loop_vinfo),
- ncopies_in, vectype, NULL_TREE);
+ vect_record_mask (loop_vinfo, ncopies_in, vectype, NULL_TREE);
else
{
/* When there is no explicit mask on the call we have
@@ -4488,9 +4479,8 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
callee_nelements = TYPE_VECTOR_SUBPARTS (masktype);
}
auto o = vector_unroll_factor (nunits, callee_nelements);
- vect_record_loop_mask (loop_vinfo,
- &LOOP_VINFO_MASKS (loop_vinfo),
- ncopies * o, masktype, NULL_TREE);
+ vect_record_mask (loop_vinfo, ncopies * o, masktype,
+ NULL_TREE);
}
}
else if ((bestn->simdclone->args[i].arg_type
@@ -4688,12 +4678,9 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
if (loop_vinfo
&& LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
{
- vec_loop_masks *loop_masks
- = &LOOP_VINFO_MASKS (loop_vinfo);
tree loop_mask
- = vect_get_loop_mask (loop_vinfo, gsi,
- loop_masks, ncopies_in,
- vectype, j);
+ = vect_get_mask (loop_vinfo, gsi, ncopies_in,
+ vectype, j);
vec_oprnd0
= prepare_vec_mask (loop_vinfo,
TREE_TYPE (loop_mask),
@@ -4701,7 +4688,6 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
gsi);
loop_vinfo->vec_cond_masked_set.add ({ vec_oprnd0,
loop_mask
});
-
}
vec_oprnd0
= build3 (VEC_COND_EXPR, atype, vec_oprnd0,
@@ -4749,12 +4735,9 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
if (loop_vinfo
&& LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
{
- vec_loop_masks *loop_masks
- = &LOOP_VINFO_MASKS (loop_vinfo);
tree loop_mask
- = vect_get_loop_mask (loop_vinfo, gsi,
- loop_masks, ncopies_in,
- vectype, j);
+ = vect_get_mask (loop_vinfo, gsi, ncopies_in,
+ vectype, j);
vec_oprnd0
= prepare_vec_mask (loop_vinfo,
TREE_TYPE (loop_mask),
@@ -4901,11 +4884,8 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
for (m = j * o; m < (j + 1) * o; m++)
{
if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
- {
- vec_loop_masks *loop_masks = &LOOP_VINFO_MASKS (loop_vinfo);
- mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies * o, mask_vectype, m);
- }
+ mask = vect_get_mask (loop_vinfo, gsi, ncopies * o,
+ mask_vectype, m);
else
mask = vect_build_all_ones_mask (vinfo, stmt_info,
mask_argtype);
@@ -6780,8 +6760,6 @@ vectorizable_operation (vec_info *vinfo,
}
int reduc_idx = SLP_TREE_REDUC_IDX (slp_node);
- vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
- vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
internal_fn cond_fn = get_conditional_internal_fn (code);
internal_fn cond_len_fn = get_conditional_len_internal_fn (code);
@@ -6805,13 +6783,11 @@ vectorizable_operation (vec_info *vinfo,
if (cond_len_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_len_fn, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo, lens, vec_num, vectype,
- 1);
+ vect_record_len (loop_vinfo, vec_num, vectype, 1);
else if (cond_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_fn, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_mask (loop_vinfo, masks, vec_num,
- vectype, NULL);
+ vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
else
{
if (dump_enabled_p ())
@@ -7071,8 +7047,7 @@ vectorizable_operation (vec_info *vinfo,
{
tree mask;
if (masked_loop_p)
- mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype, i);
+ mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
else
/* Dummy mask. */
mask = build_minus_one_cst (truth_type_for (vectype));
@@ -7098,8 +7073,8 @@ vectorizable_operation (vec_info *vinfo,
}
if (len_loop_p)
{
- tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
- vec_num, vectype, i, 1, true);
+ tree len
+ = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, 1, true);
signed char biasval
= LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
tree bias = build_int_cst (intQI_type_node, biasval);
@@ -7128,18 +7103,14 @@ vectorizable_operation (vec_info *vinfo,
{
if (loop_vinfo->scalar_cond_masked_set.contains ({ op0, vec_num
}))
{
- mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype, i);
-
+ mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
vop0 = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
vop0, gsi);
}
if (loop_vinfo->scalar_cond_masked_set.contains ({ op1, vec_num
}))
{
- mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype, i);
-
+ mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
vop1 = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
vop1, gsi);
}
@@ -8667,14 +8638,8 @@ vectorizable_store (vec_info *vinfo,
}
gcc_assert (alignment_support_scheme);
- vec_loop_masks *loop_masks
- = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
- ? &LOOP_VINFO_MASKS (loop_vinfo)
- : NULL);
- vec_loop_lens *loop_lens
- = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
- ? &LOOP_VINFO_LENS (loop_vinfo)
- : NULL);
+ bool masks_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+ bool lens_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
/* The vect_transform_stmt and vect_analyze_stmt will go here but there
are some difference here. We cannot enable both the lens and masks
@@ -8682,14 +8647,14 @@ vectorizable_store (vec_info *vinfo,
Shouldn't go with length-based approach if fully masked. */
if (cost_vec == NULL)
/* The cost_vec is NULL during transform. */
- gcc_assert ((!loop_lens || !loop_masks));
+ gcc_assert ((!lens_p || !masks_p));
/* Targets with store-lane instructions must not require explicit
realignment. vect_supportable_dr_alignment always returns either
dr_aligned or dr_unaligned_supported for masked operations. */
gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
&& !mask_node
- && !loop_masks)
+ && !masks_p)
|| alignment_support_scheme == dr_aligned
|| alignment_support_scheme == dr_unaligned_supported);
@@ -8827,20 +8792,19 @@ vectorizable_store (vec_info *vinfo,
tree final_mask = NULL;
tree final_len = NULL;
tree bias = NULL;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies, vectype, j);
+ if (masks_p)
+ final_mask
+ = vect_get_mask (loop_vinfo, gsi, ncopies, vectype, j);
if (vec_mask)
final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
vec_mask, gsi);
if (lanes_ifn == IFN_MASK_LEN_STORE_LANES)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- ncopies, vectype, j, 1, true);
- else
- final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
+ if (lens_p)
+ final_len = vect_get_len (loop_vinfo, gsi, ncopies, vectype, j,
+ 1, true);
+ else final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
signed char biasval
= LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
bias = build_int_cst (intQI_type_node, biasval);
@@ -8969,10 +8933,9 @@ vectorizable_store (vec_info *vinfo,
tree bias = NULL_TREE;
if (!costing_p)
{
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi,
- loop_masks, num_stmts,
- vectype, j);
+ if (masks_p)
+ final_mask
+ = vect_get_mask (loop_vinfo, gsi, num_stmts, vectype, j);
if (vec_mask)
final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
final_mask, vec_mask, gsi);
@@ -9038,10 +9001,9 @@ vectorizable_store (vec_info *vinfo,
if (ls.gs.ifn == IFN_MASK_LEN_SCATTER_STORE)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi,
- loop_lens, num_stmts,
- vectype, j, 1, true);
+ if (lens_p)
+ final_len = vect_get_len (loop_vinfo, gsi, num_stmts,
+ vectype, j, 1, true);
else
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
@@ -9333,7 +9295,7 @@ vectorizable_store (vec_info *vinfo,
= STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) != 0;
if (!costing_p
&& simd_lane_access_p
- && !loop_masks
+ && !lens_p
&& TREE_CODE (DR_BASE_ADDRESS (first_dr_info->dr)) == ADDR_EXPR
&& VAR_P (TREE_OPERAND (DR_BASE_ADDRESS (first_dr_info->dr), 0))
&& integer_zerop (get_dr_vinfo_offset (vinfo, first_dr_info))
@@ -9387,9 +9349,8 @@ vectorizable_store (vec_info *vinfo,
tree final_mask = NULL_TREE;
tree final_len = NULL_TREE;
tree bias = NULL_TREE;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (masks_p)
+ final_mask = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
if (vec_mask)
vec_mask = vec_masks[i];
if (vec_mask)
@@ -9418,19 +9379,19 @@ vectorizable_store (vec_info *vinfo,
set_ptr_info_alignment (get_ptr_info (dataref_ptr), align, misalign);
align = least_bit_hwi (misalign | align);
- /* Compute IFN when LOOP_LENS or final_mask valid. */
+ /* Compute IFN when LEN_LOOP_P or final_mask valid. */
machine_mode vmode = TYPE_MODE (vectype);
machine_mode new_vmode = vmode;
internal_fn partial_ifn = IFN_LAST;
- if (loop_lens)
+ if (lens_p)
{
opt_machine_mode new_ovmode
= get_len_load_store_mode (vmode, false, &partial_ifn);
new_vmode = new_ovmode.require ();
unsigned factor
= (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, factor, true);
+ final_len
+ = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, factor, true);
}
else if (final_mask)
{
@@ -9445,7 +9406,7 @@ vectorizable_store (vec_info *vinfo,
if (!final_len)
{
/* Pass VF value to 'len' argument of
- MASK_LEN_STORE if LOOP_LENS is invalid. */
+ MASK_LEN_STORE if lens_p is false. */
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
}
if (!final_mask)
@@ -10576,14 +10537,8 @@ vectorizable_load (vec_info *vinfo,
ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr));
}
- vec_loop_masks *loop_masks
- = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
- ? &LOOP_VINFO_MASKS (loop_vinfo)
- : NULL);
- vec_loop_lens *loop_lens
- = (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
- ? &LOOP_VINFO_LENS (loop_vinfo)
- : NULL);
+ bool masks_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+ bool lens_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
/* The vect_transform_stmt and vect_analyze_stmt will go here but there
are some difference here. We cannot enable both the lens and masks
@@ -10591,7 +10546,7 @@ vectorizable_load (vec_info *vinfo,
Shouldn't go with length-based approach if fully masked. */
if (cost_vec == NULL)
/* The cost_vec is NULL during transform. */
- gcc_assert ((!loop_lens || !loop_masks));
+ gcc_assert ((!lens_p || !masks_p));
/* Targets with store-lane instructions must not require explicit
realignment. vect_supportable_dr_alignment always returns either
@@ -10599,7 +10554,7 @@ vectorizable_load (vec_info *vinfo,
operations. */
gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
&& !mask_node
- && !loop_masks)
+ && !masks_p)
|| mat_gather_scatter_p (memory_access_type)
|| alignment_support_scheme == dr_aligned
|| alignment_support_scheme == dr_unaligned_supported);
@@ -10803,18 +10758,18 @@ vectorizable_load (vec_info *vinfo,
tree final_mask = NULL_TREE;
tree final_len = NULL_TREE;
tree bias = NULL_TREE;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies, vectype, j);
+ if (masks_p)
+ final_mask
+ = vect_get_mask (loop_vinfo, gsi, ncopies, vectype, j);
if (vec_mask)
final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
vec_mask, gsi);
if (lanes_ifn == IFN_MASK_LEN_LOAD_LANES)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- ncopies, vectype, j, 1, true);
+ if (lens_p)
+ final_len
+ = vect_get_len (loop_vinfo, gsi, ncopies, vectype, j, 1,
true);
else
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
signed char biasval
@@ -10953,9 +10908,9 @@ vectorizable_load (vec_info *vinfo,
{
if (mask_node)
vec_mask = vec_masks[i];
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (masks_p)
+ final_mask
+ = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
if (vec_mask)
final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
final_mask, vec_mask, gsi);
@@ -11025,9 +10980,9 @@ vectorizable_load (vec_info *vinfo,
if (ls.gs.ifn == IFN_MASK_LEN_GATHER_LOAD)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, 1,
true);
+ if (lens_p)
+ final_len = vect_get_len (loop_vinfo, gsi, vec_num, vectype,
+ i, 1, true);
else
final_len = build_int_cst (sizetype,
TYPE_VECTOR_SUBPARTS (vectype));
@@ -11427,9 +11382,9 @@ vectorizable_load (vec_info *vinfo,
{
if (mask_node)
vec_mask = vec_masks[i];
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (masks_p)
+ final_mask
+ = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
if (vec_mask)
final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
final_mask, vec_mask, gsi);
@@ -11466,19 +11421,19 @@ vectorizable_load (vec_info *vinfo,
misalign);
align = least_bit_hwi (misalign | align);
- /* Compute IFN when LOOP_LENS or final_mask valid. */
+ /* Compute IFN when LEN_LOOP_P or final_mask valid. */
machine_mode vmode = TYPE_MODE (vectype);
machine_mode new_vmode = vmode;
internal_fn partial_ifn = IFN_LAST;
- if (loop_lens)
+ if (lens_p)
{
opt_machine_mode new_ovmode
= get_len_load_store_mode (vmode, true, &partial_ifn);
new_vmode = new_ovmode.require ();
unsigned factor
= (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, factor,
true);
+ final_len = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i,
+ factor, true);
}
else if (final_mask)
{
@@ -11494,7 +11449,7 @@ vectorizable_load (vec_info *vinfo,
if (!final_len)
{
/* Pass VF value to 'len' argument of
- MASK_LEN_LOAD if LOOP_LENS is invalid. */
+ MASK_LEN_LOAD if LEN_LOOP_P is false. */
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
}
if (!final_mask)
@@ -12346,13 +12301,9 @@ vectorizable_condition (vec_info *vinfo,
{
if (direct_internal_fn_supported_p (IFN_LEN_FOLD_EXTRACT_LAST,
vectype, OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo,
- &LOOP_VINFO_LENS (loop_vinfo),
- vec_num, vectype, 1);
+ vect_record_len (loop_vinfo, vec_num, vectype, 1);
else
- vect_record_loop_mask (loop_vinfo,
- &LOOP_VINFO_MASKS (loop_vinfo),
- vec_num, vectype, NULL);
+ vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
}
/* Extra inactive lanes should be safe for vect_nested_cycle. */
else if (!nested_cycle_p)
@@ -12382,22 +12333,22 @@ vectorizable_condition (vec_info *vinfo,
/* See whether another part of the vectorized code applies a loop
mask to the condition, or to its inverse. */
- vec_loop_masks *masks = NULL;
- vec_loop_lens *lens = NULL;
+ bool fully_masked_p = false;
+ bool fully_with_length_p = false;
if (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo))
{
if (reduction_type == EXTRACT_LAST_REDUCTION)
- lens = &LOOP_VINFO_LENS (loop_vinfo);
+ fully_with_length_p = true;
}
else if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
{
if (reduction_type == EXTRACT_LAST_REDUCTION)
- masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ fully_masked_p = true;
else
{
scalar_cond_masked_key cond (cond_expr, 1);
if (loop_vinfo->scalar_cond_masked_set.contains (cond))
- masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ fully_masked_p = true;
else
{
bool honor_nans = HONOR_NANS (TREE_TYPE (cond.op0));
@@ -12405,7 +12356,7 @@ vectorizable_condition (vec_info *vinfo,
cond.code = invert_tree_comparison (cond.code, honor_nans);
if (!masked && loop_vinfo->scalar_cond_masked_set.contains (cond))
{
- masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ fully_masked_p = true;
cond_code = cond.code;
swap_cond_operands = true;
}
@@ -12418,7 +12369,7 @@ vectorizable_condition (vec_info *vinfo,
cond.code = orig_code;
if (loop_vinfo->scalar_cond_masked_set.contains (cond))
{
- masks = &LOOP_VINFO_MASKS (loop_vinfo);
+ fully_masked_p = true;
cond_code = cond.code;
swap_cond_operands = true;
must_invert_cmp_result = true;
@@ -12531,7 +12482,8 @@ vectorizable_condition (vec_info *vinfo,
in cases where that's necessary. */
tree len = NULL_TREE, bias = NULL_TREE;
- if (masks || lens || reduction_type == EXTRACT_LAST_REDUCTION)
+ if (fully_masked_p || fully_with_length_p
+ || reduction_type == EXTRACT_LAST_REDUCTION)
{
if (!is_gimple_val (vec_compare))
{
@@ -12555,12 +12507,12 @@ vectorizable_condition (vec_info *vinfo,
if (direct_internal_fn_supported_p (IFN_LEN_FOLD_EXTRACT_LAST,
vectype, OPTIMIZE_FOR_SPEED))
{
- if (lens)
+ if (fully_with_length_p)
{
/* ??? Do we really want the adjusted LEN here? Isn't this
based on number of elements? */
- len = vect_get_loop_len (loop_vinfo, gsi, lens,
- vec_num, vectype, i, 1, true);
+ len = vect_get_len (loop_vinfo, gsi, vec_num, vectype, i, 1,
+ true);
signed char biasval
= LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
bias = build_int_cst (intQI_type_node, biasval);
@@ -12571,11 +12523,10 @@ vectorizable_condition (vec_info *vinfo,
bias = build_int_cst (intQI_type_node, 0);
}
}
- if (masks)
+ if (fully_masked_p)
{
tree loop_mask
- = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
- vectype, i);
+ = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
tree tmp2 = make_ssa_name (vec_cmp_type);
gassign *g
= gimple_build_assign (tmp2, BIT_AND_EXPR, vec_compare,
@@ -13008,8 +12959,6 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
machine_mode mode = TYPE_MODE (vectype);
int vec_num = vect_get_num_copies (loop_vinfo, slp_node);
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
- vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
bool masked_loop_p = LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
bool len_loop_p = LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
@@ -13083,9 +13032,9 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
{
if (direct_internal_fn_supported_p (IFN_VCOND_MASK_LEN, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo, lens, vec_num, vectype, 1);
+ vect_record_len (loop_vinfo, vec_num, vectype, 1);
else
- vect_record_loop_mask (loop_vinfo, masks, vec_num, vectype, NULL);
+ vect_record_mask (loop_vinfo, vec_num, vectype, NULL_TREE);
}
if (!vect_compute_type_for_early_break_scalar_iv (loop_vinfo))
@@ -13153,19 +13102,17 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
for (unsigned i = 0; i < stmts.length (); i++)
{
tree stmt_mask
- = vect_get_loop_mask (loop_vinfo, gsi, masks, vec_num,
- vectype, i);
- stmt_mask
- = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask), stmt_mask,
- stmts[i], &cond_gsi);
+ = vect_get_mask (loop_vinfo, gsi, vec_num, vectype, i);
+ stmt_mask = prepare_vec_mask (loop_vinfo, TREE_TYPE (stmt_mask),
+ stmt_mask, stmts[i], &cond_gsi);
workset.quick_push (stmt_mask);
}
else if (len_loop_p)
for (unsigned i = 0; i < stmts.length (); i++)
{
- tree len_mask = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi,
- lens, vec_num,
- vectype, stmts[i], i, 1);
+ tree len_mask
+ = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, vec_num,
+ vectype, stmts[i], i, 1);
workset.quick_push (len_mask);
}
@@ -13200,14 +13147,13 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
new_temp = stmts[0];
if (masked_loop_p)
{
- tree mask
- = vect_get_loop_mask (loop_vinfo, gsi, masks, 1, vectype, 0);
+ tree mask = vect_get_mask (loop_vinfo, gsi, 1, vectype, 0);
new_temp = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
new_temp, &cond_gsi);
}
else if (len_loop_p)
- new_temp = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, lens,
- 1, vectype, new_temp, 0, 1);
+ new_temp = vect_gen_loop_len_mask (loop_vinfo, gsi, &cond_gsi, 1,
+ vectype, new_temp, 0, 1);
}
gcc_assert (new_temp);
diff --git a/gcc/tree-vectorizer.cc b/gcc/tree-vectorizer.cc
index 8c824e4ee33..87204ba54dc 100644
--- a/gcc/tree-vectorizer.cc
+++ b/gcc/tree-vectorizer.cc
@@ -470,7 +470,12 @@ vec_info::vec_info (vec_info::vec_kind kind_in,
vec_info_shared *shared_)
stmt_vec_info_ro (false),
bbs (NULL),
nbbs (0),
- inv_pattern_def_seq (NULL)
+ inv_pattern_def_seq (NULL),
+ vectorization_factor (1),
+ rgroup_compare_type (NULL_TREE),
+ rgroup_iv_type (NULL_TREE),
+ partial_vector_style (vect_partial_vectors_none),
+ partial_load_store_bias (0)
{
stmt_vec_infos.create (50);
}
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index d4eb20f1a10..66fe0ef0ae6 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -498,7 +498,7 @@ enum vect_partial_vector_style {
/* Key for map that records association between
scalar conditions and corresponding loop mask, and
- is populated by vect_record_loop_mask. */
+ is populated by vect_record_mask. */
struct scalar_cond_masked_key
{
@@ -617,104 +617,6 @@ public:
vec<ddr_p> ddrs;
};
-/* Vectorizer state common between loop and basic-block vectorization. */
-class vec_info {
-public:
- typedef hash_set<int_hash<machine_mode, E_VOIDmode, E_BLKmode> > mode_set;
- enum vec_kind { bb, loop };
-
- vec_info (vec_kind, vec_info_shared *);
- ~vec_info ();
-
- stmt_vec_info add_stmt (gimple *);
- stmt_vec_info add_pattern_stmt (gimple *, stmt_vec_info);
- stmt_vec_info resync_stmt_addr (gimple *);
- stmt_vec_info lookup_stmt (gimple *);
- stmt_vec_info lookup_def (tree);
- stmt_vec_info lookup_single_use (tree);
- class dr_vec_info *lookup_dr (data_reference *);
- void move_dr (stmt_vec_info, stmt_vec_info);
- void remove_stmt (stmt_vec_info);
- void replace_stmt (gimple_stmt_iterator *, stmt_vec_info, gimple *);
- void insert_on_entry (stmt_vec_info, gimple *);
- void insert_seq_on_entry (stmt_vec_info, gimple_seq);
-
- /* The type of vectorization. */
- vec_kind kind;
-
- /* Shared vectorizer state. */
- vec_info_shared *shared;
-
- /* The mapping of GIMPLE UID to stmt_vec_info. */
- vec<stmt_vec_info> stmt_vec_infos;
- /* Whether the above mapping is complete. */
- bool stmt_vec_info_ro;
-
- /* Whether we've done a transform we think OK to not update virtual
- SSA form. */
- bool any_known_not_updated_vssa;
-
- /* The SLP graph. */
- auto_vec<slp_instance> slp_instances;
-
- /* Maps base addresses to an innermost_loop_behavior that gives the maximum
- known alignment for that base. */
- vec_base_alignments base_alignments;
-
- /* All interleaving chains of stores, represented by the first
- stmt in the chain. */
- auto_vec<stmt_vec_info> grouped_stores;
-
- /* The set of vector modes used in the vectorized region. */
- mode_set used_vector_modes;
-
- /* The argument we should pass to related_vector_mode when looking up
- the vector mode for a scalar mode, or VOIDmode if we haven't yet
- made any decisions about which vector modes to use. */
- machine_mode vector_mode;
-
- /* The basic blocks in the vectorization region. For _loop_vec_info,
- the memory is internally managed, while for _bb_vec_info, it points
- to element space of an external auto_vec<>. This inconsistency is
- not a good class design pattern. TODO: improve it with an unified
- auto_vec<> whose lifetime is confined to vec_info object. */
- basic_block *bbs;
-
- /* The count of the basic blocks in the vectorization region. */
- unsigned int nbbs;
-
- /* Used to keep a sequence of def stmts of a pattern stmt that are loop
- invariant if they exists.
- The sequence is emitted in the loop preheader should the loop be vectorized
- and are reset when undoing patterns. */
- gimple_seq inv_pattern_def_seq;
-
-private:
- stmt_vec_info new_stmt_vec_info (gimple *stmt);
- void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true);
- void free_stmt_vec_infos ();
- void free_stmt_vec_info (stmt_vec_info);
-};
-
-class _loop_vec_info;
-class _bb_vec_info;
-
-template<>
-template<>
-inline bool
-is_a_helper <_loop_vec_info *>::test (vec_info *i)
-{
- return i->kind == vec_info::loop;
-}
-
-template<>
-template<>
-inline bool
-is_a_helper <_bb_vec_info *>::test (vec_info *i)
-{
- return i->kind == vec_info::bb;
-}
-
/* In general, we can divide the vector statements in a vectorized loop
into related groups ("rgroups") and say that for each rgroup there is
some nS such that the rgroup operates on nS values from one scalar
@@ -830,7 +732,7 @@ struct rgroup_controls {
tree bias_adjusted_ctrl;
};
-struct vec_loop_masks
+struct vec_masks
{
bool is_empty () const { return mask_set.is_empty (); }
@@ -842,7 +744,146 @@ struct vec_loop_masks
auto_vec<rgroup_controls> rgc_vec;
};
-typedef auto_vec<rgroup_controls> vec_loop_lens;
+typedef auto_vec<rgroup_controls> vec_lens;
+
+/* Vectorizer state common between loop and basic-block vectorization. */
+class vec_info {
+public:
+ typedef hash_set<int_hash<machine_mode, E_VOIDmode, E_BLKmode> > mode_set;
+ enum vec_kind { bb, loop };
+
+ vec_info (vec_kind, vec_info_shared *);
+ ~vec_info ();
+
+ stmt_vec_info add_stmt (gimple *);
+ stmt_vec_info add_pattern_stmt (gimple *, stmt_vec_info);
+ stmt_vec_info resync_stmt_addr (gimple *);
+ stmt_vec_info lookup_stmt (gimple *);
+ stmt_vec_info lookup_def (tree);
+ stmt_vec_info lookup_single_use (tree);
+ class dr_vec_info *lookup_dr (data_reference *);
+ void move_dr (stmt_vec_info, stmt_vec_info);
+ void remove_stmt (stmt_vec_info);
+ void replace_stmt (gimple_stmt_iterator *, stmt_vec_info, gimple *);
+ void insert_on_entry (stmt_vec_info, gimple *);
+ void insert_seq_on_entry (stmt_vec_info, gimple_seq);
+
+ /* The type of vectorization. */
+ vec_kind kind;
+
+ /* Shared vectorizer state. */
+ vec_info_shared *shared;
+
+ /* The mapping of GIMPLE UID to stmt_vec_info. */
+ vec<stmt_vec_info> stmt_vec_infos;
+ /* Whether the above mapping is complete. */
+ bool stmt_vec_info_ro;
+
+ /* Whether we've done a transform we think OK to not update virtual
+ SSA form. */
+ bool any_known_not_updated_vssa;
+
+ /* The SLP graph. */
+ auto_vec<slp_instance> slp_instances;
+
+ /* Maps base addresses to an innermost_loop_behavior that gives the maximum
+ known alignment for that base. */
+ vec_base_alignments base_alignments;
+
+ /* All interleaving chains of stores, represented by the first
+ stmt in the chain. */
+ auto_vec<stmt_vec_info> grouped_stores;
+
+ /* The set of vector modes used in the vectorized region. */
+ mode_set used_vector_modes;
+
+ /* The argument we should pass to related_vector_mode when looking up
+ the vector mode for a scalar mode, or VOIDmode if we haven't yet
+ made any decisions about which vector modes to use. */
+ machine_mode vector_mode;
+
+ /* The basic blocks in the vectorization region. For _loop_vec_info,
+ the memory is internally managed, while for _bb_vec_info, it points
+ to element space of an external auto_vec<>. This inconsistency is
+ not a good class design pattern. TODO: improve it with an unified
+ auto_vec<> whose lifetime is confined to vec_info object. */
+ basic_block *bbs;
+
+ /* The count of the basic blocks in the vectorization region. */
+ unsigned int nbbs;
+
+ /* Used to keep a sequence of def stmts of a pattern stmt that are loop
+ invariant if they exists.
+ The sequence is emitted in the loop preheader should the loop be vectorized
+ and are reset when undoing patterns. */
+ gimple_seq inv_pattern_def_seq;
+
+ /* Set of vector conditions that have loop mask applied. */
+ vec_cond_masked_set_type vec_cond_masked_set;
+
+ /* The masks that a vectorization region should use to avoid operating
+ on inactive scalars. */
+ vec_masks masks;
+
+ /* The lengths that a vectorization region should use to avoid operating
+ on inactive scalars. */
+ vec_lens lens;
+
+ /* Unrolling factor. In case of suitable super-word parallelism
+ it can be that no unrolling is needed, and thus this is 1. */
+ poly_uint64 vectorization_factor;
+
+ /* The type that the loop control IV should be converted to before
+ testing which of the VF scalars are active and inactive.
+ For _loop_vec_info, this is only meaningful if
+ LOOP_VINFO_USING_PARTIAL_VECTORS_P is true.
+ For _bb_vec_info, this is the type of the active lane count.
+ */
+ tree rgroup_compare_type;
+
+ /* The type that the vector loop control IV should have.
+ For _loop_vec_info, this is only meaningful if
+ LOOP_VINFO_USING_PARTIAL_VECTORS_P is true.
+ For _bb_vec_info, this is not actually the type of a
+ vector loop control IV but facilitates code reuse.
+ */
+ tree rgroup_iv_type;
+
+ /* The style used for implementing partial vectors.
+ For _loop_vec_info, this is only meaningful if
+ LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
+ vect_partial_vector_style partial_vector_style;
+
+ /* The bias for len_load and len_store. For now, only 0 and -1 are
+ supported. -1 must be used when a backend does not support
+ len_load/len_store with a length of zero. */
+ signed char partial_load_store_bias;
+
+private:
+ stmt_vec_info new_stmt_vec_info (gimple *stmt);
+ void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true);
+ void free_stmt_vec_infos ();
+ void free_stmt_vec_info (stmt_vec_info);
+};
+
+class _loop_vec_info;
+class _bb_vec_info;
+
+template<>
+template<>
+inline bool
+is_a_helper <_loop_vec_info *>::test (vec_info *i)
+{
+ return i->kind == vec_info::loop;
+}
+
+template<>
+template<>
+inline bool
+is_a_helper <_bb_vec_info *>::test (vec_info *i)
+{
+ return i->kind == vec_info::bb;
+}
typedef auto_vec<std::pair<data_reference*, tree> > drs_init_vec;
@@ -974,10 +1015,6 @@ public:
used. */
poly_uint64 versioning_threshold;
- /* Unrolling factor. In case of suitable super-word parallelism
- it can be that no unrolling is needed, and thus this is 1. */
- poly_uint64 vectorization_factor;
-
/* Gimple operand for the number of scalar iteration handed per loop
iteration, and therefore how much to increment each IV by. */
tree iv_increment;
@@ -1017,20 +1054,9 @@ public:
if there is no particular limit. */
unsigned HOST_WIDE_INT max_vectorization_factor;
- /* The masks that a fully-masked loop should use to avoid operating
- on inactive scalars. */
- vec_loop_masks masks;
-
- /* The lengths that a loop with length should use to avoid operating
- on inactive scalars. */
- vec_loop_lens lens;
-
/* Set of scalar conditions that have loop mask applied. */
scalar_cond_masked_set_type scalar_cond_masked_set;
- /* Set of vector conditions that have loop mask applied. */
- vec_cond_masked_set_type vec_cond_masked_set;
-
/* If we are using a loop mask to align memory addresses, this variable
contains the number of vector elements that we should skip in the
first iteration of the vector loop (i.e. the number of leading
@@ -1042,11 +1068,6 @@ public:
were skipped during the initial iteration of the loop. */
tree mask_skip_niters_pfa_offset;
- /* The type that the loop control IV should be converted to before
- testing which of the VF scalars are active and inactive.
- Only meaningful if LOOP_VINFO_USING_PARTIAL_VECTORS_P. */
- tree rgroup_compare_type;
-
/* For #pragma omp simd if (x) loops the x expression. If constant 0,
the loop should not be vectorized, if constant non-zero, simd_if_cond
shouldn't be set and loop vectorized normally, if SSA_NAME, the loop
@@ -1054,14 +1075,6 @@ public:
is false and vectorized loop otherwise. */
tree simd_if_cond;
- /* The type that the vector loop control IV should have when
- LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
- tree rgroup_iv_type;
-
- /* The style used for implementing partial vectors when
- LOOP_VINFO_USING_PARTIAL_VECTORS_P is true. */
- vect_partial_vector_style partial_vector_style;
-
/* Unknown DRs according to which loop was peeled. */
class dr_vec_info *unaligned_dr;
@@ -1163,11 +1176,6 @@ public:
without peeling. */
bool allow_mutual_alignment;
- /* The bias for len_load and len_store. For now, only 0 and -1 are
- supported. -1 must be used when a backend does not support
- len_load/len_store with a length of zero. */
- signed char partial_load_store_bias;
-
/* When we have grouped data accesses with gaps, we may introduce invalid
memory accesses. We peel the last iteration of the loop to prevent
this. */
@@ -2694,20 +2702,18 @@ extern void vect_gen_vector_loop_niters (loop_vec_info,
tree, tree *,
extern tree vect_get_loop_iv_increment (loop_vec_info);
extern tree vect_halve_mask_nunits (tree, machine_mode);
extern tree vect_double_mask_nunits (tree, machine_mode);
-extern void vect_record_loop_mask (loop_vec_info, vec_loop_masks *,
- unsigned int, tree, tree);
-extern tree vect_get_loop_mask (loop_vec_info, gimple_stmt_iterator *,
- vec_loop_masks *,
- unsigned int, tree, unsigned int);
-extern void vect_record_loop_len (loop_vec_info, vec_loop_lens *, unsigned int,
- tree, unsigned int);
-extern tree vect_get_loop_len (loop_vec_info, gimple_stmt_iterator *,
- vec_loop_lens *, unsigned int, tree,
- unsigned int, unsigned int, bool);
+extern void vect_record_mask (vec_info *, unsigned int, tree, tree,
+ slp_tree = nullptr);
+extern tree vect_get_mask (vec_info *, gimple_stmt_iterator *, unsigned int,
+ tree, unsigned int, slp_tree = nullptr);
+extern void vect_record_len (vec_info *, unsigned int, tree, unsigned int,
+ slp_tree = nullptr);
+extern tree vect_get_len (vec_info *, gimple_stmt_iterator *, unsigned int,
+ tree, unsigned int, unsigned int, bool,
+ slp_tree = nullptr);
extern tree vect_gen_loop_len_mask (loop_vec_info, gimple_stmt_iterator *,
- gimple_stmt_iterator *, vec_loop_lens *,
- unsigned int, tree, tree, unsigned int,
- unsigned int);
+ gimple_stmt_iterator *, unsigned int, tree,
+ tree, unsigned int, unsigned int);
extern gimple_seq vect_gen_len (tree, tree, tree, tree);
extern vect_reduc_info info_for_reduction (loop_vec_info, slp_tree);
extern bool reduction_fn_for_scalar_code (code_helper, internal_fn *);
@@ -2800,7 +2806,7 @@ extern void vect_free_slp_tree (slp_tree);
extern bool compatible_calls_p (gcall *, gcall *, bool);
extern int vect_slp_child_index_for_operand (const stmt_vec_info, int op);
-extern tree prepare_vec_mask (loop_vec_info, tree, tree, tree,
+extern tree prepare_vec_mask (vec_info *, tree, tree, tree,
gimple_stmt_iterator *);
extern tree vect_get_mask_load_else (int, tree);
extern bool vect_load_perm_consecutive_p (slp_tree, unsigned = UINT_MAX);
--
2.43.0
