Calls to vect_(get|record)_loop_(mask|len) are replaced
with calls to new wrappers that have an extra (SLP node)
parameter and which can operate on any vec_info, not just
a loop_vec_info. These wrappers pass calls through to the
original functions (and ignore the SLP node) when invoked
with a loop_vec_info; otherwise, vect_record_(len|mask)
do nothing (for now).
Direct use of LOOP_VINFO_FULLY_(MASKED|WITH_LENGTH)_P,
and LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P is replaced
with wrappers named vect_(fully_masked|fully_with_length)_p,
vect_can_use_partial_vectors_p (for queries) and
vect_cannot_use_partial_vectors (for updates). For BB
SLP, vect_cannot_use_partial_vectors does nothing and
the getters return false (for now).
To minimize code churn, the new wrappers are only used in
code shared between loop vectorization and BB SLP.
gcc/ChangeLog:
* tree-vect-stmts.cc (vect_record_mask): New function
that wraps calls to vect_record_loop_mask.
(vect_get_mask): New function that wraps calls to
vect_get_loop_mask.
(vect_record_len): New function that wraps calls to
vect_record_loop_len.
(vect_get_len): New function that wraps calls to
vect_get_loop_len.
(check_load_store_for_partial_vectors): Substitute a parameter
of type vec_info * for loop_vec_info.
Use vect_record_len instead of vect_record_loop_len.
Use vect_record_mask instead of vect_record_loop_mask.
Use vect_cannot_use_partial_vectors instead of
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
Delete local aliases for LOOP_VINFO_MASKS and LOOP_VINFO_LENS.
(prepare_vec_mask): Substitute a parameter of type vec_info *
for loop_vec_info. Rename the loop_mask parameter.
(vectorizable_call): Delete local aliases for LOOP_VINFO_MASKS
and LOOP_VINFO_LENS.
Rename masked_loop_p as masked_p and remove requirement for
non-null loop_vinfo.
Rename len_loop_p as len_p and remove requirement for non-null
loop_vinfo.
Use vect_cannot_use_partial_vectors instead of
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
Use vect_record_len instead of vect_record_loop_len.
Use vect_record_mask instead of vect_record_loop_mask.
Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P.
Use vect_fully_with_length_p instead of
LOOP_VINFO_FULLY_WITH_LENGTH_P.
Use vect_get_mask instead of vect_get_loop_mask.
Use vect_get_len instead of vect_get_loop_len.
Pass vec_info * instead of loop_vec_info to prepare_vec_mask.
(vectorizable_simd_clone_call): Use
vect_can_use_partial_vectors_p and
vect_cannot_use_partial_vectors instead of
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
Use vect_record_mask instead of vect_record_loop_mask.
Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P.
Use vect_get_mask instead of vect_get_loop_mask.
Pass vec_info * instead of loop_vec_info to prepare_vec_mask.
Delete local aliases for LOOP_VINFO_MASKS.
(vectorizable_conversion): Use
vect_can_use_partial_vectors_p and
vect_cannot_use_partial_vectors instead of
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
(vectorizable_operation): Delete local alias for
LOOP_VINFO_LENS.
Rename masked_loop_p as masked_p and remove requirement for
non-null loop_vinfo.
Rename len_loop_p as len_p and remove requirement for non-null
loop_vinfo.
Use vect_can_use_partial_vectors_p and
vect_cannot_use_partial_vectors instead of
LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
Use vect_record_len instead of vect_record_loop_len.
Use vect_record_mask instead of vect_record_loop_mask.
Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P.
Use vect_fully_with_length_p instead of
LOOP_VINFO_FULLY_WITH_LENGTH_P.
Use vect_get_mask instead of vect_get_loop_mask.
Use vect_get_len instead of vect_get_loop_len.
(vectorizable_store): Use vect_can_use_partial_vectors_p
instead of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P.
Pass vec_info * instead of loop_vec_info to
check_load_store_for_partial_vectors.
Delete local alias for LOOP_VINFO_MASKS.
Use vect_fully_masked_p instead of LOOP_VINFO_FULLY_MASKED_P.
Use vect_fully_with_length_p instead of
LOOP_VINFO_FULLY_WITH_LENGTH_P.
Pass vec_info * instead of loop_vec_info to prepare_vec_mask.
(vectorizable_load): As above.
* tree-vectorizer.h (class vec_info): Moved declaration of
the vec_cond_masked_set member from class _loop_vec_info.
(class _loop_vec_info): Moved declaration of the
vec_cond_masked_set member to class vec_info.
(prepare_vec_mask): Substitute a parameter of type vec_info *
for loop_vec_info.
(vect_can_use_partial_vectors_p): New function that wraps use
of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P as a getter.
(vect_cannot_use_partial_vectors): New function that wraps use
of LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P to clear the flag.
(vect_fully_with_length_p): New function that wraps use of
LOOP_VINFO_FULLY_WITH_LENGTH_P.
(vect_fully_masked_p): New function that wraps use of
LOOP_VINFO_FULLY_MASKED_P.
---
gcc/tree-vect-stmts.cc | 394 ++++++++++++++++++++++-------------------
gcc/tree-vectorizer.h | 74 +++++++-
2 files changed, 284 insertions(+), 184 deletions(-)
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 748b3bcb0ab..2c0d3404a69 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -1405,12 +1405,88 @@ vectorizable_internal_function (combined_fn cfn, tree
fndecl,
return IFN_LAST;
}
+/* Record that a complete set of masks associated with VINFO would need to
+ contain a sequence of NVECTORS masks that each control a vector of type
+ VECTYPE. If SCALAR_MASK is nonnull, the fully-masked loop would AND
+ these vector masks with the vector version of SCALAR_MASK. */
+static void
+vect_record_mask (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors,
+ tree vectype, tree scalar_mask)
+{
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ vect_record_loop_mask (loop_vinfo, &LOOP_VINFO_MASKS (loop_vinfo),
nvectors,
+ vectype, scalar_mask);
+ else
+ (void) slp_node; /* FORNOW */
+}
+
+/* Given a complete set of masks associated with VINFO, extract mask number
+ INDEX for an rgroup that operates on NVECTORS vectors of type VECTYPE, where
+ 0 <= INDEX < NVECTORS. Alternatively, if doing basic block vectorization,
+ materialize an equivalent mask for SLP_NODE. Insert any set-up statements
+ before GSI. */
+static tree
+vect_get_mask (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi,
+ unsigned int nvectors, tree vectype, unsigned int index)
+{
+ gcc_assert (vect_fully_masked_p (vinfo, slp_node));
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ return vect_get_loop_mask (loop_vinfo, gsi, &LOOP_VINFO_MASKS (loop_vinfo),
+ nvectors, vectype, index);
+ else
+ {
+ (void) slp_node; /* FORNOW */
+ return NULL_TREE;
+ }
+}
+
+/* Record that a complete set of lengths associated with VINFO would need to
+ contain a sequence of NVECTORS lengths for controlling an operation on
+ VECTYPE. The operation splits each element of VECTYPE into FACTOR separate
+ subelements, measuring the length as a number of these subelements. */
+static void
+vect_record_len (vec_info *vinfo, slp_tree slp_node, unsigned int nvectors,
+ tree vectype, unsigned int factor)
+{
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ vect_record_loop_len (loop_vinfo, &LOOP_VINFO_LENS (loop_vinfo), nvectors,
+ vectype, factor);
+ else
+ (void) slp_node; /* FORNOW */
+}
+
+/* Given a complete set of lengths associated with VINFO, extract length number
+ INDEX for an rgroup that operates on NVECTORS vectors of type VECTYPE, where
+ 0 <= INDEX < NVECTORS. Alternatively, if doing basic block vectorization,
+ materialize an equivalent length for SLP_NODE. Return a value that contains
+ FACTOR multiplied by the number of elements that should be processed.
Insert
+ any set-up statements before GSI. */
+
+static tree
+vect_get_len (vec_info *vinfo, slp_tree slp_node, gimple_stmt_iterator *gsi,
+ unsigned int nvectors, tree vectype, unsigned int index,
+ unsigned int factor)
+{
+ gcc_assert (vect_fully_with_length_p (vinfo, slp_node));
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ return vect_get_loop_len (loop_vinfo, gsi, &LOOP_VINFO_LENS (loop_vinfo),
+ nvectors, vectype, index, factor);
+ else
+ {
+ (void) slp_node; /* FORNOW */
+ return NULL_TREE;
+ }
+}
static tree permute_vec_elements (vec_info *, tree, tree, tree, stmt_vec_info,
gimple_stmt_iterator *);
-/* Check whether a load or store statement in the loop described by
- LOOP_VINFO is possible in a loop using partial vectors. This is
+/* Check whether a load or store statement in the loop or SLP group described
by
+ VINFO is possible using partial vectors. This is
testing whether the vectorizer pass has the appropriate support,
as well as whether the target does.
@@ -1425,13 +1501,14 @@ static tree permute_vec_elements (vec_info *, tree,
tree, tree, stmt_vec_info,
Clear LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P if a loop using partial
vectors is not supported, otherwise record the required rgroup control
- types.
+ types. For basic block SLP, simply record which style of partial vectors
+ is available (if any).
If partial vectors can be used and ELSVALS is nonzero the supported
else values will be added to the vector ELSVALS points to. */
static void
-check_load_store_for_partial_vectors (loop_vec_info loop_vinfo, tree vectype,
+check_load_store_for_partial_vectors (vec_info *vinfo, tree vectype,
slp_tree slp_node,
vec_load_store_type vls_type,
int group_size,
@@ -1439,6 +1516,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
slp_tree mask_node,
vec<int> *elsvals = nullptr)
{
+ loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
vect_memory_access_type memory_access_type = ls->memory_access_type;
/* Invariant loads need no special support. */
@@ -1462,9 +1540,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
scalar_mask = def;
}
- unsigned int nvectors = vect_get_num_copies (loop_vinfo, slp_node);
- vec_loop_masks *masks = &LOOP_VINFO_MASKS (loop_vinfo);
- vec_loop_lens *lens = &LOOP_VINFO_LENS (loop_vinfo);
+ unsigned int nvectors = vect_get_num_copies (vinfo, slp_node);
machine_mode vecmode = TYPE_MODE (vectype);
bool is_load = (vls_type == VLS_LOAD);
if (memory_access_type == VMAT_LOAD_STORE_LANES)
@@ -1475,10 +1551,10 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
elsvals)
: vect_store_lanes_supported (vectype, group_size, true));
if (ifn == IFN_MASK_LEN_LOAD_LANES || ifn == IFN_MASK_LEN_STORE_LANES)
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
+ vect_record_len (vinfo, slp_node, nvectors, vectype, 1);
else if (ifn == IFN_MASK_LOAD_LANES || ifn == IFN_MASK_STORE_LANES)
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
- scalar_mask);
+ vect_record_mask (vinfo, slp_node, nvectors, vectype,
+ scalar_mask);
else
{
if (dump_enabled_p ())
@@ -1486,7 +1562,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
"can't operate on partial vectors because"
" the target doesn't have an appropriate"
" load/store-lanes instruction.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
return;
}
@@ -1520,14 +1596,13 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
memory_type,
off_vectype, scale,
elsvals))
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, 1);
+ vect_record_len (vinfo, slp_node, nvectors, vectype, 1);
else if (internal_gather_scatter_fn_supported_p (ifn, vectype,
memory_type,
off_vectype, scale,
elsvals)
|| memory_access_type == VMAT_GATHER_SCATTER_LEGACY)
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
- scalar_mask);
+ vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask);
else
{
if (dump_enabled_p ())
@@ -1535,7 +1610,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
"can't operate on partial vectors because"
" the target doesn't have an appropriate"
" gather load or scatter store instruction.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
return;
}
@@ -1548,7 +1623,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"can't operate on partial vectors because an"
" access isn't contiguous.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
return;
}
@@ -1558,7 +1633,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"can't operate on partial vectors when emulating"
" vector operations.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
return;
}
@@ -1583,7 +1658,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
{
nvectors = group_memory_nvectors (group_size * vf, nunits);
unsigned factor = (vecmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vecmode);
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype, factor);
+ vect_record_len (vinfo, slp_node, nvectors, vectype, factor);
using_partial_vectors_p = true;
}
else if (targetm.vectorize.get_mask_mode (vecmode).exists (&mask_mode)
@@ -1591,7 +1666,7 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
elsvals))
{
nvectors = group_memory_nvectors (group_size * vf, nunits);
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype,
scalar_mask);
+ vect_record_mask (vinfo, slp_node, nvectors, vectype, scalar_mask);
using_partial_vectors_p = true;
}
@@ -1602,35 +1677,35 @@ check_load_store_for_partial_vectors (loop_vec_info
loop_vinfo, tree vectype,
"can't operate on partial vectors because the"
" target doesn't have the appropriate partial"
" vectorization load or store.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
}
/* Return the mask input to a masked load or store. VEC_MASK is the vectorized
- form of the scalar mask condition and LOOP_MASK, if nonnull, is the mask
- that needs to be applied to all loads and stores in a vectorized loop.
- Return VEC_MASK if LOOP_MASK is null or if VEC_MASK is already masked,
- otherwise return VEC_MASK & LOOP_MASK.
+ form of the scalar mask condition and LOOP_OR_TAIL_MASK, if nonnull, is the
+ mask that needs to be applied to all loads and stores in a vectorized loop.
+ Return VEC_MASK if LOOP_OR_TAIL_MASK is null or if VEC_MASK is already
+ masked, otherwise return VEC_MASK & LOOP_OR_TAIL_MASK.
MASK_TYPE is the type of both masks. If new statements are needed,
insert them before GSI. */
tree
-prepare_vec_mask (loop_vec_info loop_vinfo, tree mask_type, tree loop_mask,
+prepare_vec_mask (vec_info *vinfo, tree mask_type, tree loop_or_tail_mask,
tree vec_mask, gimple_stmt_iterator *gsi)
{
gcc_assert (useless_type_conversion_p (mask_type, TREE_TYPE (vec_mask)));
- if (!loop_mask)
+ if (!loop_or_tail_mask)
return vec_mask;
- gcc_assert (TREE_TYPE (loop_mask) == mask_type);
+ gcc_assert (TREE_TYPE (loop_or_tail_mask) == mask_type);
- if (loop_vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_mask }))
+ if (vinfo->vec_cond_masked_set.contains ({ vec_mask, loop_or_tail_mask }))
return vec_mask;
tree and_res = make_temp_ssa_name (mask_type, NULL, "vec_mask_and");
- gimple *and_stmt = gimple_build_assign (and_res, BIT_AND_EXPR,
- vec_mask, loop_mask);
+ gimple *and_stmt
+ = gimple_build_assign (and_res, BIT_AND_EXPR, vec_mask, loop_or_tail_mask);
gsi_insert_before (gsi, and_stmt, GSI_SAME_STMT);
return and_res;
@@ -3692,8 +3767,6 @@ vectorizable_call (vec_info *vinfo,
? ifn : get_conditional_internal_fn (ifn));
internal_fn cond_len_fn = get_len_internal_fn (ifn);
int len_opno = internal_fn_len_index (cond_len_fn);
- vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
- vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
unsigned int nvectors = vect_get_num_copies (vinfo, slp_node);
if (cost_vec) /* transformation not required. */
{
@@ -3711,8 +3784,7 @@ vectorizable_call (vec_info *vinfo,
DUMP_VECT_SCOPE ("vectorizable_call");
vect_model_simple_cost (vinfo, 1, slp_node, cost_vec);
- if (loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
+ if (vect_can_use_partial_vectors_p (vinfo, slp_node)
&& (reduc_idx >= 0 || mask_opno >= 0))
{
if (reduc_idx >= 0
@@ -3727,7 +3799,7 @@ vectorizable_call (vec_info *vinfo,
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"can't use a fully-masked loop because no"
" conditional operation is available.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
else
{
@@ -3737,11 +3809,10 @@ vectorizable_call (vec_info *vinfo,
if (cond_len_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_len_fn, vectype_out,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo, lens, nvectors, vectype_out,
- 1);
+ vect_record_len (vinfo, slp_node, nvectors, vectype_out, 1);
else
- vect_record_loop_mask (loop_vinfo, masks, nvectors, vectype_out,
- scalar_mask);
+ vect_record_mask (vinfo, slp_node, nvectors, vectype_out,
+ scalar_mask);
}
}
return true;
@@ -3756,10 +3827,10 @@ vectorizable_call (vec_info *vinfo,
scalar_dest = gimple_call_lhs (stmt);
vec_dest = vect_create_destination_var (scalar_dest, vectype_out);
- bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
- bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
+ bool masked_p = vect_fully_masked_p (vinfo, slp_node);
+ bool len_p = vect_fully_with_length_p (vinfo, slp_node);
unsigned int vect_nargs = nargs;
- if (len_loop_p)
+ if (len_p)
{
if (len_opno >= 0)
{
@@ -3770,7 +3841,7 @@ vectorizable_call (vec_info *vinfo,
else if (reduc_idx >= 0)
gcc_unreachable ();
}
- else if (masked_loop_p && mask_opno == -1 && reduc_idx >= 0)
+ else if (masked_p && mask_opno == -1 && reduc_idx >= 0)
{
ifn = cond_fn;
vect_nargs += 2;
@@ -3814,12 +3885,12 @@ vectorizable_call (vec_info *vinfo,
{
int varg = 0;
/* Add the mask if necessary. */
- if (masked_loop_p && mask_opno == -1 && reduc_idx >= 0)
+ if (masked_p && mask_opno == -1 && reduc_idx >= 0)
{
gcc_assert (internal_fn_mask_index (ifn) == varg);
unsigned int vec_num = vec_oprnds0.length ();
- vargs[varg++] = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_out, i);
+ vargs[varg++] = vect_get_mask (vinfo, slp_node, gsi, vec_num,
+ vectype_out, i);
}
size_t k;
for (k = 0; k < nargs; k++)
@@ -3828,7 +3899,7 @@ vectorizable_call (vec_info *vinfo,
vargs[varg++] = vec_oprndsk[i];
}
/* Add the else value if necessary. */
- if (masked_loop_p && mask_opno == -1 && reduc_idx >= 0)
+ if (masked_p && mask_opno == -1 && reduc_idx >= 0)
{
gcc_assert (internal_fn_else_index (ifn) == varg);
vargs[varg++] = vargs[reduc_idx + 1];
@@ -3859,24 +3930,26 @@ vectorizable_call (vec_info *vinfo,
}
else
{
- if (len_opno >= 0 && len_loop_p)
+ if (len_opno >= 0 && len_p)
{
unsigned int vec_num = vec_oprnds0.length ();
- tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
- vec_num, vectype_out, i, 1);
+ tree len = vect_get_len (vinfo, slp_node, gsi, vec_num,
+ vectype_out, i, 1);
signed char biasval
- = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
+ = loop_vinfo
+ ? LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo)
+ : 0;
tree bias = build_int_cst (intQI_type_node, biasval);
vargs[len_opno] = len;
vargs[len_opno + 1] = bias;
}
- else if (mask_opno >= 0 && masked_loop_p)
+ else if (mask_opno >= 0 && masked_p)
{
unsigned int vec_num = vec_oprnds0.length ();
- tree mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype_out, i);
+ tree mask = vect_get_mask (vinfo, slp_node, gsi, vec_num,
+ vectype_out, i);
vargs[mask_opno]
- = prepare_vec_mask (loop_vinfo, TREE_TYPE (mask), mask,
+ = prepare_vec_mask (vinfo, TREE_TYPE (mask), mask,
vargs[mask_opno], gsi);
}
@@ -4489,8 +4562,7 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
}
break;
case SIMD_CLONE_ARG_TYPE_MASK:
- if (loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ if (vect_can_use_partial_vectors_p (vinfo, slp_node))
{
tree arg_vectype;
if (SCALAR_INT_MODE_P
@@ -4500,23 +4572,21 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
TYPE_MODE (bestn->simdclone->args[i].vector_type));
else
arg_vectype = bestn->simdclone->args[i].vector_type;
- vect_record_loop_mask (loop_vinfo,
- &LOOP_VINFO_MASKS (loop_vinfo),
- ncopies * num_mask_args, arg_vectype,
- op);
+ vect_record_mask (vinfo, slp_node, ncopies * num_mask_args,
+ arg_vectype, op);
}
break;
}
}
- if (!bestn->simdclone->inbranch && loop_vinfo)
+ if (!bestn->simdclone->inbranch)
{
if (dump_enabled_p ()
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ && vect_can_use_partial_vectors_p (vinfo, slp_node))
dump_printf_loc (MSG_NOTE, vect_location,
"can't use a fully-masked loop because a"
" non-masked simd clone was selected.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
SLP_TREE_TYPE (slp_node) = call_simd_clone_vec_info_type;
@@ -4687,23 +4757,15 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
if (m == 0)
vec_oprnds_i[i] = 0;
vec_oprnd0 = vec_oprnds[i][vec_oprnds_i[i]++];
- if (loop_vinfo
- && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
+ if (vect_fully_masked_p (vinfo, slp_node))
{
- vec_loop_masks *loop_masks
- = &LOOP_VINFO_MASKS (loop_vinfo);
- tree loop_mask
- = vect_get_loop_mask (loop_vinfo, gsi,
- loop_masks, ncopies,
- vectype, j);
+ tree mask = vect_get_mask (vinfo, slp_node, gsi,
+ ncopies, vectype, j);
vec_oprnd0
- = prepare_vec_mask (loop_vinfo,
- TREE_TYPE (loop_mask),
- loop_mask, vec_oprnd0,
- gsi);
- loop_vinfo->vec_cond_masked_set.add ({ vec_oprnd0,
- loop_mask
});
-
+ = prepare_vec_mask (vinfo, TREE_TYPE (mask),
+ mask, vec_oprnd0, gsi);
+ vinfo->vec_cond_masked_set.add (
+ {vec_oprnd0, mask});
}
vec_oprnd0
= build3 (VEC_COND_EXPR, atype, vec_oprnd0,
@@ -4881,12 +4943,9 @@ vectorizable_simd_clone_call (vec_info *vinfo,
stmt_vec_info stmt_info,
o = vector_unroll_factor (nunits, callee_nelements);
for (m = j * o; m < (j + 1) * o; m++)
{
- if (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo))
- {
- vec_loop_masks *loop_masks = &LOOP_VINFO_MASKS (loop_vinfo);
- mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies, masktype, j);
- }
+ if (vect_fully_masked_p (vinfo, slp_node))
+ mask
+ = vect_get_mask (vinfo, slp_node, gsi, ncopies, masktype, j);
else
mask = vect_build_all_ones_mask (vinfo, stmt_info, masktype);
@@ -5297,7 +5356,6 @@ vectorizable_conversion (vec_info *vinfo,
vec<tree> vec_oprnds1 = vNULL;
tree vop0;
bb_vec_info bb_vinfo = dyn_cast <bb_vec_info> (vinfo);
- loop_vec_info loop_vinfo = dyn_cast <loop_vec_info> (vinfo);
int multi_step_cvt = 0;
vec<tree> interm_types = vNULL;
tree intermediate_type, cvt_type = NULL_TREE;
@@ -5641,8 +5699,7 @@ vectorizable_conversion (vec_info *vinfo,
}
if (modifier == WIDEN
- && loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
+ && vect_can_use_partial_vectors_p (vinfo, slp_node)
&& (code1 == VEC_WIDEN_MULT_EVEN_EXPR
|| widening_evenodd_fn_p (code1)))
{
@@ -5651,7 +5708,7 @@ vectorizable_conversion (vec_info *vinfo,
"can't use a fully-masked loop because"
" widening operation on even/odd elements"
" mixes up lanes.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
if (cost_vec) /* transformation not required. */
@@ -6731,7 +6788,6 @@ vectorizable_operation (vec_info *vinfo,
int reduc_idx = SLP_TREE_REDUC_IDX (slp_node);
vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
- vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
internal_fn cond_fn = get_conditional_internal_fn (code);
internal_fn cond_len_fn = get_conditional_len_internal_fn (code);
@@ -6748,27 +6804,24 @@ vectorizable_operation (vec_info *vinfo,
if (cost_vec) /* transformation not required. */
{
- if (loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
+ if (vect_can_use_partial_vectors_p (vinfo, slp_node)
&& mask_out_inactive)
{
if (cond_len_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_len_fn, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_len (loop_vinfo, lens, vec_num, vectype,
- 1);
+ vect_record_len (vinfo, slp_node, vec_num, vectype, 1);
else if (cond_fn != IFN_LAST
&& direct_internal_fn_supported_p (cond_fn, vectype,
OPTIMIZE_FOR_SPEED))
- vect_record_loop_mask (loop_vinfo, masks, vec_num,
- vectype, NULL);
+ vect_record_mask (vinfo, slp_node, vec_num, vectype, NULL);
else
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"can't use a fully-masked loop because no"
" conditional operation is available.\n");
- LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ vect_cannot_use_partial_vectors (vinfo, slp_node);
}
}
@@ -6828,8 +6881,8 @@ vectorizable_operation (vec_info *vinfo,
dump_printf_loc (MSG_NOTE, vect_location,
"transform binary/unary operation.\n");
- bool masked_loop_p = loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
- bool len_loop_p = loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
+ bool masked_p = vect_fully_masked_p (vinfo, slp_node);
+ bool len_p = vect_fully_with_length_p (vinfo, slp_node);
/* POINTER_DIFF_EXPR has pointer arguments which are vectorized as
vectors with unsigned elements, but the result is signed. So, we
@@ -7017,12 +7070,11 @@ vectorizable_operation (vec_info *vinfo,
gimple_assign_set_lhs (new_stmt, new_temp);
vect_finish_stmt_generation (vinfo, stmt_info, new_stmt, gsi);
}
- else if ((masked_loop_p || len_loop_p) && mask_out_inactive)
+ else if ((masked_p || len_p) && mask_out_inactive)
{
tree mask;
- if (masked_loop_p)
- mask = vect_get_loop_mask (loop_vinfo, gsi, masks,
- vec_num, vectype, i);
+ if (masked_p)
+ mask = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i);
else
/* Dummy mask. */
mask = build_minus_one_cst (truth_type_for (vectype));
@@ -7046,19 +7098,19 @@ vectorizable_operation (vec_info *vinfo,
(cond_fn, vectype, vops.length () - 1, &vops[1]);
vops.quick_push (else_value);
}
- if (len_loop_p)
+ if (len_p)
{
- tree len = vect_get_loop_len (loop_vinfo, gsi, lens,
- vec_num, vectype, i, 1);
+ tree len
+ = vect_get_len (vinfo, slp_node, gsi, vec_num, vectype, i, 1);
signed char biasval
- = LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo);
+ = loop_vinfo ? LOOP_VINFO_PARTIAL_LOAD_STORE_BIAS (loop_vinfo)
+ : 0;
tree bias = build_int_cst (intQI_type_node, biasval);
vops.quick_push (len);
vops.quick_push (bias);
}
gcall *call
- = gimple_build_call_internal_vec (masked_loop_p ? cond_fn
- : cond_len_fn,
+ = gimple_build_call_internal_vec (masked_p ? cond_fn : cond_len_fn,
vops);
new_temp = make_ssa_name (vec_dest, call);
gimple_call_set_lhs (call, new_temp);
@@ -7072,8 +7124,7 @@ vectorizable_operation (vec_info *vinfo,
/* When combining two masks check if either of them is elsewhere
combined with a loop mask, if that's the case we can mark that the
new combined mask doesn't need to be combined with a loop mask. */
- if (masked_loop_p
- && code == BIT_AND_EXPR
+ if (loop_vinfo && masked_p && code == BIT_AND_EXPR
&& VECTOR_BOOLEAN_TYPE_P (vectype))
{
if (loop_vinfo->scalar_cond_masked_set.contains ({ op0, vec_num
}))
@@ -7105,7 +7156,7 @@ vectorizable_operation (vec_info *vinfo,
/* Enter the combined value into the vector cond hash so we don't
AND it with a loop mask again. */
if (mask)
- loop_vinfo->vec_cond_masked_set.add ({ new_temp, mask });
+ vinfo->vec_cond_masked_set.add ({ new_temp, mask });
}
if (vec_cvt_dest)
@@ -8241,9 +8292,8 @@ vectorizable_store (vec_info *vinfo,
bool costing_p = cost_vec;
if (costing_p) /* transformation not required. */
{
- if (loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
- check_load_store_for_partial_vectors (loop_vinfo, vectype, slp_node,
+ if (vect_can_use_partial_vectors_p (vinfo, slp_node))
+ check_load_store_for_partial_vectors (vinfo, vectype, slp_node,
vls_type, group_size, &ls,
mask_node);
@@ -8596,10 +8646,6 @@ vectorizable_store (vec_info *vinfo,
}
gcc_assert (alignment_support_scheme);
- vec_loop_masks *loop_masks
- = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
- ? &LOOP_VINFO_MASKS (loop_vinfo)
- : NULL);
vec_loop_lens *loop_lens
= (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
? &LOOP_VINFO_LENS (loop_vinfo)
@@ -8611,14 +8657,14 @@ vectorizable_store (vec_info *vinfo,
Shouldn't go with length-based approach if fully masked. */
if (cost_vec == NULL)
/* The cost_vec is NULL during transfrom. */
- gcc_assert ((!loop_lens || !loop_masks));
+ gcc_assert ((!vect_fully_with_length_p (vinfo, slp_node)
+ || !vect_fully_masked_p (vinfo, slp_node)));
/* Targets with store-lane instructions must not require explicit
realignment. vect_supportable_dr_alignment always returns either
dr_aligned or dr_unaligned_supported for masked operations. */
- gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
- && !mask_node
- && !loop_masks)
+ gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES && !mask_node
+ && !vect_fully_masked_p (vinfo, slp_node))
|| alignment_support_scheme == dr_aligned
|| alignment_support_scheme == dr_unaligned_supported);
@@ -8748,18 +8794,18 @@ vectorizable_store (vec_info *vinfo,
tree final_mask = NULL;
tree final_len = NULL;
tree bias = NULL;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies, vectype, j);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask
+ = vect_get_mask (vinfo, slp_node, gsi, ncopies, vectype, j);
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask,
vec_mask, gsi);
if (lanes_ifn == IFN_MASK_LEN_STORE_LANES)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- ncopies, vectype, j, 1);
+ if (vect_fully_with_length_p (vinfo, slp_node))
+ final_len
+ = vect_get_len (vinfo, slp_node, gsi, ncopies, vectype, j, 1);
else
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
signed char biasval
@@ -8887,12 +8933,11 @@ vectorizable_store (vec_info *vinfo,
tree bias = NULL_TREE;
if (!costing_p)
{
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi,
- loop_masks, num_stmts,
- vectype, j);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask
+ = vect_get_mask (vinfo, slp_node, gsi, num_stmts, vectype, j);
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype,
final_mask, vec_mask, gsi);
}
@@ -8950,9 +8995,8 @@ vectorizable_store (vec_info *vinfo,
if (ls.gs.ifn == IFN_MASK_LEN_SCATTER_STORE)
{
if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi,
- loop_lens, num_stmts,
- vectype, j, 1);
+ final_len = vect_get_len (vinfo, slp_node, gsi, num_stmts,
+ vectype, j, 1);
else
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
@@ -9237,9 +9281,7 @@ vectorizable_store (vec_info *vinfo,
|| useless_type_conversion_p (vectype, TREE_TYPE (vec_oprnd)));
bool simd_lane_access_p
= STMT_VINFO_SIMD_LANE_ACCESS_P (stmt_info) != 0;
- if (!costing_p
- && simd_lane_access_p
- && !loop_masks
+ if (!costing_p && simd_lane_access_p && !vect_fully_masked_p (vinfo,
slp_node)
&& TREE_CODE (DR_BASE_ADDRESS (first_dr_info->dr)) == ADDR_EXPR
&& VAR_P (TREE_OPERAND (DR_BASE_ADDRESS (first_dr_info->dr), 0))
&& integer_zerop (get_dr_vinfo_offset (vinfo, first_dr_info))
@@ -9293,13 +9335,12 @@ vectorizable_store (vec_info *vinfo,
tree final_mask = NULL_TREE;
tree final_len = NULL_TREE;
tree bias = NULL_TREE;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i);
if (vec_mask)
vec_mask = vec_masks[i];
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask,
vec_mask, gsi);
if (i > 0)
@@ -9335,8 +9376,8 @@ vectorizable_store (vec_info *vinfo,
new_vmode = new_ovmode.require ();
unsigned factor
= (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, factor);
+ final_len
+ = vect_get_len (vinfo, slp_node, gsi, vec_num, vectype, i, factor);
}
else if (final_mask)
{
@@ -9859,9 +9900,8 @@ vectorizable_load (vec_info *vinfo,
return false;
}
- if (loop_vinfo
- && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
- check_load_store_for_partial_vectors (loop_vinfo, vectype, slp_node,
+ if (vect_can_use_partial_vectors_p (vinfo, slp_node))
+ check_load_store_for_partial_vectors (vinfo, vectype, slp_node,
VLS_LOAD, group_size, &ls,
mask_node, &ls.elsvals);
@@ -10439,14 +10479,10 @@ vectorizable_load (vec_info *vinfo,
ref_type = reference_alias_ptr_type (DR_REF (first_dr_info->dr));
}
- vec_loop_masks *loop_masks
- = (loop_vinfo && LOOP_VINFO_FULLY_MASKED_P (loop_vinfo)
- ? &LOOP_VINFO_MASKS (loop_vinfo)
- : NULL);
vec_loop_lens *loop_lens
= (loop_vinfo && LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo)
- ? &LOOP_VINFO_LENS (loop_vinfo)
- : NULL);
+ ? &LOOP_VINFO_LENS (loop_vinfo)
+ : NULL);
/* The vect_transform_stmt and vect_analyze_stmt will go here but there
are some difference here. We cannot enable both the lens and masks
@@ -10454,15 +10490,15 @@ vectorizable_load (vec_info *vinfo,
Shouldn't go with length-based approach if fully masked. */
if (cost_vec == NULL)
/* The cost_vec is NULL during transfrom. */
- gcc_assert ((!loop_lens || !loop_masks));
+ gcc_assert ((!vect_fully_with_length_p (vinfo, slp_node)
+ || !vect_fully_masked_p (vinfo, slp_node)));
/* Targets with store-lane instructions must not require explicit
realignment. vect_supportable_dr_alignment always returns either
dr_aligned or dr_unaligned_supported for (non-length) masked
operations. */
- gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES
- && !mask_node
- && !loop_masks)
+ gcc_assert ((memory_access_type != VMAT_LOAD_STORE_LANES && !mask_node
+ && !vect_fully_masked_p (vinfo, slp_node))
|| mat_gather_scatter_p (memory_access_type)
|| alignment_support_scheme == dr_aligned
|| alignment_support_scheme == dr_unaligned_supported);
@@ -10661,18 +10697,18 @@ vectorizable_load (vec_info *vinfo,
tree final_mask = NULL_TREE;
tree final_len = NULL_TREE;
tree bias = NULL_TREE;
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- ncopies, vectype, j);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask
+ = vect_get_mask (vinfo, slp_node, gsi, ncopies, vectype, j);
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype, final_mask,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype, final_mask,
vec_mask, gsi);
if (lanes_ifn == IFN_MASK_LEN_LOAD_LANES)
{
- if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- ncopies, vectype, j, 1);
+ if (vect_fully_with_length_p (vinfo, slp_node))
+ final_len
+ = vect_get_len (vinfo, slp_node, gsi, ncopies, vectype, j, 1);
else
final_len = size_int (TYPE_VECTOR_SUBPARTS (vectype));
signed char biasval
@@ -10808,11 +10844,11 @@ vectorizable_load (vec_info *vinfo,
{
if (mask_node)
vec_mask = vec_masks[i];
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask
+ = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i);
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype,
final_mask, vec_mask, gsi);
if (i > 0 && !STMT_VINFO_GATHER_SCATTER_P (stmt_info))
@@ -10874,8 +10910,8 @@ vectorizable_load (vec_info *vinfo,
if (ls.gs.ifn == IFN_MASK_LEN_GATHER_LOAD)
{
if (loop_lens)
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, 1);
+ final_len = vect_get_len (vinfo, slp_node, gsi, vec_num,
+ vectype, i, 1);
else
final_len = build_int_cst (sizetype,
TYPE_VECTOR_SUBPARTS (vectype));
@@ -11276,11 +11312,11 @@ vectorizable_load (vec_info *vinfo,
{
if (mask_node)
vec_mask = vec_masks[i];
- if (loop_masks)
- final_mask = vect_get_loop_mask (loop_vinfo, gsi, loop_masks,
- vec_num, vectype, i);
+ if (vect_fully_masked_p (vinfo, slp_node))
+ final_mask
+ = vect_get_mask (vinfo, slp_node, gsi, vec_num, vectype, i);
if (vec_mask)
- final_mask = prepare_vec_mask (loop_vinfo, mask_vectype,
+ final_mask = prepare_vec_mask (vinfo, mask_vectype,
final_mask, vec_mask, gsi);
if (i > 0)
@@ -11326,8 +11362,8 @@ vectorizable_load (vec_info *vinfo,
new_vmode = new_ovmode.require ();
unsigned factor
= (new_ovmode == vmode) ? 1 : GET_MODE_UNIT_SIZE (vmode);
- final_len = vect_get_loop_len (loop_vinfo, gsi, loop_lens,
- vec_num, vectype, i, factor);
+ final_len = vect_get_len (vinfo, slp_node, gsi, vec_num,
+ vectype, i, factor);
}
else if (final_mask)
{
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index eda5275586d..474b426e845 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -687,6 +687,9 @@ public:
and are reset when undoing patterns. */
gimple_seq inv_pattern_def_seq;
+ /* Set of vector conditions that have loop mask applied. */
+ vec_cond_masked_set_type vec_cond_masked_set;
+
private:
stmt_vec_info new_stmt_vec_info (gimple *stmt);
void set_vinfo_for_stmt (gimple *, stmt_vec_info, bool = true);
@@ -1018,9 +1021,6 @@ public:
/* Set of scalar conditions that have loop mask applied. */
scalar_cond_masked_set_type scalar_cond_masked_set;
- /* Set of vector conditions that have loop mask applied. */
- vec_cond_masked_set_type vec_cond_masked_set;
-
/* If we are using a loop mask to align memory addresses, this variable
contains the number of vector elements that we should skip in the
first iteration of the vector loop (i.e. the number of leading
@@ -2798,8 +2798,7 @@ extern slp_tree vect_create_new_slp_node (unsigned,
tree_code);
extern void vect_free_slp_tree (slp_tree);
extern bool compatible_calls_p (gcall *, gcall *, bool);
extern int vect_slp_child_index_for_operand (const gimple *, int op, bool);
-
-extern tree prepare_vec_mask (loop_vec_info, tree, tree, tree,
+extern tree prepare_vec_mask (vec_info *, tree, tree, tree,
gimple_stmt_iterator *);
extern tree vect_get_mask_load_else (int, tree);
extern bool vect_load_perm_consecutive_p (slp_tree, unsigned = 0);
@@ -2935,6 +2934,71 @@ vect_is_reduction (slp_tree slp_node)
return SLP_TREE_REDUC_IDX (slp_node) != -1;
}
+/* Return true if VINFO is vectorizer state for loop vectorization and we
+ still have the option of vectorizing the loop using partially-populated
+ vectors; or, true if VINFO is for basic-block vectorization and we still
have
+ the option of vectorizing the given SLP_NODE using partial vectors;
+ otherwise, return false. */
+inline bool
+vect_can_use_partial_vectors_p (vec_info *vinfo, slp_tree slp_node)
+{
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ return LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo);
+
+ (void) slp_node; /* FORNOW */
+ return false;
+}
+
+/* If VINFO is vectorizer state for loop vectorization then record that we no
+ longer have the option of vectorizing the loop using partially-populated
+ vectors; or, if VINFO is for basic-block vectorization then record that we
no
+ longer have the option of vectorizing the given SLP_NODE using partial
+ vectors. */
+inline void
+vect_cannot_use_partial_vectors (vec_info *vinfo, slp_tree slp_node)
+{
+ loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo);
+ if (loop_vinfo)
+ LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo) = false;
+ else
+ (void) slp_node; /* FORNOW */
+}
+
+/* Return true if VINFO is vectorizer state for loop vectorization, we've
+ decided to use partially-populated vectors so that the loop can handle fewer
+ than VF scalars, and at least one length has been recorded; or, if VINFO is
+ for basic-block vectorization and SLP_NODE will be vectorized by using
+ lengths to prevent use of inactive scalar lanes. */
+inline bool
+vect_fully_with_length_p (vec_info *vinfo, slp_tree slp_node)
+{
+ if (loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo))
+ return LOOP_VINFO_FULLY_WITH_LENGTH_P (loop_vinfo);
+ else
+ {
+ (void) slp_node; /* FORNOW */
+ return false;
+ }
+}
+
+/* Return true if VINFO is vectorizer state for loop vectorization, we've
+ decided to use partially-populated vectors so that the loop can handle fewer
+ than VF scalars, and at least one mask has been recorded; or, if VINFO is
+ for basic-block vectorization and SLP_NODE will be vectorized by using masks
+ to prevent use of inactive scalar lanes. */
+inline bool
+vect_fully_masked_p (vec_info *vinfo, slp_tree slp_node)
+{
+ if (loop_vec_info loop_vinfo = dyn_cast<loop_vec_info> (vinfo))
+ return LOOP_VINFO_FULLY_MASKED_P (loop_vinfo);
+ else
+ {
+ (void) slp_node; /* FORNOW */
+ return false;
+ }
+}
+
/* If STMT_INFO describes a reduction, return the vect_reduction_type
of the reduction it describes, otherwise return -1. */
inline int
--
2.43.0
