Hi,
I have been playing around with a vec_predicate RTX locally, moving parts of
the RVV backend over to it. Its main purpose is to lay out common operands in
a fixed order. This helps rtl passes to reason about the predicate as a whole,
as well as backends which needed to carry index information if they wanted to
access particular fields.
The staging I have in mind is roughly, each at least one patch:
(1) Add basic vec_predicate plumbing for non-"move" insns.
(2) Handle vec_predicate in simplify-rtx to allow canonicalization, merging,
and elision.
(3) Convert a self-contained subset of RVV instructions to the new format.
A vec_predicate would look like this:
(define_insn "pred_add"
[(set (match_operand:V 0 "register_operand" "=vd")
(vec_predicate:V plus
[(match_operand:V 3 "register_operand" " v")
(match_operand:V 4 "register_operand" " v")]
(match_operand:V 2 "else_operand" " velse")
(match_operand:<VM> 1 "mask_operand" " vm")
(match_operand 5 "vector_length_operand" " vl")
(const_int 0) # Bias
(match_operand 6 "const_int_operand" " i") # Length else
policy
(match_operand 7 "const_int_operand" " i")))]) # Mask else
policy
Unfortunately, there's no way to capture RVV's full capabilities without the
policy bits. With two masks, RVV, again unfortunately :), allows two different
values for inactive elements, none of them being "zero". Therefore, the
vectorizer, ifn, and optabs are going to need to query the policies rather than
the else operand for a vec_predicate. So even if the else operand is "real",
e.g. tail masking can still be something else.
As AVX and SVE don't need operands 5-7, and most autovec patterns might not
even need the else/mask, I added default-fill parsing in read-rtl: If a
vec_predicate ends prematurely, the remaining operands are filled with
"autovec" defaults like zero-fill policies, no bias, full length, etc.
One issue with default filling is the mode of else and mask. I hope that
can work if we use dedicated sentinels like VEC_UNDEF and VEC_ALL_TRUE
(which target predicates would need to allow)?
(define_insn "pred_add"
[(set (match_operand:V 0 "register_operand" "=vd")
(vec_predicate:V plus
[(match_operand:V 3 "register_operand" " v")
(match_operand:V 4 "register_operand" " v")]
What's not yet covered but should be in the future:
- Predicated vector "copy" insns (register copy, load, store).
During initial brainstorming we couldn't come up with a good way to
represent predicated vector stores.
What could perhaps work is something like
[(set (reg:V) (vec_predicate:V vec_copy [reg:V] ...))]
and
[(set (reg:V) (vec_predicate:V vec_copy [mem:V] ...))]
[(set (mem:V) (vec_predicate:V vec_copy [reg:V] ...))]
where the mem attributes could still be stored in dest or src.
This obviously would not inherit any regular "move" handling we have.
I'm not sure this is a good thing :) But at least it would be a clear cut.
Once we have predicated loads, the question of vectorizer interaction needs to
come up, which brings us to another work or staging item:
(4?) Add handling for the new shape to ifn and optabs. So far, I couldn't
think of a nicer way of identifying "vec_predicate"-style optabs/insns
than adding a field to idata and have it be populated by genoutput and
friends. Then, we could do something like
bool
insn_has_vec_predicate (enum insn_code icode)
{
return insn_data[icode].vec_predicate_opno >= 0;
}
The attached patch is item (1) from above. FWIW I have tested it with RVV to
a certain degree and was able to get rid of a number of synthetic combine
patterns already.
Regards
Robin
This patch adds a new RTL vec_predicate object which represents
predicated/masked vector operations. A vec_predicate contains the
operation itself, mask and else operand, vector length, as well as
length and mask policy bits. This is what a predicated add would look
like with the current proposal:
(define_insn "pred_add"
[(set (match_operand:V 0 "register_operand" "=vd")
(vec_predicate:V plus
[(match_operand:V 3 "register_operand" " v")
(match_operand:V 4 "register_operand" " v")]
(match_operand:V 2 "else_operand" " velse")
(match_operand:<VM> 1 "mask_operand" " vm")
(match_operand 5 "vector_length_operand" " vl")
(const_int 0) # Bias
(match_operand 6 "const_int_operand" " i") # Length else
policy
(match_operand 7 "const_int_operand" " i")))]) # Mask else
policy
Once we have a common representation of the predicate, RTL passes can
reason about it, allowing combination of two or more predicated insn,
insn elision, as well as simplification in target-code handling.
gcc/ChangeLog:
* cse.cc (hash_rtx): Hash predicate code.
(exp_equiv_p): Return false for unequal code.
* cselib.cc (rtx_equal_for_cselib_1): Ditto.
(cselib_hash_rtx): Add predicate code to hash.
* doc/rtl.texi: Document vector predicate.
* dwarf2out.cc (mem_loc_descriptor): Break for vec_predicate.
* emit-rtl.cc (init_emit_once): Init vec_all_true and vec_undef.
(rtx_expander::get_rtx): Set vec_predicate code.
(gen_vec_predicate): New function to generate a default
vector predicate.
(verify_vec_predicate): Verify vector predicate.
* genemit.cc (generator::add_exp): Encode vec_predicate code.
* genrecog.cc (rtx_test::vec_pred_code_field): Add.
(safe_to_hoist_p): Ditto.
(transition_parameter_type): Ditto.
(match_pattern_2): Ditto.
(print_nonbool_test): Print vec_predicate.
(print_test): Handle vec_predicate.
* print-rtl.cc (rtx_writer::print_rtx): Ditto.
* read-rtl.cc (apply_code_iterator): Ditto.
(rtx_reader::read_rtx_code): Ditto.
* rtl.cc (DEF_RTL_EXPR): Declare vec_all_true and vec_undef.
(rtx_equal_p): Compare vec_predicate code.
* rtl.def (VEC_ALL_TRUE): Declare.
(VEC_UNDEF): Ditto.
(VEC_PREDICATE): Declare.
* rtl.h (struct GTY): Add vec_pred_code.
(VEC_PRED_CODE): Add.
(PUT_VEC_PRED_CODE): Add.
(VEC_PRED_OPERANDS): Add.
(VEC_PRED_NOPERANDS): Add.
(VEC_PRED_OPERAND): Add.
(VEC_PRED_ELSE): Add.
(VEC_PRED_MASK): Add.
(VEC_PRED_LENGTH): Add.
(VEC_PRED_BIAS): Add.
(VEC_PRED_TAIL_POLICY): Add.
(VEC_PRED_MASK_POLICY): Add.
(VEC_PRED_FIRST_OPTIONAL): Add.
(VEC_PRED_UNDEF_ELSE_P): Add.
(VEC_PRED_FULL_LEN): Add.
(VEC_PRED_ALL_TRUE_P): Add.
(enum vec_pred_policy): Add.
(VEC_PRED_POLICY_MAX): Add.
(VEC_PRED_POLICY_VALID_P): Add.
(VEC_PRED_POLICY_PRESERVE_P): Add.
(gen_vec_predicate): Declare.
(verify_vec_predicate): Declare.
(GTY): Declare vec_all_true and vec_undefined.
---
gcc/cse.cc | 9 +++++
gcc/cselib.cc | 9 +++++
gcc/doc/rtl.texi | 91 ++++++++++++++++++++++++++++++++++++++++++++++++
gcc/dwarf2out.cc | 1 +
gcc/emit-rtl.cc | 69 ++++++++++++++++++++++++++++++++++++
gcc/genemit.cc | 4 +++
gcc/genrecog.cc | 25 +++++++++++++
gcc/print-rtl.cc | 3 ++
gcc/read-rtl.cc | 72 ++++++++++++++++++++++++++++++++++++--
gcc/rtl.cc | 11 ++++++
gcc/rtl.def | 18 ++++++++++
gcc/rtl.h | 55 +++++++++++++++++++++++++++++
12 files changed, 365 insertions(+), 2 deletions(-)
diff --git a/gcc/cse.cc b/gcc/cse.cc
index b4b39e3ebf8..ee4e8b86f1c 100644
--- a/gcc/cse.cc
+++ b/gcc/cse.cc
@@ -2494,6 +2494,10 @@ hash_rtx (const_rtx x, machine_mode mode,
}
break;
+ case VEC_PREDICATE:
+ hash += (unsigned) VEC_PRED_CODE (x);
+ break;
+
default:
break;
}
@@ -2623,6 +2627,11 @@ exp_equiv_p (const_rtx x, const_rtx y, int validate,
bool for_gcse)
return false;
break;
+ case VEC_PREDICATE:
+ if (VEC_PRED_CODE (x) != VEC_PRED_CODE (y))
+ return false;
+ break;
+
case LABEL_REF:
return label_ref_label (x) == label_ref_label (y);
diff --git a/gcc/cselib.cc b/gcc/cselib.cc
index 8a6983f4c69..769530dab6c 100644
--- a/gcc/cselib.cc
+++ b/gcc/cselib.cc
@@ -1100,6 +1100,11 @@ rtx_equal_for_cselib_1 (rtx x, rtx y, machine_mode
memmode, int depth)
return rtx_equal_for_cselib_1 (XEXP (x, 0), XEXP (y, 0), GET_MODE (x),
depth);
+ case VEC_PREDICATE:
+ if (VEC_PRED_CODE (x) != VEC_PRED_CODE (y))
+ return false;
+ break;
+
default:
break;
}
@@ -1503,6 +1508,10 @@ cselib_hash_rtx (rtx x, int create, machine_mode memmode)
case UNSPEC_VOLATILE:
return 0;
+ case VEC_PREDICATE:
+ hash.add_int (VEC_PRED_CODE (x));
+ break;
+
case ASM_OPERANDS:
if (MEM_VOLATILE_P (x))
return 0;
diff --git a/gcc/doc/rtl.texi b/gcc/doc/rtl.texi
index 7fd656b70dc..a5397a6f837 100644
--- a/gcc/doc/rtl.texi
+++ b/gcc/doc/rtl.texi
@@ -2970,8 +2970,99 @@ parts.
@item (vec_series:@var{m} @var{base} @var{step})
This operation creates a vector in which element @var{i} is equal to
@samp{@var{base} + @var{i}*@var{step}}. @var{m} must be a vector integer mode.
+
+@findex vec_all_true
+@item (vec_all_true)
+A value that denotes an all-ones vector mask. It is used as
+the default mask of a @code{vec_predicate} to mean ``no lane
+is masked out'', without having to materialize an actual mask vector or
+commit to a particular mask mode. The unique shared rtx is available as
+@code{vec_all_true_rtx}, and the predicate @code{VEC_PRED_ALL_TRUE_P}
+tests for it.
+
+@findex vec_undef
+@item (vec_undef)
+A value that denotes an undefined vector value. Used as else-operand
+in a @code{vec_predicate} when the inactive elements of the result are
+undefined or ``don't care''.
+
+@findex vec_predicate
+@item (vec_predicate:@var{m} [@var{op0} @var{op1} @dots{}] @var{else}
@var{mask} @var{length} @var{bias} @var{tail_policy} @var{mask_policy})
+This describes a predicated vector operation. A @code{vec_predicate}
+contains the operation itself, stored as RTX code in @code{u2.vec_pred_code},
+its respective operands in an @code{rtvec} (the ``operand vector''), and
+the predication state which consists of the following properties.
+
+@itemize @bullet
+@item
+Else value: Either a value the target supports or @code{vec_undef}.
+In the former case, inactive elements are overwritten with the value.
+The latter case indicates that the operation does not care about which
+values are stored in the inactive elements.
+The else operand can be accessed through @code{VEC_PRED_ELSE} and must
+have the same mod @var{m} as the vec_predicate.
+
+@item
+Mask: Either a target-supported value or @code{vec_all_true}, which is
+a placeholder for ``no masking''. Consequently, an individual mask value
+of 1 means "active", and 0 means "inactive".
+The mask operand can be accessed through @code{VEC_PRED_MASK}.
+
+@item
+Length: Either a target-supported scalar value or @code{VEC_PRED_FULL_LEN}
+(equivalently @code{constm1_rtx}). The former specifies the number of elements
+to operate on, the latter indicates ``no length masking'', meaning all
+(non-masked, mask-inactive) elements are being processed. Conceptually, both
+masks are @code{AND}ed, so neither can invert the other.
+The length operand can be accessed through @code{VEC_PRED_LENGTH}.
+
+@item
+Bias: A @code{const_int} determining which value to add to the length in order
+to compute the ``biased'' (or complete) length. Most targets only handle 0
+here, with s390 being the exception that just supports @code{constm1_rtx}.
+On s390, when e.g.
+loading one element, the vectorizer will build an internal function with the
+biased length, i.e. length + bias = @code{const0_rtx}. This is due
+to performance reasons and means that target expanders need not compute
+the complete length. Only when the unbiased length is needed, optimization
+passes should invert the operation locally.
+The bias operand can be accessed through @code{VEC_PRED_BIAS}.
+
+@item
+Tail policy: Determines which value is stored in inactive tail elements.
+Tail elements are elements whose element index is no smaller than the
+(biased) length.
+
+The tail policy and the mask policy are @code{const_int}s. Their
+value is one of @code{VEC_PRED_ZERO} (0), @code{VEC_PRED_MINUS_ONE} (1),
+@code{VEC_PRED_PRESERVE} (2) or @code{VEC_PRED_UNDEF} (3).
+When either policy is @code{VEC_PRED_PRESERVE}, the else operand must not
+be @code{(vec_undef)}.
+The tail policy operand can be accessed through @code{VEC_PRED_TAIL_POLICY}.
+
+@item
+Mask policy: Determines which value is stored in inactive mask elements that
+are not inside the tail.
+The mask policy operand can be accessed through @code{VEC_PRED_MASK_POLICY}.
+@end itemize
+
+All predicate properties mentioned above are optional and will be filled with
+non-masked defaults when omitted during parsing.
+
+The result mode @var{m} is the mode of the wrapped operation's result, while
+the mode of the input operands are dependent on the operation.
+
+The operand code can be @code{unspec}, in which case the first element of the
+operand vector denotes the @code{unspec}'s identifier as a @code{(const_int)}.
+
@end table
+A @code{vec_predicate} with all optional operands left at their
+defaults is equivalent to the unpredicated operation. The operation code
+is read and written with @code{VEC_PRED_CODE} and @code{PUT_VEC_PRED_CODE}.
+@code{verify_vec_predicate} sanity-checks a @code{vec_predicate} when
+@code{flag_checking} is set.
+
@node Conversions
@section Conversions
@cindex conversions
diff --git a/gcc/dwarf2out.cc b/gcc/dwarf2out.cc
index bf342e468bf..afd35ec2d90 100644
--- a/gcc/dwarf2out.cc
+++ b/gcc/dwarf2out.cc
@@ -17275,6 +17275,7 @@ mem_loc_descriptor (rtx rtl, machine_mode mode,
case VEC_CONCAT:
case VEC_DUPLICATE:
case VEC_SERIES:
+ case VEC_PREDICATE:
case HIGH:
case FMA:
case STRICT_LOW_PART:
diff --git a/gcc/emit-rtl.cc b/gcc/emit-rtl.cc
index e41ec2283b8..6c05238acd7 100644
--- a/gcc/emit-rtl.cc
+++ b/gcc/emit-rtl.cc
@@ -6588,6 +6588,8 @@ init_emit_once (void)
pc_rtx = gen_rtx_fmt_ (PC, VOIDmode);
ret_rtx = gen_rtx_fmt_ (RETURN, VOIDmode);
simple_return_rtx = gen_rtx_fmt_ (SIMPLE_RETURN, VOIDmode);
+ vec_all_true_rtx = gen_rtx_fmt_ (VEC_ALL_TRUE, VOIDmode);
+ vec_undefined_rtx = gen_rtx_fmt_ (VEC_UNDEF, VOIDmode);
invalid_insn_rtx = gen_rtx_INSN (VOIDmode,
/*prev_insn=*/NULL,
/*next_insn=*/NULL,
@@ -7043,6 +7045,9 @@ rtx_expander::get_rtx (rtx_code code, machine_mode mode)
gcc_unreachable ();
}
+ if (code == VEC_PREDICATE)
+ PUT_VEC_PRED_CODE (x, (enum rtx_code) get_uint ());
+
return x;
}
@@ -7100,4 +7105,68 @@ rtl_data::init_stack_alignment ()
}
+/* Build a VEC_PREDICATE rtx with "no predication" defaults for else, mask,
+ length, bias and policies. */
+
+rtx
+gen_vec_predicate (enum rtx_code pred_code, machine_mode mode,
+ rtx *ops, int nops)
+{
+ rtvec v = rtvec_alloc (nops);
+ for (int i = 0; i < nops; i++)
+ RTVEC_ELT (v, i) = ops[i];
+
+ rtx x = gen_rtx_fmt_Eeeeeee (VEC_PREDICATE, mode,
+ v,
+ vec_undefined_rtx,
+ vec_all_true_rtx,
+ VEC_PRED_FULL_LEN,
+ const0_rtx,
+ const0_rtx,
+ const0_rtx);
+ PUT_VEC_PRED_CODE (x, pred_code);
+ if (flag_checking)
+ verify_vec_predicate (x);
+ return x;
+}
+
+/* Sanity-check a VEC_PREDICATE. */
+
+void
+verify_vec_predicate (const_rtx x)
+{
+ gcc_checking_assert (GET_CODE (x) == VEC_PREDICATE);
+
+ machine_mode mode = GET_MODE (x);
+
+ gcc_checking_assert ((unsigned int) VEC_PRED_CODE (x) < NUM_RTX_CODE);
+
+ gcc_checking_assert (VEC_PRED_OPERANDS (x) != NULL);
+ gcc_checking_assert (VEC_PRED_NOPERANDS (x) > 0);
+
+ rtx els = VEC_PRED_ELSE (x);
+ gcc_checking_assert (VEC_PRED_UNDEF_ELSE_P (els)
+ || GET_MODE (els) == mode);
+
+ rtx mask = VEC_PRED_MASK (x);
+ gcc_checking_assert (known_eq (GET_MODE_NUNITS (GET_MODE (mask)),
+ GET_MODE_NUNITS (GET_MODE (els))));
+
+ rtx len = VEC_PRED_LENGTH (x);
+ gcc_checking_assert (CONST_INT_P (len)
+ || (REG_P (len)
+ && SCALAR_INT_MODE_P (GET_MODE (len))));
+
+ gcc_checking_assert (CONST_INT_P (VEC_PRED_BIAS (x)));
+
+ rtx tail_policy = VEC_PRED_TAIL_POLICY (x);
+ rtx mask_policy = VEC_PRED_MASK_POLICY (x);
+ gcc_checking_assert (VEC_PRED_POLICY_VALID_P (tail_policy));
+ gcc_checking_assert (VEC_PRED_POLICY_VALID_P (mask_policy));
+
+ if (VEC_PRED_POLICY_PRESERVE_P (tail_policy)
+ || VEC_PRED_POLICY_PRESERVE_P (mask_policy))
+ gcc_checking_assert (!VEC_PRED_UNDEF_ELSE_P (els));
+}
+
#include "gt-emit-rtl.h"
diff --git a/gcc/genemit.cc b/gcc/genemit.cc
index 6323aeb19fc..a405728aadf 100644
--- a/gcc/genemit.cc
+++ b/gcc/genemit.cc
@@ -240,6 +240,10 @@ generator::add_exp (rtx x)
gcc_unreachable ();
}
}
+
+ /* Encode the operation for VEC_PREDICATE. */
+ if (code == VEC_PREDICATE)
+ add_uint (VEC_PRED_CODE (x));
}
/* Add the expansion of rtx vector VEC to the encoding. */
diff --git a/gcc/genrecog.cc b/gcc/genrecog.cc
index b428c595417..8a08907daac 100644
--- a/gcc/genrecog.cc
+++ b/gcc/genrecog.cc
@@ -1138,6 +1138,9 @@ public:
/* Check XWINT (X, u.opno) == LABEL. */
WIDE_INT_FIELD,
+ /* Check VEC_PRED_CODE (X) == LABEL. */
+ VEC_PRED_CODE_FIELD,
+
/* Check XVECLEN (X, 0) == LABEL. */
VECLEN,
@@ -1215,6 +1218,7 @@ public:
static rtx_test subreg_field (position *);
static rtx_test int_field (position *, int);
static rtx_test wide_int_field (position *, int);
+ static rtx_test vec_pred_code_field (position *);
static rtx_test veclen (position *);
static rtx_test peep2_count (int);
static rtx_test veclen_ge (position *, int);
@@ -1280,6 +1284,12 @@ rtx_test::wide_int_field (position *pos, int opno)
return res;
}
+rtx_test
+rtx_test::vec_pred_code_field (position *pos)
+{
+ return rtx_test (pos, rtx_test::VEC_PRED_CODE_FIELD);
+}
+
rtx_test
rtx_test::veclen (position *pos)
{
@@ -1386,6 +1396,7 @@ operator == (const rtx_test &a, const rtx_test &b)
case rtx_test::MODE:
case rtx_test::REGNO_FIELD:
case rtx_test::SUBREG_FIELD:
+ case rtx_test::VEC_PRED_CODE_FIELD:
case rtx_test::VECLEN:
case rtx_test::HAVE_NUM_CLOBBERS:
return true;
@@ -1851,6 +1862,7 @@ safe_to_hoist_p (decision *d, const rtx_test &test,
known_conditions *kc)
case rtx_test::SUBREG_FIELD:
case rtx_test::INT_FIELD:
case rtx_test::WIDE_INT_FIELD:
+ case rtx_test::VEC_PRED_CODE_FIELD:
case rtx_test::VECLEN:
case rtx_test::VECLEN_GE:
/* These tests access a specific part of an rtx, so are only safe
@@ -2050,6 +2062,7 @@ transition_parameter_type (rtx_test::kind_enum kind)
switch (kind)
{
case rtx_test::CODE:
+ case rtx_test::VEC_PRED_CODE_FIELD:
return parameter::CODE;
case rtx_test::MODE:
@@ -4017,6 +4030,11 @@ match_pattern_2 (state *s, md_rtx_info *info, position
*pos, rtx pattern)
/* Check that the rtx has the right code. */
s = add_decision (s, rtx_test::code (pos), code, false);
+ /* For VEC_PREDICATE, also check the wrapped operation code. */
+ if (code == VEC_PREDICATE)
+ s = add_decision (s, rtx_test::vec_pred_code_field (pos),
+ VEC_PRED_CODE (pattern), false);
+
/* Queue a test for the mode if one is specified. */
if (GET_MODE (pattern) != VOIDmode)
pred_and_mode_tests.safe_push (pattern_pos (pattern, pos));
@@ -4611,6 +4629,12 @@ print_nonbool_test (FILE *f, output_state *os, const
rtx_test &test)
fprintf (f, ", %d)", test.u.opno);
break;
+ case rtx_test::VEC_PRED_CODE_FIELD:
+ fprintf (f, "VEC_PRED_CODE (");
+ print_test_rtx (f, os, test);
+ fprintf (f, ")");
+ break;
+
case rtx_test::REGNO_FIELD:
fprintf (f, "REGNO (");
print_test_rtx (f, os, test);
@@ -4700,6 +4724,7 @@ print_test (FILE *f, output_state *os, const rtx_test
&test, bool is_param,
case rtx_test::REGNO_FIELD:
case rtx_test::INT_FIELD:
case rtx_test::WIDE_INT_FIELD:
+ case rtx_test::VEC_PRED_CODE_FIELD:
case rtx_test::PATTERN:
print_nonbool_test (f, os, test);
fprintf (f, " %s ", invert_p ? "!=" : "==");
diff --git a/gcc/print-rtl.cc b/gcc/print-rtl.cc
index 57ef67b08f8..7731a3083fa 100644
--- a/gcc/print-rtl.cc
+++ b/gcc/print-rtl.cc
@@ -926,6 +926,9 @@ rtx_writer::print_rtx (const_rtx in_rtx)
m_sawclose = true;
idx = GET_RTX_LENGTH (VAR_LOCATION);
}
+
+ if (GET_CODE (in_rtx) == VEC_PREDICATE)
+ fprintf (m_outfile, " %s", GET_RTX_NAME (VEC_PRED_CODE (in_rtx)));
#endif
}
diff --git a/gcc/read-rtl.cc b/gcc/read-rtl.cc
index be9f2da37e1..17d6dbe6c02 100644
--- a/gcc/read-rtl.cc
+++ b/gcc/read-rtl.cc
@@ -267,7 +267,11 @@ find_code (const char *name)
static void
apply_code_iterator (rtx x, unsigned int, HOST_WIDE_INT code)
{
- PUT_CODE (x, (enum rtx_code) code);
+ /* VEC_PREDICATE has the code in u2.vec_pred_code. */
+ if (GET_CODE (x) == VEC_PREDICATE)
+ PUT_VEC_PRED_CODE (x, (enum rtx_code) code);
+ else
+ PUT_CODE (x, (enum rtx_code) code);
}
static const char *
@@ -1790,9 +1794,73 @@ rtx_reader::read_rtx_code (const char *code_name)
INSN_UID (return_rtx) = atoi (name.string);
}
+ /* VEC_PREDICATE has a mode and a modeless code after it. */
+ if (code == VEC_PREDICATE)
+ {
+ file_location loc = read_name (&name);
+ record_potential_iterator_use (&codes, loc, return_rtx, 0, name.string);
+ }
+
/* Use the format_ptr to parse the various operands of this rtx. */
for (int idx = 0; format_ptr[idx] != 0; idx++)
- return_rtx = read_rtx_operand (return_rtx, idx);
+ {
+ /* For VEC_PREDICATE, operands past the actual operands can
+ be filled with defaults. Once we hit a closing parenthesis past
+ operand 1, fill the rest. */
+ if (code == VEC_PREDICATE && idx >= VEC_PRED_FIRST_OPTIONAL)
+ {
+ c = read_skip_spaces ();
+ if (c == ')')
+ {
+ for (int i = idx; format_ptr[i] != 0; i++)
+ switch (i)
+ {
+ case 1:
+ /* Else = VEC_UNDEF. */
+ if (!vec_undefined_rtx)
+ vec_undefined_rtx = rtx_alloc (VEC_UNDEF);
+ XEXP (return_rtx, i) = vec_undefined_rtx;
+ break;
+ case 2:
+ /* Mask = VEC_ALL_TRUE. */
+ if (!vec_all_true_rtx)
+ vec_all_true_rtx = rtx_alloc (VEC_ALL_TRUE);
+ XEXP (return_rtx, i) = vec_all_true_rtx;
+ break;
+ case 3:
+ {
+ /* Full length. */
+ rtx d = rtx_alloc (CONST_INT);
+ PUT_MODE (d, VOIDmode);
+ XWINT (d, 0) = -1;
+ XEXP (return_rtx, i) = d;
+ break;
+ }
+ case 4:
+ /* Bias. */
+ /* FALLTHRU */
+ case 5:
+ /* Tail policy. */
+ /* FALLTHRU */
+ case 6:
+ {
+ /* Mask policy. */
+ rtx d = rtx_alloc (CONST_INT);
+ PUT_MODE (d, VOIDmode);
+ XWINT (d, 0) = 0;
+ XEXP (return_rtx, i) = d;
+ break;
+ }
+ default:
+ gcc_unreachable ();
+ }
+ unread_char (c);
+ break;
+ }
+ unread_char (c);
+ }
+ return_rtx = read_rtx_operand (return_rtx, idx);
+ }
/* Handle any additional information that after the regular fields
(e.g. when parsing function dumps). */
diff --git a/gcc/rtl.cc b/gcc/rtl.cc
index 3839c364571..97eb140f385 100644
--- a/gcc/rtl.cc
+++ b/gcc/rtl.cc
@@ -134,6 +134,12 @@ const unsigned char rtx_code_size[NUM_RTX_CODE] = {
#undef DEF_RTL_EXPR
};
+/* Shared VEC_ALL_TRUE. */
+rtx vec_all_true_rtx;
+
+/* Shared VEC_UNDEF. */
+rtx vec_undefined_rtx;
+
/* Names for kinds of NOTEs and REG_NOTEs. */
const char * const note_insn_name[NOTE_INSN_MAX] =
@@ -478,6 +484,11 @@ rtx_equal_p (const_rtx x, const_rtx y,
rtx_equal_p_callback_function cb)
return false;
break;
+ case VEC_PREDICATE:
+ if (VEC_PRED_CODE (x) != VEC_PRED_CODE (y))
+ return false;
+ break;
+
case DEBUG_IMPLICIT_PTR:
return DEBUG_IMPLICIT_PTR_DECL (x)
== DEBUG_IMPLICIT_PTR_DECL (y);
diff --git a/gcc/rtl.def b/gcc/rtl.def
index ad98fd59172..8f505c6e6fb 100644
--- a/gcc/rtl.def
+++ b/gcc/rtl.def
@@ -239,6 +239,24 @@ DEF_RTL_EXPR(UNSPEC, "unspec", "Ei", RTX_EXTRA)
/* Similar, but a volatile operation and one which may trap. */
DEF_RTL_EXPR(UNSPEC_VOLATILE, "unspec_volatile", "Ei", RTX_EXTRA)
+/* An all-true vector mask. */
+DEF_RTL_EXPR(VEC_ALL_TRUE, "vec_all_true", "", RTX_OBJ)
+
+/* An undefined vector value. */
+DEF_RTL_EXPR(VEC_UNDEF, "vec_undef", "", RTX_OBJ)
+
+/* A predicated vector operation. The wrapped operation's rtx_code
+ is stored in u2.vec_pred_code.
+
+ Operand 0 (E): rtvec of inputs to the operation.
+ Operands 1-6 are optional (default filled).
+ Operand 1 (e): else value.
+ Operand 2 (e): mask.
+ Operand 3 (e): length.
+ Operand 4 (e): bias.
+ Operand 5 (e): tail policy.
+ Operand 6 (e): mask policy. */
+DEF_RTL_EXPR(VEC_PREDICATE, "vec_predicate", "Eeeeeee", RTX_EXTRA)
/* ----------------------------------------------------------------------
Table jump addresses.
---------------------------------------------------------------------- */
diff --git a/gcc/rtl.h b/gcc/rtl.h
index d60587dc5ce..4adbc2cb59e 100644
--- a/gcc/rtl.h
+++ b/gcc/rtl.h
@@ -438,6 +438,9 @@ struct GTY((desc("0"), tag("0"),
/* For future expansion. */
unsigned int unused : 8;
} const_vector;
+
+ /* The code of a VEC_PREDICATE. */
+ unsigned int vec_pred_code;
} GTY ((skip)) u2;
/* The first element of the operands of this rtx.
@@ -2061,6 +2064,55 @@ const_vector_encoded_nelts (const_rtx x)
/* For a CONST_VECTOR, return the number of elements in a vector. */
#define CONST_VECTOR_NUNITS(RTX) GET_MODE_NUNITS (GET_MODE (RTX))
+/* VEC_PREDICATE accessors. */
+
+#define VEC_PRED_CODE(RTX) \
+ ((enum rtx_code) \
+ (RTL_FLAG_CHECK1 ("VEC_PRED_CODE", (RTX), VEC_PREDICATE) \
+ ->u2.vec_pred_code))
+
+#define PUT_VEC_PRED_CODE(RTX, CODE) \
+ (RTL_FLAG_CHECK1 ("PUT_VEC_PRED_CODE", (RTX), VEC_PREDICATE) \
+ ->u2.vec_pred_code = (unsigned int) (CODE))
+
+#define VEC_PRED_OPERANDS(RTX) XVEC (RTX, 0)
+#define VEC_PRED_NOPERANDS(RTX) XVECLEN (RTX, 0)
+#define VEC_PRED_OPERAND(RTX, N) XVECEXP (RTX, 0, N)
+
+#define VEC_PRED_ELSE(RTX) XCEXP (RTX, 1, VEC_PREDICATE)
+#define VEC_PRED_MASK(RTX) XCEXP (RTX, 2, VEC_PREDICATE)
+#define VEC_PRED_LENGTH(RTX) XCEXP (RTX, 3, VEC_PREDICATE)
+#define VEC_PRED_BIAS(RTX) XCEXP (RTX, 4, VEC_PREDICATE)
+#define VEC_PRED_TAIL_POLICY(RTX) XCEXP (RTX, 5, VEC_PREDICATE)
+#define VEC_PRED_MASK_POLICY(RTX) XCEXP (RTX, 6, VEC_PREDICATE)
+
+/* Index of the first optional predicate operand. */
+#define VEC_PRED_FIRST_OPTIONAL 1
+
+#define VEC_PRED_UNDEF_ELSE_P(X) (GET_CODE (X) == VEC_UNDEF)
+#define VEC_PRED_FULL_LEN constm1_rtx
+#define VEC_PRED_ALL_TRUE_P(X) (GET_CODE (X) == VEC_ALL_TRUE)
+
+/* Tail/mask policy. */
+enum vec_pred_policy {
+ VEC_PRED_ZERO = 0,
+ VEC_PRED_MINUS_ONE = 1,
+ VEC_PRED_PRESERVE = 2,
+ VEC_PRED_UNDEF = 3
+};
+#define VEC_PRED_POLICY_MAX VEC_PRED_UNDEF
+
+#define VEC_PRED_POLICY_VALID_P(X) \
+ (CONST_INT_P (X) && IN_RANGE (INTVAL (X), 0, VEC_PRED_POLICY_MAX))
+
+#define VEC_PRED_POLICY_PRESERVE_P(X) \
+ (CONST_INT_P (X) && INTVAL (X) == VEC_PRED_PRESERVE)
+
+extern rtx gen_vec_predicate (enum rtx_code pred_code, machine_mode mode,
+ rtx *ops, int nops);
+
+extern void verify_vec_predicate (const_rtx);
+
/* For a SUBREG rtx, SUBREG_REG extracts the value we want a subreg of.
SUBREG_BYTE extracts the byte-number. */
@@ -3894,6 +3946,9 @@ extern GTY(()) rtx ret_rtx;
extern GTY(()) rtx simple_return_rtx;
extern GTY(()) rtx_insn *invalid_insn_rtx;
+extern GTY(()) rtx vec_all_true_rtx;
+extern GTY(()) rtx vec_undefined_rtx;
+
/* If HARD_FRAME_POINTER_REGNUM is defined, then a special dummy reg
is used to represent the frame pointer. This is because the
hard frame pointer and the automatic variables are separated by an amount
--
2.53.0
