This patch adds the IFN and optabs needed by the later no-epilogue early-break
patches. The vectorizer can already build predicates that describe the active
lanes of a vector iteration, but it did not have a common IFN/optab interface
for asking a target to count those lanes, find the first active lane, build
inclusive or exclusive prefixes, or extract the selected scalar value.
The new IFNs are grouped as follows:
Unconditional forms:
COUNT_ACTIVE, INDEX_FIRST_ACTIVE, TAKE_INCL_ACTIVE,
TAKE_EXCL_ACTIVE and EXTRACT_FIRST operate on the full predicate. On
Adv.SIMD these enable ADDHN/UMOV compression followed by RBIT/CLZ and TBL
extraction. On SVE they enable CNTP, BRK*, FIRSTP/PFIRST and LASTB-style
extraction.
Conditional forms:
COND_COUNT_ACTIVE, COND_INDEX_FIRST_ACTIVE, COND_TAKE_INCL_ACTIVE and
COND_TAKE_EXCL_ACTIVE apply the same operations under an existing mask.
These are used when an early exit is governed by a loop mask or a previous
vector predicate.
Length-limited forms:
COND_LEN_COUNT_ACTIVE, COND_LEN_INDEX_FIRST_ACTIVE,
COND_LEN_TAKE_INCL_ACTIVE and COND_LEN_TAKE_EXCL_ACTIVE add the length
and bias operands used by partial-vector loops. These let targets account
for both the governing predicate and the active vector length in one
operation.
For example, consider:
unsigned char a[1008], b[1008];
unsigned char f1 (void)
{
unsigned char res = 0;
for (unsigned int i = 0; i < 1008; ++i)
{
unsigned char x = a[i] & b[i];
res = x;
if (x > 5)
break;
}
return res;
}
Used to generate on GCC trunk using -O3 -march=armv8-a
cmhi v30.16b, v31.16b, v29.16b
umaxp v30.4s, v30.4s, v30.4s
fmov x0, d30
cbz x0, .L2
ldrb w0, [x2, w1, uxtw]
cmp w0, 5
bhi .L3
... repeated scalar lane checks ...
which branches to scalar to find the element. With this branch we get
with Adv.SIMD:
cmhi v31.16b, v30.16b, v29.16b
addhn v31.8b, v31.8h, v31.8h
umov x0, v31.d[0]
cbz x0, .L8
rbit x0, x0
clz x0, x0
... construction of tbl index from x0
tbl v30.16b, {v30.16b}, v31.16b
fmov w0, s30
Or with SVE2:
cmphi p6.b, p7/z, z31.b, #5
b.none .L8
splice z31.b, p6, z31.b, z31.b
fmov w0, s31
ret
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
-O3, profiled, profiled -O3 and -O3 -LTO bootstrap tested and no issues.
Also SPECCPU 2017 and SPECCPU 2026 tested.
OK for master?
Thanks,
Tamar
gcc/ChangeLog:
* doc/ifn.texi: Document the IFNs.
* doc/md.texi: Document the optabs..
* internal-fn.cc (cond_mask_unary_direct): Add conditional
active-predicate direct optab descriptors.
(cond_len_mask_unary_direct): Add length-limited descriptors.
(expand_cond_mask_unary_optab_fn): Expand conditional unary IFNs.
(expand_cond_len_mask_unary_optab_fn): Expand length forms.
(direct_cond_mask_unary_optab_supported_p): Query optab support.
(direct_cond_len_mask_unary_optab_supported_p): Likewise.
(FOR_EACH_COND_FN_PAIR): Add active-predicate IFN mappings.
(get_conditional_internal_fn): Handle active-predicate IFNs.
(get_len_internal_fn): Likewise.
(get_unconditional_internal_fn): Likewise.
(internal_fn_len_index): Return length operand positions.
(internal_fn_else_index): Return else operand positions.
* internal-fn.def (COUNT_ACTIVE): New internal function.
(COND_COUNT_ACTIVE): Likewise.
(COND_LEN_COUNT_ACTIVE): Likewise.
(INDEX_FIRST_ACTIVE): Likewise.
(COND_INDEX_FIRST_ACTIVE): Likewise.
(COND_LEN_INDEX_FIRST_ACTIVE): Likewise.
(TAKE_INCL_ACTIVE): Likewise.
(COND_TAKE_INCL_ACTIVE): Likewise.
(COND_LEN_TAKE_INCL_ACTIVE): Likewise.
(TAKE_EXCL_ACTIVE): Likewise.
(COND_TAKE_EXCL_ACTIVE): Likewise.
(COND_LEN_TAKE_EXCL_ACTIVE): Likewise.
(EXTRACT_FIRST): Likewise.
* optabs.def (extract_first_optab, count_active_optab,
cond_count_active_optab, cond_len_count_active_optab,
index_first_active_optab, cond_index_first_active_optab,
cond_len_index_first_active_optab, take_incl_active_optab,
take_excl_active_optab, cond_take_incl_active_optab,
cond_take_excl_active_optab, cond_len_take_incl_active_optab,
cond_len_take_excl_active_optab): New.
---
diff --git a/gcc/doc/ifn.texi b/gcc/doc/ifn.texi
index
7cfc80eecf262db33b62e02d3d2d2552c69e5e82..9f130998ec7351f9820576cc309a6eec8798372f
100644
--- a/gcc/doc/ifn.texi
+++ b/gcc/doc/ifn.texi
@@ -457,6 +457,107 @@ The @code{IFN_COND_NOT} internal function directly maps
to the
The @code{IFN_COND_LEN_NOT} internal function directly maps to the
@code{cond_len_one_cmpl} instruction pattern.
+@cindex @code{IFN_COUNT_ACTIVE} internal function
+@item @samp{IFN_COUNT_ACTIVE}
+The @code{IFN_COUNT_ACTIVE} internal function directly maps to the
+@code{count_active} instruction pattern. It returns the number of active
+elements in its argument.
+
+@cindex @code{IFN_COND_COUNT_ACTIVE} internal function
+@item @samp{IFN_COND_COUNT_ACTIVE}
+The @code{IFN_COND_COUNT_ACTIVE} internal function directly maps to the
+@code{cond_count_active} instruction pattern. It returns the number of
+active elements in its second argument under the mask in its first
+argument.
+
+@cindex @code{IFN_COND_LEN_COUNT_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_COUNT_ACTIVE}
+The @code{IFN_COND_LEN_COUNT_ACTIVE} internal function directly maps to the
+@code{cond_len_count_active} instruction pattern. It is like
+@code{IFN_COND_COUNT_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_INDEX_FIRST_ACTIVE}
+The @code{IFN_INDEX_FIRST_ACTIVE} internal function directly maps to the
+@code{index_first_active} instruction pattern. It returns the index of the
+first active element in its argument. The result is undefined if the
+predicate has no active elements.
+
+@cindex @code{IFN_COND_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_COND_INDEX_FIRST_ACTIVE}
+The @code{IFN_COND_INDEX_FIRST_ACTIVE} internal function directly maps to the
+@code{cond_index_first_active} instruction pattern. It returns the index of
+the first active element in its second argument under the mask in its first
+argument. The result is undefined if the selected predicate has no active
+elements.
+
+@cindex @code{IFN_COND_LEN_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_INDEX_FIRST_ACTIVE}
+The @code{IFN_COND_LEN_INDEX_FIRST_ACTIVE} internal function directly maps to
+the @code{cond_len_index_first_active} instruction pattern. It is like
+@code{IFN_COND_INDEX_FIRST_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_TAKE_INCL_ACTIVE}
+The @code{IFN_TAKE_INCL_ACTIVE} internal function directly maps to the
+@code{take_incl_active} instruction pattern. It returns a predicate
+containing the active elements up to and including the first active element
+of its argument.
+
+@cindex @code{IFN_COND_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_COND_TAKE_INCL_ACTIVE}
+The @code{IFN_COND_TAKE_INCL_ACTIVE} internal function directly maps to the
+@code{cond_take_incl_active} instruction pattern. It returns a predicate
+containing the active elements up to and including the first active element
+of the predicate formed by selecting its second argument under the mask in
+its first argument, and its third argument elsewhere.
+
+@cindex @code{IFN_COND_LEN_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_TAKE_INCL_ACTIVE}
+The @code{IFN_COND_LEN_TAKE_INCL_ACTIVE} internal function directly maps to
+the @code{cond_len_take_incl_active} instruction pattern. It is like
+@code{IFN_COND_TAKE_INCL_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_TAKE_EXCL_ACTIVE}
+The @code{IFN_TAKE_EXCL_ACTIVE} internal function directly maps to the
+@code{take_excl_active} instruction pattern. It returns a predicate
+containing the active elements before, but excluding, the first active
+element of its argument.
+
+@cindex @code{IFN_COND_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_COND_TAKE_EXCL_ACTIVE}
+The @code{IFN_COND_TAKE_EXCL_ACTIVE} internal function directly maps to the
+@code{cond_take_excl_active} instruction pattern. It returns a predicate
+containing the active elements before, but excluding, the first active element
+of the predicate formed by selecting its second argument under the mask in
+its first argument, and its third argument elsewhere.
+
+@cindex @code{IFN_COND_LEN_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_TAKE_EXCL_ACTIVE}
+The @code{IFN_COND_LEN_TAKE_EXCL_ACTIVE} internal function directly maps to
+the @code{cond_len_take_excl_active} instruction pattern. It is like
+@code{IFN_COND_TAKE_EXCL_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_COND_VEC_CMP} internal function
+@item @samp{IFN_COND_VEC_CMP}
+The @code{IFN_COND_VEC_CMP} internal function directly maps to the
+@code{cond_vec_cmp} instruction pattern. It performs the comparison
+identified by its first argument on its third and fourth arguments, then
+returns the comparison result after applying the predicate mask in its
+second argument.
+
+@cindex @code{IFN_COND_LEN_VEC_CMP} internal function
+@item @samp{IFN_COND_LEN_VEC_CMP}
+The @code{IFN_COND_LEN_VEC_CMP} internal function directly maps to the
+@code{cond_len_vec_cmp} instruction pattern. It is like
+@code{IFN_COND_VEC_CMP}, but also ignores elements whose index is greater
+than or equal to the sum of the length and bias arguments.
+
@cindex @code{IFN_RSQRT} internal function
@item @samp{IFN_RSQRT}
The @code{IFN_RSQRT} internal function directly maps to the @code{rsqrt}
@@ -512,6 +613,11 @@ The @code{IFN_REDUC_SBOOL_XOR} internal function directly
maps to the
The @code{IFN_EXTRACT_LAST} internal function directly maps to the
@code{extract_last} instruction pattern.
+@cindex @code{IFN_EXTRACT_FIRST} internal function
+@item @samp{IFN_EXTRACT_FIRST}
+The @code{IFN_EXTRACT_FIRST} internal function directly maps to the
+@code{extract_first} instruction pattern.
+
@cindex @code{IFN_FOLD_EXTRACT_LAST} internal function
@item @samp{IFN_FOLD_EXTRACT_LAST}
The @code{IFN_FOLD_EXTRACT_LAST} internal function directly maps to the
@@ -1553,4 +1659,3 @@ definition is still being built by the current pass. The
pass that creates
it must replace it before finishing; the verifier rejects any remaining calls.
@end table
-
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index
f86fa8038e8a0a4498325ea12e041b9622b4e7d5..629fb18b362770a74347441f178d865983dd4e4a
100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7905,6 +7905,34 @@ has vector mode @var{m} while operand 0 has the mode
appropriate for one
element of @var{m}. Operand 1 has the usual mask mode for vectors of mode
@var{m}; see @code{TARGET_VECTORIZE_GET_MASK_MODE}.
+@mdindex extract_first_@var{m}
+@item @code{extract_first_@var{m}}
+Find the first set bit in mask operand 1 and extract the associated element
+of vector operand 2. Store the result in scalar operand 0. Operand 2
+has vector mode @var{m} while operand 0 has the mode appropriate for one
+element of @var{m}. Operand 1 has the usual mask mode for vectors of mode
+@var{m}; see @code{TARGET_VECTORIZE_GET_MASK_MODE}.
+
+@mdindex count_active_@var{m}
+@item @code{count_active_@var{m}}
+Count the set bits in mask operand 1 and store the result in scalar
+operand 0.
+
+@mdindex index_first_active_@var{m}
+@item @code{index_first_active_@var{m}}
+Find the first set bit in mask operand 1 and store the zero-based element
+index in scalar operand 0. The result is undefined if no bits are set.
+
+@mdindex take_incl_active_@var{m}
+@item @code{take_incl_active_@var{m}}
+Set mask operand 0 to the active elements of mask operand 1 up to and
+including the first active element of operand 1.
+
+@mdindex take_excl_active_@var{m}
+@item @code{take_excl_active_@var{m}}
+Set mask operand 0 to the active elements of mask operand 1 before, but
+excluding, the first active element of operand 1.
+
@mdindex fold_extract_last_@var{m}
@item @code{fold_extract_last_@var{m}}
If any bits of mask operand 2 are set, find the last set bit, extract
@@ -8675,6 +8703,76 @@ for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
: op5[i]);
@end smallexample
+@mdindex cond_count_active_@var{mode}
+@item @samp{cond_count_active_@var{mode}}
+Set operand 0 to the number of active elements in operand 2 under the mask
+in operand 1. Operand 0 has a scalar integer mode. Operands 1 and 2 have
+mode @var{mode}.
+
+@mdindex cond_len_count_active_@var{mode}
+@item @samp{cond_len_count_active_@var{mode}}
+Like @samp{cond_count_active_@var{mode}}, but additionally ignore elements
+whose index is greater than or equal to the sum of operands 3 and 4.
+Operand 3 has whichever integer mode the target prefers, and operand 4 is
+a bias.
+
+@mdindex cond_index_first_active_@var{mode}
+@item @samp{cond_index_first_active_@var{mode}}
+Set operand 0 to the index of the first active element in operand 2 under
+the mask in operand 1. Operand 0 has a scalar integer mode. Operands 1
+and 2 have mode @var{mode}. The result is undefined if the selected
+predicate has no
+active elements.
+
+@mdindex cond_len_index_first_active_@var{mode}
+@item @samp{cond_len_index_first_active_@var{mode}}
+Like @samp{cond_index_first_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 3
+and 4. Operand 3 has whichever integer mode the target prefers, and
+operand 4 is a bias.
+
+@mdindex cond_take_incl_active_@var{mode}
+@item @samp{cond_take_incl_active_@var{mode}}
+Set operand 0 to a predicate that contains the active elements up to and
+including the first active element in the predicate formed by selecting
+operand 2 under operand 1, and operand 3 elsewhere. Operands 0, 1, 2
+and 3 have mode @var{mode}.
+
+@mdindex cond_len_take_incl_active_@var{mode}
+@item @samp{cond_len_take_incl_active_@var{mode}}
+Like @samp{cond_take_incl_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 4
+and 5. Operand 4 has whichever integer mode the target prefers, and
+operand 5 is a bias.
+
+@mdindex cond_take_excl_active_@var{mode}
+@item @samp{cond_take_excl_active_@var{mode}}
+Set operand 0 to a predicate that contains the active elements before,
+but excluding, the first active element in the predicate formed by selecting
+operand 2 under operand 1, and operand 3 elsewhere. Operands 0, 1, 2
+and 3 have mode @var{mode}.
+
+@mdindex cond_len_take_excl_active_@var{mode}
+@item @samp{cond_len_take_excl_active_@var{mode}}
+Like @samp{cond_take_excl_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 4
+and 5. Operand 4 has whichever integer mode the target prefers, and
+operand 5 is a bias.
+
+@mdindex cond_vec_cmp@var{m}@var{n}
+@item @samp{cond_vec_cmp@var{m}@var{n}}
+Set operand 0 to the result of applying comparison operand 1 to operands
+3 and 4, after applying predicate mask operand 2. Operand 0 has mode
+@var{n}; operands 3 and 4 have mode @var{m}. Operand 2 has the mode
+returned by @code{TARGET_VECTORIZE_GET_MASK_MODE} for @var{m}.
+
+@mdindex cond_len_vec_cmp@var{m}@var{n}
+@item @samp{cond_len_vec_cmp@var{m}@var{n}}
+Like @samp{cond_vec_cmp@var{m}@var{n}}, but additionally ignore elements
+whose index is greater than or equal to the sum of operands 5 and 6.
+Operand 5 has whichever integer mode the target prefers, and operand 6 is
+a bias.
+
@mdindex cbranch@var{mode}4
@item @samp{cbranch@var{mode}4}
Conditional branch instruction combined with a compare instruction.
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index
0138c6f7ef0117f43afa5c81406a6564a425d304..0030db0c29bf62c4cb4aebb250be9fb5e598dbfa
100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -186,6 +186,8 @@ init_internal_fns ()
#define cond_len_unary_direct { 1, 1, true }
#define cond_len_binary_direct { 1, 1, true }
#define cond_len_ternary_direct { 1, 1, true }
+#define cond_mask_unary_direct { 0, 0, true }
+#define cond_len_mask_unary_direct { 0, 0, true }
#define while_direct { 0, 2, false }
#define fold_extract_direct { 2, 2, false }
#define fold_len_extract_direct { 2, 2, false }
@@ -4294,6 +4296,12 @@ expand_reduc_sbool_optab_fn (internal_fn fn, gcall
*stmt, direct_optab optab)
#define expand_cond_len_ternary_optab_fn(FN, STMT, OPTAB) \
expand_direct_optab_fn (FN, STMT, OPTAB, 7)
+#define expand_cond_mask_unary_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 2)
+
+#define expand_cond_len_mask_unary_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 4)
+
#define expand_fold_extract_optab_fn(FN, STMT, OPTAB) \
expand_direct_optab_fn (FN, STMT, OPTAB, 3)
@@ -4404,6 +4412,8 @@ multi_vector_optab_supported_p (convert_optab optab,
tree_pair types,
#define direct_cond_len_unary_optab_supported_p direct_optab_supported_p
#define direct_cond_len_binary_optab_supported_p direct_optab_supported_p
#define direct_cond_len_ternary_optab_supported_p direct_optab_supported_p
+#define direct_cond_mask_unary_optab_supported_p direct_optab_supported_p
+#define direct_cond_len_mask_unary_optab_supported_p direct_optab_supported_p
#define direct_crc_optab_supported_p convert_optab_supported_p
#define direct_mask_load_optab_supported_p convert_optab_supported_p
#define direct_load_lanes_optab_supported_p multi_vector_optab_supported_p
@@ -4892,6 +4902,8 @@ get_conditional_len_internal_fn (tree_code code)
/* Invoke T(IFN) for each internal function IFN that also has an
IFN_COND_* form. */
#define FOR_EACH_COND_FN_PAIR(T) \
+ T (TAKE_INCL_ACTIVE) \
+ T (TAKE_EXCL_ACTIVE) \
T (FMAX) \
T (FMIN) \
T (FMA) \
@@ -4923,6 +4935,10 @@ get_conditional_internal_fn (internal_fn fn)
{
switch (fn)
{
+ case IFN_COUNT_ACTIVE:
+ return IFN_COND_COUNT_ACTIVE;
+ case IFN_INDEX_FIRST_ACTIVE:
+ return IFN_COND_INDEX_FIRST_ACTIVE;
#define CASE(NAME) case IFN_##NAME: return IFN_COND_##NAME;
FOR_EACH_COND_FN_PAIR(CASE)
#undef CASE
@@ -4958,6 +4974,14 @@ get_len_internal_fn (internal_fn fn)
return IFN_MASK_LEN_LOAD_LANES;
case IFN_MASK_GATHER_LOAD:
return IFN_MASK_LEN_GATHER_LOAD;
+ case IFN_COND_COUNT_ACTIVE:
+ return IFN_COND_LEN_COUNT_ACTIVE;
+ case IFN_COND_INDEX_FIRST_ACTIVE:
+ return IFN_COND_LEN_INDEX_FIRST_ACTIVE;
+ case IFN_COND_TAKE_INCL_ACTIVE:
+ return IFN_COND_LEN_TAKE_INCL_ACTIVE;
+ case IFN_COND_TAKE_EXCL_ACTIVE:
+ return IFN_COND_LEN_TAKE_EXCL_ACTIVE;
default:
return IFN_LAST;
}
@@ -4971,6 +4995,12 @@ get_unconditional_internal_fn (internal_fn ifn)
{
switch (ifn)
{
+ case IFN_COND_COUNT_ACTIVE:
+ case IFN_COND_LEN_COUNT_ACTIVE:
+ return IFN_COUNT_ACTIVE;
+ case IFN_COND_INDEX_FIRST_ACTIVE:
+ case IFN_COND_LEN_INDEX_FIRST_ACTIVE:
+ return IFN_INDEX_FIRST_ACTIVE;
#define CASE(NAME)
\
case IFN_COND_##NAME:
\
case IFN_COND_LEN_##NAME:
\
@@ -5177,6 +5207,17 @@ internal_fn_len_index (internal_fn fn)
case IFN_MASK_LEN_STRIDED_STORE:
return 4;
+ case IFN_COND_LEN_COUNT_ACTIVE:
+ case IFN_COND_LEN_INDEX_FIRST_ACTIVE:
+ return 2;
+
+ case IFN_COND_LEN_TAKE_INCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_EXCL_ACTIVE:
+ return 3;
+
+ case IFN_COND_LEN_VEC_CMP:
+ return 4;
+
case IFN_COND_LEN_NEG:
case IFN_MASK_LEN_STORE:
case IFN_MASK_LEN_STORE_LANES:
@@ -5214,6 +5255,10 @@ internal_fn_else_index (internal_fn fn)
case IFN_COND_LEN_FLOOR:
case IFN_COND_LEN_ROUND:
case IFN_COND_LEN_RINT:
+ case IFN_COND_TAKE_INCL_ACTIVE:
+ case IFN_COND_TAKE_EXCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_INCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_EXCL_ACTIVE:
return 2;
case IFN_LEN_LOAD:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index
af9f92950c73dd09356a625cf2fd60a5abbde321..8ec060415e60dfb6d073a7041c866a1fdf05e800
100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -89,6 +89,10 @@ along with GCC; see the file COPYING3. If not see
- cond_len_binary: a conditional binary optab, such as cond_len_add<mode>
- cond_len_ternary: a conditional ternary optab, such as
cond_len_fma_rev<mode>
+ - cond_mask_unary: a unary optab that takes a mask and a predicate operand,
+ such as cond_count_active<mode>
+ - cond_len_mask_unary: a unary optab that takes a mask, a predicate
+ operand, a length and a bias, such as cond_len_count_active<mode>
DEF_INTERNAL_SIGNED_OPTAB_FN defines an internal function that
maps to one of two optabs, depending on the signedness of an input.
@@ -337,6 +341,28 @@ DEF_INTERNAL_COND_FN (FNMS, ECF_CONST, fnms, ternary)
DEF_INTERNAL_COND_FN (NEG, ECF_CONST, neg, unary)
DEF_INTERNAL_COND_FN (NOT, ECF_CONST, one_cmpl, unary)
+DEF_INTERNAL_OPTAB_FN (COUNT_ACTIVE, ECF_CONST, count_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_COUNT_ACTIVE, ECF_CONST, cond_count_active,
+ cond_mask_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_COUNT_ACTIVE, ECF_CONST,
+ cond_len_count_active, cond_len_mask_unary)
+DEF_INTERNAL_OPTAB_FN (INDEX_FIRST_ACTIVE, ECF_CONST, index_first_active,
+ unary)
+DEF_INTERNAL_OPTAB_FN (COND_INDEX_FIRST_ACTIVE, ECF_CONST,
+ cond_index_first_active, cond_mask_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_INDEX_FIRST_ACTIVE, ECF_CONST,
+ cond_len_index_first_active, cond_len_mask_unary)
+DEF_INTERNAL_OPTAB_FN (TAKE_INCL_ACTIVE, ECF_CONST, take_incl_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_TAKE_INCL_ACTIVE, ECF_CONST,
+ cond_take_incl_active, cond_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_TAKE_INCL_ACTIVE, ECF_CONST,
+ cond_len_take_incl_active, cond_len_unary)
+DEF_INTERNAL_OPTAB_FN (TAKE_EXCL_ACTIVE, ECF_CONST, take_excl_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_TAKE_EXCL_ACTIVE, ECF_CONST,
+ cond_take_excl_active, cond_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_TAKE_EXCL_ACTIVE, ECF_CONST,
+ cond_len_take_excl_active, cond_len_unary)
+
DEF_INTERNAL_OPTAB_FN (RSQRT, ECF_CONST, rsqrt, unary)
DEF_INTERNAL_OPTAB_FN (REDUC_PLUS, ECF_CONST | ECF_NOTHROW,
@@ -363,9 +389,11 @@ DEF_INTERNAL_OPTAB_FN (REDUC_SBOOL_XOR, ECF_CONST |
ECF_NOTHROW,
reduc_sbool_xor_scal, reduc_sbool)
-/* Extract the last active element from a vector. */
+/* Extract the first or last active element from a vector. */
DEF_INTERNAL_OPTAB_FN (EXTRACT_LAST, ECF_CONST | ECF_NOTHROW,
extract_last, fold_left)
+DEF_INTERNAL_OPTAB_FN (EXTRACT_FIRST, ECF_CONST | ECF_NOTHROW,
+ extract_first, fold_left)
/* Same, but return the first argument if no elements are active. */
DEF_INTERNAL_OPTAB_FN (FOLD_EXTRACT_LAST, ECF_CONST | ECF_NOTHROW,
diff --git a/gcc/optabs.def b/gcc/optabs.def
index
327e3efaf8a9eb217db7276848fee1676de02e80..73571fe700c6cb73b35f7f25796bb1b2d090d244
100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -427,8 +427,21 @@ OPTAB_D (mask_fold_left_plus_optab,
"mask_fold_left_plus_$a")
OPTAB_D (mask_len_fold_left_plus_optab, "mask_len_fold_left_plus_$a")
OPTAB_D (extract_last_optab, "extract_last_$a")
+OPTAB_D (extract_first_optab, "extract_first_$a")
+OPTAB_D (count_active_optab, "count_active_$a")
+OPTAB_D (cond_count_active_optab, "cond_count_active_$a")
+OPTAB_D (cond_len_count_active_optab, "cond_len_count_active_$a")
+OPTAB_D (index_first_active_optab, "index_first_active_$a")
+OPTAB_D (cond_index_first_active_optab, "cond_index_first_active_$a")
+OPTAB_D (cond_len_index_first_active_optab, "cond_len_index_first_active_$a")
OPTAB_D (fold_extract_last_optab, "fold_extract_last_$a")
OPTAB_D (len_fold_extract_last_optab, "len_fold_extract_last_$a")
+OPTAB_D (take_incl_active_optab, "take_incl_active_$a")
+OPTAB_D (take_excl_active_optab, "take_excl_active_$a")
+OPTAB_D (cond_take_incl_active_optab, "cond_take_incl_active_$a")
+OPTAB_D (cond_take_excl_active_optab, "cond_take_excl_active_$a")
+OPTAB_D (cond_len_take_incl_active_optab, "cond_len_take_incl_active_$a")
+OPTAB_D (cond_len_take_excl_active_optab, "cond_len_take_excl_active_$a")
OPTAB_D (uabd_optab, "uabd$a3")
OPTAB_D (sabd_optab, "sabd$a3")
--
diff --git a/gcc/doc/ifn.texi b/gcc/doc/ifn.texi
index 7cfc80eecf262db33b62e02d3d2d2552c69e5e82..9f130998ec7351f9820576cc309a6eec8798372f 100644
--- a/gcc/doc/ifn.texi
+++ b/gcc/doc/ifn.texi
@@ -457,6 +457,107 @@ The @code{IFN_COND_NOT} internal function directly maps to the
The @code{IFN_COND_LEN_NOT} internal function directly maps to the
@code{cond_len_one_cmpl} instruction pattern.
+@cindex @code{IFN_COUNT_ACTIVE} internal function
+@item @samp{IFN_COUNT_ACTIVE}
+The @code{IFN_COUNT_ACTIVE} internal function directly maps to the
+@code{count_active} instruction pattern. It returns the number of active
+elements in its argument.
+
+@cindex @code{IFN_COND_COUNT_ACTIVE} internal function
+@item @samp{IFN_COND_COUNT_ACTIVE}
+The @code{IFN_COND_COUNT_ACTIVE} internal function directly maps to the
+@code{cond_count_active} instruction pattern. It returns the number of
+active elements in its second argument under the mask in its first
+argument.
+
+@cindex @code{IFN_COND_LEN_COUNT_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_COUNT_ACTIVE}
+The @code{IFN_COND_LEN_COUNT_ACTIVE} internal function directly maps to the
+@code{cond_len_count_active} instruction pattern. It is like
+@code{IFN_COND_COUNT_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_INDEX_FIRST_ACTIVE}
+The @code{IFN_INDEX_FIRST_ACTIVE} internal function directly maps to the
+@code{index_first_active} instruction pattern. It returns the index of the
+first active element in its argument. The result is undefined if the
+predicate has no active elements.
+
+@cindex @code{IFN_COND_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_COND_INDEX_FIRST_ACTIVE}
+The @code{IFN_COND_INDEX_FIRST_ACTIVE} internal function directly maps to the
+@code{cond_index_first_active} instruction pattern. It returns the index of
+the first active element in its second argument under the mask in its first
+argument. The result is undefined if the selected predicate has no active
+elements.
+
+@cindex @code{IFN_COND_LEN_INDEX_FIRST_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_INDEX_FIRST_ACTIVE}
+The @code{IFN_COND_LEN_INDEX_FIRST_ACTIVE} internal function directly maps to
+the @code{cond_len_index_first_active} instruction pattern. It is like
+@code{IFN_COND_INDEX_FIRST_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_TAKE_INCL_ACTIVE}
+The @code{IFN_TAKE_INCL_ACTIVE} internal function directly maps to the
+@code{take_incl_active} instruction pattern. It returns a predicate
+containing the active elements up to and including the first active element
+of its argument.
+
+@cindex @code{IFN_COND_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_COND_TAKE_INCL_ACTIVE}
+The @code{IFN_COND_TAKE_INCL_ACTIVE} internal function directly maps to the
+@code{cond_take_incl_active} instruction pattern. It returns a predicate
+containing the active elements up to and including the first active element
+of the predicate formed by selecting its second argument under the mask in
+its first argument, and its third argument elsewhere.
+
+@cindex @code{IFN_COND_LEN_TAKE_INCL_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_TAKE_INCL_ACTIVE}
+The @code{IFN_COND_LEN_TAKE_INCL_ACTIVE} internal function directly maps to
+the @code{cond_len_take_incl_active} instruction pattern. It is like
+@code{IFN_COND_TAKE_INCL_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_TAKE_EXCL_ACTIVE}
+The @code{IFN_TAKE_EXCL_ACTIVE} internal function directly maps to the
+@code{take_excl_active} instruction pattern. It returns a predicate
+containing the active elements before, but excluding, the first active
+element of its argument.
+
+@cindex @code{IFN_COND_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_COND_TAKE_EXCL_ACTIVE}
+The @code{IFN_COND_TAKE_EXCL_ACTIVE} internal function directly maps to the
+@code{cond_take_excl_active} instruction pattern. It returns a predicate
+containing the active elements before, but excluding, the first active element
+of the predicate formed by selecting its second argument under the mask in
+its first argument, and its third argument elsewhere.
+
+@cindex @code{IFN_COND_LEN_TAKE_EXCL_ACTIVE} internal function
+@item @samp{IFN_COND_LEN_TAKE_EXCL_ACTIVE}
+The @code{IFN_COND_LEN_TAKE_EXCL_ACTIVE} internal function directly maps to
+the @code{cond_len_take_excl_active} instruction pattern. It is like
+@code{IFN_COND_TAKE_EXCL_ACTIVE}, but also ignores elements whose index is
+greater than or equal to the sum of the length and bias arguments.
+
+@cindex @code{IFN_COND_VEC_CMP} internal function
+@item @samp{IFN_COND_VEC_CMP}
+The @code{IFN_COND_VEC_CMP} internal function directly maps to the
+@code{cond_vec_cmp} instruction pattern. It performs the comparison
+identified by its first argument on its third and fourth arguments, then
+returns the comparison result after applying the predicate mask in its
+second argument.
+
+@cindex @code{IFN_COND_LEN_VEC_CMP} internal function
+@item @samp{IFN_COND_LEN_VEC_CMP}
+The @code{IFN_COND_LEN_VEC_CMP} internal function directly maps to the
+@code{cond_len_vec_cmp} instruction pattern. It is like
+@code{IFN_COND_VEC_CMP}, but also ignores elements whose index is greater
+than or equal to the sum of the length and bias arguments.
+
@cindex @code{IFN_RSQRT} internal function
@item @samp{IFN_RSQRT}
The @code{IFN_RSQRT} internal function directly maps to the @code{rsqrt}
@@ -512,6 +613,11 @@ The @code{IFN_REDUC_SBOOL_XOR} internal function directly maps to the
The @code{IFN_EXTRACT_LAST} internal function directly maps to the
@code{extract_last} instruction pattern.
+@cindex @code{IFN_EXTRACT_FIRST} internal function
+@item @samp{IFN_EXTRACT_FIRST}
+The @code{IFN_EXTRACT_FIRST} internal function directly maps to the
+@code{extract_first} instruction pattern.
+
@cindex @code{IFN_FOLD_EXTRACT_LAST} internal function
@item @samp{IFN_FOLD_EXTRACT_LAST}
The @code{IFN_FOLD_EXTRACT_LAST} internal function directly maps to the
@@ -1553,4 +1659,3 @@ definition is still being built by the current pass. The pass that creates
it must replace it before finishing; the verifier rejects any remaining calls.
@end table
-
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index f86fa8038e8a0a4498325ea12e041b9622b4e7d5..629fb18b362770a74347441f178d865983dd4e4a 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7905,6 +7905,34 @@ has vector mode @var{m} while operand 0 has the mode appropriate for one
element of @var{m}. Operand 1 has the usual mask mode for vectors of mode
@var{m}; see @code{TARGET_VECTORIZE_GET_MASK_MODE}.
+@mdindex extract_first_@var{m}
+@item @code{extract_first_@var{m}}
+Find the first set bit in mask operand 1 and extract the associated element
+of vector operand 2. Store the result in scalar operand 0. Operand 2
+has vector mode @var{m} while operand 0 has the mode appropriate for one
+element of @var{m}. Operand 1 has the usual mask mode for vectors of mode
+@var{m}; see @code{TARGET_VECTORIZE_GET_MASK_MODE}.
+
+@mdindex count_active_@var{m}
+@item @code{count_active_@var{m}}
+Count the set bits in mask operand 1 and store the result in scalar
+operand 0.
+
+@mdindex index_first_active_@var{m}
+@item @code{index_first_active_@var{m}}
+Find the first set bit in mask operand 1 and store the zero-based element
+index in scalar operand 0. The result is undefined if no bits are set.
+
+@mdindex take_incl_active_@var{m}
+@item @code{take_incl_active_@var{m}}
+Set mask operand 0 to the active elements of mask operand 1 up to and
+including the first active element of operand 1.
+
+@mdindex take_excl_active_@var{m}
+@item @code{take_excl_active_@var{m}}
+Set mask operand 0 to the active elements of mask operand 1 before, but
+excluding, the first active element of operand 1.
+
@mdindex fold_extract_last_@var{m}
@item @code{fold_extract_last_@var{m}}
If any bits of mask operand 2 are set, find the last set bit, extract
@@ -8675,6 +8703,76 @@ for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
: op5[i]);
@end smallexample
+@mdindex cond_count_active_@var{mode}
+@item @samp{cond_count_active_@var{mode}}
+Set operand 0 to the number of active elements in operand 2 under the mask
+in operand 1. Operand 0 has a scalar integer mode. Operands 1 and 2 have
+mode @var{mode}.
+
+@mdindex cond_len_count_active_@var{mode}
+@item @samp{cond_len_count_active_@var{mode}}
+Like @samp{cond_count_active_@var{mode}}, but additionally ignore elements
+whose index is greater than or equal to the sum of operands 3 and 4.
+Operand 3 has whichever integer mode the target prefers, and operand 4 is
+a bias.
+
+@mdindex cond_index_first_active_@var{mode}
+@item @samp{cond_index_first_active_@var{mode}}
+Set operand 0 to the index of the first active element in operand 2 under
+the mask in operand 1. Operand 0 has a scalar integer mode. Operands 1
+and 2 have mode @var{mode}. The result is undefined if the selected
+predicate has no
+active elements.
+
+@mdindex cond_len_index_first_active_@var{mode}
+@item @samp{cond_len_index_first_active_@var{mode}}
+Like @samp{cond_index_first_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 3
+and 4. Operand 3 has whichever integer mode the target prefers, and
+operand 4 is a bias.
+
+@mdindex cond_take_incl_active_@var{mode}
+@item @samp{cond_take_incl_active_@var{mode}}
+Set operand 0 to a predicate that contains the active elements up to and
+including the first active element in the predicate formed by selecting
+operand 2 under operand 1, and operand 3 elsewhere. Operands 0, 1, 2
+and 3 have mode @var{mode}.
+
+@mdindex cond_len_take_incl_active_@var{mode}
+@item @samp{cond_len_take_incl_active_@var{mode}}
+Like @samp{cond_take_incl_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 4
+and 5. Operand 4 has whichever integer mode the target prefers, and
+operand 5 is a bias.
+
+@mdindex cond_take_excl_active_@var{mode}
+@item @samp{cond_take_excl_active_@var{mode}}
+Set operand 0 to a predicate that contains the active elements before,
+but excluding, the first active element in the predicate formed by selecting
+operand 2 under operand 1, and operand 3 elsewhere. Operands 0, 1, 2
+and 3 have mode @var{mode}.
+
+@mdindex cond_len_take_excl_active_@var{mode}
+@item @samp{cond_len_take_excl_active_@var{mode}}
+Like @samp{cond_take_excl_active_@var{mode}}, but additionally ignore
+elements whose index is greater than or equal to the sum of operands 4
+and 5. Operand 4 has whichever integer mode the target prefers, and
+operand 5 is a bias.
+
+@mdindex cond_vec_cmp@var{m}@var{n}
+@item @samp{cond_vec_cmp@var{m}@var{n}}
+Set operand 0 to the result of applying comparison operand 1 to operands
+3 and 4, after applying predicate mask operand 2. Operand 0 has mode
+@var{n}; operands 3 and 4 have mode @var{m}. Operand 2 has the mode
+returned by @code{TARGET_VECTORIZE_GET_MASK_MODE} for @var{m}.
+
+@mdindex cond_len_vec_cmp@var{m}@var{n}
+@item @samp{cond_len_vec_cmp@var{m}@var{n}}
+Like @samp{cond_vec_cmp@var{m}@var{n}}, but additionally ignore elements
+whose index is greater than or equal to the sum of operands 5 and 6.
+Operand 5 has whichever integer mode the target prefers, and operand 6 is
+a bias.
+
@mdindex cbranch@var{mode}4
@item @samp{cbranch@var{mode}4}
Conditional branch instruction combined with a compare instruction.
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 0138c6f7ef0117f43afa5c81406a6564a425d304..0030db0c29bf62c4cb4aebb250be9fb5e598dbfa 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -186,6 +186,8 @@ init_internal_fns ()
#define cond_len_unary_direct { 1, 1, true }
#define cond_len_binary_direct { 1, 1, true }
#define cond_len_ternary_direct { 1, 1, true }
+#define cond_mask_unary_direct { 0, 0, true }
+#define cond_len_mask_unary_direct { 0, 0, true }
#define while_direct { 0, 2, false }
#define fold_extract_direct { 2, 2, false }
#define fold_len_extract_direct { 2, 2, false }
@@ -4294,6 +4296,12 @@ expand_reduc_sbool_optab_fn (internal_fn fn, gcall *stmt, direct_optab optab)
#define expand_cond_len_ternary_optab_fn(FN, STMT, OPTAB) \
expand_direct_optab_fn (FN, STMT, OPTAB, 7)
+#define expand_cond_mask_unary_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 2)
+
+#define expand_cond_len_mask_unary_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 4)
+
#define expand_fold_extract_optab_fn(FN, STMT, OPTAB) \
expand_direct_optab_fn (FN, STMT, OPTAB, 3)
@@ -4404,6 +4412,8 @@ multi_vector_optab_supported_p (convert_optab optab, tree_pair types,
#define direct_cond_len_unary_optab_supported_p direct_optab_supported_p
#define direct_cond_len_binary_optab_supported_p direct_optab_supported_p
#define direct_cond_len_ternary_optab_supported_p direct_optab_supported_p
+#define direct_cond_mask_unary_optab_supported_p direct_optab_supported_p
+#define direct_cond_len_mask_unary_optab_supported_p direct_optab_supported_p
#define direct_crc_optab_supported_p convert_optab_supported_p
#define direct_mask_load_optab_supported_p convert_optab_supported_p
#define direct_load_lanes_optab_supported_p multi_vector_optab_supported_p
@@ -4892,6 +4902,8 @@ get_conditional_len_internal_fn (tree_code code)
/* Invoke T(IFN) for each internal function IFN that also has an
IFN_COND_* form. */
#define FOR_EACH_COND_FN_PAIR(T) \
+ T (TAKE_INCL_ACTIVE) \
+ T (TAKE_EXCL_ACTIVE) \
T (FMAX) \
T (FMIN) \
T (FMA) \
@@ -4923,6 +4935,10 @@ get_conditional_internal_fn (internal_fn fn)
{
switch (fn)
{
+ case IFN_COUNT_ACTIVE:
+ return IFN_COND_COUNT_ACTIVE;
+ case IFN_INDEX_FIRST_ACTIVE:
+ return IFN_COND_INDEX_FIRST_ACTIVE;
#define CASE(NAME) case IFN_##NAME: return IFN_COND_##NAME;
FOR_EACH_COND_FN_PAIR(CASE)
#undef CASE
@@ -4958,6 +4974,14 @@ get_len_internal_fn (internal_fn fn)
return IFN_MASK_LEN_LOAD_LANES;
case IFN_MASK_GATHER_LOAD:
return IFN_MASK_LEN_GATHER_LOAD;
+ case IFN_COND_COUNT_ACTIVE:
+ return IFN_COND_LEN_COUNT_ACTIVE;
+ case IFN_COND_INDEX_FIRST_ACTIVE:
+ return IFN_COND_LEN_INDEX_FIRST_ACTIVE;
+ case IFN_COND_TAKE_INCL_ACTIVE:
+ return IFN_COND_LEN_TAKE_INCL_ACTIVE;
+ case IFN_COND_TAKE_EXCL_ACTIVE:
+ return IFN_COND_LEN_TAKE_EXCL_ACTIVE;
default:
return IFN_LAST;
}
@@ -4971,6 +4995,12 @@ get_unconditional_internal_fn (internal_fn ifn)
{
switch (ifn)
{
+ case IFN_COND_COUNT_ACTIVE:
+ case IFN_COND_LEN_COUNT_ACTIVE:
+ return IFN_COUNT_ACTIVE;
+ case IFN_COND_INDEX_FIRST_ACTIVE:
+ case IFN_COND_LEN_INDEX_FIRST_ACTIVE:
+ return IFN_INDEX_FIRST_ACTIVE;
#define CASE(NAME) \
case IFN_COND_##NAME: \
case IFN_COND_LEN_##NAME: \
@@ -5177,6 +5207,17 @@ internal_fn_len_index (internal_fn fn)
case IFN_MASK_LEN_STRIDED_STORE:
return 4;
+ case IFN_COND_LEN_COUNT_ACTIVE:
+ case IFN_COND_LEN_INDEX_FIRST_ACTIVE:
+ return 2;
+
+ case IFN_COND_LEN_TAKE_INCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_EXCL_ACTIVE:
+ return 3;
+
+ case IFN_COND_LEN_VEC_CMP:
+ return 4;
+
case IFN_COND_LEN_NEG:
case IFN_MASK_LEN_STORE:
case IFN_MASK_LEN_STORE_LANES:
@@ -5214,6 +5255,10 @@ internal_fn_else_index (internal_fn fn)
case IFN_COND_LEN_FLOOR:
case IFN_COND_LEN_ROUND:
case IFN_COND_LEN_RINT:
+ case IFN_COND_TAKE_INCL_ACTIVE:
+ case IFN_COND_TAKE_EXCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_INCL_ACTIVE:
+ case IFN_COND_LEN_TAKE_EXCL_ACTIVE:
return 2;
case IFN_LEN_LOAD:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index af9f92950c73dd09356a625cf2fd60a5abbde321..8ec060415e60dfb6d073a7041c866a1fdf05e800 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -89,6 +89,10 @@ along with GCC; see the file COPYING3. If not see
- cond_len_binary: a conditional binary optab, such as cond_len_add<mode>
- cond_len_ternary: a conditional ternary optab, such as
cond_len_fma_rev<mode>
+ - cond_mask_unary: a unary optab that takes a mask and a predicate operand,
+ such as cond_count_active<mode>
+ - cond_len_mask_unary: a unary optab that takes a mask, a predicate
+ operand, a length and a bias, such as cond_len_count_active<mode>
DEF_INTERNAL_SIGNED_OPTAB_FN defines an internal function that
maps to one of two optabs, depending on the signedness of an input.
@@ -337,6 +341,28 @@ DEF_INTERNAL_COND_FN (FNMS, ECF_CONST, fnms, ternary)
DEF_INTERNAL_COND_FN (NEG, ECF_CONST, neg, unary)
DEF_INTERNAL_COND_FN (NOT, ECF_CONST, one_cmpl, unary)
+DEF_INTERNAL_OPTAB_FN (COUNT_ACTIVE, ECF_CONST, count_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_COUNT_ACTIVE, ECF_CONST, cond_count_active,
+ cond_mask_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_COUNT_ACTIVE, ECF_CONST,
+ cond_len_count_active, cond_len_mask_unary)
+DEF_INTERNAL_OPTAB_FN (INDEX_FIRST_ACTIVE, ECF_CONST, index_first_active,
+ unary)
+DEF_INTERNAL_OPTAB_FN (COND_INDEX_FIRST_ACTIVE, ECF_CONST,
+ cond_index_first_active, cond_mask_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_INDEX_FIRST_ACTIVE, ECF_CONST,
+ cond_len_index_first_active, cond_len_mask_unary)
+DEF_INTERNAL_OPTAB_FN (TAKE_INCL_ACTIVE, ECF_CONST, take_incl_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_TAKE_INCL_ACTIVE, ECF_CONST,
+ cond_take_incl_active, cond_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_TAKE_INCL_ACTIVE, ECF_CONST,
+ cond_len_take_incl_active, cond_len_unary)
+DEF_INTERNAL_OPTAB_FN (TAKE_EXCL_ACTIVE, ECF_CONST, take_excl_active, unary)
+DEF_INTERNAL_OPTAB_FN (COND_TAKE_EXCL_ACTIVE, ECF_CONST,
+ cond_take_excl_active, cond_unary)
+DEF_INTERNAL_OPTAB_FN (COND_LEN_TAKE_EXCL_ACTIVE, ECF_CONST,
+ cond_len_take_excl_active, cond_len_unary)
+
DEF_INTERNAL_OPTAB_FN (RSQRT, ECF_CONST, rsqrt, unary)
DEF_INTERNAL_OPTAB_FN (REDUC_PLUS, ECF_CONST | ECF_NOTHROW,
@@ -363,9 +389,11 @@ DEF_INTERNAL_OPTAB_FN (REDUC_SBOOL_XOR, ECF_CONST | ECF_NOTHROW,
reduc_sbool_xor_scal, reduc_sbool)
-/* Extract the last active element from a vector. */
+/* Extract the first or last active element from a vector. */
DEF_INTERNAL_OPTAB_FN (EXTRACT_LAST, ECF_CONST | ECF_NOTHROW,
extract_last, fold_left)
+DEF_INTERNAL_OPTAB_FN (EXTRACT_FIRST, ECF_CONST | ECF_NOTHROW,
+ extract_first, fold_left)
/* Same, but return the first argument if no elements are active. */
DEF_INTERNAL_OPTAB_FN (FOLD_EXTRACT_LAST, ECF_CONST | ECF_NOTHROW,
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 327e3efaf8a9eb217db7276848fee1676de02e80..73571fe700c6cb73b35f7f25796bb1b2d090d244 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -427,8 +427,21 @@ OPTAB_D (mask_fold_left_plus_optab, "mask_fold_left_plus_$a")
OPTAB_D (mask_len_fold_left_plus_optab, "mask_len_fold_left_plus_$a")
OPTAB_D (extract_last_optab, "extract_last_$a")
+OPTAB_D (extract_first_optab, "extract_first_$a")
+OPTAB_D (count_active_optab, "count_active_$a")
+OPTAB_D (cond_count_active_optab, "cond_count_active_$a")
+OPTAB_D (cond_len_count_active_optab, "cond_len_count_active_$a")
+OPTAB_D (index_first_active_optab, "index_first_active_$a")
+OPTAB_D (cond_index_first_active_optab, "cond_index_first_active_$a")
+OPTAB_D (cond_len_index_first_active_optab, "cond_len_index_first_active_$a")
OPTAB_D (fold_extract_last_optab, "fold_extract_last_$a")
OPTAB_D (len_fold_extract_last_optab, "len_fold_extract_last_$a")
+OPTAB_D (take_incl_active_optab, "take_incl_active_$a")
+OPTAB_D (take_excl_active_optab, "take_excl_active_$a")
+OPTAB_D (cond_take_incl_active_optab, "cond_take_incl_active_$a")
+OPTAB_D (cond_take_excl_active_optab, "cond_take_excl_active_$a")
+OPTAB_D (cond_len_take_incl_active_optab, "cond_len_take_incl_active_$a")
+OPTAB_D (cond_len_take_excl_active_optab, "cond_len_take_excl_active_$a")
OPTAB_D (uabd_optab, "uabd$a3")
OPTAB_D (sabd_optab, "sabd$a3")
