The AArch64 ACLE folder needs to resolve MASK_CALL requests using the vector
types selected by the compiler. These can have the same element layout as a
public ACLE type without having its type identity or register ABI.
For example, this GNU GIMPLE input requests AES on a compiler vector type:
#include <arm_neon.h>
typedef unsigned char v16u8 __attribute__((vector_size(16)));
extern uint8x16_t vaeseq_u8 (uint8x16_t, uint8x16_t);
v16u8 __GIMPLE
masked_aese (v16u8 a, v16u8 b, v16u8 mask)
{
v16u8 r;
r = .MASK_CALL (&vaeseq_u8, a, b, mask);
return r;
}
We register a companion vector_decl for each vector-returning function. It has
a separate builtin identity and a signature using compiler vector types, while
the public declaration retains its existing ACLE signature. compiler_type
preserves the data mode and selects the predicate mode for the element size.
gimple_folder::redirect_mask_call selects an implementation and checks its
required extensions without emitting diagnostics for failed probes. It checks
the full argument and result types, inserting view conversions for permitted
fixed-length SVE/Adv. SIMD mode differences with matching element layouts.
Unsupported signatures remain unresolved.
With -O2 -march=armv8-a+aes -fgimple -fdump-tree-optimized -S, this folds to:
r_4 = __builtin_aarch64_vaeseq_u8 (a_1(D), b_2(D));
return r_4;
and generates:
masked_aese:
aese v0.16b, v1.16b
ret
This is an unpredicated 128-bit Adv. SIMD operation. The mask only identifies
elements needing valid results; it does not require zeroing or merging of
inactive results. A 256-bit AES request is rejected.
Expansion also accepts element-sized predicate modes. Anonymous full-SVE
compiler vectors return in memory instead of acquiring the public ACLE
register ABI. The LTO test checks that the companion declarations can be
recreated with the same identities.
This patch adds scaffolding and an example of how to perform the folding.
The way this is intended to work is:
1. Gated behind a flag +hvla or something that we turn on when vectorization
is enabled (note this may interact poorly with function level attributes and
pragmas, but the flag should be there at least for debugging) we register
builtins not just under their ACLE types but the internal compiler anonymous
types. e.g. not just int64x2_t but also vector (4) int. This exposes the
types to the middle-end for vectorization.
2. When SVE is enabled, the same intrinsics also registers the appropriate SVE
internal types. This also solves the issue where to get SVE transformation
the user would have had to include both arm_neon.h and arm_sve.h, but by
using
the internal types we only need arm_neon.h. The additional types should only
be registered when their requirements have been met, e.g. +sve etc.
3. Folding from masked to unmasked happens through IFN_MASK_CALL. The
intrinsics
should support folding this IFN to indicate they support masking. It is up
to
the intrinsics framework to place the mark in the right place.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Any feedback?
Thanks,
Tamar
gcc/ChangeLog:
PR tree-optimization/119187
* config/aarch64/aarch64.cc (aarch64_gimple_fold_builtin): Dispatch ACLE
MASK_CALL requests.
(aarch64_return_in_memory_1): Handle anonymous SVE vector results.
* config/aarch64/aarch64-acle-builtins.cc (registered_function): Record
the compiler-vector declaration.
(compiler_type): New.
(function_builder::add_function): Initialize the vector declaration.
(function_builder::add_unique_function): Register compiler-vector
overloads.
(gimple_folder::redirect_call): Preserve compiler-vector signatures.
(gimple_folder::redirect_mask_call): New.
(gimple_folder::fold): Dispatch masked implementation requests.
(function_expander::add_input_operand): Accept element-sized predicates.
* config/aarch64/aarch64-acle-builtins.h
(gimple_folder::redirect_mask_call): Declare.
(function_base::fold_mask_call): New virtual method.
* config/aarch64/aarch64-neon-builtins-base.cc
(gimple_exact_insn::fold_mask_call): Resolve requests with the original
vector width.
* config/aarch64/aarch64-builtins.cc
(aarch64_check_required_extensions): Allow silent feature checks for
implementation requests.
* config/aarch64/aarch64-protos.h
(aarch64_check_required_extensions): Add a defaulted diagnostic flag.
gcc/testsuite/ChangeLog:
PR tree-optimization/119187
* gcc.target/aarch64/acle/vect-revectorize-types-lto.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-aes.c: New test.
---
--
diff --git a/gcc/config/aarch64/aarch64-acle-builtins.cc
b/gcc/config/aarch64/aarch64-acle-builtins.cc
index
ccfaeaafee4926477493375e683705d60113f20f..5e8dd7cef15c8adce5e1915649e285c7637c7ebc
100644
--- a/gcc/config/aarch64/aarch64-acle-builtins.cc
+++ b/gcc/config/aarch64/aarch64-acle-builtins.cc
@@ -88,6 +88,10 @@ public:
/* The decl itself. */
tree decl;
+ /* The implementation with compiler vector types, if the result is a
+ vector. This has its own builtin code but uses the same instance. */
+ tree vector_decl;
+
/* The architecture extensions that the function requires. */
aarch64_required_extensions required_extensions;
@@ -774,6 +778,7 @@ function_builder::add_function (const function_instance
&instance,
registered_function &rfn = *ggc_alloc <registered_function> ();
rfn.instance = instance;
rfn.decl = decl;
+ rfn.vector_decl = NULL_TREE;
rfn.required_extensions = required_extensions;
rfn.overloaded_p = overloaded_p;
if (m_function_index >= length)
@@ -785,6 +790,16 @@ function_builder::add_function (const function_instance
&instance,
return rfn;
}
+/* Return the compiler's equivalent of TYPE. Preserve the declared vector
+ mode even when it is unavailable under the current target options. */
+static tree
+compiler_type (tree type)
+{
+ return (VECTOR_TYPE_P (type)
+ ? build_vector_type_for_mode (TREE_TYPE (type), TYPE_MODE_RAW (type))
+ : type);
+}
+
/* Add a built-in function for INSTANCE, with the argument types given
by ARGUMENT_TYPES and the return type given by RETURN_TYPE.
REQUIRED_EXTENSIONS describes the architecture extensions that the
@@ -830,6 +845,39 @@ add_unique_function (const function_instance &instance,
required_extensions, false, placeholder_p);
}
+ /* Advertise vector implementations to the middle end independently of
+ the public ACLE types. In particular, compiler predicates have one
+ logical lane per data element rather than svbool_t's lane per byte.
+ This preserves INSTANCE's vector mode. SVE implementations get their
+ own compiler declarations when the SVE functions are registered. */
+ if (VECTOR_TYPE_P (return_type))
+ {
+ /* Keep Adv. SIMD and SVE modes distinct during type interning, even
+ when one of them is unavailable under the current options. */
+ aarch64_target_switcher switcher (AARCH64_FL_SVE);
+ auto_vec<tree> types (argument_types.length ());
+ for (tree type : argument_types)
+ types.quick_push (compiler_type (type));
+ if (instance.gp_index >= 0
+ && instance.type_suffix (0).vector_p
+ && VECTOR_BOOLEAN_TYPE_P (types[instance.gp_index]))
+ {
+ machine_mode mode = instance.gp_mode (0);
+ types[instance.gp_index]
+ = build_truth_vector_type_for_mode (GET_MODE_NUNITS (mode), mode);
+ }
+
+ tree fntype
+ = build_function_type_array (compiler_type (return_type),
+ types.length (), types.address ());
+ tree attrs = get_attributes (instance, required_extensions);
+ char *vector_name = concat ("__builtin_aarch64_", name, NULL);
+ rfn.vector_decl
+ = add_function (instance, vector_name, fntype, attrs,
+ required_extensions, false, false).decl;
+ XDELETEVEC (vector_name);
+ }
+
obstack_free (&m_string_obstack, name);
}
@@ -2891,11 +2939,116 @@ gimple_folder::redirect_call (const function_instance
&instance)
if (!rfn)
return NULL;
- gimple_call_set_fndecl (call, rfn->decl);
- gimple_call_set_fntype (call, TREE_TYPE (rfn->decl));
+ tree type = TREE_TYPE (TREE_TYPE (fndecl));
+ tree decl
+ = (VECTOR_TYPE_P (type) && !builtin_type_p (type)
+ ? rfn->vector_decl : rfn->decl);
+ if (!decl)
+ return nullptr;
+
+ gimple_call_set_fndecl (call, decl);
+ gimple_call_set_fntype (call, TREE_TYPE (decl));
return call;
}
+/* Resolve a MASK_CALL to INSTANCE. The caller selects INSTANCE; looking it
+ up does not search for other implementations with the requested types.
+ Check the complete signature, since the vectorizer also uses this to probe
+ for implementations. The caller decides whether it is safe to ignore the
+ execution mask for an unpredicated call. */
+gimple *
+gimple_folder::redirect_mask_call (const function_instance &instance)
+{
+ registered_function *rfn
+ = function_table->find_with_hash (instance, instance.hash ());
+ if (!rfn)
+ return nullptr;
+
+ if (!aarch64_check_required_extensions (gimple_location (call), rfn->decl,
+ rfn->required_extensions, false))
+ return nullptr;
+
+ tree decl
+ = (builtin_type_p (TREE_TYPE (lhs)) ? rfn->decl : rfn->vector_decl);
+ if (!decl)
+ return nullptr;
+
+ tree fntype = TREE_TYPE (decl);
+ auto compatible_type = [] (tree a, tree b)
+ {
+ if ((VECTOR_TYPE_P (a) && !VECTOR_MODE_P (TYPE_MODE (a)))
+ || (VECTOR_TYPE_P (b) && !VECTOR_MODE_P (TYPE_MODE (b))))
+ return false;
+
+ /* Compatible types need not be the same tree node, for example when
+ one is a typedef. Also allow equal element layouts in different
+ compiler vector modes: with -msve-vector-bits=128, a request using
+ VNx4SI can use a V4SI Adv. SIMD implementation. Such mode changes
+ need explicit view conversions, even though the lane counts match.
+ Keep this extra case limited to compiler vectors. */
+ return useless_type_conversion_p (a, b)
+ || (VECTOR_TYPE_P (a) && VECTOR_TYPE_P (b)
+ && !builtin_type_p (a) && !builtin_type_p (b)
+ && TYPE_MODE (a) != TYPE_MODE (b)
+ && known_eq (TYPE_VECTOR_SUBPARTS (a), TYPE_VECTOR_SUBPARTS (b))
+ && types_compatible_p (TREE_TYPE (a), TREE_TYPE (b)));
+ };
+ if (!compatible_type (TREE_TYPE (fntype), TREE_TYPE (lhs)))
+ return nullptr;
+
+ /* MASK_CALL has the callee first and the execution mask last. Pass the
+ mask as a leading argument only if INSTANCE has a governing predicate. */
+ unsigned int nargs = gimple_call_num_args (call) - 2;
+ auto_vec<tree> args (nargs + 1);
+ if (instance.shape->has_gp_argument_p (instance))
+ args.quick_push (gimple_call_arg (call, nargs + 1));
+
+ for (unsigned int i = 0; i < nargs; ++i)
+ args.quick_push (gimple_call_arg (call, i + 1));
+
+ /* Validate all arguments before emitting conversions, so a failed probe
+ leaves no extra statements behind. */
+ tree argtypes = TYPE_ARG_TYPES (fntype);
+ for (tree arg : args)
+ {
+ if (!argtypes || TREE_VALUE (argtypes) == void_type_node
+ || !compatible_type (TREE_VALUE (argtypes), TREE_TYPE (arg)))
+ return nullptr;
+
+ argtypes = TREE_CHAIN (argtypes);
+ }
+
+ if (!argtypes || TREE_VALUE (argtypes) != void_type_node)
+ return nullptr;
+
+ argtypes = TYPE_ARG_TYPES (fntype);
+ for (tree &arg : args)
+ {
+ tree type = TREE_VALUE (argtypes);
+ if (!useless_type_conversion_p (type, TREE_TYPE (arg)))
+ arg = gimple_build (&stmts, VIEW_CONVERT_EXPR, type, arg);
+
+ argtypes = TREE_CHAIN (argtypes);
+ }
+
+ gcall *new_call = gimple_build_call_vec (decl, args);
+ tree result_type = TREE_TYPE (fntype);
+ if (useless_type_conversion_p (TREE_TYPE (lhs), result_type))
+ {
+ gimple_call_set_lhs (new_call, lhs);
+ return new_call;
+ }
+
+ /* The call must produce its declared result mode. Convert that result
+ back to the requested mode instead of attaching LHS to an invalid call.
*/
+ tree result
+ = (gimple_in_ssa_p (cfun) ? make_ssa_name (result_type)
+ : create_tmp_var (result_type));
+ gimple_call_set_lhs (new_call, result);
+ gimple_seq_add_stmt_without_update (&stmts, new_call);
+ return fold_call_to (result);
+}
+
/* Redirect _z and _m calls to _x functions if the predicate is all-true.
This allows us to use unpredicated instructions, where available. */
gimple *
@@ -3125,15 +3278,28 @@ gimple_folder::fold ()
/* Create a placeholder variable for the result.
This allows implementers of `fold` to assume `lhs` will never be null.
If the result is unused, it will be optimized away later. */
- if (!lhs && TREE_TYPE (gimple_call_fntype (call)) != void_type_node)
+ if (!lhs && !gimple_call_internal_p (call)
+ && TREE_TYPE (gimple_call_fntype (call)) != void_type_node)
lhs = create_tmp_var (TREE_TYPE (gimple_call_fntype (call)));
- /* First try some simplifications that are common to many functions. */
- gimple *res = redirect_pred_x ();
- if (!res)
- res = fold_pfalse ();
- if (!res)
- res = base->fold (*this);
+ gimple *res;
+ if (gimple_call_internal_p (call, IFN_MASK_CALL))
+ {
+ if (!lhs || !VECTOR_TYPE_P (TREE_TYPE (lhs)))
+ return nullptr;
+
+ res = base->fold_mask_call (*this);
+ }
+ else
+ {
+ /* First try some simplifications that are common to many functions. */
+ res = redirect_pred_x ();
+ if (!res)
+ res = fold_pfalse ();
+
+ if (!res)
+ res = base->fold (*this);
+ }
if (res == call)
/* Folds can only reuse the existing call if there are no extra vops
@@ -3361,8 +3527,8 @@ function_expander::add_output_operand (insn_code icode)
vector of the appropriate mode.
- Otherwise, if the operand is a predicate, coerce X to have the
- mode that the instruction expects. In this case X is known to be
- VNx16BImode (the mode of svbool_t).
+ mode that the instruction expects. X already has that mode or
+ is VNx16BImode (the mode of svbool_t).
- Otherwise use X directly. The expand machinery checks that X has
the right mode for the instruction. */
@@ -3391,7 +3557,7 @@ function_expander::add_input_operand (insn_code icode,
rtx x)
x = expand_vector_broadcast (mode, x);
else if (GET_MODE_CLASS (mode) == MODE_VECTOR_BOOL)
{
- gcc_assert (GET_MODE (x) == VNx16BImode);
+ gcc_assert (GET_MODE (x) == mode || GET_MODE (x) == VNx16BImode);
x = gen_lowpart (mode, x);
}
m_ops.safe_grow (m_ops.length () + 1, true);
diff --git a/gcc/config/aarch64/aarch64-acle-builtins.h
b/gcc/config/aarch64/aarch64-acle-builtins.h
index
3260140f1e8e893fd6e219068cac97bbcdbc9d1d..1262e981903f30518427ba7a55685858b437d735
100644
--- a/gcc/config/aarch64/aarch64-acle-builtins.h
+++ b/gcc/config/aarch64/aarch64-acle-builtins.h
@@ -654,6 +654,7 @@ public:
tree load_store_cookie (tree);
gcall *redirect_call (const function_instance &);
+ gimple *redirect_mask_call (const function_instance &);
gimple *redirect_pred_x ();
gimple *fold_pfalse ();
gimple *convert_and_fold (tree, gimple *(*) (gimple_folder &,
@@ -790,6 +791,11 @@ public:
on success, otherwise return null. */
virtual gimple *fold (gimple_folder &) const { return NULL; }
+ /* Resolve a MASK_CALL implementation request. The result and operand
+ types specify the requested vector width, and inactive result lanes
+ are unspecified. Return null if no suitable implementation exists. */
+ virtual gimple *fold_mask_call (gimple_folder &) const { return nullptr; }
+
/* Expand the given call into rtl. Return the result of the function,
or an arbitrary value if the function doesn't return a result. */
virtual rtx expand (function_expander &) const = 0;
diff --git a/gcc/config/aarch64/aarch64-builtins.cc
b/gcc/config/aarch64/aarch64-builtins.cc
index
1d8baaef5c41a012cb4888e865b5318e738319b6..b501f173348424f2bbc73c63a99debebd37a56a6
100644
--- a/gcc/config/aarch64/aarch64-builtins.cc
+++ b/gcc/config/aarch64/aarch64-builtins.cc
@@ -2613,20 +2613,23 @@ aarch64_report_missing_registers (location_t location,
tree fndecl)
}
/* Check whether the requirements in REQUIRED_EXTENSIONS are met, given that
- those requirements come from calling function FNDECL. Report an error
- against LOCATION if not. */
+ those requirements come from calling function FNDECL. If COMPLAIN is true,
+ report an error against LOCATION if not. */
bool
aarch64_check_required_extensions (location_t location, tree fndecl,
aarch64_required_extensions
- required_extensions)
+ required_extensions,
+ bool complain)
{
aarch64_feature_flags sm_state_extensions = 0;
if (!TARGET_STREAMING)
{
if (required_extensions.sm_off == 0)
{
- error_at (location, "ACLE function %qD can only be called when"
- " SME streaming mode is enabled", fndecl);
+ if (complain)
+ error_at (location, "ACLE function %qD can only be called when"
+ " SME streaming mode is enabled", fndecl);
+
return false;
}
sm_state_extensions |= required_extensions.sm_off & ~AARCH64_FL_SM_OFF;
@@ -2635,8 +2638,10 @@ aarch64_check_required_extensions (location_t location,
tree fndecl,
{
if (required_extensions.sm_on == 0)
{
- error_at (location, "ACLE function %qD cannot be called when"
- " SME streaming mode is enabled", fndecl);
+ if (complain)
+ error_at (location, "ACLE function %qD cannot be called when"
+ " SME streaming mode is enabled", fndecl);
+
return false;
}
sm_state_extensions |= required_extensions.sm_on & ~AARCH64_FL_SM_ON;
@@ -2645,6 +2650,9 @@ aarch64_check_required_extensions (location_t location,
tree fndecl,
if ((sm_state_extensions & ~aarch64_isa_flags) == 0)
return true;
+ if (!complain)
+ return false;
+
auto missing_extensions = sm_state_extensions & ~aarch64_asm_isa_flags;
if (missing_extensions == 0)
{
diff --git a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
index
4dd26fd102da2a2d06eae80bcb6c95c8434078e8..839791288bf25c2ee0c7fe09520d3d1409663b3d
100644
--- a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
+++ b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
@@ -203,6 +203,11 @@ public:
template <insn_code insn> struct gimple_exact_insn : gimple_function_base
{
gimple *fold (gimple_folder &) const override { return nullptr; }
+ gimple *fold_mask_call (gimple_folder &f) const override
+ {
+ return f.redirect_mask_call (f);
+ }
+
rtx expand (function_expander &e) const override
{
return e.use_exact_insn (insn);
diff --git a/gcc/config/aarch64/aarch64-protos.h
b/gcc/config/aarch64/aarch64-protos.h
index
500886568f04817ca0526ff068f576e8182c845a..d2cc508df9558505524b760f34fc93ee7ccbf021
100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -1164,7 +1164,8 @@ void handle_arm_acle_h (void);
void init_arm_neon_builtins (void);
bool aarch64_check_required_extensions (location_t, tree,
- aarch64_required_extensions);
+ aarch64_required_extensions,
+ bool = true);
bool aarch64_general_check_builtin_call (location_t, vec<location_t>,
unsigned int, tree, unsigned int,
tree *);
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index
566d65903b915b70db85ae2e5ae9d9466d4f0cdb..d31933fa1c63f3a30a36aba74c72cb15d5f56edc
100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -7647,6 +7647,16 @@ aarch64_return_in_memory_1 (const_tree type)
/* Simple scalar types always returned in registers. */
return false;
+ /* Compiler vectors in full SVE modes have no scalable ABI identity.
+ This also applies when -msve-vector-bits=128 gives them the same
+ size as a short vector. Partial SVE modes keep their existing ABI. */
+ if (VECTOR_TYPE_P (type))
+ {
+ unsigned int vec_flags = aarch64_classify_vector_mode (TYPE_MODE (type));
+ if ((vec_flags & VEC_ANY_SVE) && !(vec_flags & VEC_PARTIAL))
+ return true;
+ }
+
if (aarch64_vfp_is_call_or_return_candidate (TYPE_MODE (type), type,
&ag_mode, &count, NULL, false))
return false;
@@ -17159,15 +17169,20 @@ static bool
aarch64_gimple_fold_builtin (gimple_stmt_iterator *gsi)
{
gcall *stmt = as_a <gcall *> (gsi_stmt (*gsi));
- if (gimple_call_internal_p (stmt))
- return false;
tree fndecl = gimple_call_fndecl (stmt);
+ bool mask_call = gimple_call_internal_p (stmt, IFN_MASK_CALL);
+ if (mask_call)
+ fndecl = gimple_call_addr_fndecl (gimple_call_arg (stmt, 0));
+
unsigned int code = DECL_MD_FUNCTION_CODE (fndecl);
unsigned int subcode = code >> AARCH64_BUILTIN_SHIFT;
gimple *new_stmt = NULL;
switch (code & AARCH64_BUILTIN_CLASS)
{
case AARCH64_BUILTIN_GENERAL:
+ if (mask_call)
+ return false;
+
new_stmt = aarch64_general_gimple_fold_builtin (subcode, stmt, gsi);
break;
diff --git a/gcc/testsuite/gcc.target/aarch64/acle/vect-revectorize-types-lto.c
b/gcc/testsuite/gcc.target/aarch64/acle/vect-revectorize-types-lto.c
new file mode 100644
index
0000000000000000000000000000000000000000..e78816925e4168b16e52392b5456df5bef0f7b64
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/acle/vect-revectorize-types-lto.c
@@ -0,0 +1,16 @@
+/* { dg-do link } */
+/* { dg-require-effective-target lto } */
+/* { dg-options "-O2 -flto -march=armv8-a+sve -msve-vector-bits=128" } */
+
+#include "../../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_sme.h>
+
+/* Recreating the builtin declarations in LTO must preserve vector modes,
+ including the predicate arguments of SME functions. */
+int
+main (void)
+{
+ check_vect ();
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-aes.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-aes.c
new file mode 100644
index
0000000000000000000000000000000000000000..b05a65fa7d128655382aed734496a0fd84b5d50c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-aes.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=armv8-a+aes -fgimple -fdump-tree-optimized" } */
+
+#include <arm_neon.h>
+
+typedef unsigned char v16u8 __attribute__ ((vector_size (16)));
+
+extern uint8x16_t vaeseq_u8 (uint8x16_t, uint8x16_t);
+extern uint8x16_t vaesmcq_u8 (uint8x16_t);
+
+/* These intrinsics already belong to the ACLE framework. */
+v16u8 __GIMPLE
+masked_aese (v16u8 a, v16u8 b, v16u8 mask)
+{
+ v16u8 r;
+ r = .MASK_CALL (&vaeseq_u8, a, b, mask);
+ return r;
+}
+
+v16u8 __GIMPLE
+masked_aesmc (v16u8 a, v16u8 mask)
+{
+ v16u8 r;
+ r = .MASK_CALL (&vaesmcq_u8, a, mask);
+ return r;
+}
+
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL|VIEW_CONVERT_EXPR"
"optimized" } } */
+/* { dg-final { scan-tree-dump {r_[0-9]+ = __builtin_aarch64_vaeseq_u8
\(a_[0-9]+\(D\), b_[0-9]+\(D\)\);} "optimized" } } */
+/* { dg-final { scan-tree-dump {r_[0-9]+ = __builtin_aarch64_vaesmcq_u8
\(a_[0-9]+\(D\)\);} "optimized" } } */
+/* { dg-final { scan-assembler {aese\tv[0-9]+\.16b, v[0-9]+\.16b} } } */
+/* { dg-final { scan-assembler {aesmc\tv[0-9]+\.16b, v[0-9]+\.16b} } } */
