Port the vld{1,2,3,4}_dup family of intrinsics to the pragma-based
framework, and add corresponding assembly tests for each variant.
Porting the load-dup builtin functions away from the old framework makes
`aarch64_builtin_signatures::{load,store}` and
`SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX` obsolete, so all code dealing
with them has been deleted.
The `__builtin_aarch64_simd_*` types are also now all obsolete (they
were only used for casting pointers to the correct type in load/store
functions). So all code dealing with them has also been deleted.
gcc/ChangeLog:
* config/aarch64/aarch64-builtins.h
(qualifier_struct_load_store_lane_index): Delete now-unused enum
member.
* config/aarch64/aarch64-builtins.cc (FLAG_LOAD):
Delete now-unused global variable.
(TYPES_LOADSTRUCT, TYPES_LOADSTRUCT_U, TYPES_LOADSTRUCT_P):
Delete now-unused macros.
(aarch64_scalar_builtin_types, aarch64_simd_intOI_type_node,
aarch64_simd_intCI_type_node, aarch64_simd_intXI_type_node):
Delete now-unused global variables.
(aarch64_mangle_builtin_scalar_type): Delete now-unused
function.
(aarch64_int_or_fp_type): Delete cases handling `E_OImode`,
`E_CImode` and `E_XImode`.
(aarch64_init_simd_builtin_types): Delete code to register
`__builtin_aarch64_simd_{oi,ci,xi}` types.
(aarch64_init_simd_builtin_scalar_types): Delete now-unused
function.
(enum class aarch64_builtin_signatures): Delete now-unused enum
members `load` and `store`.
(aarch64_fntype): Delete cases to handle
`aarch64_builtin_signatures::load` and
`aarch64_builtin_signatures::store`
(SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX): Delete now-unused enum
member.
(aarch64_simd_expand_args): Delete now-unused `builtin_mode`
argument; delete case to handle
`SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX`.
(aarch64_simd_expand_builtin): Delete handling of
`qualifier_struct_load_store_lane_index`; remove last argument
of `aarch64_simd_expand_args`.
(aarch64_dereference_pointer, aarch64_convert_to_lane_mask):
Delete now-unused functions.
(aarch64_expand_pragma_builtin): Delete cases to handle
`UNSPEC_DUP`, `UNSPEC_LD2_DUP`, `UNSPEC_LD3_DUP` and
`UNSPEC_LD4_DUP`.
(aarch64_force_gimple_val, aarch64_fold_to_val,
aarch64_get_lane_bit_index, aarch64_set_lane,
aarch64_gimple_fold_pragma_builtin): Delete now-unused
functions.
(aarch64_general_gimple_fold_builtin): Delete call to
`aarch64_gimple_fold_pragma_builtin`.
* config/aarch64/aarch64-neon-builtins-base.cc (tuple_len): New
function.
(struct gimple_load_dup): New struct.
* config/aarch64/aarch64-neon-builtins-base.def (vld1_dup,
vld1q_dup, vld2_dup, vld2q_dup, vld3_dup, vld3q_dup, vld4_dup,
vld4q_dup): Declare new intrinsics.
* config/aarch64/aarch64-simd-builtins.def: Delete builtin
function declaractions.
* config/aarch64/aarch64-simd-pragma-builtins.def: Delete
intrinsic function definitions.
* config/aarch64/aarch64-simd.md
(@aarch64_simd_ld2r<vstruct_elt>,
@aarch64_simd_ld3r<vstruct_elt>,
@aarch64_simd_ld4r<vstruct_elt>,
aarch64_ld<nregs>r<vstruct_elt>): Replace with...
(@aarch64_simd_load_dup<mode>): ...this.
* config/aarch64/aarch64.md (UNSPEC_DUP): Delete now-unused
UNSPEC.
(UNSPEC_LD1_DUP, UNSPEC_LD2_DUP, UNSPEC_LD3_DUP,
UNSPEC_LD4_DUP): Replace with...
(UNSPEC_LOAD_DUP): ...this.
* config/aarch64/arm_neon.h
(vld1_dup_f16, vld1_dup_f32, vld1_dup_f64, vld1_dup_p8,
vld1_dup_p16, vld1_dup_p64, vld1_dup_s8, vld1_dup_s16,
vld1_dup_s32, vld1_dup_s64, vld1_dup_u8, vld1_dup_u16,
vld1_dup_u32, vld1_dup_u64, vld1q_dup_f16, vld1q_dup_f32,
vld1q_dup_f64, vld1q_dup_p8, vld1q_dup_p16, vld1q_dup_p64,
vld1q_dup_s8, vld1q_dup_s16, vld1q_dup_s32, vld1q_dup_s64,
vld1q_dup_u8, vld1q_dup_u16, vld1q_dup_u32, vld1q_dup_u64,
vld2_dup_s8, vld2_dup_s16, vld2_dup_s32, vld2_dup_f16,
vld2_dup_f32, vld2_dup_f64, vld2_dup_u8, vld2_dup_u16,
vld2_dup_u32, vld2_dup_p8, vld2_dup_p16, vld2_dup_p64,
vld2_dup_s64, vld2_dup_u64, vld2q_dup_s8, vld2q_dup_p8,
vld2q_dup_s16, vld2q_dup_p16, vld2q_dup_s32, vld2q_dup_s64,
vld2q_dup_u8, vld2q_dup_u16, vld2q_dup_u32, vld2q_dup_u64,
vld2q_dup_f16, vld2q_dup_f32, vld2q_dup_f64, vld2q_dup_p64,
vld3_dup_s64, vld3_dup_u64, vld3_dup_f64, vld3_dup_s8,
vld3_dup_p8, vld3_dup_s16, vld3_dup_p16, vld3_dup_s32,
vld3_dup_u8, vld3_dup_u16, vld3_dup_u32, vld3_dup_f16,
vld3_dup_f32, vld3_dup_p64, vld3q_dup_s8, vld3q_dup_p8,
vld3q_dup_s16, vld3q_dup_p16, vld3q_dup_s32, vld3q_dup_s64,
vld3q_dup_u8, vld3q_dup_u16, vld3q_dup_u32, vld3q_dup_u64,
vld3q_dup_f16, vld3q_dup_f32, vld3q_dup_f64, vld3q_dup_p64,
vld4_dup_s64, vld4_dup_u64, vld4_dup_f64, vld4_dup_s8,
vld4_dup_p8, vld4_dup_s16, vld4_dup_p16, vld4_dup_s32,
vld4_dup_u8, vld4_dup_u16, vld4_dup_u32, vld4_dup_f16,
vld4_dup_f32, vld4_dup_p64, vld4q_dup_s8, vld4q_dup_p8,
vld4q_dup_s16, vld4q_dup_p16, vld4q_dup_s32, vld4q_dup_s64,
vld4q_dup_u8, vld4q_dup_u16, vld4q_dup_u32, vld4q_dup_u64,
vld4q_dup_f16, vld4q_dup_f32, vld4q_dup_f64, vld4q_dup_p64,
vld1_dup_bf16, vld1q_dup_bf16, vld3_dup_bf16, vld3q_dup_bf16,
vld4_dup_bf16, vld4q_dup_bf16): Delete function definitions.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/neon/vld1_dup.c: New test.
* gcc.target/aarch64/neon/vld2_dup.c: New test.
* gcc.target/aarch64/neon/vld3_dup.c: New test.
* gcc.target/aarch64/neon/vld4_dup.c: New test.
---
gcc/config/aarch64/aarch64-builtins.cc | 268 +-----
gcc/config/aarch64/aarch64-builtins.h | 2 -
.../aarch64/aarch64-neon-builtins-base.cc | 57 ++
.../aarch64/aarch64-neon-builtins-base.def | 10 +
gcc/config/aarch64/aarch64-simd-builtins.def | 11 -
.../aarch64/aarch64-simd-pragma-builtins.def | 22 -
gcc/config/aarch64/aarch64-simd.md | 58 +-
gcc/config/aarch64/aarch64.md | 6 +-
gcc/config/aarch64/arm_neon.h | 872 ------------------
.../gcc.target/aarch64/neon/vld1_dup.c | 228 +++++
.../gcc.target/aarch64/neon/vld2_dup.c | 228 +++++
.../gcc.target/aarch64/neon/vld3_dup.c | 228 +++++
.../gcc.target/aarch64/neon/vld4_dup.c | 228 +++++
13 files changed, 999 insertions(+), 1219 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/neon/vld1_dup.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/neon/vld2_dup.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/neon/vld3_dup.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/neon/vld4_dup.c
diff --git a/gcc/config/aarch64/aarch64-builtins.cc
b/gcc/config/aarch64/aarch64-builtins.cc
index 44e434e62dd..dc0f9005ca8 100644
--- a/gcc/config/aarch64/aarch64-builtins.cc
+++ b/gcc/config/aarch64/aarch64-builtins.cc
@@ -210,7 +210,6 @@ const unsigned int FLAG_AUTO_FP = 1U << 6;
const unsigned int FLAG_QUIET = 0;
const unsigned int FLAG_DEFAULT = FLAG_AUTO_FP;
const unsigned int FLAG_FP = FLAG_READ_FPCR | FLAG_RAISE_FP_EXCEPTIONS;
-const unsigned int FLAG_LOAD = FLAG_READ_MEMORY;
const unsigned int FLAG_FP8 = FLAG_FP | FLAG_USES_FPMR;
typedef struct
@@ -342,19 +341,6 @@
aarch64_types_fcvt_from_unsigned_qualifiers[SIMD_MAX_BUILTIN_ARGS]
= { qualifier_none, qualifier_unsigned, qualifier_immediate };
#define TYPES_FCVTIMM_SUS (aarch64_types_fcvt_from_unsigned_qualifiers)
-static enum aarch64_type_qualifiers
-aarch64_types_load1_qualifiers[SIMD_MAX_BUILTIN_ARGS]
- = { qualifier_none, qualifier_const_pointer_map_mode };
-#define TYPES_LOADSTRUCT (aarch64_types_load1_qualifiers)
-static enum aarch64_type_qualifiers
-aarch64_types_load1_u_qualifiers[SIMD_MAX_BUILTIN_ARGS]
- = { qualifier_unsigned, qualifier_const_pointer_map_mode };
-#define TYPES_LOADSTRUCT_U (aarch64_types_load1_u_qualifiers)
-static enum aarch64_type_qualifiers
-aarch64_types_load1_p_qualifiers[SIMD_MAX_BUILTIN_ARGS]
- = { qualifier_poly, qualifier_const_pointer_map_mode };
-#define TYPES_LOADSTRUCT_P (aarch64_types_load1_p_qualifiers)
-
constexpr insn_code CODE_FOR_aarch64_sdot_prodv8qi
= CODE_FOR_sdot_prodv2siv8qi;
constexpr insn_code CODE_FOR_aarch64_udot_prodv8qi
@@ -703,34 +689,6 @@ static GTY(()) tree
aarch64_builtin_decls[AARCH64_BUILTIN_MAX];
#define NUM_DREG_TYPES 6
#define NUM_QREG_TYPES 6
-/* Internal scalar builtin types. These types are used to support
- neon intrinsic builtins. They are _not_ user-visible types. Therefore
- the mangling for these types are implementation defined. */
-const char *aarch64_scalar_builtin_types[] = {
- "__builtin_aarch64_simd_qi",
- "__builtin_aarch64_simd_hi",
- "__builtin_aarch64_simd_si",
- "__builtin_aarch64_simd_hf",
- "__builtin_aarch64_simd_sf",
- "__builtin_aarch64_simd_di",
- "__builtin_aarch64_simd_df",
- "__builtin_aarch64_simd_poly8",
- "__builtin_aarch64_simd_poly16",
- "__builtin_aarch64_simd_poly64",
- "__builtin_aarch64_simd_poly128",
- "__builtin_aarch64_simd_ti",
- "__builtin_aarch64_simd_uqi",
- "__builtin_aarch64_simd_uhi",
- "__builtin_aarch64_simd_usi",
- "__builtin_aarch64_simd_udi",
- "__builtin_aarch64_simd_ei",
- "__builtin_aarch64_simd_oi",
- "__builtin_aarch64_simd_ci",
- "__builtin_aarch64_simd_xi",
- "__builtin_aarch64_simd_bf",
- NULL
-};
-
extern const aarch64_simd_type_info aarch64_simd_types[];
extern GTY(()) aarch64_simd_type_info_trees aarch64_simd_types_trees[];
@@ -750,10 +708,6 @@ aarch64_simd_types_trees[ARRAY_SIZE (aarch64_simd_types)];
static machine_mode aarch64_simd_tuple_modes[ARM_NEON_H_TYPES_LAST][3];
GTY (()) tree aarch64_simd_tuple_types[ARM_NEON_H_TYPES_LAST][3];
-static GTY(()) tree aarch64_simd_intOI_type_node = NULL_TREE;
-static GTY(()) tree aarch64_simd_intCI_type_node = NULL_TREE;
-static GTY(()) tree aarch64_simd_intXI_type_node = NULL_TREE;
-
/* The user-visible __mfp8 type, and a pointer to that type. Used
across the back-end. */
tree aarch64_mfp8_type_node = NULL_TREE;
@@ -790,24 +744,6 @@ aarch64_general_simulate_builtin (const char *name, tree
fntype,
code, NULL, attrs);
}
-static const char *
-aarch64_mangle_builtin_scalar_type (const_tree type)
-{
- int i = 0;
-
- while (aarch64_scalar_builtin_types[i] != NULL)
- {
- const char *name = aarch64_scalar_builtin_types[i];
-
- if (TREE_CODE (TYPE_NAME (type)) == TYPE_DECL
- && DECL_NAME (TYPE_NAME (type))
- && !strcmp (IDENTIFIER_POINTER (DECL_NAME (TYPE_NAME (type))), name))
- return aarch64_scalar_builtin_types[i];
- i++;
- }
- return NULL;
-}
-
static const char *
aarch64_mangle_builtin_vector_type (const_tree type)
{
@@ -824,14 +760,9 @@ aarch64_mangle_builtin_vector_type (const_tree type)
const char *
aarch64_general_mangle_builtin_type (const_tree type)
{
- const char *mangle;
/* Walk through all the AArch64 builtins types tables to filter out the
incoming type. */
- if ((mangle = aarch64_mangle_builtin_vector_type (type))
- || (mangle = aarch64_mangle_builtin_scalar_type (type)))
- return mangle;
-
- return NULL;
+ return aarch64_mangle_builtin_vector_type (type);
}
/* Helper function for aarch64_simd_builtin_type. */
@@ -855,12 +786,6 @@ aarch64_int_or_fp_type (machine_mode mode,
return QUAL_TYPE (DI);
case E_TImode:
return QUAL_TYPE (TI);
- case E_OImode:
- return aarch64_simd_intOI_type_node;
- case E_CImode:
- return aarch64_simd_intCI_type_node;
- case E_XImode:
- return aarch64_simd_intXI_type_node;
case E_HFmode:
return aarch64_fp16_type_node;
case E_SFmode:
@@ -1023,64 +948,6 @@ aarch64_init_simd_builtin_types (void)
aarch64_simd_types_trees[i].itype);
TYPE_NAME (aarch64_simd_types_trees[i].itype) = tdecl;
}
-
-#define AARCH64_BUILD_SIGNED_TYPE(mode) \
- make_signed_type (GET_MODE_PRECISION (mode));
- aarch64_simd_intOI_type_node = AARCH64_BUILD_SIGNED_TYPE (OImode);
- aarch64_simd_intCI_type_node = AARCH64_BUILD_SIGNED_TYPE (CImode);
- aarch64_simd_intXI_type_node = AARCH64_BUILD_SIGNED_TYPE (XImode);
-#undef AARCH64_BUILD_SIGNED_TYPE
-
- tdecl = add_builtin_type
- ("__builtin_aarch64_simd_oi" , aarch64_simd_intOI_type_node);
- TYPE_NAME (aarch64_simd_intOI_type_node) = tdecl;
- tdecl = add_builtin_type
- ("__builtin_aarch64_simd_ci" , aarch64_simd_intCI_type_node);
- TYPE_NAME (aarch64_simd_intCI_type_node) = tdecl;
- tdecl = add_builtin_type
- ("__builtin_aarch64_simd_xi" , aarch64_simd_intXI_type_node);
- TYPE_NAME (aarch64_simd_intXI_type_node) = tdecl;
-}
-
-static void
-aarch64_init_simd_builtin_scalar_types (void)
-{
- /* Define typedefs for all the standard scalar types. */
- (*lang_hooks.types.register_builtin_type) (intQI_type_node,
- "__builtin_aarch64_simd_qi");
- (*lang_hooks.types.register_builtin_type) (intHI_type_node,
- "__builtin_aarch64_simd_hi");
- (*lang_hooks.types.register_builtin_type) (aarch64_fp16_type_node,
- "__builtin_aarch64_simd_hf");
- (*lang_hooks.types.register_builtin_type) (intSI_type_node,
- "__builtin_aarch64_simd_si");
- (*lang_hooks.types.register_builtin_type) (float_type_node,
- "__builtin_aarch64_simd_sf");
- (*lang_hooks.types.register_builtin_type) (intDI_type_node,
- "__builtin_aarch64_simd_di");
- (*lang_hooks.types.register_builtin_type) (double_type_node,
- "__builtin_aarch64_simd_df");
- (*lang_hooks.types.register_builtin_type) (unsigned_intQI_type_node,
- "__builtin_aarch64_simd_poly8");
- (*lang_hooks.types.register_builtin_type) (unsigned_intHI_type_node,
- "__builtin_aarch64_simd_poly16");
- (*lang_hooks.types.register_builtin_type) (unsigned_intDI_type_node,
- "__builtin_aarch64_simd_poly64");
- (*lang_hooks.types.register_builtin_type) (unsigned_intTI_type_node,
- "__builtin_aarch64_simd_poly128");
- (*lang_hooks.types.register_builtin_type) (intTI_type_node,
- "__builtin_aarch64_simd_ti");
- (*lang_hooks.types.register_builtin_type) (bfloat16_type_node,
- "__builtin_aarch64_simd_bf");
- /* Unsigned integer types for various mode sizes. */
- (*lang_hooks.types.register_builtin_type) (unsigned_intQI_type_node,
- "__builtin_aarch64_simd_uqi");
- (*lang_hooks.types.register_builtin_type) (unsigned_intHI_type_node,
- "__builtin_aarch64_simd_uhi");
- (*lang_hooks.types.register_builtin_type) (unsigned_intSI_type_node,
- "__builtin_aarch64_simd_usi");
- (*lang_hooks.types.register_builtin_type) (unsigned_intDI_type_node,
- "__builtin_aarch64_simd_udi");
}
/* If MODE is a single Advanced SIMD vector, return the number of lanes in the
@@ -1367,8 +1234,6 @@ enum class aarch64_builtin_signatures
binary,
binary_lane,
binary_two_lanes,
- load,
- store,
ternary,
ternary_lane,
unary,
@@ -1477,19 +1342,12 @@ aarch64_fntype (const aarch64_pragma_builtins_data
&builtin_data)
arg_types.quick_push (integer_type_node);
break;
- case aarch64_builtin_signatures::load:
case aarch64_builtin_signatures::unary:
case aarch64_builtin_signatures::unary_lane:
return_type = builtin_data.types[0].type ();
arg_types.quick_push (builtin_data.types[1].type ());
break;
- case aarch64_builtin_signatures::store:
- return_type = void_type_node;
- for (int i = 0; i <= 1; ++i)
- arg_types.quick_push (builtin_data.types[i].type ());
- break;
-
case aarch64_builtin_signatures::ternary:
case aarch64_builtin_signatures::ternary_lane:
return_type = builtin_data.types[0].type ();
@@ -1664,12 +1522,6 @@ aarch64_init_simd_builtins (void)
{
aarch64_init_simd_builtin_types ();
- /* Strong-typing hasn't been implemented for all AdvSIMD builtin intrinsics.
- Therefore we need to preserve the old __builtin scalar types. It can be
- removed once all the intrinsics become strongly typed using the qualifier
- system. */
- aarch64_init_simd_builtin_scalar_types ();
-
aarch64_init_simd_builtin_functions (false);
if (in_lto_p)
init_arm_neon_builtins ();
@@ -2793,7 +2645,6 @@ typedef enum
SIMD_ARG_COPY_TO_REG,
SIMD_ARG_CONSTANT,
SIMD_ARG_LANE_INDEX,
- SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX,
SIMD_ARG_LANE_PAIR_INDEX,
SIMD_ARG_LANE_QUADTUP_INDEX,
SIMD_ARG_STOP
@@ -2802,8 +2653,7 @@ typedef enum
static rtx
aarch64_simd_expand_args (rtx target, int icode, int have_retval,
- tree exp, builtin_simd_arg *args,
- machine_mode builtin_mode)
+ tree exp, builtin_simd_arg *args)
{
rtx pat;
rtx op[SIMD_MAX_BUILTIN_ARGS + 1]; /* First element for result operand. */
@@ -2842,19 +2692,6 @@ aarch64_simd_expand_args (rtx target, int icode, int
have_retval,
op[opc] = copy_to_mode_reg (mode, op[opc]);
break;
- case SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX:
- gcc_assert (opc > 1);
- if (CONST_INT_P (op[opc]))
- {
- unsigned int nunits
- = GET_MODE_NUNITS (builtin_mode).to_constant ();
- aarch64_simd_lane_bounds (op[opc], 0, nunits, exp);
- /* Keep to GCC-vector-extension lane indices in the RTL. */
- op[opc] = aarch64_endian_lane_rtx (builtin_mode,
- INTVAL (op[opc]));
- }
- goto constant_arg;
-
case SIMD_ARG_LANE_INDEX:
/* Must be a previous operand into which this is an index. */
gcc_assert (opc > 0);
@@ -3022,8 +2859,6 @@ aarch64_simd_expand_builtin (int fcode, tree exp, rtx
target)
args[k] = SIMD_ARG_LANE_PAIR_INDEX;
else if (d->qualifiers[qualifiers_k] & qualifier_lane_quadtup_index)
args[k] = SIMD_ARG_LANE_QUADTUP_INDEX;
- else if (d->qualifiers[qualifiers_k] &
qualifier_struct_load_store_lane_index)
- args[k] = SIMD_ARG_STRUCT_LOAD_STORE_LANE_INDEX;
else if (d->qualifiers[qualifiers_k] & qualifier_immediate)
args[k] = SIMD_ARG_CONSTANT;
else if (d->qualifiers[qualifiers_k] & qualifier_maybe_immediate)
@@ -3046,8 +2881,7 @@ aarch64_simd_expand_builtin (int fcode, tree exp, rtx
target)
/* The interface to aarch64_simd_expand_args expects a 0 if
the function is void, and a 1 if it is not. */
- return aarch64_simd_expand_args
- (target, icode, !is_void, exp, &args[1], d->mode);
+ return aarch64_simd_expand_args (target, icode, !is_void, exp, &args[1]);
}
rtx
@@ -3764,25 +3598,6 @@ aarch64_convert_address (expand_operand *op)
op->mode = Pmode;
}
-/* Dereference the pointer in OP, turning it into a memory reference to
- NELTS instances of MEM_MODE. */
-static void
-aarch64_dereference_pointer (expand_operand *op, machine_mode mem_mode,
- unsigned int nelts = 1)
-{
- if (nelts == 1)
- {
- op->value = gen_rtx_MEM (mem_mode, op->value);
- op->mode = mem_mode;
- }
- else
- {
- op->value = gen_rtx_MEM (BLKmode, op->value);
- op->mode = BLKmode;
- set_mem_size (op->value, GET_MODE_SIZE (mem_mode) * nelts);
- }
-}
-
/* OP contains an integer index into a vector or tuple of mode VEC_MODE.
Convert OP from an architectural lane number to a GCC lane number. */
static void
@@ -4089,11 +3904,6 @@ aarch64_expand_pragma_builtin (tree exp, rtx target,
insn_code icode;
switch (builtin_data.unspec)
{
- case UNSPEC_DUP:
- if (builtin_data.signature == aarch64_builtin_signatures::load)
- aarch64_dereference_pointer (&ops[1], GET_MODE_INNER (ops[0].mode));
- return expand_vector_broadcast (ops[0].mode, ops[1].value);
-
case UNSPEC_FAMAX:
case UNSPEC_FAMIN:
case UNSPEC_FMMLA:
@@ -4157,21 +3967,6 @@ aarch64_expand_pragma_builtin (tree exp, rtx target,
gcc_unreachable ();
break;
- case UNSPEC_LD2_DUP:
- aarch64_dereference_pointer (&ops[1], GET_MODE_INNER (ops[0].mode), 2);
- icode = code_for_aarch64_simd_ld2r (ops[0].mode);
- break;
-
- case UNSPEC_LD3_DUP:
- aarch64_dereference_pointer (&ops[1], GET_MODE_INNER (ops[0].mode), 3);
- icode = code_for_aarch64_simd_ld3r (ops[0].mode);
- break;
-
- case UNSPEC_LD4_DUP:
- aarch64_dereference_pointer (&ops[1], GET_MODE_INNER (ops[0].mode), 4);
- icode = code_for_aarch64_simd_ld4r (ops[0].mode);
- break;
-
case UNSPEC_LUTI2:
case UNSPEC_LUTI4:
create_integer_operand (ops.safe_push ({}),
@@ -4586,32 +4381,6 @@ aarch64_record_vector_load_arg (tree addr)
cfun->machine->vector_load_decls->add (decl);
}
-/* Force VAL into a valid gimple value, creating a new SSA_NAME if
- necessary. Insert any new statements before GSI. */
-static tree
-aarch64_force_gimple_val (gimple_stmt_iterator *gsi, tree val)
-{
- if (is_gimple_val (val))
- return val;
-
- tree tmp = make_ssa_name (TREE_TYPE (val));
- gsi_insert_before_without_update (gsi, gimple_build_assign (tmp, val),
- GSI_SAME_STMT);
- return tmp;
-}
-
-/* Fold STMT (at GSI) to VAL, with SEQ setting up the value of VAL.
- Return the replacement statement. */
-static gimple *
-aarch64_fold_to_val (gcall *stmt, gimple_stmt_iterator *gsi,
- gimple *seq, tree val)
-{
- auto *assign = gimple_build_assign (gimple_call_lhs (stmt), val);
- gimple_seq_add_stmt_without_update (&seq, assign);
- gsi_replace_with_seq_vops (gsi, seq);
- return assign;
-}
-
/* Dereference pointer ADDR, giving a memory reference of type TYPE. */
tree
aarch64_dereference (tree addr, tree type)
@@ -4626,34 +4395,6 @@ aarch64_dereference (tree addr, tree type)
return fold_build2 (MEM_REF, access_type, addr, zero);
}
-/* Try to fold STMT (at GSI), given that it is a call to the builtin
- described by BUILTIN_DATA. Return the new statement on success,
- otherwise return null. */
-static gimple *
-aarch64_gimple_fold_pragma_builtin
- (gcall *stmt, gimple_stmt_iterator *gsi,
- const aarch64_pragma_builtins_data &builtin_data)
-{
- auto &types = builtin_data.types;
-
- switch (builtin_data.unspec)
- {
- case UNSPEC_DUP:
- {
- tree arg = gimple_call_arg (stmt, 0);
- tree type = types[0].type ();
- if (builtin_data.signature == aarch64_builtin_signatures::load)
- arg = aarch64_dereference (arg, TREE_TYPE (type));
- arg = aarch64_force_gimple_val (gsi, arg);
-
- tree dup = build_vector_from_val (type, arg);
- return aarch64_fold_to_val (stmt, gsi, nullptr, dup);
- }
- default:
- return nullptr;
- }
-}
-
/* Return the fndecl of the builtin paired with FCODE_LO if one
exists (see aarch64-builtin-pairs.def), or NULL_TREE if not. */
static inline tree
@@ -4931,9 +4672,6 @@ aarch64_general_gimple_fold_builtin (unsigned int fcode,
gcall *stmt,
}
break;
default:
- if (auto builtin_data = aarch64_get_pragma_builtin (fcode))
- new_stmt = aarch64_gimple_fold_pragma_builtin (stmt, gsi,
- *builtin_data);
break;
}
diff --git a/gcc/config/aarch64/aarch64-builtins.h
b/gcc/config/aarch64/aarch64-builtins.h
index 87fc0dcb2d5..ff5c1495fb0 100644
--- a/gcc/config/aarch64/aarch64-builtins.h
+++ b/gcc/config/aarch64/aarch64-builtins.h
@@ -47,8 +47,6 @@ enum aarch64_type_qualifiers
qualifier_poly = 1 << 8,
/* Lane indices - must be in range, and flipped for bigendian. */
qualifier_lane_index = 1 << 9,
- /* Lane indices for single lane structure loads and stores. */
- qualifier_struct_load_store_lane_index = 1 << 10,
/* Lane indices selected in pairs. - must be in range, and flipped for
bigendian. */
qualifier_lane_pair_index = 1 << 11,
diff --git a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
index 0d6f11fba6e..ba68bbb0b09 100644
--- a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
+++ b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
@@ -144,6 +144,22 @@ build_shift_right (tree type, tree val, tree amount)
build_int_cst (TREE_TYPE (amount), bits - 1));
}
+/* Get the number of vectors in TUPLE, which can either be a type or an
+ expression.
+ For convenience, a vector is treated as a single-element tuple. */
+unsigned int
+tuple_len (tree tuple)
+{
+ tree type = TYPE_P (tuple) ? tuple : TREE_TYPE (tuple);
+
+ if (VECTOR_TYPE_P (type))
+ return 1;
+
+ auto field = tuple_type_field (type);
+ auto array_type = TREE_TYPE (field);
+ return int_cst_value (TYPE_MAX_VALUE (TYPE_DOMAIN (array_type))) + 1;
+}
+
/* Build `TUPLE.val`. */
tree
build_tuple_get_array (tree tuple)
@@ -1748,6 +1764,43 @@ public:
gimple *fold (gimple_folder &) const override { return nullptr; }
};
+struct gimple_load_dup : public gimple_function_base
+{
+ unsigned int call_properties (const function_instance &) const override
+ {
+ return CP_READ_MEMORY;
+ }
+
+ rtx expand (function_expander &e) const override
+ {
+ auto tuple_mode = e.result_mode ();
+ auto &ptr_arg = e.args[0];
+ ptr_arg = gen_rtx_MEM (BLKmode, e.convert_to_pmode (ptr_arg));
+ set_mem_size (ptr_arg, GET_MODE_SIZE (GET_MODE_INNER (tuple_mode))
+ * targetm.hard_regno_nregs (V0_REGNUM,
tuple_mode));
+ auto icode = code_for_aarch64_simd_load_dup (tuple_mode);
+ gcc_assert (icode != CODE_FOR_nothing);
+ return e.use_exact_insn (icode);
+ }
+
+ gimple *fold (gimple_folder &f) const override
+ {
+ auto ptr = gimple_call_arg (f.call, 0);
+ auto ret_type = TREE_TYPE (f.lhs);
+ auto elem_type = this->memory_scalar_type (f);
+ auto vec_count = tuple_len (ret_type);
+
+ if (vec_count == 1)
+ {
+ /* lhs = dup (*ptr) */
+ auto load = f.force_val (aarch64_dereference (ptr, elem_type));
+ return gimple_build_assign (f.lhs, build_vec_dup (ret_type, load));
+ }
+
+ return nullptr;
+ }
+};
+
// Reinterpret
NEON_FUNCTION (vreinterpret, vreinterpretq, gimple_reinterpret,)
@@ -1981,6 +2034,10 @@ NEON_FUNCTION (vst1_lane, vst1q_lane, vst2_lane,
vst2q_lane,
NEON_FUNCTION (vldap1_lane, vldap1q_lane, gimple_load_store_lane_atomic<true>,)
NEON_FUNCTION (vstl1_lane, vstl1q_lane,
gimple_load_store_lane_atomic<false>,)
+// Load and broadcast
+NEON_FUNCTION (vld1_dup, vld1q_dup, vld2_dup, vld2q_dup,
+ vld3_dup, vld3q_dup, vld4_dup, vld4q_dup, gimple_load_dup,)
+
// AES
NEON_FUNCTION (vaeseq, gimple_exact_insn<CODE_FOR_aarch64_crypto_aesev16qi>,)
NEON_FUNCTION (vaesdq, gimple_exact_insn<CODE_FOR_aarch64_crypto_aesdv16qi>,)
diff --git a/gcc/config/aarch64/aarch64-neon-builtins-base.def
b/gcc/config/aarch64/aarch64-neon-builtins-base.def
index a68c67acaf6..44e9a07bee6 100644
--- a/gcc/config/aarch64/aarch64-neon-builtins-base.def
+++ b/gcc/config/aarch64/aarch64-neon-builtins-base.def
@@ -628,6 +628,16 @@ DEF_NEON_FUNCTION (vld3_lane, all_neon,
("D0x3,al,D0x3,ss32", lane<2>))
DEF_NEON_FUNCTION (vld3q_lane, all_neon, ("Q0x3,al,Q0x3,ss32", lane<2>))
DEF_NEON_FUNCTION (vld4_lane, all_neon, ("D0x4,al,D0x4,ss32", lane<2>))
DEF_NEON_FUNCTION (vld4q_lane, all_neon, ("Q0x4,al,Q0x4,ss32", lane<2>))
+
+// Load and broadcast
+DEF_NEON_FUNCTION (vld1_dup, all_neon, ("D0,al"))
+DEF_NEON_FUNCTION (vld1q_dup, all_neon, ("Q0,al"))
+DEF_NEON_FUNCTION (vld2_dup, all_neon, ("D0x2,al"))
+DEF_NEON_FUNCTION (vld2q_dup, all_neon, ("Q0x2,al"))
+DEF_NEON_FUNCTION (vld3_dup, all_neon, ("D0x3,al"))
+DEF_NEON_FUNCTION (vld3q_dup, all_neon, ("Q0x3,al"))
+DEF_NEON_FUNCTION (vld4_dup, all_neon, ("D0x4,al"))
+DEF_NEON_FUNCTION (vld4q_dup, all_neon, ("Q0x4,al"))
#undef REQUIRED_EXTENSIONS
// Stores
diff --git a/gcc/config/aarch64/aarch64-simd-builtins.def
b/gcc/config/aarch64/aarch64-simd-builtins.def
index 5ceee439b2e..f317358321d 100644
--- a/gcc/config/aarch64/aarch64-simd-builtins.def
+++ b/gcc/config/aarch64/aarch64-simd-builtins.def
@@ -60,17 +60,6 @@
BUILTIN_VSDQ_I (BINOP_SSU, suqadd, 0, DEFAULT)
BUILTIN_VSDQ_I (BINOP_UUS, usqadd, 0, DEFAULT)
- /* Implemented by aarch64_ld<nregs>r<vstruct_elt>. */
- BUILTIN_VALLDIF (LOADSTRUCT, ld2r, 0, LOAD)
- BUILTIN_VSDQ_I_DI (LOADSTRUCT_U, ld2r, 0, LOAD)
- BUILTIN_VALLP (LOADSTRUCT_P, ld2r, 0, LOAD)
- BUILTIN_VALLDIF (LOADSTRUCT, ld3r, 0, LOAD)
- BUILTIN_VSDQ_I_DI (LOADSTRUCT_U, ld3r, 0, LOAD)
- BUILTIN_VALLP (LOADSTRUCT_P, ld3r, 0, LOAD)
- BUILTIN_VALLDIF (LOADSTRUCT, ld4r, 0, LOAD)
- BUILTIN_VSDQ_I_DI (LOADSTRUCT_U, ld4r, 0, LOAD)
- BUILTIN_VALLP (LOADSTRUCT_P, ld4r, 0, LOAD)
-
BUILTIN_VQW (BINOP, saddl2, 0, DEFAULT)
BUILTIN_VQW (BINOPU, uaddl2, 0, DEFAULT)
BUILTIN_VQW (BINOP, ssubl2, 0, DEFAULT)
diff --git a/gcc/config/aarch64/aarch64-simd-pragma-builtins.def
b/gcc/config/aarch64/aarch64-simd-pragma-builtins.def
index 873a451624b..d049165828c 100644
--- a/gcc/config/aarch64/aarch64-simd-pragma-builtins.def
+++ b/gcc/config/aarch64/aarch64-simd-pragma-builtins.def
@@ -30,10 +30,6 @@
#define ENTRY_BINARY_TWO_LANES(N, T0, T1, T2, U, F) \
ENTRY (N, binary_two_lanes, T0, T1, T2, none, U, F)
-#undef ENTRY_LOAD
-#define ENTRY_LOAD(N, T0, T1, U) \
- ENTRY (N, load, T0, T1, none, none, U, LOAD)
-
#undef ENTRY_TERNARY
#define ENTRY_TERNARY(N, T0, T1, T2, T3, U, F) \
ENTRY (N, ternary, T0, T1, T2, T3, U, F)
@@ -140,24 +136,6 @@ ENTRY_FMA_FPM (vmlalltb, f32, UNSPEC_FMLALLTB_FP8)
ENTRY_FMA_FPM (vmlalltt, f32, UNSPEC_FMLALLTT_FP8)
#undef REQUIRED_EXTENSIONS
-// ld1
-#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_SIMD)
-ENTRY_LOAD (vld1_dup_mf8, mf8, mf8_scalar_const_ptr, UNSPEC_DUP)
-ENTRY_LOAD (vld1q_dup_mf8, mf8q, mf8_scalar_const_ptr, UNSPEC_DUP)
-#undef REQUIRED_EXTENSIONS
-
-// ld<n>
-#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_SIMD)
-ENTRY_LOAD (vld2_dup_mf8, mf8x2, mf8_scalar_const_ptr, UNSPEC_LD2_DUP)
-ENTRY_LOAD (vld2q_dup_mf8, mf8qx2, mf8_scalar_const_ptr, UNSPEC_LD2_DUP)
-
-ENTRY_LOAD (vld3_dup_mf8, mf8x3, mf8_scalar_const_ptr, UNSPEC_LD3_DUP)
-ENTRY_LOAD (vld3q_dup_mf8, mf8qx3, mf8_scalar_const_ptr, UNSPEC_LD3_DUP)
-
-ENTRY_LOAD (vld4_dup_mf8, mf8x4, mf8_scalar_const_ptr, UNSPEC_LD4_DUP)
-ENTRY_LOAD (vld4q_dup_mf8, mf8qx4, mf8_scalar_const_ptr, UNSPEC_LD4_DUP)
-#undef REQUIRED_EXTENSIONS
-
// mmla f16 mf8
#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_F8F16MM)
ENTRY_TERNARY (vmmlaq_f16_mf8, f16q, f16q, mf8q, mf8q, UNSPEC_FMMLA, FP8)
diff --git a/gcc/config/aarch64/aarch64-simd.md
b/gcc/config/aarch64/aarch64-simd.md
index 8de8d3d5d34..cb8894c1b8f 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -8405,38 +8405,6 @@ (define_insn "*sqrt<mode>2<vczle><vczbe>"
[(set_attr "type" "neon_fp_sqrt_<stype><q>")]
)
-;; Patterns for vector struct loads and stores.
-
-(define_insn "@aarch64_simd_ld2r<vstruct_elt>"
- [(set (match_operand:VSTRUCT_2QD 0 "register_operand" "=w")
- (unspec:VSTRUCT_2QD [
- (match_operand:BLK 1 "aarch64_simd_struct_operand" "Utv")]
- UNSPEC_LD2_DUP))]
- "TARGET_SIMD"
- "ld2r\\t{%S0.<Vtype> - %T0.<Vtype>}, %1"
- [(set_attr "type" "neon_load2_all_lanes<q>")]
-)
-
-(define_insn "@aarch64_simd_ld3r<vstruct_elt>"
- [(set (match_operand:VSTRUCT_3QD 0 "register_operand" "=w")
- (unspec:VSTRUCT_3QD [
- (match_operand:BLK 1 "aarch64_simd_struct_operand" "Utv")]
- UNSPEC_LD3_DUP))]
- "TARGET_SIMD"
- "ld3r\\t{%S0.<Vtype> - %U0.<Vtype>}, %1"
- [(set_attr "type" "neon_load3_all_lanes<q>")]
-)
-
-(define_insn "@aarch64_simd_ld4r<vstruct_elt>"
- [(set (match_operand:VSTRUCT_4QD 0 "register_operand" "=w")
- (unspec:VSTRUCT_4QD [
- (match_operand:BLK 1 "aarch64_simd_struct_operand" "Utv")]
- UNSPEC_LD4_DUP))]
- "TARGET_SIMD"
- "ld4r\\t{%S0.<Vtype> - %V0.<Vtype>}, %1"
- [(set_attr "type" "neon_load4_all_lanes<q>")]
-)
-
;; Patterns for rcpc3 vector lane loads and stores.
(define_insn "aarch64_vec_stl1_lanes<mode>_lane<Vel>"
@@ -8859,17 +8827,23 @@ (define_split
FAIL;
})
-(define_expand "aarch64_ld<nregs>r<vstruct_elt>"
- [(match_operand:VSTRUCT_QD 0 "register_operand")
- (match_operand:DI 1 "register_operand")]
+(define_insn "@aarch64_simd_load_dup<mode>"
+ [(set (match_operand:VSTRUCT_QD 0 "register_operand" "=w")
+ (unspec:VSTRUCT_QD [
+ (match_operand:BLK 1 "aarch64_simd_struct_operand" "Utv")]
+ UNSPEC_LOAD_DUP))]
"TARGET_SIMD"
-{
- rtx mem = gen_rtx_MEM (BLKmode, operands[1]);
- set_mem_size (mem, GET_MODE_SIZE (GET_MODE_INNER (<MODE>mode)) * <nregs>);
-
- emit_insn (gen_aarch64_simd_ld<nregs>r<vstruct_elt> (operands[0], mem));
- DONE;
-})
+ {
+ switch (<nregs>)
+ {
+ case 2: return "ld2r\t{%S0.<Vtype> - %T0.<Vtype>}, %1";
+ case 3: return "ld3r\t{%S0.<Vtype> - %U0.<Vtype>}, %1";
+ case 4: return "ld4r\t{%S0.<Vtype> - %V0.<Vtype>}, %1";
+ default: gcc_unreachable ();
+ }
+ }
+ [(set_attr "type" "neon_load<nregs>_all_lanes<q>")]
+)
(define_insn "@aarch64_simd_load_lane<mode>"
[(set (match_operand:VSTRUCT_QD 0 "register_operand" "=w")
diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
index f55342db924..cf7b1f18be0 100644
--- a/gcc/config/aarch64/aarch64.md
+++ b/gcc/config/aarch64/aarch64.md
@@ -239,7 +239,6 @@ (define_c_enum "unspec" [
UNSPEC_CRC32H
UNSPEC_CRC32W
UNSPEC_CRC32X
- UNSPEC_DUP
UNSPEC_FCVTZS
UNSPEC_FCVTZU
UNSPEC_FJCVTZS
@@ -269,10 +268,7 @@ (define_c_enum "unspec" [
UNSPEC_LOAD_MULTIPLE
UNSPEC_LOAD_INTERLEAVED
UNSPEC_LOAD_LANE
- UNSPEC_LD1_DUP
- UNSPEC_LD2_DUP
- UNSPEC_LD3_DUP
- UNSPEC_LD4_DUP
+ UNSPEC_LOAD_DUP
UNSPEC_LD64B
UNSPEC_ST64B
UNSPEC_ST64BV
diff --git a/gcc/config/aarch64/arm_neon.h b/gcc/config/aarch64/arm_neon.h
index e9ae7a6938b..5996ef92c4e 100644
--- a/gcc/config/aarch64/arm_neon.h
+++ b/gcc/config/aarch64/arm_neon.h
@@ -5587,832 +5587,6 @@ vcvtpq_u64_f64 (float64x2_t __a)
return __builtin_aarch64_lceiluv2dfv2di_us (__a);
}
-/* vld1_dup */
-
-__extension__ extern __inline float16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_f16 (const float16_t* __a)
-{
- return vdup_n_f16 (*__a);
-}
-
-__extension__ extern __inline float32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_f32 (const float32_t* __a)
-{
- return vdup_n_f32 (*__a);
-}
-
-__extension__ extern __inline float64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_f64 (const float64_t* __a)
-{
- return vdup_n_f64 (*__a);
-}
-
-__extension__ extern __inline poly8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_p8 (const poly8_t* __a)
-{
- return vdup_n_p8 (*__a);
-}
-
-__extension__ extern __inline poly16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_p16 (const poly16_t* __a)
-{
- return vdup_n_p16 (*__a);
-}
-
-__extension__ extern __inline poly64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_p64 (const poly64_t* __a)
-{
- return vdup_n_p64 (*__a);
-}
-
-__extension__ extern __inline int8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_s8 (const int8_t* __a)
-{
- return vdup_n_s8 (*__a);
-}
-
-__extension__ extern __inline int16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_s16 (const int16_t* __a)
-{
- return vdup_n_s16 (*__a);
-}
-
-__extension__ extern __inline int32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_s32 (const int32_t* __a)
-{
- return vdup_n_s32 (*__a);
-}
-
-__extension__ extern __inline int64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_s64 (const int64_t* __a)
-{
- return vdup_n_s64 (*__a);
-}
-
-__extension__ extern __inline uint8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_u8 (const uint8_t* __a)
-{
- return vdup_n_u8 (*__a);
-}
-
-__extension__ extern __inline uint16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_u16 (const uint16_t* __a)
-{
- return vdup_n_u16 (*__a);
-}
-
-__extension__ extern __inline uint32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_u32 (const uint32_t* __a)
-{
- return vdup_n_u32 (*__a);
-}
-
-__extension__ extern __inline uint64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_u64 (const uint64_t* __a)
-{
- return vdup_n_u64 (*__a);
-}
-
-/* vld1q_dup */
-
-__extension__ extern __inline float16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_f16 (const float16_t* __a)
-{
- return vdupq_n_f16 (*__a);
-}
-
-__extension__ extern __inline float32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_f32 (const float32_t* __a)
-{
- return vdupq_n_f32 (*__a);
-}
-
-__extension__ extern __inline float64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_f64 (const float64_t* __a)
-{
- return vdupq_n_f64 (*__a);
-}
-
-__extension__ extern __inline poly8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_p8 (const poly8_t* __a)
-{
- return vdupq_n_p8 (*__a);
-}
-
-__extension__ extern __inline poly16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_p16 (const poly16_t* __a)
-{
- return vdupq_n_p16 (*__a);
-}
-
-__extension__ extern __inline poly64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_p64 (const poly64_t* __a)
-{
- return vdupq_n_p64 (*__a);
-}
-
- __extension__ extern __inline int8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_s8 (const int8_t* __a)
-{
- return vdupq_n_s8 (*__a);
-}
-
-__extension__ extern __inline int16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_s16 (const int16_t* __a)
-{
- return vdupq_n_s16 (*__a);
-}
-
-__extension__ extern __inline int32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_s32 (const int32_t* __a)
-{
- return vdupq_n_s32 (*__a);
-}
-
-__extension__ extern __inline int64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_s64 (const int64_t* __a)
-{
- return vdupq_n_s64 (*__a);
-}
-
-__extension__ extern __inline uint8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_u8 (const uint8_t* __a)
-{
- return vdupq_n_u8 (*__a);
-}
-
-__extension__ extern __inline uint16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_u16 (const uint16_t* __a)
-{
- return vdupq_n_u16 (*__a);
-}
-
-__extension__ extern __inline uint32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_u32 (const uint32_t* __a)
-{
- return vdupq_n_u32 (*__a);
-}
-
-__extension__ extern __inline uint64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_u64 (const uint64_t* __a)
-{
- return vdupq_n_u64 (*__a);
-}
-
-/* vldn_dup */
-
-__extension__ extern __inline int8x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld2rv8qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld2rv4hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld2rv2si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline float16x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld2rv4hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld2rv2sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline float64x1x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld2rdf ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline uint8x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld2rv8qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld2rv4hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld2rv2si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline poly8x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld2rv8qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly16x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld2rv4hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly64x1x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld2rdi_ps ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int64x1x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld2rdi ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint64x1x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld2rdi_us ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int8x16x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld2rv16qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly8x16x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld2rv16qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld2rv8hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly16x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld2rv8hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld2rv4si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline int64x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld2rv2di ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint8x16x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld2rv16qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld2rv8hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld2rv4si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline uint64x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld2rv2di_us (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline float16x8x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld2rv8hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x4x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld2rv4sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline float64x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld2rv2df ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline poly64x2x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld2q_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld2rv2di_ps (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int64x1x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld3rdi ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint64x1x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld3rdi_us ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline float64x1x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld3rdf ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline int8x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld3rv8qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly8x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld3rv8qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld3rv4hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly16x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld3rv4hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld3rv2si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline uint8x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld3rv8qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld3rv4hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld3rv2si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline float16x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld3rv4hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld3rv2sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline poly64x1x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld3rdi_ps ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int8x16x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld3rv16qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly8x16x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld3rv16qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld3rv8hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly16x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld3rv8hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld3rv4si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline int64x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld3rv2di ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint8x16x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld3rv16qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld3rv8hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld3rv4si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline uint64x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld3rv2di_us (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline float16x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld3rv8hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld3rv4sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline float64x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld3rv2df ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline poly64x2x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld3rv2di_ps (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int64x1x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld4rdi ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint64x1x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld4rdi_us ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline float64x1x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld4rdf ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline int8x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld4rv8qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly8x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld4rv8qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld4rv4hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly16x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld4rv4hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld4rv2si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline uint8x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld4rv8qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld4rv4hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld4rv2si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline float16x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld4rv4hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld4rv2sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline poly64x1x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld4rdi_ps ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline int8x16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_s8 (const int8_t * __a)
-{
- return __builtin_aarch64_ld4rv16qi ((const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline poly8x16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_p8 (const poly8_t * __a)
-{
- return __builtin_aarch64_ld4rv16qi_ps (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline int16x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_s16 (const int16_t * __a)
-{
- return __builtin_aarch64_ld4rv8hi ((const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline poly16x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_p16 (const poly16_t * __a)
-{
- return __builtin_aarch64_ld4rv8hi_ps (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline int32x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_s32 (const int32_t * __a)
-{
- return __builtin_aarch64_ld4rv4si ((const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline int64x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_s64 (const int64_t * __a)
-{
- return __builtin_aarch64_ld4rv2di ((const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline uint8x16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_u8 (const uint8_t * __a)
-{
- return __builtin_aarch64_ld4rv16qi_us (
- (const __builtin_aarch64_simd_qi *) __a);
-}
-
-__extension__ extern __inline uint16x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_u16 (const uint16_t * __a)
-{
- return __builtin_aarch64_ld4rv8hi_us (
- (const __builtin_aarch64_simd_hi *) __a);
-}
-
-__extension__ extern __inline uint32x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_u32 (const uint32_t * __a)
-{
- return __builtin_aarch64_ld4rv4si_us (
- (const __builtin_aarch64_simd_si *) __a);
-}
-
-__extension__ extern __inline uint64x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_u64 (const uint64_t * __a)
-{
- return __builtin_aarch64_ld4rv2di_us (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
-__extension__ extern __inline float16x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_f16 (const float16_t * __a)
-{
- return __builtin_aarch64_ld4rv8hf ((const __builtin_aarch64_simd_hf *) __a);
-}
-
-__extension__ extern __inline float32x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_f32 (const float32_t * __a)
-{
- return __builtin_aarch64_ld4rv4sf ((const __builtin_aarch64_simd_sf *) __a);
-}
-
-__extension__ extern __inline float64x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_f64 (const float64_t * __a)
-{
- return __builtin_aarch64_ld4rv2df ((const __builtin_aarch64_simd_df *) __a);
-}
-
-__extension__ extern __inline poly64x2x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_p64 (const poly64_t * __a)
-{
- return __builtin_aarch64_ld4rv2di_ps (
- (const __builtin_aarch64_simd_di *) __a);
-}
-
/* vmulx */
__extension__ extern __inline float32x2_t
@@ -9813,52 +8987,6 @@ vrnd64xq_f64 (float64x2_t __a)
#pragma GCC push_options
#pragma GCC target ("+nothing+bf16")
-/* vld */
-
-__extension__ extern __inline bfloat16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1_dup_bf16 (const bfloat16_t* __a)
-{
- return vdup_n_bf16 (*__a);
-}
-
-__extension__ extern __inline bfloat16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld1q_dup_bf16 (const bfloat16_t* __a)
-{
- return vdupq_n_bf16 (*__a);
-}
-
-__extension__ extern __inline bfloat16x4x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3_dup_bf16 (const bfloat16_t * __a)
-{
- return __builtin_aarch64_ld3rv4bf ((const __builtin_aarch64_simd_bf *) __a);
-}
-
-__extension__ extern __inline bfloat16x8x3_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld3q_dup_bf16 (const bfloat16_t * __a)
-{
- return __builtin_aarch64_ld3rv8bf ((const __builtin_aarch64_simd_bf *) __a);
-}
-
-__extension__ extern __inline bfloat16x4x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4_dup_bf16 (const bfloat16_t * __a)
-{
- return __builtin_aarch64_ld4rv4bf ((const __builtin_aarch64_simd_bf *) __a);
-}
-
-__extension__ extern __inline bfloat16x8x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vld4q_dup_bf16 (const bfloat16_t * __a)
-{
- return __builtin_aarch64_ld4rv8bf ((const __builtin_aarch64_simd_bf *) __a);
-}
-
-/* vst */
-
__extension__ extern __inline float32x2_t
__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
vbfdot_f32 (float32x2_t __r, bfloat16x4_t __a, bfloat16x4_t __b)
diff --git a/gcc/testsuite/gcc.target/aarch64/neon/vld1_dup.c
b/gcc/testsuite/gcc.target/aarch64/neon/vld1_dup.c
new file mode 100644
index 00000000000..c7cb7ec1415
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/neon/vld1_dup.c
@@ -0,0 +1,228 @@
+/* { dg-do compile } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include "arm_neon_test.h"
+
+/*
+** test_vld1_dup_f16:
+** ld1r {v0\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_f16, float16x4_t, const float16_t *)
+
+/*
+** test_vld1_dup_bf16:
+** ld1r {v0\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_bf16, bfloat16x4_t, const bfloat16_t *)
+
+/*
+** test_vld1_dup_f32:
+** ld1r {v0\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_f32, float32x2_t, const float32_t *)
+
+/*
+** test_vld1_dup_f64:
+** ldr d0, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_f64, float64x1_t, const float64_t *)
+
+/*
+** test_vld1_dup_mf8:
+** ld1r {v0\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_mf8, mfloat8x8_t, const mfloat8_t *)
+
+/*
+** test_vld1_dup_s8:
+** ld1r {v0\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_s8, int8x8_t, const int8_t *)
+
+/*
+** test_vld1_dup_s16:
+** ld1r {v0\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_s16, int16x4_t, const int16_t *)
+
+/*
+** test_vld1_dup_s32:
+** ld1r {v0\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_s32, int32x2_t, const int32_t *)
+
+/*
+** test_vld1_dup_s64:
+** ldr d0, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_s64, int64x1_t, const int64_t *)
+
+/*
+** test_vld1_dup_u8:
+** ld1r {v0\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_u8, uint8x8_t, const uint8_t *)
+
+/*
+** test_vld1_dup_u16:
+** ld1r {v0\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_u16, uint16x4_t, const uint16_t *)
+
+/*
+** test_vld1_dup_u32:
+** ld1r {v0\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_u32, uint32x2_t, const uint32_t *)
+
+/*
+** test_vld1_dup_u64:
+** ldr d0, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_u64, uint64x1_t, const uint64_t *)
+
+/*
+** test_vld1_dup_p8:
+** ld1r {v0\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_p8, poly8x8_t, const poly8_t *)
+
+/*
+** test_vld1_dup_p16:
+** ld1r {v0\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_p16, poly16x4_t, const poly16_t *)
+
+/*
+** test_vld1_dup_p64:
+** ldr d0, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1_dup_p64, poly64x1_t, const poly64_t *)
+
+/*
+** test_vld1q_dup_bf16:
+** ld1r {v0\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_bf16, bfloat16x8_t, const bfloat16_t *)
+
+/*
+** test_vld1q_dup_f16:
+** ld1r {v0\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_f16, float16x8_t, const float16_t *)
+
+/*
+** test_vld1q_dup_f32:
+** ld1r {v0\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_f32, float32x4_t, const float32_t *)
+
+/*
+** test_vld1q_dup_f64:
+** ld1r {v0\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_f64, float64x2_t, const float64_t *)
+
+/*
+** test_vld1q_dup_mf8:
+** ld1r {v0\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_mf8, mfloat8x16_t, const mfloat8_t *)
+
+/*
+** test_vld1q_dup_s8:
+** ld1r {v0\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_s8, int8x16_t, const int8_t *)
+
+/*
+** test_vld1q_dup_s16:
+** ld1r {v0\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_s16, int16x8_t, const int16_t *)
+
+/*
+** test_vld1q_dup_s32:
+** ld1r {v0\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_s32, int32x4_t, const int32_t *)
+
+/*
+** test_vld1q_dup_s64:
+** ld1r {v0\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_s64, int64x2_t, const int64_t *)
+
+/*
+** test_vld1q_dup_u8:
+** ld1r {v0\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_u8, uint8x16_t, const uint8_t *)
+
+/*
+** test_vld1q_dup_u16:
+** ld1r {v0\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_u16, uint16x8_t, const uint16_t *)
+
+/*
+** test_vld1q_dup_u32:
+** ld1r {v0\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_u32, uint32x4_t, const uint32_t *)
+
+/*
+** test_vld1q_dup_u64:
+** ld1r {v0\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_u64, uint64x2_t, const uint64_t *)
+
+/*
+** test_vld1q_dup_p8:
+** ld1r {v0\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_p8, poly8x16_t, const poly8_t *)
+
+/*
+** test_vld1q_dup_p16:
+** ld1r {v0\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_p16, poly16x8_t, const poly16_t *)
+
+/*
+** test_vld1q_dup_p64:
+** ld1r {v0\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld1q_dup_p64, poly64x2_t, const poly64_t *)
diff --git a/gcc/testsuite/gcc.target/aarch64/neon/vld2_dup.c
b/gcc/testsuite/gcc.target/aarch64/neon/vld2_dup.c
new file mode 100644
index 00000000000..e259371062a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/neon/vld2_dup.c
@@ -0,0 +1,228 @@
+/* { dg-do compile } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include "arm_neon_test.h"
+
+/*
+** test_vld2_dup_f16:
+** ld2r {v0\.4h - v1\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_f16, float16x4x2_t, const float16_t *)
+
+/*
+** test_vld2_dup_bf16:
+** ld2r {v0\.4h - v1\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_bf16, bfloat16x4x2_t, const bfloat16_t *)
+
+/*
+** test_vld2_dup_f32:
+** ld2r {v0\.2s - v1\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_f32, float32x2x2_t, const float32_t *)
+
+/*
+** test_vld2_dup_f64:
+** ld2r {v0\.1d - v1\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_f64, float64x1x2_t, const float64_t *)
+
+/*
+** test_vld2_dup_mf8:
+** ld2r {v0\.8b - v1\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_mf8, mfloat8x8x2_t, const mfloat8_t *)
+
+/*
+** test_vld2_dup_s8:
+** ld2r {v0\.8b - v1\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_s8, int8x8x2_t, const int8_t *)
+
+/*
+** test_vld2_dup_s16:
+** ld2r {v0\.4h - v1\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_s16, int16x4x2_t, const int16_t *)
+
+/*
+** test_vld2_dup_s32:
+** ld2r {v0\.2s - v1\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_s32, int32x2x2_t, const int32_t *)
+
+/*
+** test_vld2_dup_s64:
+** ld2r {v0\.1d - v1\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_s64, int64x1x2_t, const int64_t *)
+
+/*
+** test_vld2_dup_u8:
+** ld2r {v0\.8b - v1\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_u8, uint8x8x2_t, const uint8_t *)
+
+/*
+** test_vld2_dup_u16:
+** ld2r {v0\.4h - v1\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_u16, uint16x4x2_t, const uint16_t *)
+
+/*
+** test_vld2_dup_u32:
+** ld2r {v0\.2s - v1\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_u32, uint32x2x2_t, const uint32_t *)
+
+/*
+** test_vld2_dup_u64:
+** ld2r {v0\.1d - v1\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_u64, uint64x1x2_t, const uint64_t *)
+
+/*
+** test_vld2_dup_p8:
+** ld2r {v0\.8b - v1\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_p8, poly8x8x2_t, const poly8_t *)
+
+/*
+** test_vld2_dup_p16:
+** ld2r {v0\.4h - v1\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_p16, poly16x4x2_t, const poly16_t *)
+
+/*
+** test_vld2_dup_p64:
+** ld2r {v0\.1d - v1\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2_dup_p64, poly64x1x2_t, const poly64_t *)
+
+/*
+** test_vld2q_dup_bf16:
+** ld2r {v0\.8h - v1\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_bf16, bfloat16x8x2_t, const bfloat16_t *)
+
+/*
+** test_vld2q_dup_f16:
+** ld2r {v0\.8h - v1\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_f16, float16x8x2_t, const float16_t *)
+
+/*
+** test_vld2q_dup_f32:
+** ld2r {v0\.4s - v1\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_f32, float32x4x2_t, const float32_t *)
+
+/*
+** test_vld2q_dup_f64:
+** ld2r {v0\.2d - v1\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_f64, float64x2x2_t, const float64_t *)
+
+/*
+** test_vld2q_dup_mf8:
+** ld2r {v0\.16b - v1\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_mf8, mfloat8x16x2_t, const mfloat8_t *)
+
+/*
+** test_vld2q_dup_s8:
+** ld2r {v0\.16b - v1\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_s8, int8x16x2_t, const int8_t *)
+
+/*
+** test_vld2q_dup_s16:
+** ld2r {v0\.8h - v1\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_s16, int16x8x2_t, const int16_t *)
+
+/*
+** test_vld2q_dup_s32:
+** ld2r {v0\.4s - v1\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_s32, int32x4x2_t, const int32_t *)
+
+/*
+** test_vld2q_dup_s64:
+** ld2r {v0\.2d - v1\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_s64, int64x2x2_t, const int64_t *)
+
+/*
+** test_vld2q_dup_u8:
+** ld2r {v0\.16b - v1\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_u8, uint8x16x2_t, const uint8_t *)
+
+/*
+** test_vld2q_dup_u16:
+** ld2r {v0\.8h - v1\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_u16, uint16x8x2_t, const uint16_t *)
+
+/*
+** test_vld2q_dup_u32:
+** ld2r {v0\.4s - v1\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_u32, uint32x4x2_t, const uint32_t *)
+
+/*
+** test_vld2q_dup_u64:
+** ld2r {v0\.2d - v1\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_u64, uint64x2x2_t, const uint64_t *)
+
+/*
+** test_vld2q_dup_p8:
+** ld2r {v0\.16b - v1\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_p8, poly8x16x2_t, const poly8_t *)
+
+/*
+** test_vld2q_dup_p16:
+** ld2r {v0\.8h - v1\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_p16, poly16x8x2_t, const poly16_t *)
+
+/*
+** test_vld2q_dup_p64:
+** ld2r {v0\.2d - v1\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld2q_dup_p64, poly64x2x2_t, const poly64_t *)
diff --git a/gcc/testsuite/gcc.target/aarch64/neon/vld3_dup.c
b/gcc/testsuite/gcc.target/aarch64/neon/vld3_dup.c
new file mode 100644
index 00000000000..1f7bfd808ee
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/neon/vld3_dup.c
@@ -0,0 +1,228 @@
+/* { dg-do compile } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include "arm_neon_test.h"
+
+/*
+** test_vld3_dup_f16:
+** ld3r {v0\.4h - v2\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_f16, float16x4x3_t, const float16_t *)
+
+/*
+** test_vld3_dup_bf16:
+** ld3r {v0\.4h - v2\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_bf16, bfloat16x4x3_t, const bfloat16_t *)
+
+/*
+** test_vld3_dup_f32:
+** ld3r {v0\.2s - v2\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_f32, float32x2x3_t, const float32_t *)
+
+/*
+** test_vld3_dup_f64:
+** ld3r {v0\.1d - v2\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_f64, float64x1x3_t, const float64_t *)
+
+/*
+** test_vld3_dup_mf8:
+** ld3r {v0\.8b - v2\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_mf8, mfloat8x8x3_t, const mfloat8_t *)
+
+/*
+** test_vld3_dup_s8:
+** ld3r {v0\.8b - v2\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_s8, int8x8x3_t, const int8_t *)
+
+/*
+** test_vld3_dup_s16:
+** ld3r {v0\.4h - v2\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_s16, int16x4x3_t, const int16_t *)
+
+/*
+** test_vld3_dup_s32:
+** ld3r {v0\.2s - v2\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_s32, int32x2x3_t, const int32_t *)
+
+/*
+** test_vld3_dup_s64:
+** ld3r {v0\.1d - v2\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_s64, int64x1x3_t, const int64_t *)
+
+/*
+** test_vld3_dup_u8:
+** ld3r {v0\.8b - v2\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_u8, uint8x8x3_t, const uint8_t *)
+
+/*
+** test_vld3_dup_u16:
+** ld3r {v0\.4h - v2\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_u16, uint16x4x3_t, const uint16_t *)
+
+/*
+** test_vld3_dup_u32:
+** ld3r {v0\.2s - v2\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_u32, uint32x2x3_t, const uint32_t *)
+
+/*
+** test_vld3_dup_u64:
+** ld3r {v0\.1d - v2\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_u64, uint64x1x3_t, const uint64_t *)
+
+/*
+** test_vld3_dup_p8:
+** ld3r {v0\.8b - v2\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_p8, poly8x8x3_t, const poly8_t *)
+
+/*
+** test_vld3_dup_p16:
+** ld3r {v0\.4h - v2\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_p16, poly16x4x3_t, const poly16_t *)
+
+/*
+** test_vld3_dup_p64:
+** ld3r {v0\.1d - v2\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3_dup_p64, poly64x1x3_t, const poly64_t *)
+
+/*
+** test_vld3q_dup_bf16:
+** ld3r {v0\.8h - v2\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_bf16, bfloat16x8x3_t, const bfloat16_t *)
+
+/*
+** test_vld3q_dup_f16:
+** ld3r {v0\.8h - v2\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_f16, float16x8x3_t, const float16_t *)
+
+/*
+** test_vld3q_dup_f32:
+** ld3r {v0\.4s - v2\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_f32, float32x4x3_t, const float32_t *)
+
+/*
+** test_vld3q_dup_f64:
+** ld3r {v0\.2d - v2\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_f64, float64x2x3_t, const float64_t *)
+
+/*
+** test_vld3q_dup_mf8:
+** ld3r {v0\.16b - v2\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_mf8, mfloat8x16x3_t, const mfloat8_t *)
+
+/*
+** test_vld3q_dup_s8:
+** ld3r {v0\.16b - v2\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_s8, int8x16x3_t, const int8_t *)
+
+/*
+** test_vld3q_dup_s16:
+** ld3r {v0\.8h - v2\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_s16, int16x8x3_t, const int16_t *)
+
+/*
+** test_vld3q_dup_s32:
+** ld3r {v0\.4s - v2\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_s32, int32x4x3_t, const int32_t *)
+
+/*
+** test_vld3q_dup_s64:
+** ld3r {v0\.2d - v2\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_s64, int64x2x3_t, const int64_t *)
+
+/*
+** test_vld3q_dup_u8:
+** ld3r {v0\.16b - v2\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_u8, uint8x16x3_t, const uint8_t *)
+
+/*
+** test_vld3q_dup_u16:
+** ld3r {v0\.8h - v2\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_u16, uint16x8x3_t, const uint16_t *)
+
+/*
+** test_vld3q_dup_u32:
+** ld3r {v0\.4s - v2\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_u32, uint32x4x3_t, const uint32_t *)
+
+/*
+** test_vld3q_dup_u64:
+** ld3r {v0\.2d - v2\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_u64, uint64x2x3_t, const uint64_t *)
+
+/*
+** test_vld3q_dup_p8:
+** ld3r {v0\.16b - v2\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_p8, poly8x16x3_t, const poly8_t *)
+
+/*
+** test_vld3q_dup_p16:
+** ld3r {v0\.8h - v2\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_p16, poly16x8x3_t, const poly16_t *)
+
+/*
+** test_vld3q_dup_p64:
+** ld3r {v0\.2d - v2\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld3q_dup_p64, poly64x2x3_t, const poly64_t *)
diff --git a/gcc/testsuite/gcc.target/aarch64/neon/vld4_dup.c
b/gcc/testsuite/gcc.target/aarch64/neon/vld4_dup.c
new file mode 100644
index 00000000000..d33e71dbcca
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/neon/vld4_dup.c
@@ -0,0 +1,228 @@
+/* { dg-do compile } */
+/* { dg-final { check-function-bodies "**" "" } } */
+
+#include "arm_neon_test.h"
+
+/*
+** test_vld4_dup_f16:
+** ld4r {v0\.4h - v3\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_f16, float16x4x4_t, const float16_t *)
+
+/*
+** test_vld4_dup_bf16:
+** ld4r {v0\.4h - v3\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_bf16, bfloat16x4x4_t, const bfloat16_t *)
+
+/*
+** test_vld4_dup_f32:
+** ld4r {v0\.2s - v3\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_f32, float32x2x4_t, const float32_t *)
+
+/*
+** test_vld4_dup_f64:
+** ld4r {v0\.1d - v3\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_f64, float64x1x4_t, const float64_t *)
+
+/*
+** test_vld4_dup_mf8:
+** ld4r {v0\.8b - v3\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_mf8, mfloat8x8x4_t, const mfloat8_t *)
+
+/*
+** test_vld4_dup_s8:
+** ld4r {v0\.8b - v3\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_s8, int8x8x4_t, const int8_t *)
+
+/*
+** test_vld4_dup_s16:
+** ld4r {v0\.4h - v3\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_s16, int16x4x4_t, const int16_t *)
+
+/*
+** test_vld4_dup_s32:
+** ld4r {v0\.2s - v3\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_s32, int32x2x4_t, const int32_t *)
+
+/*
+** test_vld4_dup_s64:
+** ld4r {v0\.1d - v3\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_s64, int64x1x4_t, const int64_t *)
+
+/*
+** test_vld4_dup_u8:
+** ld4r {v0\.8b - v3\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_u8, uint8x8x4_t, const uint8_t *)
+
+/*
+** test_vld4_dup_u16:
+** ld4r {v0\.4h - v3\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_u16, uint16x4x4_t, const uint16_t *)
+
+/*
+** test_vld4_dup_u32:
+** ld4r {v0\.2s - v3\.2s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_u32, uint32x2x4_t, const uint32_t *)
+
+/*
+** test_vld4_dup_u64:
+** ld4r {v0\.1d - v3\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_u64, uint64x1x4_t, const uint64_t *)
+
+/*
+** test_vld4_dup_p8:
+** ld4r {v0\.8b - v3\.8b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_p8, poly8x8x4_t, const poly8_t *)
+
+/*
+** test_vld4_dup_p16:
+** ld4r {v0\.4h - v3\.4h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_p16, poly16x4x4_t, const poly16_t *)
+
+/*
+** test_vld4_dup_p64:
+** ld4r {v0\.1d - v3\.1d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4_dup_p64, poly64x1x4_t, const poly64_t *)
+
+/*
+** test_vld4q_dup_bf16:
+** ld4r {v0\.8h - v3\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_bf16, bfloat16x8x4_t, const bfloat16_t *)
+
+/*
+** test_vld4q_dup_f16:
+** ld4r {v0\.8h - v3\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_f16, float16x8x4_t, const float16_t *)
+
+/*
+** test_vld4q_dup_f32:
+** ld4r {v0\.4s - v3\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_f32, float32x4x4_t, const float32_t *)
+
+/*
+** test_vld4q_dup_f64:
+** ld4r {v0\.2d - v3\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_f64, float64x2x4_t, const float64_t *)
+
+/*
+** test_vld4q_dup_mf8:
+** ld4r {v0\.16b - v3\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_mf8, mfloat8x16x4_t, const mfloat8_t *)
+
+/*
+** test_vld4q_dup_s8:
+** ld4r {v0\.16b - v3\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_s8, int8x16x4_t, const int8_t *)
+
+/*
+** test_vld4q_dup_s16:
+** ld4r {v0\.8h - v3\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_s16, int16x8x4_t, const int16_t *)
+
+/*
+** test_vld4q_dup_s32:
+** ld4r {v0\.4s - v3\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_s32, int32x4x4_t, const int32_t *)
+
+/*
+** test_vld4q_dup_s64:
+** ld4r {v0\.2d - v3\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_s64, int64x2x4_t, const int64_t *)
+
+/*
+** test_vld4q_dup_u8:
+** ld4r {v0\.16b - v3\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_u8, uint8x16x4_t, const uint8_t *)
+
+/*
+** test_vld4q_dup_u16:
+** ld4r {v0\.8h - v3\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_u16, uint16x8x4_t, const uint16_t *)
+
+/*
+** test_vld4q_dup_u32:
+** ld4r {v0\.4s - v3\.4s}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_u32, uint32x4x4_t, const uint32_t *)
+
+/*
+** test_vld4q_dup_u64:
+** ld4r {v0\.2d - v3\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_u64, uint64x2x4_t, const uint64_t *)
+
+/*
+** test_vld4q_dup_p8:
+** ld4r {v0\.16b - v3\.16b}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_p8, poly8x16x4_t, const poly8_t *)
+
+/*
+** test_vld4q_dup_p16:
+** ld4r {v0\.8h - v3\.8h}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_p16, poly16x8x4_t, const poly16_t *)
+
+/*
+** test_vld4q_dup_p64:
+** ld4r {v0\.2d - v3\.2d}, \[x0\]
+** ret
+*/
+TEST_UNARY (vld4q_dup_p64, poly64x2x4_t, const poly64_t *)
--
2.56.0
