This is more of a PoC than something I expect to get in, but it's here to show
how it's supposed to work and to test the code paths in the middle-end.
Re-vectorization supports folding from one ISA to another.
It is up to the target to implement this however. For SVE this can mean
creating a predicate for the operation. To do this I have picked an operation
that hasn't been lowered to GIMPLE, as GIMPLE lowering isn't needed for this to
work and can actually make it harder.
The example is to allow the vectorized to fold usqadd from Adv. SIMD to SVE
based on costing.
e.g.
#include <arm_neon.h>
#include <arm_sve.h>
void mixed_add (uint8x8_t *restrict out, const uint8x8_t *a,
const int8x8_t *b, unsigned n)
{
for (unsigned i = 0; i < n; ++i)
out[i] = vsqadd_u8 (a[i], b[i]);
}
mixed_sign_sat_add::fold_mask_call selects vsqadd/vuqadd, their q forms, or
predicated svsqadd/svuqadd from the requested mode and signedness. The shared
resolver checks the argument types and required extensions. This allows two
eight-byte Adv. SIMD calls to combine into one wider call, or an SVE2 call to
use the loop mask for a partial final vector.
Including arm_sve.h registers the SVE instances used here.
But we should have a think about how to properly organize this. See patch 12.
With -O3 -march=armv8.5-a+sve2 -msve-vector-bits=256 we generated before:
ldr d31, [x1, x4]
ldr d30, [x2, x4]
usqadd v31.8b, v30.8b
str d31, [x0, x4]
After these patches we get:
ld1b z31.b, p7/z, [x1, x4]
ld1b z30.b, p7/z, [x2, x4]
usqadd z31.b, p7/m, z31.b, z30.b
st1b z31.b, p7, [x0, x4]
The optimized GIMPLE uses __builtin_aarch64_svsqadd_u8_x with the loop mask
and signed/unsigned inputs. The loop advances by 32 bytes rather than eight,
with whilelo handling the tail.
We also move integer pairwise min/max into the ACLE framework. These must keep
their original width: combining two vpmax_u8 calls requires:
pairmax(A0), pairmax(B0), pairmax(A1), pairmax(B1)
but widening the instruction on concatenated inputs would produce:
pairmax(A0), pairmax(A1), pairmax(B0), pairmax(B1)
pairwise_maxmin therefore redirects to the same function instance and lets
signature checking reject a width change.
Both classes use the existing instructions through parameterized RTL patterns.
Registered builtins replace the arm_neon.h wrappers with the same public
signatures. V1DI uses the scalar DI operand mode of its existing instruction
and restores the vector view afterwards.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Any feedback?
Thanks,
Tamar
Assisted-by: Codex GPT 6.1 Sol
AI was used to generate test cases for various types and inputs.
gcc/ChangeLog:
PR tree-optimization/119187
* config/aarch64/aarch64-neon-builtins-base.cc (mixed_sign_sat_add,
pairwise_maxmin): New.
(vsqadd, vsqaddq, vuqadd, vuqaddq, vpmax, vpmaxq, vpmin, vpminq):
Register.
* config/aarch64/aarch64-neon-builtins-base.def: Register mixed-sign
saturating adds and pairwise integer min/max.
* config/aarch64/aarch64-simd.md: Expose parameterized patterns for
mixed-sign saturating adds and pairwise integer min/max.
* config/aarch64/arm_neon.h: Remove wrappers for the registered
intrinsic implementations.
gcc/testsuite/ChangeLog:
PR tree-optimization/119187
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-11.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-4.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-6.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-7.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-8.c: New test.
* g++.target/aarch64/vect-revectorize-intrinsics.C: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-1.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-10.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-12.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-13.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-15.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-2.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-3.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-5.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-9.c: New test.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-widen.h: New test
support.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold-widths.h: New test
support.
* gcc.target/aarch64/vect-revectorize-intrinsics-fold.h: New test
support.
---
--
diff --git a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
index
839791288bf25c2ee0c7fe09520d3d1409663b3d..210546f3e7cd4c1ed043a1b13dfdc26146ac7d2c
100644
--- a/gcc/config/aarch64/aarch64-neon-builtins-base.cc
+++ b/gcc/config/aarch64/aarch64-neon-builtins-base.cc
@@ -41,6 +41,9 @@
#include "vec-perm-indices.h"
#include "aarch64-acle-builtins.h"
#include "aarch64-neon-builtins-base.h"
+#include "aarch64-neon-builtins-shapes.h"
+#include "aarch64-sve-builtins-sve2.h"
+#include "aarch64-sve-builtins-shapes.h"
#include "aarch64-neon-builtins-functions.h"
#include "gimple-fold.h"
#include "ssa.h"
@@ -214,6 +217,91 @@ template <insn_code insn> struct gimple_exact_insn :
gimple_function_base
}
};
+/* Mixed-sign saturating adds act independently on each lane. Link the
+ two AdvSIMD widths and the SVE2 form here, alongside RTL expansion. */
+class mixed_sign_sat_add : public function_base
+{
+public:
+ gimple *fold_mask_call (gimple_folder &f) const override
+ {
+ function_instance instance (f);
+ machine_mode mode = TYPE_MODE (TREE_TYPE (f.lhs));
+ bool unsigned_p = f.type_suffix (0).unsigned_p;
+ if (aarch64_sve_mode_p (mode))
+ {
+ instance.base_name = unsigned_p ? "svsqadd" : "svuqadd";
+ instance.base = unsigned_p ? functions::svsqadd : functions::svuqadd;
+ instance.shape
+ = unsigned_p ? shapes::binary_int_opt_n : shapes::binary_uint_opt_n;
+ instance.pred = PRED_x;
+ }
+ else if (known_eq (GET_MODE_SIZE (mode), 8))
+ {
+ instance.base_name = unsigned_p ? "vsqadd" : "vuqadd";
+ instance.base = unsigned_p ? functions::vsqadd : functions::vuqadd;
+ instance.shape
+ = unsigned_p ? shapes::vsqadd_all_unsigned_shape
+ : shapes::vuqadd_all_signed_shape;
+ }
+ else if (known_eq (GET_MODE_SIZE (mode), 16))
+ {
+ instance.base_name = unsigned_p ? "vsqaddq" : "vuqaddq";
+ instance.base = unsigned_p ? functions::vsqaddq : functions::vuqaddq;
+ instance.shape
+ = unsigned_p ? shapes::vsqaddq_all_unsigned_shape
+ : shapes::vuqaddq_all_signed_shape;
+ }
+ else
+ return nullptr;
+
+ return f.redirect_mask_call (instance);
+ }
+
+ rtx expand (function_expander &e) const override
+ {
+ machine_mode mode = e.result_mode ();
+ if (mode == V1DImode)
+ {
+ mode = DImode;
+ for (rtx &arg : e.args)
+ arg = gen_lowpart (mode, arg);
+ }
+
+ insn_code icode
+ = code_for_aarch64_qadd (e.type_suffix (0).unsigned_p
+ ? UNSPEC_USQADD : UNSPEC_SUQADD,
+ mode);
+ return gen_lowpart (e.result_mode (), e.use_exact_insn (icode));
+ }
+};
+
+/* Pairwise operations can execute inactive lanes, but changing the vector
+ width would change the order of the results. Signature checking in
+ redirect_mask_call therefore keeps the original vector width. */
+class pairwise_maxmin : public function_base
+{
+ bool m_max_p;
+
+public:
+ constexpr pairwise_maxmin (bool max_p) : m_max_p (max_p) {}
+
+ gimple *fold_mask_call (gimple_folder &f) const override
+ {
+ return f.redirect_mask_call (f);
+ }
+
+ rtx expand (function_expander &e) const override
+ {
+ machine_mode mode = e.result_mode ();
+ bool unsigned_p = e.type_suffix (0).unsigned_p;
+ int unspec
+ = m_max_p ? (unsigned_p ? UNSPEC_UMAXV : UNSPEC_SMAXV)
+ : (unsigned_p ? UNSPEC_UMINV : UNSPEC_SMINV);
+ insn_code icode = code_for_aarch64_p (unspec, mode);
+ return e.use_exact_insn (icode);
+ }
+};
+
/* For intrinsics that map to a single GIMPLE expression with no argument
preparation necessary. */
class gimple_expr : public gimple_function_base
@@ -992,6 +1080,10 @@ NEON_FUNCTION (vmul_lane, vmul_laneq,
vmuld_lane, vmuld_laneq, gimple_arith_lane, (MULT_EXPR))
// Saturating arithmetic
+NEON_FUNCTION (vsqadd, vsqaddq, vuqadd, vuqaddq, mixed_sign_sat_add, )
+NEON_FUNCTION (vpmax, vpmaxq, pairwise_maxmin, (true))
+NEON_FUNCTION (vpmin, vpminq, pairwise_maxmin, (false))
+
NEON_FUNCTION (vqaddb, vqaddh, vqadds, vqaddd, vqadd, vqaddq, gimple_ifn,
(IFN_SAT_ADD))
NEON_FUNCTION (vqsubb, vqsubh, vqsubs, vqsubd, vqsub, vqsubq, gimple_ifn,
(IFN_SAT_SUB))
diff --git a/gcc/config/aarch64/aarch64-neon-builtins-base.def
b/gcc/config/aarch64/aarch64-neon-builtins-base.def
index
a42a25135a0ce86215c74e9c8868fd7c70ca4dac..f9e5888c6819ba9bdb9f241e5b7b99c6be830da9
100644
--- a/gcc/config/aarch64/aarch64-neon-builtins-base.def
+++ b/gcc/config/aarch64/aarch64-neon-builtins-base.def
@@ -177,6 +177,22 @@ DEF_NEON_FUNCTION (vqsub, all_integer, ("D0,D0,D0"))
DEF_NEON_FUNCTION (vqsubq, all_integer, ("Q0,Q0,Q0"))
#undef REQUIRED_EXTENSIONS
+// Mixed-sign saturating arithmetic
+#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_SIMD)
+DEF_NEON_FUNCTION (vsqadd, all_unsigned, ("D0,D0,Ds0"))
+DEF_NEON_FUNCTION (vsqaddq, all_unsigned, ("Q0,Q0,Qs0"))
+DEF_NEON_FUNCTION (vuqadd, all_signed, ("D0,D0,Du0"))
+DEF_NEON_FUNCTION (vuqaddq, all_signed, ("Q0,Q0,Qu0"))
+#undef REQUIRED_EXTENSIONS
+
+// Pairwise integer max/min
+#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_SIMD)
+DEF_NEON_FUNCTION (vpmax, bhs_integer, ("D0,D0,D0"))
+DEF_NEON_FUNCTION (vpmaxq, bhs_integer, ("Q0,Q0,Q0"))
+DEF_NEON_FUNCTION (vpmin, bhs_integer, ("D0,D0,D0"))
+DEF_NEON_FUNCTION (vpminq, bhs_integer, ("Q0,Q0,Q0"))
+#undef REQUIRED_EXTENSIONS
+
// Max/min
#define REQUIRED_EXTENSIONS nonstreaming_only (AARCH64_FL_SIMD)
DEF_NEON_FUNCTION (vmax, bhs_integer, ("D0,D0,D0"))
diff --git a/gcc/config/aarch64/aarch64-simd.md
b/gcc/config/aarch64/aarch64-simd.md
index
11705bc9073801dbc1a25ce3d9723ba9a96c446f..fae331687c1db01f2b806320263c1960003d3464
100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -2150,7 +2150,7 @@ (define_expand "<su><maxmin>v2di3"
})
;; Pairwise Integer Max/Min operations.
-(define_insn "aarch64_<optab>p<mode><vczle><vczbe>"
+(define_insn "@aarch64_<optab>p<mode><vczle><vczbe>"
[(set (match_operand:VDQ_BHSI 0 "register_operand" "=w")
(unspec:VDQ_BHSI [(match_operand:VDQ_BHSI 1 "register_operand" "w")
(match_operand:VDQ_BHSI 2 "register_operand" "w")]
@@ -6229,7 +6229,7 @@ (define_insn_and_split
"<su_optab>s<addsub><mode>3<vczle><vczbe>"
;; suqadd and usqadd
-(define_insn "aarch64_<sur>qadd<mode><vczle><vczbe>"
+(define_insn "@aarch64_<sur>qadd<mode><vczle><vczbe>"
[(set (match_operand:VSDQ_I 0 "register_operand" "=w")
(unspec:VSDQ_I [(match_operand:VSDQ_I 1 "register_operand" "0")
(match_operand:VSDQ_I 2 "register_operand" "w")]
diff --git a/gcc/config/aarch64/arm_neon.h b/gcc/config/aarch64/arm_neon.h
index
9caa0a2088d89804015e67754cc6da99efc6de47..3a0121f3d787dac15c88a1819df31d8294b97f16
100644
--- a/gcc/config/aarch64/arm_neon.h
+++ b/gcc/config/aarch64/arm_neon.h
@@ -11001,96 +11001,6 @@ vmulxd_laneq_f64 (float64_t __a, float64x2_t __v,
const int __lane)
/* vpmax */
-__extension__ extern __inline int8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_s8 (int8x8_t __a, int8x8_t __b)
-{
- return __builtin_aarch64_smaxpv8qi (__a, __b);
-}
-
-__extension__ extern __inline int16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_s16 (int16x4_t __a, int16x4_t __b)
-{
- return __builtin_aarch64_smaxpv4hi (__a, __b);
-}
-
-__extension__ extern __inline int32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_s32 (int32x2_t __a, int32x2_t __b)
-{
- return __builtin_aarch64_smaxpv2si (__a, __b);
-}
-
-__extension__ extern __inline uint8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_u8 (uint8x8_t __a, uint8x8_t __b)
-{
- return (uint8x8_t) __builtin_aarch64_umaxpv8qi ((int8x8_t) __a,
- (int8x8_t) __b);
-}
-
-__extension__ extern __inline uint16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_u16 (uint16x4_t __a, uint16x4_t __b)
-{
- return (uint16x4_t) __builtin_aarch64_umaxpv4hi ((int16x4_t) __a,
- (int16x4_t) __b);
-}
-
-__extension__ extern __inline uint32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmax_u32 (uint32x2_t __a, uint32x2_t __b)
-{
- return (uint32x2_t) __builtin_aarch64_umaxpv2si ((int32x2_t) __a,
- (int32x2_t) __b);
-}
-
-__extension__ extern __inline int8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_s8 (int8x16_t __a, int8x16_t __b)
-{
- return __builtin_aarch64_smaxpv16qi (__a, __b);
-}
-
-__extension__ extern __inline int16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_s16 (int16x8_t __a, int16x8_t __b)
-{
- return __builtin_aarch64_smaxpv8hi (__a, __b);
-}
-
-__extension__ extern __inline int32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_s32 (int32x4_t __a, int32x4_t __b)
-{
- return __builtin_aarch64_smaxpv4si (__a, __b);
-}
-
-__extension__ extern __inline uint8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_u8 (uint8x16_t __a, uint8x16_t __b)
-{
- return (uint8x16_t) __builtin_aarch64_umaxpv16qi ((int8x16_t) __a,
- (int8x16_t) __b);
-}
-
-__extension__ extern __inline uint16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_u16 (uint16x8_t __a, uint16x8_t __b)
-{
- return (uint16x8_t) __builtin_aarch64_umaxpv8hi ((int16x8_t) __a,
- (int16x8_t) __b);
-}
-
-__extension__ extern __inline uint32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmaxq_u32 (uint32x4_t __a, uint32x4_t __b)
-{
- return (uint32x4_t) __builtin_aarch64_umaxpv4si ((int32x4_t) __a,
- (int32x4_t) __b);
-}
-
__extension__ extern __inline float32x2_t
__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
vpmax_f32 (float32x2_t __a, float32x2_t __b)
@@ -11165,96 +11075,6 @@ vpmaxnms_f32 (float32x2_t __a)
/* vpmin */
-__extension__ extern __inline int8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_s8 (int8x8_t __a, int8x8_t __b)
-{
- return __builtin_aarch64_sminpv8qi (__a, __b);
-}
-
-__extension__ extern __inline int16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_s16 (int16x4_t __a, int16x4_t __b)
-{
- return __builtin_aarch64_sminpv4hi (__a, __b);
-}
-
-__extension__ extern __inline int32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_s32 (int32x2_t __a, int32x2_t __b)
-{
- return __builtin_aarch64_sminpv2si (__a, __b);
-}
-
-__extension__ extern __inline uint8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_u8 (uint8x8_t __a, uint8x8_t __b)
-{
- return (uint8x8_t) __builtin_aarch64_uminpv8qi ((int8x8_t) __a,
- (int8x8_t) __b);
-}
-
-__extension__ extern __inline uint16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_u16 (uint16x4_t __a, uint16x4_t __b)
-{
- return (uint16x4_t) __builtin_aarch64_uminpv4hi ((int16x4_t) __a,
- (int16x4_t) __b);
-}
-
-__extension__ extern __inline uint32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpmin_u32 (uint32x2_t __a, uint32x2_t __b)
-{
- return (uint32x2_t) __builtin_aarch64_uminpv2si ((int32x2_t) __a,
- (int32x2_t) __b);
-}
-
-__extension__ extern __inline int8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_s8 (int8x16_t __a, int8x16_t __b)
-{
- return __builtin_aarch64_sminpv16qi (__a, __b);
-}
-
-__extension__ extern __inline int16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_s16 (int16x8_t __a, int16x8_t __b)
-{
- return __builtin_aarch64_sminpv8hi (__a, __b);
-}
-
-__extension__ extern __inline int32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_s32 (int32x4_t __a, int32x4_t __b)
-{
- return __builtin_aarch64_sminpv4si (__a, __b);
-}
-
-__extension__ extern __inline uint8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_u8 (uint8x16_t __a, uint8x16_t __b)
-{
- return (uint8x16_t) __builtin_aarch64_uminpv16qi ((int8x16_t) __a,
- (int8x16_t) __b);
-}
-
-__extension__ extern __inline uint16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_u16 (uint16x8_t __a, uint16x8_t __b)
-{
- return (uint16x8_t) __builtin_aarch64_uminpv8hi ((int16x8_t) __a,
- (int16x8_t) __b);
-}
-
-__extension__ extern __inline uint32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vpminq_u32 (uint32x4_t __a, uint32x4_t __b)
-{
- return (uint32x4_t) __builtin_aarch64_uminpv4si ((int32x4_t) __a,
- (int32x4_t) __b);
-}
-
__extension__ extern __inline float32x2_t
__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
vpmin_f32 (float32x2_t __a, float32x2_t __b)
@@ -15037,62 +14857,6 @@ vslid_n_u64 (uint64_t __a, uint64_t __b, const int __c)
/* vsqadd */
-__extension__ extern __inline uint8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqadd_u8 (uint8x8_t __a, int8x8_t __b)
-{
- return __builtin_aarch64_usqaddv8qi_uus (__a, __b);
-}
-
-__extension__ extern __inline uint16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqadd_u16 (uint16x4_t __a, int16x4_t __b)
-{
- return __builtin_aarch64_usqaddv4hi_uus (__a, __b);
-}
-
-__extension__ extern __inline uint32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqadd_u32 (uint32x2_t __a, int32x2_t __b)
-{
- return __builtin_aarch64_usqaddv2si_uus (__a, __b);
-}
-
-__extension__ extern __inline uint64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqadd_u64 (uint64x1_t __a, int64x1_t __b)
-{
- return (uint64x1_t) {__builtin_aarch64_usqadddi_uus (__a[0], __b[0])};
-}
-
-__extension__ extern __inline uint8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqaddq_u8 (uint8x16_t __a, int8x16_t __b)
-{
- return __builtin_aarch64_usqaddv16qi_uus (__a, __b);
-}
-
-__extension__ extern __inline uint16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqaddq_u16 (uint16x8_t __a, int16x8_t __b)
-{
- return __builtin_aarch64_usqaddv8hi_uus (__a, __b);
-}
-
-__extension__ extern __inline uint32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqaddq_u32 (uint32x4_t __a, int32x4_t __b)
-{
- return __builtin_aarch64_usqaddv4si_uus (__a, __b);
-}
-
-__extension__ extern __inline uint64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vsqaddq_u64 (uint64x2_t __a, int64x2_t __b)
-{
- return __builtin_aarch64_usqaddv2di_uus (__a, __b);
-}
-
__extension__ extern __inline uint8_t
__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
vsqaddb_u8 (uint8_t __a, int8_t __b)
@@ -17213,62 +16977,6 @@ vtstd_u64 (uint64_t __a, uint64_t __b)
/* vuqadd */
-__extension__ extern __inline int8x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqadd_s8 (int8x8_t __a, uint8x8_t __b)
-{
- return __builtin_aarch64_suqaddv8qi_ssu (__a, __b);
-}
-
-__extension__ extern __inline int16x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqadd_s16 (int16x4_t __a, uint16x4_t __b)
-{
- return __builtin_aarch64_suqaddv4hi_ssu (__a, __b);
-}
-
-__extension__ extern __inline int32x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqadd_s32 (int32x2_t __a, uint32x2_t __b)
-{
- return __builtin_aarch64_suqaddv2si_ssu (__a, __b);
-}
-
-__extension__ extern __inline int64x1_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqadd_s64 (int64x1_t __a, uint64x1_t __b)
-{
- return (int64x1_t) {__builtin_aarch64_suqadddi_ssu (__a[0], __b[0])};
-}
-
-__extension__ extern __inline int8x16_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqaddq_s8 (int8x16_t __a, uint8x16_t __b)
-{
- return __builtin_aarch64_suqaddv16qi_ssu (__a, __b);
-}
-
-__extension__ extern __inline int16x8_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqaddq_s16 (int16x8_t __a, uint16x8_t __b)
-{
- return __builtin_aarch64_suqaddv8hi_ssu (__a, __b);
-}
-
-__extension__ extern __inline int32x4_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqaddq_s32 (int32x4_t __a, uint32x4_t __b)
-{
- return __builtin_aarch64_suqaddv4si_ssu (__a, __b);
-}
-
-__extension__ extern __inline int64x2_t
-__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
-vuqaddq_s64 (int64x2_t __a, uint64x2_t __b)
-{
- return __builtin_aarch64_suqaddv2di_ssu (__a, __b);
-}
-
__extension__ extern __inline int8_t
__attribute__ ((__always_inline__, __gnu_inline__, __artificial__))
vuqaddb_s8 (int8_t __a, uint8_t __b)
diff --git a/gcc/testsuite/g++.target/aarch64/vect-revectorize-intrinsics.C
b/gcc/testsuite/g++.target/aarch64/vect-revectorize-intrinsics.C
new file mode 100644
index
0000000000000000000000000000000000000000..c47765f472ac7f4c1dfe7f4c66d19d74f6f1da93
--- /dev/null
+++ b/gcc/testsuite/g++.target/aarch64/vect-revectorize-intrinsics.C
@@ -0,0 +1,11 @@
+/* { dg-do run { target aarch64_sve2_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8.5-a+sve2 -fno-vect-cost-model -Wno-narrowing
-fdump-tree-vect-details -fdump-tree-optimized" } */
+
+#include <arm_sve.h>
+#include "../../gcc.target/aarch64/vect-revectorize-intrinsics-fold-widths.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using variable length
vectors" 8 "vect" } } */
+/* { dg-final { scan-tree-dump "svsqadd_u64_x" "optimized" } } */
+/* { dg-final { scan-tree-dump-not {VIEW_CONVERT_EXPR<sv} "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..f277ca6ced0490866e330c0267a4aedda351f3de
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-1.c
@@ -0,0 +1,9 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-vect-cost-model -fdump-tree-vect-details
-fdump-tree-optimized" } */
+
+/* Revectorize calls that survive ordinary builtin folding. */
+#include "vect-revectorize-intrinsics-fold.h"
+
+/* { dg-final { scan-tree-dump "vectorizable_vect_call" "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-10.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-10.c
new file mode 100644
index
0000000000000000000000000000000000000000..ea96502b94d6af64200c2c78cfd494e4050cf72c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-10.c
@@ -0,0 +1,16 @@
+/* { dg-do run { target aarch64_sve2_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8.5-a+sve2 -fno-vect-cost-model
-fdump-tree-vect-details -fdump-tree-optimized -save-temps" } */
+
+#include <arm_sve.h>
+#include "vect-revectorize-intrinsics-fold-widen.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using variable length
vectors" 4 "vect" } } */
+/* { dg-final { scan-tree-dump "vectorizable_vect_call" "vect" } } */
+/* { dg-final { scan-tree-dump "svsqadd_u8_x" "optimized" } } */
+/* { dg-final { scan-tree-dump "svuqadd_s8_x" "optimized" } } */
+/* The implementation takes compiler vectors and predicates directly. */
+/* { dg-final { scan-tree-dump-not {VIEW_CONVERT_EXPR<sv} "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tz[0-9]+\.b, p[0-9]+/m} } } */
+/* { dg-final { scan-assembler {suqadd\tz[0-9]+\.b, p[0-9]+/m} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-11.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-11.c
new file mode 100644
index
0000000000000000000000000000000000000000..7c23ff15b8c6ffeeb2282928c22cc5fb41f24680
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-11.c
@@ -0,0 +1,91 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O2 -fgimple -fdump-tree-optimized -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+typedef unsigned char v16u8 __attribute__ ((vector_size (16)));
+typedef signed char v16s8 __attribute__ ((vector_size (16)));
+typedef unsigned int v4u32 __attribute__ ((vector_size (16)));
+
+/* Request implementations using anonymous vector types. */
+extern uint8x8_t vsqadd_u8 (uint8x8_t, int8x8_t);
+extern uint32x4_t vpminq_u32 (uint32x4_t, uint32x4_t);
+
+__attribute__ ((noipa)) v16u8 __GIMPLE
+unsigned_add (v16u8 a, v16s8 b, v16u8 mask)
+{
+ v16u8 r;
+ r = .MASK_CALL (&vsqadd_u8, a, b, mask);
+ return r;
+}
+
+__attribute__ ((noipa)) v4u32 __GIMPLE
+pairwise_min (v4u32 a, v4u32 b, v4u32 mask)
+{
+ v4u32 r;
+ r = .MASK_CALL (&vpminq_u32, a, b, mask);
+ return r;
+}
+
+static const v16u8 masks[] = {
+ { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
+ 255 },
+ { 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255 },
+ { 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0 },
+ { 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255 }
+};
+static const v4u32 masks32[] = {
+ { 0, -1U, -1U, 0 },
+ { -1U, -1U, 0, -1U },
+ { -1U, 0, -1U, -1U },
+ { 0, -1U, -1U, 0 },
+ { -1U, -1U, 0, -1U }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ v16u8 a = { 0, 1, 20, 100, 160, 240, 254, 255,
+ 255, 254, 240, 160, 100, 20, 1, 0 };
+ v16s8 b = { -128, -1, -30, 120, 127, 20, 1, 0,
+ 127, 0, -1, -128, -120, 100, -2, 1 };
+ v4u32 x = { 4000000000U, 3000000000U, 7, 11 };
+ v4u32 y = { 19, 17, 2500000000U, 3000000000U };
+
+ /* Pairwise results use ACLE lane order. */
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+ const v4u32 expected = { 17, 2500000000U, 3000000000U, 7 };
+#else
+ const v4u32 expected = { 3000000000U, 7, 17, 2500000000U };
+#endif
+ const v16u8 expected_add = { 0, 0, 0, 220, 255, 255, 255, 255,
+ 255, 254, 239, 32, 0, 120, 0, 1 };
+ for (unsigned k = 0; k < 5; ++k)
+ {
+ v16u8 mask = masks[k];
+ v4u32 mask32 = masks32[k];
+ v16u8 r = unsigned_add (a, b, mask);
+ v4u32 s = pairwise_min (x, y, mask32);
+ for (unsigned j = 0; j < 16; ++j)
+ if (mask[j] && r[j] != expected_add[j])
+ __builtin_abort ();
+
+ for (unsigned j = 0; j < 4; ++j)
+ if (mask32[j] && s[j] != expected[j])
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* These calls need no conversions between the anonymous and ACLE types. */
+/* { dg-final { scan-tree-dump-times {r_[0-9]+ = __builtin_aarch64_vsqaddq_u8
\(a_[0-9]+\(D\), b_[0-9]+\(D\)\);} 1 "optimized" } } */
+/* { dg-final { scan-tree-dump-times {r_[0-9]+ = __builtin_aarch64_vpminq_u32
\(a_[0-9]+\(D\), b_[0-9]+\(D\)\);} 1 "optimized" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tv[0-9]+\.16b} } } */
+/* { dg-final { scan-assembler {uminp\tv[0-9]+\.4s} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-12.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-12.c
new file mode 100644
index
0000000000000000000000000000000000000000..e3bdd8ca76d3520157a45bf7227cd6ba0c81fec4
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-12.c
@@ -0,0 +1,10 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fno-vect-cost-model
-fdump-tree-vect-details -fdump-tree-optimized -save-temps" } */
+
+#include "vect-revectorize-intrinsics-fold-widths.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using 16 byte vectors
and unroll factor 2" 8 "vect" } } */
+/* { dg-final { scan-tree-dump "vsqaddq_u64" "optimized" } } */
+/* { dg-final { scan-tree-dump "vuqaddq_s64" "optimized" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-13.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-13.c
new file mode 100644
index
0000000000000000000000000000000000000000..1c88c3b8604ecf9fc9794cfea60450852598ce80
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-13.c
@@ -0,0 +1,13 @@
+/* { dg-do run { target aarch64_sve2_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8.5-a+sve2 -fno-vect-cost-model
-fdump-tree-vect-details -fdump-tree-optimized -save-temps" } */
+
+#include <arm_sve.h>
+#include "vect-revectorize-intrinsics-fold-widths.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using variable length
vectors" 8 "vect" } } */
+/* { dg-final { scan-tree-dump "svsqadd_u64_x" "optimized" } } */
+/* { dg-final { scan-tree-dump "svuqadd_s64_x" "optimized" } } */
+/* All predicate element widths must work without ACLE conversions. */
+/* { dg-final { scan-tree-dump-not {VIEW_CONVERT_EXPR<sv} "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-15.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-15.c
new file mode 100644
index
0000000000000000000000000000000000000000..2489c9531427fd9478aaf2316515a8dd3f6b44e1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-15.c
@@ -0,0 +1,15 @@
+/* { dg-do run { target { aarch64_sve128_hw && aarch64_sve2_hw } } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8.5-a+sve2 -msve-vector-bits=128
-fno-vect-cost-model -fdump-tree-vect-details -fdump-tree-optimized" } */
+
+#include <arm_sve.h>
+#include "vect-revectorize-intrinsics-fold-widths.h"
+
+/* Fixed-length SVE results need no ACLE conversions. Including arm_sve.h
+ must preserve the modes of AdvSIMD implementations. */
+/* { dg-final { scan-tree-dump "__builtin_aarch64_vsqaddq_u8" "optimized" {
target aarch64_little_endian } } } */
+/* { dg-final { scan-tree-dump "__builtin_aarch64_svsqadd_u8_x" "optimized" {
target aarch64_big_endian } } } */
+/* { dg-final { scan-tree-dump "__builtin_aarch64_svsqadd_u64_x" "optimized" }
} */
+/* { dg-final { scan-tree-dump "__builtin_aarch64_svuqadd_s64_x" "optimized" }
} */
+/* { dg-final { scan-tree-dump-not {VIEW_CONVERT_EXPR<sv} "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..732231d0f701397f26238015363560f81be21129
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-2.c
@@ -0,0 +1,13 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-vect-cost-model -march=armv8.2-a+sve
-mautovec-preference=sve-only --param=aarch64-autovec-mode=V2SI
-fdump-tree-vect-details -fdump-tree-optimized -save-temps" } */
+
+/* Generate both 64-bit copies of each saturating add. The pairwise call
+ must retain its 128-bit layout instead of splitting in the same way. */
+#include "vect-revectorize-intrinsics-fold.h"
+
+/* { dg-final { scan-tree-dump "vectorizable_vect_call" "vect" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tv[0-9]+\.8b} } } */
+/* { dg-final { scan-assembler {suqadd\tv[0-9]+\.8b} } } */
+/* { dg-final { scan-assembler {uminp\tv[0-9]+\.4s} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-3.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-3.c
new file mode 100644
index
0000000000000000000000000000000000000000..4741b442d5c254d3d9d649ff441d4cc0fcbba300
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-3.c
@@ -0,0 +1,10 @@
+/* { dg-do run { target aarch64_sve128_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-vect-cost-model -march=armv8.2-a+sve
-msve-vector-bits=128 -mautovec-preference=sve-only -fdump-tree-vect-details
-fdump-tree-optimized" } */
+
+/* Exercise mask folding and mode changes between fixed-length SVE and
+ AdvSIMD without requiring arm_sve.h declarations. */
+#include "vect-revectorize-intrinsics-fold.h"
+
+/* { dg-final { scan-tree-dump "vectorizable_vect_call" "vect" { target
aarch64_little_endian } } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-4.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-4.c
new file mode 100644
index
0000000000000000000000000000000000000000..34f4e81d70946b1447ac73a6412e22aebcf58675
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-4.c
@@ -0,0 +1,77 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O2 -fgimple -fdump-tree-optimized -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Explicit declarations let GIMPLE take builtin addresses. Test narrowing
+ the add and preserving the pairwise layout with runtime masks. */
+extern uint8x16_t vsqaddq_u8 (uint8x16_t, int8x16_t);
+extern uint32x4_t vpminq_u32 (uint32x4_t, uint32x4_t);
+
+__attribute__ ((noipa)) uint8x8_t __GIMPLE
+masked_add (uint8x8_t a, int8x8_t b, uint8x8_t mask)
+{
+ uint8x8_t r;
+ r = .MASK_CALL (&vsqaddq_u8, a, b, mask);
+ return r;
+}
+
+__attribute__ ((noipa)) uint32x4_t __GIMPLE
+masked_min (uint32x4_t a, uint32x4_t b, uint32x4_t mask)
+{
+ uint32x4_t r;
+ r = .MASK_CALL (&vpminq_u32, a, b, mask);
+ return r;
+}
+
+static const uint8x8_t masks[] = {
+ { 0, 255, 255, 0, 255, 255, 0, 255 },
+ { 255, 255, 0, 255, 255, 0, 255, 255 },
+ { 255, 0, 255, 255, 0, 255, 255, 0 }
+};
+static const uint32x4_t masks32[] = {
+ { 0, -1U, -1U, 0 },
+ { -1U, -1U, 0, -1U },
+ { -1U, 0, -1U, -1U }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ uint8x8_t a = { 0, 1, 20, 100, 160, 240, 254, 255 };
+ int8x8_t b = { -128, -1, -30, 120, 127, 20, 1, 0 };
+ uint32x4_t x = { 4000000000U, 3000000000U, 7, 11 };
+ uint32x4_t y = { 19, 17, 2500000000U, 3000000000U };
+ const uint8x8_t expected_add = { 0, 0, 0, 220, 255, 255, 255, 255 };
+
+ /* Pairwise results use ACLE lane order. */
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+ const uint32x4_t expected = { 17, 2500000000U, 3000000000U, 7 };
+#else
+ const uint32x4_t expected = { 3000000000U, 7, 17, 2500000000U };
+#endif
+ for (unsigned k = 0; k < 3; ++k)
+ {
+ uint8x8_t mask8 = masks[k];
+ uint32x4_t mask32 = masks32[k];
+ uint8x8_t r = masked_add (a, b, mask8);
+ uint32x4_t s = masked_min (x, y, mask32);
+ for (unsigned j = 0; j < 8; ++j)
+ if (mask8[j] && r[j] != expected_add[j])
+ __builtin_abort ();
+
+ for (unsigned j = 0; j < 4; ++j)
+ if (mask32[j] && s[j] != expected[j])
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tv[0-9]+\.8b} } } */
+/* { dg-final { scan-assembler {uminp\tv[0-9]+\.4s} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-5.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-5.c
new file mode 100644
index
0000000000000000000000000000000000000000..ae8ac4d7af6d8e6da9369d6216a910171b0a0c0a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-5.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -fno-vect-cost-model" } */
+
+#include <arm_neon.h>
+
+/* A loop containing a target builtin can also contain vector conversions.
+ Decline the unsupported conversion without treating its mode as scalar. */
+uint32x4_t
+pairwise_convert (uint32x4_t x, uint32x4_t y, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = vpminq_u32 (x, y);
+ y = vcvtq_u32_f32 (vcvtq_f32_u32 (x) + 1.0f);
+ }
+
+ return x;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-6.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-6.c
new file mode 100644
index
0000000000000000000000000000000000000000..75da4dac42d4a7a0893ec48e0005506038578b90
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-6.c
@@ -0,0 +1,97 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O2 -fgimple -fdump-tree-optimized -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* These arm_neon.h calls survive folding as target builtins. */
+__attribute__ ((noipa)) uint8x8_t
+neon_unsigned_add (uint8x8_t a, int8x8_t b)
+{
+ return vsqadd_u8 (a, b);
+}
+
+__attribute__ ((noipa)) int8x8_t
+neon_signed_add (int8x8_t a, uint8x8_t b)
+{
+ return vuqadd_s8 (a, b);
+}
+
+/* Let GIMPLE take builtin addresses to request 128-bit implementations. */
+extern uint8x8_t vsqadd_u8 (uint8x8_t, int8x8_t);
+extern int8x8_t vuqadd_s8 (int8x8_t, uint8x8_t);
+
+__attribute__ ((noipa)) uint8x16_t __GIMPLE
+wide_unsigned_add (uint8x16_t a, int8x16_t b, uint8x16_t mask)
+{
+ uint8x16_t r;
+ r = .MASK_CALL (&vsqadd_u8, a, b, mask);
+ return r;
+}
+
+__attribute__ ((noipa)) int8x16_t __GIMPLE
+wide_signed_add (int8x16_t a, uint8x16_t b, uint8x16_t mask)
+{
+ int8x16_t r;
+ r = .MASK_CALL (&vuqadd_s8, a, b, mask);
+ return r;
+}
+
+static const uint8x16_t masks[] = {
+ { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
+ { 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255, 255,
+ 255 },
+ { 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255 },
+ { 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0 },
+ { 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255 },
+ { 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255 },
+ { 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0 },
+ { 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255 },
+ { 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255 },
+ { 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0, 255, 255, 0 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ uint8x16_t a = { 0, 1, 20, 100, 160, 240, 254, 255,
+ 255, 254, 240, 160, 100, 20, 1, 0 };
+ int8x16_t b = { -128, -1, -30, 120, 127, 20, 1, 0,
+ 127, 0, -1, -128, -120, 100, -2, 1 };
+ const uint8x16_t expected_u = { 0, 0, 0, 220, 255, 255, 255, 255,
+ 255, 254, 239, 32, 0, 120, 0, 1 };
+ const int8x16_t expected_s = { -128, 0, -10, 127, 127, 127, 127, 127,
+ 127, 127, 127, 32, -20, 120, -1, 1 };
+ uint8x16_t native_u
+ = vcombine_u8 (neon_unsigned_add (vget_low_u8 (a), vget_low_s8 (b)),
+ neon_unsigned_add (vget_high_u8 (a), vget_high_s8 (b)));
+ int8x16_t native_s
+ = vcombine_s8 (neon_signed_add (vget_low_s8 (b), vget_low_u8 (a)),
+ neon_signed_add (vget_high_s8 (b), vget_high_u8 (a)));
+ if (__builtin_memcmp (&native_u, &expected_u, sizeof (native_u))
+ || __builtin_memcmp (&native_s, &expected_s, sizeof (native_s)))
+ __builtin_abort ();
+
+ for (unsigned k = 0; k < 10; ++k)
+ {
+ uint8x16_t mask = masks[k];
+ uint8x16_t r = wide_unsigned_add (a, b, mask);
+ int8x16_t s = wide_signed_add (b, a, mask);
+ for (unsigned j = 0; j < 16; ++j)
+ if (mask[j] && (r[j] != expected_u[j] || s[j] != expected_s[j]))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "vsqadd_u8" "optimized" } } */
+/* { dg-final { scan-tree-dump "vuqadd_s8" "optimized" } } */
+/* { dg-final { scan-tree-dump "vsqaddq_u8" "optimized" } } */
+/* { dg-final { scan-tree-dump "vuqaddq_s8" "optimized" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tv[0-9]+\.16b} } } */
+/* { dg-final { scan-assembler {suqadd\tv[0-9]+\.16b} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-7.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-7.c
new file mode 100644
index
0000000000000000000000000000000000000000..731460ab25c1db6ed5bb194f8ac3cc0f2e93ad42
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-7.c
@@ -0,0 +1,89 @@
+/* { dg-do run { target aarch64_sve2_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O2 -march=armv8.5-a+sve2 -fgimple -fdump-tree-optimized
-save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+#include <arm_sve.h>
+
+/* Request SVE implementations of 64-bit intrinsics with runtime masks. */
+extern uint8x8_t vsqadd_u8 (uint8x8_t, int8x8_t);
+extern int8x8_t vuqadd_s8 (int8x8_t, uint8x8_t);
+
+__attribute__ ((noipa)) svuint8_t __GIMPLE
+sve_unsigned_add (svuint8_t a, svint8_t b, svbool_t mask)
+{
+ svuint8_t r;
+ r = .MASK_CALL (&vsqadd_u8, a, b, mask);
+ return r;
+}
+
+__attribute__ ((noipa)) svint8_t __GIMPLE
+sve_signed_add (svint8_t a, svuint8_t b, svbool_t mask)
+{
+ svint8_t r;
+ r = .MASK_CALL (&vuqadd_s8, a, b, mask);
+ return r;
+}
+
+static const int8_t input_b[] = {
+ -128, -1, -30, 120, 127, 20, 1, 0, -1, -30, 120, 127, 20, 1, 0, -128, -30,
+ 120, 127, 20, 1, 0, -128, -1, 120, 127, 20, 1, 0, -128, -1, -30, 127, 20,
+ 1, 0, -128, -1, -30, 120, 20, 1, 0, -128, -1, -30, 120, 127, 1, 0, -128,
+ -1, -30, 120, 127, 20, 0, -128, -1, -30, 120, 127, 20, 1
+};
+static const uint8_t expected_u[] = {
+ 0, 0, 0, 220, 255, 255, 255, 255, 0, 0, 140, 227, 180, 241, 254, 127, 0,
+ 121, 147, 120, 161, 240, 126, 254, 120, 128, 40, 101, 160, 112, 253, 225,
+ 127, 21, 21, 100, 32, 239, 224, 255, 20, 2, 20, 0, 159, 210, 255, 255, 1,
+ 1, 0, 99, 130, 255, 255, 255, 0, 0, 19, 70, 255, 255, 255, 255
+};
+static const int8_t expected_s[] = {
+ -128, 0, -10, 127, 127, 127, 127, 127, -1, -29, 127, 127, 127, 127, 127,
+ 127, -30, 121, 127, 120, 127, 127, 126, 127, 120, 127, 40, 101, 127, 112,
+ 127, 127, 127, 21, 21, 100, 32, 127, 127, 127, 20, 2, 20, -28, 127, 127,
+ 127, 127, 1, 1, -108, 99, 127, 127, 127, 127, 0, -127, 19, 70, 127, 127,
+ 127, 127
+};
+
+int
+main (void)
+{
+ check_vect ();
+ static const uint8_t ua[] = { 0, 1, 20, 100, 160, 240, 254, 255 };
+ unsigned n = svcntb ();
+ uint8_t a[n], mask[n], r[n];
+ int8_t b[n], s[n];
+ svbool_t pg = svptrue_b8 ();
+ for (unsigned j = 0; j < n; ++j)
+ {
+ a[j] = ua[j % 8];
+ b[j] = input_b[j % 64];
+ }
+
+ for (unsigned k = 0; k < 10; ++k)
+ {
+ for (unsigned j = 0; j < n; ++j)
+ mask[j] = k == 0 ? 0 : k == 1 ? 1 : (k + j) % 3 != 0;
+ svbool_t active = svcmpne_n_u8 (pg, svld1_u8 (pg, mask), 0);
+ svuint8_t x = svld1_u8 (pg, a);
+ svint8_t y = svld1_s8 (pg, b);
+ svst1_u8 (pg, r, sve_unsigned_add (x, y, active));
+ svst1_s8 (pg, s, sve_signed_add (y, x, active));
+ for (unsigned j = 0; j < n; ++j)
+ {
+ if (mask[j] && (r[j] != expected_u[j % 64]
+ || s[j] != expected_s[j % 64]))
+ __builtin_abort ();
+ }
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-tree-dump "svsqadd_u8_x" "optimized" } } */
+/* { dg-final { scan-tree-dump "svuqadd_s8_x" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tz[0-9]+\.b, p[0-9]+/m} } } */
+/* { dg-final { scan-assembler {suqadd\tz[0-9]+\.b, p[0-9]+/m} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-8.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-8.c
new file mode 100644
index
0000000000000000000000000000000000000000..03afb95f240afb12aa0de1a686f3ad2fb7c7eeb2
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-8.c
@@ -0,0 +1,114 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O2 -fchecking=2" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Native saturating adds and pairwise min/max, including one-lane vectors. */
+#define CHECK_ADD(BITS, LANES, Q) \
+__attribute__ ((noipa)) static uint##BITS##x##LANES##_t \
+unsigned_add_##BITS##Q (uint##BITS##x##LANES##_t a, \
+ int##BITS##x##LANES##_t b) \
+{ return vsqadd##Q##_u##BITS (a, b); } \
+__attribute__ ((noipa)) static int##BITS##x##LANES##_t \
+signed_add_##BITS##Q (int##BITS##x##LANES##_t a, \
+ uint##BITS##x##LANES##_t b) \
+{ return vuqadd##Q##_s##BITS (a, b); } \
+static void check_add_##BITS##Q (void) \
+{ \
+ const uint##BITS##_t values_u[] = { 0, 1, -2, -1 }; \
+ const int##BITS##_t values_s[] \
+ = { INT##BITS##_MIN, -1, 0, INT##BITS##_MAX }; \
+ const uint##BITS##_t expected_u[] = { 0, 0, -2, -1 }; \
+ const int##BITS##_t expected_s[] \
+ = { INT##BITS##_MIN, 0, INT##BITS##_MAX, INT##BITS##_MAX }; \
+ for (unsigned k = 0; k < 4; ++k) \
+ { \
+ uint##BITS##x##LANES##_t a; \
+ int##BITS##x##LANES##_t b; \
+ for (unsigned j = 0; j < LANES; ++j) \
+ { \
+ unsigned i = (k + j) % 4; \
+ a[j] = values_u[i]; \
+ b[j] = values_s[i]; \
+ } \
+ uint##BITS##x##LANES##_t u = unsigned_add_##BITS##Q (a, b); \
+ int##BITS##x##LANES##_t s = signed_add_##BITS##Q (b, a); \
+ for (unsigned j = 0; j < LANES; ++j) \
+ { \
+ unsigned i = (k + j) % 4; \
+ if (u[j] != expected_u[i] || s[j] != expected_s[i]) \
+ __builtin_abort (); \
+ } \
+ } \
+}
+
+CHECK_ADD (8, 8, )
+CHECK_ADD (8, 16, q)
+CHECK_ADD (16, 4, )
+CHECK_ADD (16, 8, q)
+CHECK_ADD (32, 2, )
+CHECK_ADD (32, 4, q)
+CHECK_ADD (64, 1, )
+CHECK_ADD (64, 2, q)
+
+#define CHECK_PAIRWISE(BITS, LANES, Q, SIGN, TYPE) \
+__attribute__ ((noipa)) static TYPE##BITS##x##LANES##_t \
+pair_min_##SIGN##BITS##Q (TYPE##BITS##x##LANES##_t a, \
+ TYPE##BITS##x##LANES##_t b) \
+{ return vpmin##Q##_##SIGN##BITS (a, b); } \
+__attribute__ ((noipa)) static TYPE##BITS##x##LANES##_t \
+pair_max_##SIGN##BITS##Q (TYPE##BITS##x##LANES##_t a, \
+ TYPE##BITS##x##LANES##_t b) \
+{ return vpmax##Q##_##SIGN##BITS (a, b); } \
+static void check_pair_##SIGN##BITS##Q (void) \
+{ \
+ TYPE##BITS##x##LANES##_t a, b; \
+ for (unsigned j = 0; j < LANES; ++j) \
+ { a[j] = (j & 1) ? j : -j - 1; b[j] = (j & 1) ? -j - 3 : j + 2; } \
+ TYPE##BITS##x##LANES##_t lo = pair_min_##SIGN##BITS##Q (a, b); \
+ TYPE##BITS##x##LANES##_t hi = pair_max_##SIGN##BITS##Q (a, b); \
+ /* C vector indices run opposite to ACLE lanes on big-endian. */ \
+ TYPE##BITS##x##LANES##_t first = a, second = b; \
+ if (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__) \
+ first = b, second = a; \
+ for (unsigned j = 0; j < LANES; ++j) \
+ { \
+ TYPE##BITS##_t x = j < LANES / 2 ? first[2 * j] \
+ : second[2 * j - LANES]; \
+ TYPE##BITS##_t y = j < LANES / 2 ? first[2 * j + 1] \
+ : second[2 * j - LANES + 1]; \
+ if (lo[j] != (x < y ? x : y) || hi[j] != (x > y ? x : y)) \
+ __builtin_abort (); \
+ } \
+}
+
+#define CHECK_PAIRS(BITS, LANES, Q) \
+ CHECK_PAIRWISE (BITS, LANES, Q, s, int) \
+ CHECK_PAIRWISE (BITS, LANES, Q, u, uint)
+CHECK_PAIRS (8, 8, )
+CHECK_PAIRS (8, 16, q)
+CHECK_PAIRS (16, 4, )
+CHECK_PAIRS (16, 8, q)
+CHECK_PAIRS (32, 2, )
+CHECK_PAIRS (32, 4, q)
+
+int
+main (void)
+{
+ check_vect ();
+#define RUN_ADD(BITS) check_add_##BITS (); check_add_##BITS##q ();
+ RUN_ADD (8)
+ RUN_ADD (16)
+ RUN_ADD (32)
+ RUN_ADD (64)
+#define RUN_PAIR(BITS) \
+ check_pair_s##BITS (); check_pair_s##BITS##q (); \
+ check_pair_u##BITS (); check_pair_u##BITS##q ();
+ RUN_PAIR (8)
+ RUN_PAIR (16)
+ RUN_PAIR (32)
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-9.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-9.c
new file mode 100644
index
0000000000000000000000000000000000000000..94353223fbbcfb1429bbc4a092ba157311458e13
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-9.c
@@ -0,0 +1,13 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fdump-tree-vect-details
-fdump-tree-optimized -save-temps" } */
+
+#include "vect-revectorize-intrinsics-fold-widen.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using 16 byte vectors
and unroll factor 2" 4 "vect" } } */
+/* { dg-final { scan-tree-dump "vectorizable_vect_call" "vect" } } */
+/* { dg-final { scan-tree-dump "vsqaddq_u8" "optimized" } } */
+/* { dg-final { scan-tree-dump "vuqaddq_s8" "optimized" } } */
+/* { dg-final { scan-tree-dump-not "\\.MASK_CALL" "optimized" } } */
+/* { dg-final { scan-assembler {usqadd\tv[0-9]+\.16b} } } */
+/* { dg-final { scan-assembler {suqadd\tv[0-9]+\.16b} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widen.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widen.h
new file mode 100644
index
0000000000000000000000000000000000000000..b0f60ce26fb9a6e422b36bea34fdbe4c6acffb19
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widen.h
@@ -0,0 +1,101 @@
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Widen 64-bit target builtin calls across loop iterations. */
+__attribute__ ((noipa)) void
+unsigned_sat_add (uint8x8_t *restrict out, const uint8x8_t *a,
+ const int8x8_t *b, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = vsqadd_u8 (a[i], b[i]);
+}
+
+__attribute__ ((noipa)) void
+signed_sat_add (int8x8_t *restrict out, const int8x8_t *a,
+ const uint8x8_t *b, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = vuqadd_s8 (a[i], b[i]);
+}
+
+/* Widening must repeat all lanes of invariant and constant operands. */
+__attribute__ ((noipa)) void
+unsigned_sat_add_invariant (uint8x8_t *restrict out, const uint8x8_t *a,
+ int8x8_t b, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = vsqadd_u8 (a[i], b);
+}
+
+__attribute__ ((noipa)) void
+signed_sat_add_constant (int8x8_t *restrict out, const int8x8_t *a, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = vuqadd_s8 (a[i], (uint8x8_t) { 0, 1, 20, 100,
+ 160, 240, 254, 255 });
+}
+
+static int
+clamp (int value, int low, int high)
+{
+ return value < low ? low : value > high ? high : value;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ uint8x8_t a[67], out_u[67], out_invariant[67];
+ uint8x8_t expected_u[67], expected_invariant[67];
+ int8x8_t b[67], out_s[67], out_constant[67];
+ int8x8_t expected_s[67], expected_constant[67];
+ const int invariant_b[] = { -61, -54, -47, -40, -33, -26, -19, -12 };
+ const int constant_b[] = { 0, 1, 20, 100, 160, 240, 254, 255 };
+ const uint8x8_t guard_u = { 23, 40, 57, 74, 91, 108, 125, 142 };
+ const int8x8_t guard_s = { 41, 38, 35, 32, 29, 26, 23, 20 };
+
+ /* Widen the sums before clamping, keeping the reference scalar. */
+ for (unsigned k = 0; k < 67 * 8; ++k)
+ {
+ unsigned i = k / 8, j = k % 8;
+ int x = (17 * i + 13 * j) & 255;
+ int y = (int) ((19 * i + 7 * j) & 255) - 128;
+ a[i][j] = x;
+ b[i][j] = y;
+ expected_u[i][j] = clamp (x + y, 0, 255);
+ expected_s[i][j] = clamp (x + y, -128, 127);
+ expected_invariant[i][j] = clamp (x + invariant_b[j], 0, 255);
+ expected_constant[i][j] = clamp (y + constant_b[j], -128, 127);
+ }
+
+ for (unsigned n = 0; n <= 67; ++n)
+ {
+ for (unsigned i = 0; i < 67; ++i)
+ {
+ out_u[i] = out_invariant[i] = guard_u;
+ out_s[i] = out_constant[i] = guard_s;
+ }
+
+ unsigned_sat_add (out_u, a, b, n);
+ signed_sat_add (out_s, b, a, n);
+ unsigned_sat_add_invariant (out_invariant, a, b[17], n);
+ signed_sat_add_constant (out_constant, b, n);
+ if (__builtin_memcmp (out_u, expected_u, n * sizeof (out_u[0]))
+ || __builtin_memcmp (out_s, expected_s, n * sizeof (out_s[0]))
+ || __builtin_memcmp (out_invariant, expected_invariant,
+ n * sizeof (out_invariant[0]))
+ || __builtin_memcmp (out_constant, expected_constant,
+ n * sizeof (out_constant[0])))
+ __builtin_abort ();
+
+ for (unsigned i = n; i < 67; ++i)
+ if (__builtin_memcmp (&out_u[i], &guard_u, sizeof (guard_u))
+ || __builtin_memcmp (&out_invariant[i], &guard_u, sizeof (guard_u))
+ || __builtin_memcmp (&out_s[i], &guard_s, sizeof (guard_s))
+ || __builtin_memcmp (&out_constant[i], &guard_s, sizeof (guard_s)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widths.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widths.h
new file mode 100644
index
0000000000000000000000000000000000000000..581cb92f0d1e0a948a92a29cf6610e2e419656f2
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold-widths.h
@@ -0,0 +1,80 @@
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Widen 64-bit intrinsics for each integer and predicate element size. */
+#define DEFINE_ADDS(BITS, LANES) \
+__attribute__ ((noipa)) void \
+unsigned_add_##BITS (uint##BITS##x##LANES##_t *__restrict out, \
+ const uint##BITS##x##LANES##_t *a, \
+ const int##BITS##x##LANES##_t *b, unsigned n) \
+{ \
+ for (unsigned i = 0; i < n; ++i) \
+ out[i] = vsqadd_u##BITS (a[i], b[i]); \
+} \
+__attribute__ ((noipa)) void \
+signed_add_##BITS (int##BITS##x##LANES##_t *__restrict out, \
+ const int##BITS##x##LANES##_t *a, \
+ const uint##BITS##x##LANES##_t *b, unsigned n) \
+{ \
+ for (unsigned i = 0; i < n; ++i) \
+ out[i] = vuqadd_s##BITS (a[i], b[i]); \
+} \
+__attribute__ ((noinline, optimize ("no-tree-vectorize"))) static void \
+check_##BITS (void) \
+{ \
+ const uint##BITS##_t values_u[] \
+ = { 0, 1, -1, -2, 0, 1, -1, -2, 0, 1, -1, -2 }; \
+ const int##BITS##_t values_s[] \
+ = { INT##BITS##_MIN, -1, INT##BITS##_MAX, 0, \
+ INT##BITS##_MIN, -1, INT##BITS##_MAX, 0, \
+ INT##BITS##_MIN, -1, INT##BITS##_MAX, 0 }; \
+ const uint##BITS##_t expected_u[] \
+ = { 0, 0, -1, -2, 0, 0, -1, -2, 0, 0, -1, -2 }; \
+ const int##BITS##_t expected_s[] \
+ = { INT##BITS##_MIN, 0, INT##BITS##_MAX, INT##BITS##_MAX, \
+ INT##BITS##_MIN, 0, INT##BITS##_MAX, INT##BITS##_MAX, \
+ INT##BITS##_MIN, 0, INT##BITS##_MAX, INT##BITS##_MAX }; \
+ const uint##BITS##x##LANES##_t guard_u \
+ = (uint##BITS##x##LANES##_t) { 0 } + 23; \
+ const int##BITS##x##LANES##_t guard_s \
+ = (int##BITS##x##LANES##_t) { 0 } - 17; \
+ uint##BITS##x##LANES##_t a[37], out_u[37]; \
+ int##BITS##x##LANES##_t b[37], out_s[37]; \
+ for (unsigned i = 0; i < 37; ++i) \
+ { \
+ __builtin_memcpy (&a[i], values_u + i % 4, sizeof (a[i])); \
+ __builtin_memcpy (&b[i], values_s + i % 4, sizeof (b[i])); \
+ } \
+ for (unsigned n = 0; n <= 37; ++n) \
+ { \
+ for (unsigned i = 0; i < 37; ++i) \
+ { out_u[i] = guard_u; out_s[i] = guard_s; } \
+ unsigned_add_##BITS (out_u, a, b, n); \
+ signed_add_##BITS (out_s, b, a, n); \
+ for (unsigned i = 0; i < 37; ++i) \
+ if (__builtin_memcmp (&out_u[i], \
+ i < n ? (const void *) (expected_u + i % 4) \
+ : &guard_u, sizeof (guard_u)) \
+ || __builtin_memcmp (&out_s[i], \
+ i < n ? (const void *) (expected_s + i % 4) \
+ : &guard_s, sizeof (guard_s))) \
+ __builtin_abort (); \
+ } \
+}
+
+DEFINE_ADDS (8, 8)
+DEFINE_ADDS (16, 4)
+DEFINE_ADDS (32, 2)
+DEFINE_ADDS (64, 1)
+
+int
+main (void)
+{
+ check_vect ();
+ check_8 ();
+ check_16 ();
+ check_32 ();
+ check_64 ();
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold.h
new file mode 100644
index
0000000000000000000000000000000000000000..0945018af7dc803ab3d5e5bd7830b479cdeb33a6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-intrinsics-fold.h
@@ -0,0 +1,227 @@
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+__attribute__ ((noipa)) uint8x16_t
+unsigned_sat_add (uint8x16_t x, int8x16_t y, int8x16_t shifts, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = vsqaddq_u8 (x, y);
+ y >>= shifts;
+ }
+
+ return x;
+}
+
+__attribute__ ((noipa)) int8x16_t
+signed_sat_add (int8x16_t x, uint8x16_t y, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = vuqaddq_s8 (x, y);
+ y >>= 1;
+ }
+
+ return x;
+}
+
+__attribute__ ((noipa)) uint32x4_t
+pairwise_min (uint32x4_t x, uint32x4_t y, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = vpminq_u32 (x, y);
+ y += (uint32x4_t) { 1, 3, 7, 11 };
+ }
+
+ return x;
+}
+
+__attribute__ ((noipa)) void
+store_sat_add (uint8x16_t *restrict out, const uint8x16_t *x,
+ const int8x16_t *y, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = vsqaddq_u8 (x[i], y[i]);
+}
+
+static const uint8x16_t expected_u[] = {
+ { 0, 17, 34, 51, 68, 85, 102, 119, 136, 153, 170, 187, 204, 221, 238, 255 },
+ { 0, 0, 0, 0, 8, 42, 76, 110, 144, 178, 212, 246, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 31, 50, 105, 146, 203, 233, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 28, 24, 102, 146, 228, 243, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 27, 0, 100, 146, 253, 248, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 26, 0, 99, 146, 255, 250, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 25, 0, 98, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 24, 0, 97, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 23, 0, 96, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 22, 0, 95, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 21, 0, 94, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 20, 0, 93, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 19, 0, 92, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 18, 0, 91, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 17, 0, 90, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 16, 0, 89, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 15, 0, 88, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 14, 0, 87, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 13, 0, 86, 146, 255, 251, 255, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 0, 12, 0, 85, 146, 255, 251, 255, 255, 255, 255, 255 }
+};
+static const int8x16_t expected_s[] = {
+ { -128, -111, -94, -77, -60, -43, -26, -9, 8, 25, 42, 59, 76, 93, 110, 127 },
+ { -128, -94, -60, -26, 8, 42, 76, 110, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -86, -43, -1, 42, 84, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -82, -35, 11, 59, 105, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -80, -31, 17, 67, 115, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -29, 20, 71, 120, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 73, 122, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 },
+ { -128, -79, -28, 21, 74, 123, 127, 127, 127, 127, 127, 127, 127, 127, 127,
+ 127 }
+};
+
+/* Pairwise results use ACLE lane order. */
+#if __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
+static const uint32x4_t expected_min[] = {
+ { 4000000000, 2000000000, 3000000000, 2500000000 },
+ { 23, 43, 2000000000, 2500000000 },
+ { 26, 54, 23, 2000000000 },
+ { 29, 65, 26, 23 },
+ { 32, 76, 29, 23 },
+ { 35, 87, 32, 23 },
+ { 38, 98, 35, 23 },
+ { 41, 109, 38, 23 },
+ { 44, 120, 41, 23 },
+ { 47, 131, 44, 23 },
+ { 50, 142, 47, 23 },
+ { 53, 153, 50, 23 },
+ { 56, 164, 53, 23 },
+ { 59, 175, 56, 23 },
+ { 62, 186, 59, 23 },
+ { 65, 197, 62, 23 },
+ { 68, 208, 65, 23 },
+ { 71, 219, 68, 23 },
+ { 74, 230, 71, 23 },
+ { 77, 241, 74, 23 }
+};
+#else
+static const uint32x4_t expected_min[] = {
+ { 4000000000, 2000000000, 3000000000, 2500000000 },
+ { 2000000000, 2500000000, 23, 43 },
+ { 2000000000, 23, 26, 54 },
+ { 23, 26, 29, 65 },
+ { 23, 29, 32, 76 },
+ { 23, 32, 35, 87 },
+ { 23, 35, 38, 98 },
+ { 23, 38, 41, 109 },
+ { 23, 41, 44, 120 },
+ { 23, 44, 47, 131 },
+ { 23, 47, 50, 142 },
+ { 23, 50, 53, 153 },
+ { 23, 53, 56, 164 },
+ { 23, 56, 59, 175 },
+ { 23, 59, 62, 186 },
+ { 23, 62, 65, 197 },
+ { 23, 65, 68, 208 },
+ { 23, 68, 71, 219 },
+ { 23, 71, 74, 230 },
+ { 23, 74, 77, 241 }
+};
+#endif
+static const uint8x16_t input[] = {
+ { 0, 17, 34, 51, 68, 85, 102, 119, 136, 153, 170, 187, 204, 221, 238, 255 },
+ { 1, 18, 35, 52, 69, 86, 103, 120, 137, 154, 171, 188, 205, 222, 239, 0 },
+ { 2, 19, 36, 53, 70, 87, 104, 121, 138, 155, 172, 189, 206, 223, 240, 1 },
+ { 3, 20, 37, 54, 71, 88, 105, 122, 139, 156, 173, 190, 207, 224, 241, 2 },
+ { 4, 21, 38, 55, 72, 89, 106, 123, 140, 157, 174, 191, 208, 225, 242, 3 },
+ { 5, 22, 39, 56, 73, 90, 107, 124, 141, 158, 175, 192, 209, 226, 243, 4 },
+ { 6, 23, 40, 57, 74, 91, 108, 125, 142, 159, 176, 193, 210, 227, 244, 5 }
+};
+static const int8x16_t increment[] = {
+ { -128, -111, -94, -77, -60, -43, -26, -9, 8, 25, 42, 59, 76, 93, 110, 127 },
+ { -127, -110, -93, -76, -59, -42, -25, -8, 9, 26, 43, 60, 77, 94, 111, -128
+ },
+ { -126, -109, -92, -75, -58, -41, -24, -7, 10, 27, 44, 61, 78, 95, 112, -127
+ },
+ { -125, -108, -91, -74, -57, -40, -23, -6, 11, 28, 45, 62, 79, 96, 113, -126
+ },
+ { -124, -107, -90, -73, -56, -39, -22, -5, 12, 29, 46, 63, 80, 97, 114, -125
+ },
+ { -123, -106, -89, -72, -55, -38, -21, -4, 13, 30, 47, 64, 81, 98, 115, -124
+ },
+ { -122, -105, -88, -71, -54, -37, -20, -3, 14, 31, 48, 65, 82, 99, 116, -123
+ }
+};
+static const uint8x16_t expected_store[] = {
+ { 0, 0, 0, 0, 8, 42, 76, 110, 144, 178, 212, 246, 255, 255, 255, 255 },
+ { 0, 0, 0, 0, 10, 44, 78, 112, 146, 180, 214, 248, 255, 255, 255, 0 },
+ { 0, 0, 0, 0, 12, 46, 80, 114, 148, 182, 216, 250, 255, 255, 255, 0 },
+ { 0, 0, 0, 0, 14, 48, 82, 116, 150, 184, 218, 252, 255, 255, 255, 0 },
+ { 0, 0, 0, 0, 16, 50, 84, 118, 152, 186, 220, 254, 255, 255, 255, 0 },
+ { 0, 0, 0, 0, 18, 52, 86, 120, 154, 188, 222, 255, 255, 255, 255, 0 },
+ { 0, 0, 0, 0, 20, 54, 88, 122, 156, 190, 224, 255, 255, 255, 255, 0 }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ const uint8x16_t u = { 0, 17, 34, 51, 68, 85, 102, 119,
+ 136, 153, 170, 187, 204, 221, 238, 255 };
+ const int8x16_t s = { -128, -111, -94, -77, -60, -43, -26, -9,
+ 8, 25, 42, 59, 76, 93, 110, 127 };
+ const int8x16_t shifts = { 0, 1, 2, 0, 1, 2, 0, 1,
+ 2, 0, 1, 2, 0, 1, 2, 0 };
+
+ /* Distinct halves and large unsigned values catch incorrect splitting. */
+ const uint32x4_t x = { 4000000000U, 2000000000U,
+ 3000000000U, 2500000000U };
+ const uint32x4_t y = { 4000000017U, 23, 3000000011U, 43 };
+ for (unsigned n = 0; n < 20; ++n)
+ {
+ uint8x16_t a = unsigned_sat_add (u, s, shifts, n);
+ int8x16_t b = signed_sat_add (s, u, n);
+ uint32x4_t z = pairwise_min (x, y, n);
+ if (__builtin_memcmp (&a, &expected_u[n], sizeof (a))
+ || __builtin_memcmp (&b, &expected_s[n], sizeof (b))
+ || __builtin_memcmp (&z, &expected_min[n], sizeof (z)))
+ __builtin_abort ();
+ }
+
+ uint8x16_t output[7];
+ store_sat_add (output, input, increment, 7);
+ if (__builtin_memcmp (output, expected_store, sizeof (output)))
+ __builtin_abort ();
+
+ return 0;
+}
