> -----Original Message-----
> From: Kyrylo Tkachov <[email protected]>
> Sent: 20 August 2026 15:32
> To: Tamar Christina <[email protected]>
> Cc: [email protected]
> Subject: Re: [PATCH] aarch64: Use SVE SPLICE for scalable recurrences
>
> Hi Tamar,
>
> > On 20 Aug 2026, at 15:52, Tamar Christina <[email protected]>
> wrote:
> >
> >> -----Original Message-----
> >> From: [email protected] <[email protected]>
> >> Sent: 20 August 2026 13:34
> >> To: [email protected]
> >> Cc: Tamar Christina <[email protected]>; Kyrylo Tkachov
> >> <[email protected]>
> >> Subject: [PATCH] aarch64: Use SVE SPLICE for scalable recurrences
> >>
> >> From: Kyrylo Tkachov <[email protected]>
> >>
> >> The vectorizer represents a first-order recurrence as a scalable
> >> permutation that takes the final D elements from one vector and the
> >> remaining elements from the next vector. The AArch64 permutation
> >> expander did not recognize this variable-length shape.
> >>
> >> Recognize the exact two-input permutation and build a predicate for D
> >> elements. Reverse that predicate and use the SVE SPLICE pattern. Reject
> >> fixed-length vectors, rotations, and distances that cannot be represented
> >> at the minimum vector length.
> >>
> >> The compile test checks seven direct SPLICE sequences across byte,
> >> halfword, word, and doubleword modes. A 67-element runtime test
> covers
> >> distances 1 and 2 with two seed values, and partial final vectors.
> >
> > Nice!
> >
> >>
> >> On an example testcase:
> >>
> >> typedef __UINT32_TYPE__ uint32_t;
> >>
> >> void
> >> recur (const uint32_t *__restrict a, uint32_t *__restrict b,
> >> const uint32_t *__restrict init)
> >> {
> >> uint32_t prev = *init;
> >> for (int i = 0; i < 67; ++i)
> >> {
> >> b[i] = a[i] - prev;
> >> prev = a[i];
> >> }
> >> }
> >>
> >> With -O3 -march=armv8.2-a+sve
> >> We emitted before:
> >>
> >> ldr w4, [x2]
> >> mov w3, 0
> >> .p2align 5,,15
> >> .L2:
> >> mov w2, w4
> >> ldr w4, [x0, x3]
> >> sub w2, w4, w2
> >> str w2, [x1, x3]
> >> add x3, x3, 4
> >> cmp x3, 268
> >> bne .L2
> >>
> >> And after this patch:
> >> mov w3, 0
> >> cntw x5
> >> ptrue p5.b, all
> >> mov w4, 67
> >> ptrue p6.b, vl1
> >> mov p7.b, p5.b
> >> ld1rw z31.s, p5/z, [x2]
> >> rev p6.s, p6.s
> >> .p2align 5,,15
> >> .L2:
> >> mov z30.d, z31.d
> >> ld1w z31.s, p7/z, [x0, x3, lsl 2]
> >> splice z30.s, p6, z30.s, z31.s
> >> sub z30.s, z31.s, z30.s
> >> st1w z30.s, p7, [x1, x3, lsl 2]
> >> add x3, x3, x5
> >> whilelo p7.s, w3, w4
> >> b.any .L2
> >>
> >> This also vectorizes some loops in OpenEXR 4.0.0.
> >>
> >> Upstream PXR24, RLE, ZIP, and ZIPS tests pass, and the baseline and
> >> patched encoders produced identical files. A paired exrmetrics run on an
> >> NVIDIA Vera system measured the following write-time improvements.
> >>
> >> Flowers.exr multipart.0001.exr
> >> PXR24, float 2.28% 4.58%
> >> PXR24, original 3.22% 8.07%
> >> RLE 13.52% 23.27%
> >> ZIP 3.67% 9.59%
> >> ZIPS 3.15% 5.62%
> >>
> >> Bootstrapped and tested on aarch64-none-linux-gnu.
> >> Ok for trunk?
> >> Thanks,
> >> Kyrill
> >>
> >> gcc/ChangeLog:
> >>
> >> * config/aarch64/aarch64.cc (aarch64_evpc_splice): New function.
> >> (aarch64_expand_vec_perm_const_1): Use it.
> >>
> >> gcc/testsuite/ChangeLog:
> >>
> >> * gcc.target/aarch64/sve/perm_splice_1.c: New test.
> >> * gcc.target/aarch64/sve/perm_splice_1-run.c: Likewise.
> >>
> >> Signed-off-by: Kyrylo Tkachov <[email protected]>
> >> ---
> >> gcc/config/aarch64/aarch64.cc | 42 +++++++++
> >> .../aarch64/sve/perm_splice_1-run.c | 94 +++++++++++++++++++
> >> .../gcc.target/aarch64/sve/perm_splice_1.c | 63 +++++++++++++
> >> 3 files changed, 199 insertions(+)
> >> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-
> >> run.c
> >> create mode 100644
> gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c
> >>
> >> diff --git a/gcc/config/aarch64/aarch64.cc
> b/gcc/config/aarch64/aarch64.cc
> >> index fdffb13ad22..2e440ce916d 100644
> >> --- a/gcc/config/aarch64/aarch64.cc
> >> +++ b/gcc/config/aarch64/aarch64.cc
> >> @@ -28261,6 +28261,46 @@ aarch64_evpc_ext (struct
> expand_vec_perm_d
> >> *d)
> >> return true;
> >> }
> >>
> >> +/* Return true if D describes a scalable-vector permutation that takes the
> >> + last DIST elements from the first input and the remaining elements from
> >> + the second input. */
> >> +
> >> +static bool
> >> +aarch64_evpc_splice (struct expand_vec_perm_d *d)
> >> +{
> >> + poly_int64 nelt = d->perm.length ();
> >> + HOST_WIDE_INT dist;
> >> +
> >> + if (d->vec_flags != VEC_SVE_DATA
> >> + || d->one_vector_p
> >> + || nelt.is_constant ()
> >> + || !(nelt - d->perm[0]).is_constant (&dist)
> >> + || !IN_RANGE (dist, 1, INT_MAX)
> >> + || !d->perm.series_p (0, 1, nelt - dist, 1))
> >> + return false;
> >> +
> >> + machine_mode pred_mode = aarch64_sve_pred_mode (d->vmode);
> >> + if (aarch64_svpattern_for_vl (pred_mode, dist)
> >> + == AARCH64_NUM_SVPATTERNS)
> >> + return false;
> >> +
> >> + if (d->testing_p)
> >> + return true;
> >> +
> >> + rtx_vector_builder builder (pred_mode, dist, 2);
> >> + for (HOST_WIDE_INT i = 0; i < dist; ++i)
> >> + builder.quick_push (CONST1_RTX (BImode));
> >> + for (HOST_WIDE_INT i = 0; i < dist; ++i)
> >> + builder.quick_push (CONST0_RTX (BImode));
> >> +
> >> + rtx head = force_reg (pred_mode, builder.build ());
> >
> > Since you already have the pattern above, why not save the result of the
> > aarch64_svpattern_for...
> >
> > and then replace this builder code with
> >
> > rtx head = aarch64_ptrue_reg (pred_mode, pattern);
> >
> > which is slightly cleaner?
>
> There is no aarch64_ptrue_reg overload taking an aarch64_svpattern. The
> three are (machine_mode), (machine_mode, unsigned int), (machine_mode,
> machine_mode), so this silently selects the unsigned int one via enum-to-int
> conversion.
>
> If we add one such overload I think it’d be a separate patch that can clean up
> other opportunities like this around here?
Sure, that works for me.
Thanks,
Tamar
> Thanks,
> Kyrill
>
> >
> > LGTM with that change.
> >
> > Thanks,
> > Tamar
> >
> >> + rtx pred = gen_reg_rtx (pred_mode);
> >> + emit_insn (gen_aarch64_sve_rev (pred_mode, pred, head));
> >> + emit_insn (gen_aarch64_sve_splice (d->vmode, d->target, pred,
> >> + d->op0, d->op1));
> >> + return true;
> >> +}
> >> +
> >> /* Recognize patterns for the REV{64,32,16} insns, which reverse elements
> >> within each 64-bit, 32-bit or 16-bit granule. */
> >>
> >> @@ -28786,6 +28826,8 @@ aarch64_expand_vec_perm_const_1 (struct
> >> expand_vec_perm_d *d)
> >> return true;
> >> else if (aarch64_evpc_rev_global (d))
> >> return true;
> >> + else if (aarch64_evpc_splice (d))
> >> + return true;
> >> else if (aarch64_evpc_ext (d))
> >> return true;
> >> else if (aarch64_evpc_dup (d))
> >> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c
> >> b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c
> >> new file mode 100644
> >> index 00000000000..b2c13120c53
> >> --- /dev/null
> >> +++ b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1-run.c
> >> @@ -0,0 +1,94 @@
> >> +/* { dg-do run } */
> >> +/* { dg-require-effective-target aarch64_sve_hw } */
> >> +/* { dg-options "-O3 -march=armv8.2-a+sve -mautovec-preference=sve-
> >> only" } */
> >> +/* { dg-additional-options "-msve-vector-bits=scalable" } */
> >> +/* { dg-additional-options "-fvect-cost-model=unlimited" } */
> >> +
> >> +typedef __UINT8_TYPE__ uint8_t;
> >> +typedef __UINT16_TYPE__ uint16_t;
> >> +typedef __UINT32_TYPE__ uint32_t;
> >> +typedef __UINT64_TYPE__ uint64_t;
> >> +
> >> +#define N 67
> >> +
> >> +#define DEFINE_RECUR(TYPE, SUFFIX) \
> >> + __attribute__((noipa)) \
> >> + static void \
> >> + recur1_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \
> >> + const TYPE *__restrict init) \
> >> + { \
> >> + TYPE prev = init[0]; \
> >> + for (int i = 0; i < N; ++i) \
> >> + { \
> >> + b[i] = a[i] - prev; \
> >> + prev = a[i]; \
> >> + } \
> >> + } \
> >> + \
> >> + __attribute__((noipa)) \
> >> + static void \
> >> + recur2_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \
> >> + const TYPE *__restrict init) \
> >> + { \
> >> + TYPE prev0 = init[0]; \
> >> + TYPE prev1 = init[0]; \
> >> + for (int i = 0; i + 1 < N; i += 2) \
> >> + { \
> >> + b[i] = a[i] - prev0; \
> >> + prev0 = a[i]; \
> >> + b[i + 1] = a[i + 1] - prev1; \
> >> + prev1 = a[i + 1]; \
> >> + } \
> >> + b[N - 1] = 0; \
> >> + }
> >> +
> >> +DEFINE_RECUR (uint8_t, u8)
> >> +DEFINE_RECUR (uint16_t, u16)
> >> +DEFINE_RECUR (uint32_t, u32)
> >> +DEFINE_RECUR (uint64_t, u64)
> >> +
> >> +#define CHECK_RECUR(TYPE, SUFFIX) \
> >> + do \
> >> + { \
> >> + TYPE a[N]; \
> >> + TYPE b[N]; \
> >> + const TYPE init[2] = { (TYPE) 19, (TYPE) 43 }; \
> >> + \
> >> + for (int i = 0; i < N; ++i) \
> >> + a[i] = (TYPE) ((unsigned int) i * (unsigned int) i \
> >> + + 5U * (unsigned int) i + 11U); \
> >> + \
> >> + for (int seed = 0; seed < 2; ++seed) \
> >> + { \
> >> + recur1_##SUFFIX (a, b, &init[seed]); \
> >> + if (b[0] != (TYPE) (a[0] - init[seed])) \
> >> + __builtin_abort (); \
> >> + _Pragma ("GCC novector") \
> >> + for (int i = 1; i < N; ++i) \
> >> + if (b[i] != (TYPE) (a[i] - a[i - 1])) \
> >> + __builtin_abort (); \
> >> + \
> >> + recur2_##SUFFIX (a, b, &init[seed]); \
> >> + if (b[0] != (TYPE) (a[0] - init[seed]) \
> >> + || b[1] != (TYPE) (a[1] - init[seed])) \
> >> + __builtin_abort (); \
> >> + _Pragma ("GCC novector") \
> >> + for (int i = 2; i < N - 1; ++i) \
> >> + if (b[i] != (TYPE) (a[i] - a[i - 2])) \
> >> + __builtin_abort (); \
> >> + if (b[N - 1] != 0) \
> >> + __builtin_abort (); \
> >> + } \
> >> + } \
> >> + while (0)
> >> +
> >> +int
> >> +main (void)
> >> +{
> >> + CHECK_RECUR (uint8_t, u8);
> >> + CHECK_RECUR (uint16_t, u16);
> >> + CHECK_RECUR (uint32_t, u32);
> >> + CHECK_RECUR (uint64_t, u64);
> >> +
> >> + return 0;
> >> +}
> >> diff --git a/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c
> >> b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c
> >> new file mode 100644
> >> index 00000000000..c2a9879008b
> >> --- /dev/null
> >> +++ b/gcc/testsuite/gcc.target/aarch64/sve/perm_splice_1.c
> >> @@ -0,0 +1,63 @@
> >> +/* { dg-do compile } */
> >> +/* { dg-options "-O3 -march=armv8.2-a+sve -mautovec-preference=sve-
> >> only" } */
> >> +/* { dg-additional-options "-msve-vector-bits=scalable" } */
> >> +/* { dg-additional-options "-fvect-cost-model=unlimited -fdump-tree-
> vect-
> >> details" } */
> >> +
> >> +typedef __UINT8_TYPE__ uint8_t;
> >> +typedef __UINT16_TYPE__ uint16_t;
> >> +typedef __UINT32_TYPE__ uint32_t;
> >> +typedef __UINT64_TYPE__ uint64_t;
> >> +
> >> +#define DEF_RECUR1(TYPE, SUFFIX) \
> >> + __attribute__((noipa)) \
> >> + void \
> >> + recur1_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \
> >> + const TYPE *__restrict init) \
> >> + { \
> >> + TYPE prev = *init; \
> >> + for (int i = 0; i < 67; ++i) \
> >> + { \
> >> + b[i] = a[i] - prev; \
> >> + prev = a[i]; \
> >> + } \
> >> + }
> >> +
> >> +#define DEF_RECUR2(TYPE, SUFFIX) \
> >> + __attribute__((noipa)) \
> >> + void \
> >> + recur2_##SUFFIX (TYPE *__restrict a, TYPE *__restrict b, \
> >> + const TYPE *__restrict init) \
> >> + { \
> >> + TYPE prev0 = *init; \
> >> + TYPE prev1 = *init; \
> >> + for (int i = 0; i < 66; i += 2) \
> >> + { \
> >> + b[i] = a[i] - prev0; \
> >> + prev0 = a[i]; \
> >> + b[i + 1] = a[i + 1] - prev1; \
> >> + prev1 = a[i + 1]; \
> >> + } \
> >> + }
> >> +
> >> +DEF_RECUR1 (uint8_t, u8)
> >> +DEF_RECUR1 (uint16_t, u16)
> >> +DEF_RECUR1 (uint32_t, u32)
> >> +DEF_RECUR1 (uint64_t, u64)
> >> +
> >> +DEF_RECUR2 (uint8_t, u8)
> >> +DEF_RECUR2 (uint16_t, u16)
> >> +DEF_RECUR2 (uint32_t, u32)
> >> +
> >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.b, vl1} 4 } } */
> >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.b, vl2} 1 } } */
> >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.h, vl2} 1 } } */
> >> +/* { dg-final { scan-assembler-times {\tptrue\tp[0-9]+\.s, vl2} 1 } } */
> >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.b, p[0-9]+\.b} 2 }
> >> } */
> >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.h, p[0-9]+\.h} 2 }
> >> } */
> >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.s, p[0-9]+\.s} 2 }
> >> } */
> >> +/* { dg-final { scan-assembler-times {\trev\tp[0-9]+\.d, p[0-9]+\.d} 1 }
> >> } */
> >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.b, p[0-9]+, z[0-
> 9]+\.b,
> >> z[0-9]+\.b} 2 } } */
> >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.h, p[0-9]+, z[0-
> 9]+\.h,
> >> z[0-9]+\.h} 2 } } */
> >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.s, p[0-9]+, z[0-
> 9]+\.s,
> >> z[0-9]+\.s} 2 } } */
> >> +/* { dg-final { scan-assembler-times {\tsplice\tz[0-9]+\.d, p[0-9]+, z[0-
> 9]+\.d,
> >> z[0-9]+\.d} 1 } } */
> >> +/* { dg-final { scan-tree-dump-times "vectorized 1 loops in function" 7
> "vect"
> >> } } */
> >> --
> >> 2.50.1 (Apple Git-155)
>
