We can now lift the emulated-vector restriction and allow native fixed-length
data vectors to participate in SLP. This lets the vectorizer combine existing
operations into a wider vector type, including changing from Adv. SIMD to SVE.
For example, PR119187 has two adjacent eight-byte additions:
#include <arm_neon.h>
void foo2 (unsigned char *a, unsigned char *b, unsigned char *c)
{
uint8x8_t a0 = vld1_u8 (a), a1 = vld1_u8 (a + 8);
uint8x8_t b0 = vld1_u8 (b), b1 = vld1_u8 (b + 8);
vst1_u8 (c, vadd_u8 (a0, b0));
vst1_u8 (c + 8, vadd_u8 (a1, b1));
}
Each source addition contributes eight lanes to the SLP group. The two
additions can therefore use one vector(16) unsigned char operation, with the
load and store groups covering the same sixteen bytes.
With -O3 -march=armv8-a we generated before:
ldp d29, d31, [x0]
ldp d28, d30, [x1]
add v28.8b, v29.8b, v28.8b
add v30.8b, v31.8b, v30.8b
stp d28, d30, [x2]
ret
After these patches we get:
ldr q31, [x0]
ldr q30, [x1]
add v30.16b, v31.16b, v30.16b
str q30, [x2]
ret
To support this we seed loop-store roots with complete native source vectors
and count their elements when building grouped stores. If discovery falls back
to smaller groups, each group must end at a source-vector boundary. BB SLP
skips native groups that would keep or reduce the existing width.
For operands outside the tree, vect_create_constant_vectors concatenates whole
source vectors when their widths and element types fit. Extracting every
scalar lane instead could split existing vector loads into scalar loads.
Results needing zero padding keep the other construction path.
vect_can_revectorize_type_p accepts native data vectors but excludes boolean
vectors, whose packed layout differs from their scalar element layout.
Register aggregates are left alone when widening would require incompatible
register modes. Concatenated-subreg folding checks mode compatibility too.
Backend tests that require their original access widths disable the relevant
vectorization pass.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Any feedback?
Thanks,
Tamar
Assisted-by: Codex GPT 6.1 Sol
AI was used to generate test cases for various types and inputs.
gcc/ChangeLog:
PR tree-optimization/119187
* tree-vectorizer.cc (vect_can_revectorize_type_p): Allow native types.
* tree-vect-data-refs.cc (vect_analyze_data_refs): Analyze native vector
element types and avoid spilling register aggregates.
* tree-vect-loop.cc (vect_analyze_scalar_cycles_1): Allow native vector
PHIs.
* tree-vect-slp.cc (vect_build_slp_tree_2): Use invariant element types.
(vect_analyze_slp_instance): Count grouped source lanes and preserve
complete source vectors when splitting groups. Skip unchanged widths.
(vect_analyze_slp): Seed complete vector loop-store roots.
* simplify-rtx.cc (simplify_context::simplify_binary_operation_1):
Require compatible register modes when folding concatenated subregs.
* tree-vect-slp.cc (vect_create_constant_vectors): Concatenate whole
source vectors when their widths and types match.
* tree-vect-stmts.cc (get_vectype_for_scalar_type): Restrict source
vector handling to supported data vector types.
gcc/testsuite/ChangeLog:
PR tree-optimization/119187
* gcc.dg/vect/vect-revectorize-invariant-cyclic.c: New test.
* gcc.dg/vect/vect-revectorize-native.c: New test.
* gcc.dg/vect/vect-revectorize-group-gaps.c: New test.
* gcc.dg/vect/vect-revectorize-group-live.c: New test.
* gcc.target/aarch64/vect-revectorize-bb-native.c: New test.
* gcc.target/aarch64/vect-revectorize-group-1.c: New test.
* gcc.target/aarch64/vect-revectorize-group-2.c: New test.
* gcc.target/aarch64/vect-revectorize-group-live.c: New test.
* gcc.target/aarch64/vect-revectorize-group.h: New test.
* gcc.target/aarch64/vect-revectorize-widen-1.c: New test.
* gcc.target/aarch64/vect-revectorize-widen-2.c: New test.
* gcc.target/aarch64/vect-revectorize-widen-3.c: New test.
* gcc.target/aarch64/vect-revectorize-widen.h: New test.
* gcc.target/aarch64/pr113613.c: Disable vectorization.
* gcc.target/aarch64/stp_vec_64_1.c: Likewise.
* gcc.target/aarch64/vect-vrnd.c: Disable loop vectorization and check
SIMD availability.
* gcc.dg/vect/vect-revectorize-bb-live.c: Update test.
* gcc.target/aarch64/vect-revectorize-tuple-pack.c: New test.
* gcc.target/aarch64/vect-revectorize-concat-1.c: New test.
* gcc.target/aarch64/vect-revectorize-predicate.c: New test.
* gcc.target/i386/pr52252-core.c: Disable tree vectorization to retain
the original test width.
* gcc.target/i386/pr85323-2.c: Disable tree vectorization to retain the
original test width.
* gcc.target/i386/pr88531-1b.c: Disable tree vectorization to retain the
original test width.
* gcc.target/i386/vect-strided-1.c: Disable tree vectorization to retain
the original test width.
* gcc.target/i386/vect-strided-1b.c: Disable tree vectorization to
retain the original test width.
* gcc.target/i386/vect-strided-2b.c: Disable tree vectorization to
retain the original test width.
---
--
diff --git a/gcc/simplify-rtx.cc b/gcc/simplify-rtx.cc
index
e979d4c296411170d3de0ffa5863df7cd07c0b33..394c96ca5fbd05ca9a72b7e24f8d664cc5edb6c7
100644
--- a/gcc/simplify-rtx.cc
+++ b/gcc/simplify-rtx.cc
@@ -5730,13 +5730,16 @@ simplify_ashift:
(subreg:N/2 OP 0)
(subreg:N/2 OP N/2)) --> OP
i.e. where concatenating the first and second halves of the
- same object OP. */
+ same object OP. A register tuple can have a different layout
+ from the packed vector, so require compatible register modes. */
{
poly_uint64 offset = 0u;
rtx base0 = get_ref_base_and_offset (trueop0, &offset);
if (known_eq (offset, 0u)
&& known_eq (GET_MODE_SIZE (GET_MODE (base0)),
- GET_MODE_SIZE (mode)))
+ GET_MODE_SIZE (mode))
+ && (MEM_P (base0)
+ || targetm.modes_tieable_p (GET_MODE (base0), mode)))
{
rtx base1 = get_ref_base_and_offset (trueop1, &offset);
if (rtx_equal_p (base0, base1)
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-live.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-live.c
index
2615b2eace1991817bfbfa55c8a4d424345785e9..34737ea6e37b49643bcce6722a60049f85b25f96
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-live.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-live.c
@@ -15,20 +15,35 @@ store_and_return (V *out, const V *a, const V *b)
return value;
}
+/* Reconstruct after all pieces, even when the permutation reverses their
+ source order. The store and return must agree on every lane. */
+__attribute__ ((noipa)) V
+permute_and_return (V *out, const V *a, const V *b)
+{
+ V value = *a + *b;
+ V result = __builtin_shufflevector (value, value, 4, 5, 6, 7, 0, 1, 2, 3);
+ *out = result;
+ return result;
+}
+
int
main (void)
{
check_vect ();
- V a, b, stored;
- for (unsigned i = 0; i < 8; ++i)
- {
- a[i] = 13 * i;
- b[i] = 100 - 7 * i;
- }
+ V a = { 0, 13, 26, 39, 52, 65, 78, 91 };
+ V b = { 100, 93, 86, 79, 72, 65, 58, 51 };
+ V expected = { 101, 105, 117, 113, 117, 137, 133, 129 };
+ V expected_permute = { 124, 130, 136, 142, 100, 106, 112, 118 };
+ V stored;
V returned = store_and_return (&stored, &a, &b);
- for (unsigned i = 0; i < 8; ++i)
- if (stored[i] != ((100 + 6 * i) ^ (2 * i + 1))
- || returned[i] != stored[i])
- __builtin_abort ();
+ if (__builtin_memcmp (&stored, &expected, sizeof (stored))
+ || __builtin_memcmp (&returned, &expected, sizeof (returned)))
+ __builtin_abort ();
+
+ returned = permute_and_return (&stored, &a, &b);
+ if (__builtin_memcmp (&stored, &expected_permute, sizeof (stored))
+ || __builtin_memcmp (&returned, &expected_permute, sizeof (returned)))
+ __builtin_abort ();
+
return 0;
}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-gaps.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-gaps.c
new file mode 100644
index
0000000000000000000000000000000000000000..fae053bc2832c33b6141ab588126c1ac7f860229
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-gaps.c
@@ -0,0 +1,60 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V __attribute__ ((vector_size (8)));
+
+/* Data-reference groups have three source vectors, while the SLP nodes
+ contain two. Test both a trailing gap and a permuted group. */
+__attribute__ ((noipa)) void
+group_gaps (V *restrict out, const V *a, const V *b, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ {
+ out[2 * i] = a[3 * i] + b[3 * i + 2];
+ out[2 * i + 1] = a[3 * i + 1] + b[3 * i];
+ }
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ V a[210], b[210], out[141];
+ const V guard = { -1, -1 };
+
+ /* Different offsets keep every source group distinct. */
+ for (unsigned i = 0; i < 70; ++i)
+ {
+ a[3 * i] = (V) { 10 * i + 1, 10 * i + 2 };
+ a[3 * i + 1] = (V) { 10 * i + 3, 10 * i + 4 };
+ a[3 * i + 2] = (V) { 10 * i + 5, 10 * i + 6 };
+ b[3 * i] = (V) { 1000 * i + 10, 1000 * i + 20 };
+ b[3 * i + 1] = (V) { 1000 * i + 30, 1000 * i + 40 };
+ b[3 * i + 2] = (V) { 1000 * i + 50, 1000 * i + 60 };
+ }
+
+ for (unsigned n = 0; n <= 70; ++n)
+ {
+ out[2 * n] = guard;
+ group_gaps (out, a, b, n);
+ for (unsigned i = 0; i < n; ++i)
+ {
+ /* The two results start at { 51, 62 } and { 13, 24 }.
+ Each iteration adds 10 from A and 1000 from B. */
+ const unsigned expected[] = {
+ 1010 * i + 51, 1010 * i + 62,
+ 1010 * i + 13, 1010 * i + 24
+ };
+ if (__builtin_memcmp (&out[2 * i], expected, sizeof (expected)))
+ __builtin_abort ();
+ }
+
+ if (__builtin_memcmp (&out[2 * n], &guard, sizeof (guard)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-live.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-live.c
new file mode 100644
index
0000000000000000000000000000000000000000..a007afc1127eb140777e12cb09cf4efeae0689b8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-group-live.c
@@ -0,0 +1,68 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+#ifndef BYTES
+#define BYTES 64
+#endif
+typedef unsigned V __attribute__ ((vector_size (BYTES)));
+
+/* Both source vectors belong to the same SLP group. The second vector
+ is live after the loop and must be reconstructed from its own pieces. */
+__attribute__ ((noipa)) V
+group_live (V *restrict out, const V *a, const V *b, unsigned n)
+{
+ V x = { 0 }, y = { 0 };
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = (x + a[2 * i]) ^ b[2 * i];
+ y = (y + a[2 * i + 1]) ^ b[2 * i + 1];
+ out[2 * i] = x;
+ out[2 * i + 1] = y;
+ }
+
+ return y;
+}
+
+/* Scaling a power of two shifts the same recurrence into each lane.
+ Distinct lanes also check reconstruction of every source-vector piece. */
+static const unsigned lane_values[] = {
+ 1, 2, 4, 8, 16, 32, 64, 128,
+ 256, 512, 1024, 2048, 4096, 8192, 16384, 32768
+};
+
+/* For A = 1, 2, ..., 10 and B = 3 * A, the two recurrences produce
+ these multipliers, interleaved in the same order as the output vectors. */
+static const unsigned expected[] = { 2, 4, 12, 4, 30, 24, 48, 56, 34, 92 };
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ V a[10], b[10], out[10], lanes;
+ __builtin_memcpy (&lanes, lane_values, sizeof (V));
+ for (unsigned i = 0; i < 10; ++i)
+ {
+ a[i] = lanes * (i + 1);
+ b[i] = a[i] * 3;
+ }
+
+ for (unsigned n = 0; n <= 5; ++n)
+ {
+ V result = group_live (out, a, b, n);
+ for (unsigned i = 0; i < 2 * n; ++i)
+ {
+ V want = lanes * expected[i];
+ if (__builtin_memcmp (&out[i], &want, sizeof (V)))
+ __builtin_abort ();
+ }
+
+ V want = n ? lanes * expected[2 * n - 1] : (V) { 0 };
+ if (__builtin_memcmp (&result, &want, sizeof (V)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-invariant-cyclic.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-invariant-cyclic.c
new file mode 100644
index
0000000000000000000000000000000000000000..0483fc50828395fd61183b0927be90aae18bf7b2
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-invariant-cyclic.c
@@ -0,0 +1,63 @@
+/* { dg-additional-options "-O2" } */
+/* { dg-require-effective-target vect_int } */
+
+#include "tree-vect.h"
+
+struct record
+{
+ unsigned int a, b, c;
+};
+
+/* Packing three cyclic invariants can skip past the end of the list. */
+__attribute__ ((noipa)) void
+fill (struct record *p, int n, unsigned int a, unsigned int b, unsigned int c)
+{
+ for (int i = 0; i < n; ++i)
+ {
+ p[i].a = a;
+ p[i].b = b;
+ p[i].c = c;
+ }
+}
+
+__attribute__ ((noipa)) void
+fill_same (struct record *p, int start, int end, unsigned int a)
+{
+ for (int i = end - 1; i >= start; --i)
+ {
+ p[i].a = a;
+ p[i].b = a;
+ p[i].c = 6;
+ }
+}
+
+#define N 67
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ struct record initial[N];
+ for (unsigned i = 0; i < N; ++i)
+ initial[i] = (struct record) { 101 + i, 202 + i, 303 + i };
+
+ for (int n = 0; n < N; ++n)
+ {
+ struct record p[N], q[N];
+ __builtin_memcpy (p, initial, sizeof (p));
+ __builtin_memcpy (q, initial, sizeof (q));
+ const struct record filled = { 17 * n, 17 * n + 1, 17 * n + 2 };
+ const struct record same = { 17 * n, 17 * n, 6 };
+ fill (p, n, filled.a, filled.b, filled.c);
+ fill_same (q, n / 3, n, same.a);
+ for (int i = 0; i < N; ++i)
+ if (__builtin_memcmp (&p[i], i < n ? &filled : &initial[i],
+ sizeof (p[i]))
+ || __builtin_memcmp (&q[i],
+ i >= n / 3 && i < n ? &same : &initial[i],
+ sizeof (q[i])))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-native.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-native.c
new file mode 100644
index
0000000000000000000000000000000000000000..e89696254ed6d0240d0d11c13777b024c05b9cc8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-native.c
@@ -0,0 +1,68 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V __attribute__ ((vector_size (16)));
+
+__attribute__ ((noipa)) void
+native_vectors (V *restrict out, const V *restrict a, const V *restrict b,
+ unsigned n)
+{
+ V x = (V) { 1, 3, 5, 7 };
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x = (x + a[i]) ^ b[i];
+ out[i] = x;
+ }
+}
+
+static const V a[] = {
+ { 0, 1, 2, 3 },
+ { 11, 12, 13, 14 },
+ { 22, 23, 24, 25 },
+ { 33, 34, 35, 36 },
+ { 44, 45, 46, 47 },
+ { 55, 56, 57, 58 },
+ { 66, 67, 68, 69 },
+ { 77, 78, 79, 80 },
+ { 88, 89, 90, 91 }
+};
+static const V b[] = {
+ { 100, 97, 94, 91 },
+ { 99, 96, 93, 90 },
+ { 98, 95, 92, 89 },
+ { 97, 94, 91, 88 },
+ { 96, 93, 90, 87 },
+ { 95, 92, 89, 86 },
+ { 94, 91, 88, 85 },
+ { 93, 90, 87, 84 },
+ { 92, 89, 86, 83 }
+};
+static const V expected[] = {
+ { 101, 101, 89, 81 },
+ { 19, 17, 59, 5 },
+ { 75, 119, 15, 71 },
+ { 13, 199, 105, 51 },
+ { 89, 169, 205, 53 },
+ { 207, 189, 351, 57 },
+ { 335, 347, 507, 43 },
+ { 449, 499, 541, 47 },
+ { 581, 533, 545, 217 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ V out[9];
+ for (unsigned n = 0; n <= 9; ++n)
+ {
+ native_vectors (out, a, b, n);
+ if (__builtin_memcmp (out, expected, n * sizeof (V)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/pr113613.c
b/gcc/testsuite/gcc.target/aarch64/pr113613.c
index
382e4a11c0a3382f236ef43337d5d0ea4a67ee0c..37abd17b96a8cc0a1727203c4cd634891293c272
100644
--- a/gcc/testsuite/gcc.target/aarch64/pr113613.c
+++ b/gcc/testsuite/gcc.target/aarch64/pr113613.c
@@ -1,4 +1,4 @@
-// { dg-options "-O2" }
+// { dg-options "-O2 -fno-tree-vectorize" }
typedef float __attribute__((vector_size(8))) v2sf;
v2sf a[4];
diff --git a/gcc/testsuite/gcc.target/aarch64/stp_vec_64_1.c
b/gcc/testsuite/gcc.target/aarch64/stp_vec_64_1.c
index
11e757a245d2ac5e58df1a8ff731535a6d8591ae..e988e89d24a9518d255b1dd5cfd076e6ea6e2bee
100644
--- a/gcc/testsuite/gcc.target/aarch64/stp_vec_64_1.c
+++ b/gcc/testsuite/gcc.target/aarch64/stp_vec_64_1.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-Ofast" } */
+/* { dg-options "-Ofast -fno-tree-vectorize" } */
typedef int int32x2_t __attribute__ ((__vector_size__ ((8))));
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-native.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-native.c
new file mode 100644
index
0000000000000000000000000000000000000000..97c44b4f79f123af8d00d559faf618e8b9de3bb1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-native.c
@@ -0,0 +1,72 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fdump-tree-slp-details -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+__attribute__ ((noipa)) void
+foo (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ uint8x8_t av1 = vld1_u8 (a);
+ uint8x8_t av2 = vld1_u8 (a + 8);
+ uint8x8_t bv1 = vld1_u8 (b);
+ uint8x8_t bv2 = vld1_u8 (b + 8);
+ vst1_u8 (c, vadd_u8 (av1, bv1));
+ vst1_u8 (c + 8, vadd_u8 (av2, bv2));
+}
+
+/* Also keep the second source vector live after the grouped stores. */
+__attribute__ ((noipa)) uint8x8_t
+foo_live (uint8_t *a, uint8_t *b, uint8_t *c)
+{
+ uint8x8_t av1 = vld1_u8 (a);
+ uint8x8_t av2 = vld1_u8 (a + 8);
+ uint8x8_t bv1 = vld1_u8 (b);
+ uint8x8_t bv2 = vld1_u8 (b + 8);
+ uint8x8_t cv1 = vadd_u8 (av1, bv1);
+ uint8x8_t cv2 = vadd_u8 (av2, bv2);
+ vst1_u8 (c, cv1);
+ vst1_u8 (c + 8, cv2);
+ return cv2;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ uint8_t a[48], b[48], out[48], returned[8], expected[16];
+ for (unsigned live = 0; live < 2; ++live)
+ for (unsigned overlap = 0; overlap < 3; ++overlap)
+ for (unsigned offset = 0; offset < 16; ++offset)
+ {
+ for (unsigned i = 0; i < 48; ++i)
+ {
+ a[i] = offset + 13 * i;
+ b[i] = 255 - 7 * i;
+ }
+
+ /* Loads start at A + 1 and B + 3. */
+ for (unsigned i = 0; i < 16; ++i)
+ expected[i] = offset + 247 + 6 * i;
+
+ uint8_t *c = (overlap == 0 ? out : overlap == 1 ? a : b) + offset;
+ if (live)
+ vst1_u8 (returned, foo_live (a + 1, b + 3, c));
+ else
+ foo (a + 1, b + 3, c, 0);
+
+ if (__builtin_memcmp (c, expected, 16)
+ || (live && __builtin_memcmp (returned, expected + 8, 8)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* Both pairs must combine at the normal -O3 cost setting. */
+/* Adv. SIMD load/store folding is currently little-endian only. */
+/* { dg-final { scan-tree-dump-times "basic block part vectorized using 16
byte vectors" 2 "slp2" { target aarch64_little_endian } } } */
+/* { dg-final { scan-assembler-times {\tadd\tv[0-9]+\.16b,} 2 { target
aarch64_little_endian } } } */
+/* { dg-final { scan-assembler-not {\tadd\tv[0-9]+\.8b,} { target
aarch64_little_endian } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-concat-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-concat-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..4f3ca5dbb0f0de8a2bdb65dbceb872ebb80f5d6e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-concat-1.c
@@ -0,0 +1,26 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=armv8-a -fno-vect-cost-model" } */
+
+typedef int V __attribute__ ((vector_size (8), aligned (4)));
+
+void
+foo (V *restrict out, const V *in)
+{
+ V value = *in;
+ out[0] = value;
+ out[1] = value;
+}
+
+void
+bar (V *restrict out, const V *a, const V *b)
+{
+ V first = *a;
+ V second = *b;
+ out[0] = first;
+ out[1] = second;
+}
+
+/* Combine the stores while keeping the source loads whole. */
+/* { dg-final { scan-assembler-times {\tldr\td[0-9]+,} 2 } } */
+/* { dg-final { scan-assembler-times {\tld1\t\{v[0-9]+\.d\}\[1\],} 1 } } */
+/* { dg-final { scan-assembler-times {\tstr\tq[0-9]+,} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..ef5ca7b7fe0127043a60d25e2be7e28a85592c16
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-1.c
@@ -0,0 +1,9 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fdump-tree-vect-details -save-temps" } */
+
+#include "vect-revectorize-group.h"
+
+/* { dg-final { scan-tree-dump "loop vectorized using 16 byte vectors" "vect"
{ target aarch64_little_endian } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" { target
aarch64_little_endian } } } */
+/* { dg-final { scan-assembler {add\tv[0-9]+\.16b, v[0-9]+\.16b, v[0-9]+\.16b}
{ target aarch64_little_endian } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..e11f300ee68aaa6a2be5e2fb5e56da86c360609e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-2.c
@@ -0,0 +1,8 @@
+/* { dg-do run { target aarch64_sve_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a+sve -fdump-tree-vect-details -save-temps"
} */
+
+#include "vect-revectorize-group.h"
+
+/* { dg-final { scan-tree-dump "loop vectorized using variable length vectors"
"vect" { target aarch64_little_endian } } } */
+/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" { target
aarch64_little_endian } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-live.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-live.c
new file mode 100644
index
0000000000000000000000000000000000000000..f86b958d8810ec539fe5612306e3b95b7fd89d99
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group-live.c
@@ -0,0 +1,11 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fno-vect-cost-model
-fdump-tree-vect-details" } */
+
+#define BYTES 8
+#include "../../gcc.dg/vect/vect-revectorize-group-live.c"
+
+/* The second 64-bit vector must be extracted from the 128-bit result,
+ without retaining a duplicate of its original computation. */
+/* { dg-final { scan-tree-dump "loop vectorized using 16 byte vectors" "vect"
} } */
+/* { dg-final { scan-tree-dump-times "extracting lane for live stmt" 1 "vect"
} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group.h
new file mode 100644
index
0000000000000000000000000000000000000000..f4a6ec6b7d4dfdd5077dd12b9e5abd595569492b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-group.h
@@ -0,0 +1,132 @@
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Preserve access order when the output overlaps either input. */
+__attribute__ ((noipa)) void
+add2 (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
+ {
+ uint8x8_t a0 = vld1_u8 (a), a1 = vld1_u8 (a + 8);
+ uint8x8_t b0 = vld1_u8 (b), b1 = vld1_u8 (b + 8);
+ vst1_u8 (c, a0 + b0);
+ vst1_u8 (c + 8, a1 + b1);
+ }
+}
+
+__attribute__ ((noipa)) void
+add4 (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ for (int i = 0; i < n; i += 32, a += 32, b += 32, c += 32)
+ {
+ uint8x8_t a0 = vld1_u8 (a), a1 = vld1_u8 (a + 8);
+ uint8x8_t b0 = vld1_u8 (b), b1 = vld1_u8 (b + 8);
+ vst1_u8 (c, a0 + b0);
+ vst1_u8 (c + 8, a1 + b1);
+ uint8x8_t a2 = vld1_u8 (a + 16), a3 = vld1_u8 (a + 24);
+ uint8x8_t b2 = vld1_u8 (b + 16), b3 = vld1_u8 (b + 24);
+ vst1_u8 (c + 16, a2 + b2);
+ vst1_u8 (c + 24, a3 + b3);
+ }
+}
+
+/* Distinct invariant lanes and reversed source-vector order. */
+__attribute__ ((noipa, optimize ("vect-cost-model=unlimited"))) void
+invariants (uint8_t *a, uint8_t *b, uint8_t *c, int n,
+ uint8x8_t k0, uint8x8_t k1)
+{
+ for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
+ {
+ uint8x8_t a0 = vld1_u8 (a), a1 = vld1_u8 (a + 8);
+ uint8x8_t b0 = vld1_u8 (b + 8), b1 = vld1_u8 (b);
+ vst1_u8 (c, (a0 + b0 + k0)
+ ^ (uint8x8_t) { 1, 3, 5, 7, 9, 11, 13, 15 });
+ vst1_u8 (c + 8, (a1 + b1 + k1)
+ ^ (uint8x8_t) { 2, 4, 6, 8, 10, 12, 14, 16 });
+ }
+}
+
+/* Discovery can split a group only between complete source vectors. */
+__attribute__ ((noipa, optimize ("vect-cost-model=unlimited"))) void
+different_ops (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ for (int i = 0; i < n; i += 16, a += 16, b += 16, c += 16)
+ {
+ uint8x8_t a0 = vld1_u8 (a), a1 = vld1_u8 (a + 8);
+ uint8x8_t b0 = vld1_u8 (b), b1 = vld1_u8 (b + 8);
+ vst1_u8 (c, a0 + b0);
+ vst1_u8 (c + 8, a1 ^ b1);
+ }
+}
+
+__attribute__ ((noipa, optimize ("no-tree-vectorize"))) void
+reference (uint8_t *a, uint8_t *b, uint8_t *c, int n, unsigned kind,
+ uint8x8_t k0, uint8x8_t k1)
+{
+ unsigned step = kind == 1 ? 32 : 16;
+ for (int i = 0; i < n; i += step, a += step, b += step, c += step)
+ for (unsigned base = 0; base < step; base += 16)
+ {
+ uint8_t tmp[16];
+ for (unsigned j = 0; j < 16; ++j)
+ {
+ unsigned bpos = kind == 2 ? j ^ 8 : j;
+ unsigned sum = a[base + j] + b[base + bpos];
+ if (kind == 2)
+ {
+ /* The invariant vectors use C indices, rather than ACLE
+ lane order. */
+ unsigned lane = j % 8;
+ if (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
+ lane = 7 - lane;
+
+ sum
+ = ((sum + (j < 8 ? k0[lane] : k1[lane])) & 255)
+ ^ (2 * lane + (j < 8 ? 1 : 2));
+ }
+
+ if (kind == 3 && j >= 8)
+ sum = a[j] ^ b[j];
+
+ tmp[j] = sum;
+ }
+
+ for (unsigned j = 0; j < 16; ++j)
+ c[base + j] = tmp[j];
+ }
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ static const unsigned offsets[]
+ = { 640, 32, 33, 40, 48, 31, 24, 320, 321, 328, 336, 319, 312 };
+ uint8x8_t k0 = { 0, 1, 20, 100, 160, 240, 254, 255 };
+ uint8x8_t k1 = { 255, 254, 240, 160, 100, 20, 1, 0 };
+ uint8_t actual[1024], expected[1024];
+ for (unsigned kind = 0; kind < 4; ++kind)
+ for (unsigned k = 0; k < sizeof (offsets) / sizeof (offsets[0]); ++k)
+ for (int n = -1; n <= 193; ++n)
+ {
+ for (unsigned j = 0; j < sizeof (actual); ++j)
+ actual[j] = expected[j] = 17 * j + 13 * n;
+ reference (expected + 32, expected + 320, expected + offsets[k],
+ n, kind, k0, k1);
+ uint8_t *a = actual + 32, *b = actual + 320;
+ uint8_t *c = actual + offsets[k];
+ switch (kind)
+ {
+ case 0: add2 (a, b, c, n); break;
+ case 1: add4 (a, b, c, n); break;
+ case 2: invariants (a, b, c, n, k0, k1); break;
+ case 3: different_ops (a, b, c, n); break;
+ }
+
+ if (__builtin_memcmp (actual, expected, sizeof (actual)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-predicate.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-predicate.c
new file mode 100644
index
0000000000000000000000000000000000000000..11a0877fdaaa1acd129ebd3ba15038756a87eb11
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-predicate.c
@@ -0,0 +1,36 @@
+/* A packed predicate has one bit per lane, rather than one byte. */
+/* { dg-do run { target aarch64_sve128_hw } } */
+/* { dg-require-effective-target aarch64_little_endian } */
+/* { dg-options "-O3 -march=armv8.2-a+sve -msve-vector-bits=128
-fno-vect-cost-model -fno-tree-loop-distribute-patterns" } */
+
+#include <arm_sve.h>
+
+typedef svbool_t P __attribute__ ((arm_sve_vector_bits (128)));
+
+__attribute__ ((noipa)) void
+copy (P *restrict out, const P *restrict in, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = in[i];
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ P in[16], out[16];
+ unsigned char guard[sizeof (out)];
+ __builtin_memset (guard, 0x5a, sizeof (guard));
+ __builtin_memset (in, 0xa5, sizeof (in));
+ for (unsigned n = 0; n < 16; ++n)
+ {
+ __builtin_memset (out, 0x5a, sizeof (out));
+ copy (out, in, n);
+ unsigned bytes = n * sizeof (P);
+ if (__builtin_memcmp (out, in, bytes)
+ || __builtin_memcmp ((char *) out + bytes, guard + bytes,
+ sizeof (out) - bytes))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-tuple-pack.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-tuple-pack.c
new file mode 100644
index
0000000000000000000000000000000000000000..3d52ac4320ca89399d497c25f933c3a0260e3c8e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-tuple-pack.c
@@ -0,0 +1,22 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fno-tree-vectorize" } */
+
+#include <arm_neon.h>
+
+uint8x16_t
+pack_u8 (const uint8_t *p)
+{
+ uint8x8x2_t value = vld2_u8 (p);
+ return vcombine_u8 (value.val[0], value.val[1]);
+}
+
+float32x4_t
+pack_f32 (const float *p)
+{
+ float32x2x2_t value = vld2_f32 (p);
+ return vcombine_f32 (value.val[0], value.val[1]);
+}
+
+/* Pack D-register tuples without a memory round-trip. */
+/* { dg-final { scan-assembler-not {\t(add|sub).*sp} } } */
+/* { dg-final { scan-assembler-not {\t(ld|st)[rp]} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..5730a9c0b84dca35b8200b9729f77825328d606c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-1.c
@@ -0,0 +1,58 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fdump-tree-vect-details -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Two iterations of the original 64-bit operation should become one
+ 128-bit operation, with runtime alias checks when needed. */
+__attribute__ ((noipa)) void
+add (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
+ {
+ uint8x8_t av = vld1_u8 (a);
+ uint8x8_t bv = vld1_u8 (b);
+ vst1_u8 (c, vadd_u8 (av, bv));
+ }
+}
+
+__attribute__ ((noipa, optimize ("no-tree-vectorize"))) void
+reference (uint8_t *a, uint8_t *b, uint8_t *c, int n)
+{
+ for (int i = 0; i < n; i += 8, a += 8, b += 8, c += 8)
+ {
+ uint8_t tmp[8];
+ for (unsigned j = 0; j < 8; ++j)
+ tmp[j] = a[j] + b[j];
+ for (unsigned j = 0; j < 8; ++j)
+ c[j] = tmp[j];
+ }
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ static const unsigned offsets[]
+ = { 384, 32, 33, 40, 48, 31, 24, 208, 209, 216, 224, 207, 200 };
+ uint8_t actual[768], expected[768];
+ for (unsigned k = 0; k < sizeof (offsets) / sizeof (offsets[0]); ++k)
+ for (int n = -1; n <= 145; ++n)
+ {
+ for (unsigned j = 0; j < sizeof (actual); ++j)
+ actual[j] = expected[j] = 17 * j + 13 * n;
+ reference (expected + 32, expected + 208, expected + offsets[k], n);
+ add (actual + 32, actual + 208, actual + offsets[k], n);
+ if (__builtin_memcmp (actual, expected, sizeof (actual)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* AdvSIMD load/store folding is currently little-endian only. */
+/* { dg-final { scan-tree-dump "loop vectorized using 16 byte vectors and
unroll factor 2" "vect" { target aarch64_little_endian } } } */
+/* { dg-final { scan-assembler {add\tv[0-9]+\.16b, v[0-9]+\.16b, v[0-9]+\.16b}
{ target aarch64_little_endian } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..4a54ed872bbd95302e2c6f614e207eaa7eb53dfa
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-2.c
@@ -0,0 +1,8 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -Wno-psabi -fdump-tree-vect-details
-fdump-tree-optimized -save-temps" } */
+
+#include "vect-revectorize-widen.h"
+
+/* { dg-final { scan-tree-dump-times "loop vectorized using 16 byte vectors
and unroll factor 2" 2 "vect" } } */
+/* { dg-final { scan-tree-dump "loop vectorized using 16 byte vectors and
unroll factor 4" "vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-3.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-3.c
new file mode 100644
index
0000000000000000000000000000000000000000..a5afe8cb890492c0a320adac8f46099db4e88702
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen-3.c
@@ -0,0 +1,7 @@
+/* { dg-do run { target aarch64_sve2_hw } } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8.5-a+sve2 -Wno-psabi -fdump-tree-vect-details
-fdump-tree-optimized -save-temps" } */
+
+#include "vect-revectorize-widen.h"
+
+/* { dg-final { scan-tree-dump "loop vectorized using variable length vectors"
"vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen.h
new file mode 100644
index
0000000000000000000000000000000000000000..1555489a86eb2ffa861bc5e094ab4ae280b7c01c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-widen.h
@@ -0,0 +1,98 @@
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+__attribute__ ((noipa)) void
+invariant (uint8x8_t *restrict out, const uint8x8_t *a, uint8x8_t bias,
+ unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = (a[i] + bias) ^ (uint8x8_t) { 1, 3, 5, 7, 9, 11, 13, 15 };
+}
+
+__attribute__ ((noipa)) uint8x8_t
+live (uint8x8_t *restrict out, const uint8x8_t *a, unsigned n)
+{
+ uint8x8_t r = { 0 };
+ for (unsigned i = 0; i < n; ++i)
+ {
+ r = a[i] + (uint8x8_t) { 1, 3, 5, 7, 9, 11, 13, 15 };
+ out[i] = r;
+ }
+
+ return r;
+}
+
+/* Split a live-out and its invariant operand. Scalar stores force unrolling
+ so each iteration needs its own pieces. */
+typedef unsigned V __attribute__ ((vector_size (32)));
+
+__attribute__ ((noipa, optimize ("vect-cost-model=unlimited"))) V
+split (V *restrict out, const V *a, V bias, unsigned *restrict scalar_out,
+ const unsigned *scalar_in, unsigned n)
+{
+ V r = bias;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ r = (a[i] + bias) ^ (V) { 1, 3, 5, 7, 9, 11, 13, 15 };
+ out[i] = r;
+ scalar_out[i] = scalar_in[i] + 7;
+ }
+
+ return r;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ uint8x8_t a[67], out[67], expected_invariant[67], expected_live[67];
+ V wide[67], wide_out[67], expected_split[67];
+ unsigned scalar_in[67], scalar_out[67], expected_scalar[67];
+ const uint8x8_t bias = { 0, 1, 20, 100, 160, 240, 254, 255 };
+ const uint8x8_t zero = { 0 };
+ const V wide_bias = { 3, 104, 205, 306, 407, 508, 609, 710 };
+
+ /* Fill the lane patterns and scalar references without vectorization. */
+ for (unsigned k = 0; k < 67 * 8; ++k)
+ {
+ unsigned i = k / 8, j = k % 8;
+ a[i][j] = 17 * i + 13 * j;
+ wide[i][j] = 101 * i + 17 * j;
+ expected_invariant[i][j]
+ = (uint8_t) (a[i][j] + bias[j]) ^ (2 * j + 1);
+ expected_live[i][j] = a[i][j] + 2 * j + 1;
+ expected_split[i][j]
+ = (101 * i + 17 * j + wide_bias[j]) ^ (2 * j + 1);
+ }
+
+ for (unsigned i = 0; i < 67; ++i)
+ {
+ scalar_in[i] = 19 * i;
+ expected_scalar[i] = 19 * i + 7;
+ }
+
+ for (unsigned n = 0; n <= 67; ++n)
+ {
+ invariant (out, a, bias, n);
+ if (__builtin_memcmp (out, expected_invariant, n * sizeof (out[0])))
+ __builtin_abort ();
+
+ uint8x8_t last = live (out, a, n);
+ if (__builtin_memcmp (out, expected_live, n * sizeof (out[0]))
+ || __builtin_memcmp (&last, n ? &expected_live[n - 1] : &zero,
+ sizeof (last)))
+ __builtin_abort ();
+
+ V wide_last = split (wide_out, wide, wide_bias, scalar_out, scalar_in,
n);
+ if (__builtin_memcmp (scalar_out, expected_scalar,
+ n * sizeof (scalar_out[0]))
+ || __builtin_memcmp (wide_out, expected_split, n * sizeof (V))
+ || __builtin_memcmp (&wide_last,
+ n ? &expected_split[n - 1] : &wide_bias,
+ sizeof (V)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-vrnd.c
b/gcc/testsuite/gcc.target/aarch64/vect-vrnd.c
index
b98c389a6a8002b30f53d5fb54aa44e9f8d71a4b..4511ed30d137220ecb869a1f12bc4a6dc729d17c
100644
--- a/gcc/testsuite/gcc.target/aarch64/vect-vrnd.c
+++ b/gcc/testsuite/gcc.target/aarch64/vect-vrnd.c
@@ -1,5 +1,8 @@
/* { dg-do run } */
-/* { dg-options "-O3 --save-temps" } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-tree-loop-vectorize --save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
#include <arm_neon.h>
@@ -105,6 +108,7 @@ BUILD_VARIANTS (x, rint)
int
main (int argc, char **argv)
{
+ check_vect ();
BUILD_VARIANTS ( , trunc)
BUILD_VARIANTS (a, round)
BUILD_VARIANTS (i, nearbyint)
diff --git a/gcc/testsuite/gcc.target/i386/pr52252-core.c
b/gcc/testsuite/gcc.target/i386/pr52252-core.c
index
897026b09975dfc1056a0b23db1c9161f1a43883..d35dd15dfc3ef45d11f2141c2c33523eddedba9c
100644
--- a/gcc/testsuite/gcc.target/i386/pr52252-core.c
+++ b/gcc/testsuite/gcc.target/i386/pr52252-core.c
@@ -25,4 +25,5 @@ matrix_mul (byte *in, byte *out, int size)
}
}
-/* { dg-final { scan-assembler "pshufb" } } */
+/* AVX512VBMI can use two-source byte permutes instead. */
+/* { dg-final { scan-assembler {pshufb|vperm[ti]2b} } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr85323-2.c
b/gcc/testsuite/gcc.target/i386/pr85323-2.c
index
9fa21f525227cec243521aec8d97accd242a4152..31f98cbb3f7d06c123262cc20997c421d90afb05
100644
--- a/gcc/testsuite/gcc.target/i386/pr85323-2.c
+++ b/gcc/testsuite/gcc.target/i386/pr85323-2.c
@@ -1,6 +1,7 @@
/* PR target/85323 */
/* { dg-do compile } */
-/* { dg-options "-O2 -mavx512vl -mavx512bw -fdump-tree-optimized" } */
+/* Keep the zero stores at their original widths to check intrinsic folding.
*/
+/* { dg-options "-O2 -mavx512vl -mavx512bw -fno-tree-vectorize
-fdump-tree-optimized" } */
/* { dg-final { scan-tree-dump-times "= \{ 0, 0 \};" 12 "optimized" } } */
/* { dg-final { scan-tree-dump-times "= \{ 0, 0, 0, 0 \};" 12 "optimized" } }
*/
/* { dg-final { scan-tree-dump-times "= \{ 0, 0, 0, 0, 0, 0, 0, 0 \};" 12
"optimized" } } */
diff --git a/gcc/testsuite/gcc.target/i386/pr88531-1b.c
b/gcc/testsuite/gcc.target/i386/pr88531-1b.c
index
e6df789de9007b9f6a4331e2560b6f0c4d2fa5f4..a81e7bad2a95a21fc29b126104de874b77cc451d
100644
--- a/gcc/testsuite/gcc.target/i386/pr88531-1b.c
+++ b/gcc/testsuite/gcc.target/i386/pr88531-1b.c
@@ -5,4 +5,5 @@
/* { dg-final { scan-assembler-times "vgatherdpd" 4 { target { ! lp64 } } } }
*/
/* { dg-final { scan-assembler-times "vgatherqpd" 4 { target lp64 } } } */
-/* { dg-final { scan-assembler-times "vmulpd" 4 } } */
+/* SLP combines the two 128-bit epilogue multiplies into one 256-bit multiply.
*/
+/* { dg-final { scan-assembler-times "vmulpd" 3 } } */
diff --git a/gcc/testsuite/gcc.target/i386/vect-strided-1.c
b/gcc/testsuite/gcc.target/i386/vect-strided-1.c
index
8e232540bd448e145799a56ef7d23bb5e7a6666a..76ab90d63a717381141e6dfa68ab3432da713fcc
100644
--- a/gcc/testsuite/gcc.target/i386/vect-strided-1.c
+++ b/gcc/testsuite/gcc.target/i386/vect-strided-1.c
@@ -1,5 +1,6 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -msse2 -mno-avx --param ix86-vect-compare-costs=0" } */
+/* Check the loop vectorizer's choice without subsequent BB revectorization.
*/
+/* { dg-options "-O2 -msse2 -mno-avx -fno-tree-slp-vectorize --param
ix86-vect-compare-costs=0" } */
void foo (int * __restrict a, int *b, int s)
{
diff --git a/gcc/testsuite/gcc.target/i386/vect-strided-1b.c
b/gcc/testsuite/gcc.target/i386/vect-strided-1b.c
index
6f173eaea5c7beddf2649aadce1ca3bcb1bff5fc..3e5e918142debbfe21ddae5d49236b37b20ef6be
100644
--- a/gcc/testsuite/gcc.target/i386/vect-strided-1b.c
+++ b/gcc/testsuite/gcc.target/i386/vect-strided-1b.c
@@ -1,5 +1,6 @@
/* { dg-do compile { target { ! ia32 } } } */
-/* { dg-options "-O2 -msse2 -mno-avx --param ix86-vect-compare-costs=1" } */
+/* Check the loop vectorizer's choice without subsequent BB revectorization.
*/
+/* { dg-options "-O2 -msse2 -mno-avx -fno-tree-slp-vectorize --param
ix86-vect-compare-costs=1" } */
void foo (int * __restrict a, int *b, int s)
{
diff --git a/gcc/testsuite/gcc.target/i386/vect-strided-2b.c
b/gcc/testsuite/gcc.target/i386/vect-strided-2b.c
index
9eabfbb082b7af8e0781551be26486ebc0bd5c79..d4acdccd455796641d39e91d71bf6d38e464d16f
100644
--- a/gcc/testsuite/gcc.target/i386/vect-strided-2b.c
+++ b/gcc/testsuite/gcc.target/i386/vect-strided-2b.c
@@ -1,5 +1,6 @@
/* { dg-do compile { target { ! ia32 } } } */
-/* { dg-options "-O2 -msse2 -mno-avx --param ix86-vect-compare-costs=1" } */
+/* Check the loop vectorizer's choice without subsequent BB revectorization.
*/
+/* { dg-options "-O2 -msse2 -mno-avx -fno-tree-slp-vectorize --param
ix86-vect-compare-costs=1" } */
void foo (int * __restrict a, int *b, int s)
{
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index
d7a4472bb22ddb5cd336217dc4834ea5411953cc..c343c59ba6078d62f47edafb9f317f2a743625df
100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -5422,6 +5422,18 @@ vect_analyze_data_refs (vec_info *vinfo, bool *fatal)
/* Set vectype for STMT. */
scalar_type = TREE_TYPE (DR_REF (dr));
tree vectype = get_vectype_for_scalar_type (vinfo, scalar_type);
+
+ /* Register aggregates need not have a linear layout. Leave their
+ vector references intact if revectorization would require an
+ incompatible register mode. */
+ if (vectype && VECTOR_TYPE_P (TREE_TYPE (DR_REF (dr)))
+ && base && DECL_P (base)
+ && !is_global_var (base)
+ && !TREE_ADDRESSABLE (base)
+ && DECL_MODE (base) != BLKmode
+ && !targetm.modes_tieable_p (DECL_MODE (base), TYPE_MODE (vectype)))
+ vectype = NULL_TREE;
+
if (!vectype)
{
if (dump_enabled_p ())
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index
a6e6d3ab4a4813c0a70946973d3a3ee938c1489c..6b70c34476c55386fbc2da01f3a99761bbfdf331
100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -490,7 +490,6 @@ vect_analyze_scalar_cycles_1 (loop_vec_info loop_vinfo,
class loop *loop)
tree type = TREE_TYPE (def);
unsigned HOST_WIDE_INT nelts;
if (TYPE_VECTOR_SUBPARTS (type).is_constant (&nelts)
- && vect_emulated_vector_p (type)
&& get_vectype_for_scalar_type (loop_vinfo, type,
nelts))
{
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index
7723bc1b681aee6fe80bc8aeb57a11c2a12ee081..109cac7142f94f22dd35546e21882dc84be87ed3
100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -3141,10 +3141,15 @@ out:
uniform_val = NULL_TREE;
break;
}
+
+ tree scalar_type = TREE_TYPE (op0);
+ if (VECTOR_TYPE_P (scalar_type))
+ scalar_type = TREE_TYPE (scalar_type);
+
if (!uniform_val
&& !can_duplicate_and_interleave_p (vinfo,
oprnd_info->ops.length (),
- TREE_TYPE (op0)))
+ scalar_type))
{
for (unsigned k = 0; k < j; ++k)
matches[k] = 0;
@@ -5311,11 +5316,32 @@ vect_analyze_slp_instance (vec_info *vinfo,
gcc_assert (kind == slp_inst_kind_store);
/* Collect the stores and store them in scalar_stmts. */
+ tree type = TREE_TYPE (DR_REF (STMT_VINFO_DATA_REF (stmt_info)));
+
+ /* Reprocessing native vector stores at their original width or a smaller
+ width does not combine any source vectors. Leave them for optimization;
+ BB SLP can still widen them or split an unsupported source vector. */
+ if (is_a <bb_vec_info> (vinfo) && vect_can_revectorize_type_p (type)
+ && !vect_emulated_vector_p (type))
+ {
+ tree vectype
+ = get_related_vectype_for_scalar_type (vinfo->vector_mode,
+ TREE_TYPE (type));
+ if (vectype
+ && known_le (TYPE_VECTOR_SUBPARTS (vectype),
+ TYPE_VECTOR_SUBPARTS (type)))
+ return false;
+ }
+
+ unsigned lanes_per_stmt
+ = vect_can_revectorize_type_p (type)
+ ? TYPE_VECTOR_SUBPARTS (type).to_constant () : 1;
scalar_stmts.create (DR_GROUP_SIZE (stmt_info));
stmt_vec_info next_info = stmt_info;
while (next_info)
{
- scalar_stmts.quick_push (vect_stmt_to_vectorize (next_info));
+ for (unsigned j = 0; j < lanes_per_stmt; ++j)
+ scalar_stmts.safe_push (vect_stmt_to_vectorize (next_info));
next_info = DR_GROUP_NEXT_ELEMENT (next_info);
}
@@ -5404,6 +5430,7 @@ vect_analyze_slp_instance (vec_info *vinfo,
/* For basic block SLP, try to break the group up into multiples of
a vector size. */
if (is_a <bb_vec_info> (vinfo)
+ && lanes_per_stmt == 1
&& (i > 1 && i < group_size))
{
/* Free the allocated memory. */
@@ -5453,10 +5480,10 @@ vect_analyze_slp_instance (vec_info *vinfo,
}
}
- /* For loop vectorization split the RHS into arbitrary pieces of
- size >= 1. */
+ /* For loop vectorization split the RHS at source-statement
+ boundaries. A vector statement contributes LANES_PER_STMT lanes. */
else if (is_a <loop_vec_info> (vinfo)
- && (group_size != 1 && i < group_size))
+ && (group_size != lanes_per_stmt && i < group_size))
{
gcall *call = dyn_cast <gcall *> (stmt_info->stmt);
bool masked_p = call
@@ -5478,15 +5505,15 @@ vect_analyze_slp_instance (vec_info *vinfo,
&& ! STMT_VINFO_STRIDED_P (stmt_info)
&& ! STMT_VINFO_SLP_VECT_ONLY (stmt_info)
&& compare_step_with_zero (vinfo, stmt_info) > 0
+ && lanes_per_stmt == 1
&& vect_slp_prefer_store_lanes_p (vinfo, stmt_info, NULL_TREE,
masked_p, group_size, i));
if (want_store_lanes || force_single_lane)
- i = 1;
+ i = lanes_per_stmt;
/* A fatal discovery fail doesn't always mean single-lane SLP
isn't a possibility, so try. */
- if (i == 0)
- i = 1;
+ i = MAX (lanes_per_stmt, i / lanes_per_stmt * lanes_per_stmt);
if (dump_enabled_p ())
dump_printf_loc (MSG_NOTE, vect_location,
@@ -5516,14 +5543,14 @@ vect_analyze_slp_instance (vec_info *vinfo,
rhs_common_nlanes = 0;
start = end;
if (want_store_lanes || force_single_lane)
- end = start + 1;
+ end = start + lanes_per_stmt;
else
end = group_size;
}
else
{
substmts.release ();
- if (end - start == 1)
+ if (end - start == lanes_per_stmt)
{
/* Single-lane discovery failed. Free resources. */
for (auto node : rhs_nodes)
@@ -5540,12 +5567,15 @@ vect_analyze_slp_instance (vec_info *vinfo,
later. As we know we arrived here with a group
larger than one try a group of size one! */
if (matches[0] != 0)
- end = start + 1;
+ end = start + lanes_per_stmt;
else
for (unsigned j = start; j < end; j++)
if (matches[j - start] != 0)
{
- end = j;
+ end
+ = start + MAX (lanes_per_stmt,
+ (j - start) / lanes_per_stmt
+ * lanes_per_stmt);
break;
}
}
@@ -6197,8 +6227,14 @@ vect_analyze_slp (vec_info *vinfo, unsigned
max_tree_size,
vec<stmt_vec_info> stmts;
vec<stmt_vec_info> roots = vNULL;
vec<tree> remain = vNULL;
- stmts.create (1);
- stmts.quick_push (stmt_info);
+ tree type = TREE_TYPE (DR_REF (dr));
+ unsigned HOST_WIDE_INT nelts = 1;
+ if (vect_can_revectorize_type_p (type))
+ nelts = TYPE_VECTOR_SUBPARTS (type).to_constant ();
+
+ stmts.create (nelts);
+ for (unsigned HOST_WIDE_INT j = 0; j < nelts; ++j)
+ stmts.quick_push (stmt_info);
if (! vect_build_slp_instance (vinfo, slp_inst_kind_store,
stmts, roots, remain, max_tree_size,
&limit, bst_map, force_single_lane))
@@ -11468,9 +11504,12 @@ vect_create_constant_vectors (vec_info *vinfo,
slp_tree op_node)
/* Create 'vect_ = {op0,op1,...,opn}'. */
tree orig_op = op;
unsigned source_lane = 0;
+ unsigned source_nelts = 0;
if (VECTOR_TYPE_P (TREE_TYPE (op)))
{
- unsigned nelts = TYPE_VECTOR_SUBPARTS (TREE_TYPE
(op)).to_constant ();
+ unsigned nelts
+ = TYPE_VECTOR_SUBPARTS (TREE_TYPE (op)).to_constant ();
+ source_nelts = nelts;
tree elt_type = TREE_TYPE (TREE_TYPE (op));
source_lane = i % nelts;
@@ -11574,6 +11613,26 @@ vect_create_constant_vectors (vec_info *vinfo,
slp_tree op_node)
bitsize_int (source_lane
* vector_element_bits
(TREE_TYPE (orig_op))));
+ else if (source_nelts
+ && type_nunits.is_constant ()
+ && (excess_elts == 0 || !voprnds.is_empty ())
+ && multiple_p (nunits, source_nelts)
+ && multiple_p (group_size, source_nelts)
+ && useless_type_conversion_p
+ (TREE_TYPE (vector_type),
+ TREE_TYPE (TREE_TYPE (orig_op))))
+ {
+ /* Concatenate whole source vectors. Extracting their scalar
+ lanes can split existing vector loads into scalar
+ loads. */
+ vec<constructor_elt, va_gc> *pieces = NULL;
+ for (unsigned k = 0; k < nunits; k += source_nelts)
+ CONSTRUCTOR_APPEND_ELT (pieces, NULL_TREE,
+ SLP_TREE_SCALAR_OPS (op_node)[(i + k) % group_size]);
+ vec_cst = make_ssa_name (vector_type);
+ gimple_seq_add_stmt (&ctor_seq, gimple_build_assign
+ (vec_cst, build_constructor (vector_type, pieces)));
+ }
else if (uniform_elt)
vec_cst = gimple_build_vector_from_val (&ctor_seq, vector_type,
elts[0]);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index
6f0201e9efd7c5909adf6cfcd0ab031beecd6497..6c99b29ae6fcce540ede8d440b608f6ff1a7e134
100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -13541,7 +13541,7 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree
scalar_type,
{
if (VECTOR_TYPE_P (scalar_type))
{
- if (!TYPE_VECTOR_SUBPARTS (scalar_type).is_constant ())
+ if (!vect_can_revectorize_type_p (scalar_type))
return NULL_TREE;
scalar_type = TREE_TYPE (scalar_type);
diff --git a/gcc/tree-vectorizer.cc b/gcc/tree-vectorizer.cc
index
a0b1e3e6ab613aad612f4ab7ac65b753ad194811..dea2c49b59ed1458062594824487dbb762639daa
100644
--- a/gcc/tree-vectorizer.cc
+++ b/gcc/tree-vectorizer.cc
@@ -97,14 +97,16 @@ auto_purge_vect_location::~auto_purge_vect_location ()
vect_location = dump_user_location_t ();
}
-/* Return true if TYPE is a fixed-length emulated source vector type. */
+/* Return true if TYPE is a fixed-length vector type whose lanes can be
+ treated as scalar elements. Packed predicate vectors have a different
+ memory layout from vectors of their scalar element type. */
bool
vect_can_revectorize_type_p (tree type)
{
return (VECTOR_TYPE_P (type)
- && TYPE_VECTOR_SUBPARTS (type).is_constant ()
- && vect_emulated_vector_p (type));
+ && !VECTOR_BOOLEAN_TYPE_P (type)
+ && TYPE_VECTOR_SUBPARTS (type).is_constant ());
}
/* Dump a cost entry according to args to F. */
