SRA replaces aggregate fields with independent values, but it does not split
a vector field into vectors of the target's native width. A local aggregate
can therefore become one emulated vector value while an array used to assemble
its halves remains in memory. The vectorizer then sees stores and reloads of
the same temporary on every iteration.
e.g.
typedef float V4 __attribute__((vector_size(16)));
typedef float V8 __attribute__((vector_size(32)));
struct object { V8 v; };
void swap_halves (struct object *p, unsigned n)
{
struct object x = *p;
for (unsigned i = 0; i < n; ++i)
{
V4 d[2], tmp;
__builtin_memcpy (d, &x.v, sizeof (d));
tmp = d[0];
d[0] = d[1];
d[1] = tmp;
__builtin_memcpy (&x.v, d, sizeof (d));
}
*p = x;
}
On Adv. SIMD, SRA replaces x with one 256-bit vector value, but leaves d as an
array. Before loop vectorization, the loop body still contains:
x = PHI <initial_x, next_x>;
low = BIT_FIELD_REF <x, 128, 0>;
high = BIT_FIELD_REF <x, 128, 128>;
d[0] = high;
d[1] = low;
next_x = MEM <V8> [&d];
The stores and reload all have address step zero because d is the same local
array on each iteration. They are not independent: the reload must see both
stores, and next_x feeds the next iteration. Previously, access analysis
rejected the zero-step stores. Treating the reload as an invariant value
would also be wrong, even though its address is invariant.
We allow these source-vector accesses at loop VF 1, so each transformed
iteration still performs one original iteration. The V8 statement contributes
eight SLP lanes, which Adv. SIMD processes as two V4 values. For dependent
accesses, the new analysis path requires the same innermost-loop header and
uses vect_preserves_scalar_order_p to check that grouping retains access order.
It keeps LOOP_VINFO_NO_DATA_DEPENDENCIES false; this is permission to split
one iteration, not to remove its dependencies or combine iterations.
get_load_store_type treats the pieces within each source vector as contiguous.
A 32-byte reload is split at offsets 0 and 16 from the same base, and
vect_create_data_ref_ptr keeps that base unchanged on the next iteration.
The stores to d and its reload now use matching native pieces, so later store
forwarding can replace the reloads with high and low. With -O3
-fno-vect-cost-model -march=armv8-a, the optimized loop carries two V4 PHIs:
low = PHI <initial_low, high>;
high = PHI <initial_high, low>;
The temporary array is gone, and each iteration swaps the two register values.
The same ordering rule also applies to user memory that can alias:
typedef unsigned U8 __attribute__((vector_size(32)));
void fixed_address (U8 *p, const U8 *q, unsigned n)
{
for (unsigned i = 0; i < n; ++i)
*p = *q + (U8) {1,2,3,4,5,6,7,8};
}
For p == q this is a recurrence: every iteration adds to the previous result.
Splitting keeps both loads before both stores:
q_lo = MEM <vector(4) unsigned int> [q];
q_hi = MEM <vector(4) unsigned int> [q + 16B];
r_lo = q_lo + {1,2,3,4};
r_hi = q_hi + {5,6,7,8};
MEM <vector(4) unsigned int> [p] = r_lo;
MEM <vector(4) unsigned int> [p + 16B] = r_hi;
The next iteration reloads q after those stores; the load is not hoisted just
because q has a fixed address. An invariant vector store limits VF to 1 even
when no pairwise dependence check imposes that limit. Nonlinear source-vector
accesses and groups whose order cannot be preserved remain unsupported.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Any feedback?
Assisted-by: Codex GPT 6.1 Sol
AI was used to generate test cases for various types and inputs.
Thanks,
Tamar
gcc/ChangeLog:
PR tree-optimization/119187
* tree-vect-data-refs.cc (vect_analyze_data_ref_dependence): Allow
ordered vector splitting at VF 1.
(vect_analyze_data_ref_dependences): Limit VF for invariant stores.
(vect_analyze_group_access_1): Reject nonlinear source-vector groups.
(vect_analyze_data_ref_access): Accept invariant vector references and
reject unsupported nonlinear accesses.
(vect_create_data_ref_ptr): Reuse invariant pointers when splitting.
* tree-vect-stmts.cc (get_load_store_type): Split invariant-address
vector accesses in place at VF 1.
gcc/testsuite/ChangeLog:
PR tree-optimization/119187
* gcc.dg/vect/vect-revectorize-storage.c: New test.
* gcc.dg/vect/vect-revectorize-storage-no-sra.c: New test.
* gcc.dg/vect/vect-revectorize-storage-slp.c: New test.
* gcc.dg/vect/vect-revectorize-storage-access.c: New test.
* gcc.dg/vect/vect-revectorize-storage-boundary.c: New test.
* gcc.dg/vect/vect-revectorize-storage-lanes.c: New test.
* gcc.target/aarch64/vect-revectorize-storage-1.c: New test.
* gcc.target/aarch64/vect-revectorize-storage-2.c: New test.
* gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run.c: New
test.
* gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run-o1.c: New
test.
* gcc.dg/vect/vect-revectorize-storage-access-2.c: New test.
* gcc.dg/vect/vect-revectorize-storage-access-3.c: New test.
* gcc.dg/vect/vect-revectorize-storage-fallback.c: New test.
* gcc.dg/vect/vect-revectorize-storage-live.c: New test.
* gcc.dg/vect/vect-revectorize-storage-order.c: New test.
* gcc.dg/vect/vect-revectorize-storage-reject.c: New test.
* gcc.target/aarch64/vect-revectorize-sra-unsupported-vector-1.c: Update
test.
* gcc.target/aarch64/vect-revectorize-nonlinear-2.c: New test.
* gcc.target/aarch64/vect-revectorize-nonlinear.c: New test.
* gcc.target/aarch64/vect-revectorize-nonlinear.h: New test support.
* gcc.target/aarch64/vect-revectorize-register-tuples-2.c: New test.
* gcc.target/aarch64/vect-revectorize-register-tuples-3.c: New test.
* gcc.target/aarch64/vect-revectorize-register-tuples.c: New test.
* gcc.target/aarch64/vect-revectorize-storage-order-1.c: New test.
* gcc.target/aarch64/vect-revectorize-storage-reject-slp.c: New test.
* gcc.target/aarch64/vect-revectorize-storage-reject.c: New test.
---
--
diff --git
a/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run-o1.c
b/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run-o1.c
new file mode 100644
index
0000000000000000000000000000000000000000..db2fcd5e3a68928bcb0f9e533b80f013b7b0c3bc
--- /dev/null
+++
b/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run-o1.c
@@ -0,0 +1,6 @@
+/* Check the same wide vector copies at -O1. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-options "-O1 -fno-tree-vectorize -Wno-psabi" } */
+
+#include "vect-revectorize-sra-unsupported-vector-run.c"
diff --git
a/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run.c
b/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run.c
new file mode 100644
index
0000000000000000000000000000000000000000..f991d88b2d55733ba5a6ee4fea09d32559477dfa
--- /dev/null
+++
b/gcc/testsuite/gcc.dg/tree-ssa/vect-revectorize-sra-unsupported-vector-run.c
@@ -0,0 +1,6 @@
+/* Check wide-vector copies with vectorization disabled. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-options "-O3 -fno-tree-vectorize -Wno-psabi" } */
+
+#include "../vect/vect-revectorize-storage.c"
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-2.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..a02fa37fd2fe1c13ee82c38e689b1dacb23a19aa
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-2.c
@@ -0,0 +1,85 @@
+/* Indexed and partial updates of local vector storage. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float V128 __attribute__ ((vector_size (16)));
+typedef float V256 __attribute__ ((vector_size (32)));
+typedef struct { V256 v; } object;
+
+__attribute__ ((noipa)) static void
+indexed (object *p, unsigned n, unsigned index)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V128 halves[2];
+ __builtin_memcpy (halves, &x.v, sizeof (halves));
+ halves[index] += (V128) { 1, 2, 3, 4 };
+ __builtin_memcpy (&x.v, halves, sizeof (halves));
+ }
+
+ *p = x;
+}
+
+__attribute__ ((noipa)) static void
+partial (object *p, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V128 low = __builtin_shufflevector (x.v, x.v, 0, 1, 2, 3);
+ low += (V128) { 1, 2, 3, 4 };
+ __builtin_memcpy (&x.v, &low, sizeof (low));
+ x.v[6] += 13;
+ }
+
+ *p = x;
+}
+
+static const V256 expected_low[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 2, 4, 6, 8, 5, 6, 7, 8 },
+ { 3, 6, 9, 12, 5, 6, 7, 8 },
+ { 4, 8, 12, 16, 5, 6, 7, 8 },
+ { 5, 10, 15, 20, 5, 6, 7, 8 },
+ { 6, 12, 18, 24, 5, 6, 7, 8 }
+};
+static const V256 expected_high[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 1, 2, 3, 4, 6, 8, 10, 12 },
+ { 1, 2, 3, 4, 7, 10, 13, 16 },
+ { 1, 2, 3, 4, 8, 12, 16, 20 },
+ { 1, 2, 3, 4, 9, 14, 19, 24 },
+ { 1, 2, 3, 4, 10, 16, 22, 28 }
+};
+static const V256 expected_part[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 2, 4, 6, 8, 5, 6, 20, 8 },
+ { 3, 6, 9, 12, 5, 6, 33, 8 },
+ { 4, 8, 12, 16, 5, 6, 46, 8 },
+ { 5, 10, 15, 20, 5, 6, 59, 8 },
+ { 6, 12, 18, 24, 5, 6, 72, 8 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 6; ++n)
+ {
+ object low = { { 1, 2, 3, 4, 5, 6, 7, 8 } };
+ object high = low, part = low;
+ indexed (&low, n, 0);
+ indexed (&high, n, 1);
+ partial (&part, n);
+ if (__builtin_memcmp (&low.v, &expected_low[n], sizeof (low.v))
+ || __builtin_memcmp (&high.v, &expected_high[n], sizeof (high.v))
+ || __builtin_memcmp (&part.v, &expected_part[n], sizeof (part.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-3.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-3.c
new file mode 100644
index
0000000000000000000000000000000000000000..4ebc7c0067258a30776c8ef06822dbedd7731371
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access-3.c
@@ -0,0 +1,97 @@
+/* Volatile, zero-initialized and scalar views of local vector storage. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float V128 __attribute__ ((vector_size (16)));
+typedef float V256 __attribute__ ((vector_size (32)));
+typedef struct { V256 v; } object;
+
+__attribute__ ((noipa)) static void
+volatiles (object *p, unsigned n)
+{
+ volatile object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V256 value = x.v;
+ V128 high = __builtin_shufflevector (value, value, 4, 5, 6, 7);
+ __builtin_memcpy (&value, &high, sizeof (high));
+ x.v = value;
+ }
+
+ *p = x;
+}
+
+__attribute__ ((noipa)) static void
+initialized (object *p, unsigned n)
+{
+ object x = { 0 };
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V128 high = __builtin_shufflevector (x.v, x.v, 4, 5, 6, 7);
+ high += (V128) { 1, 2, 3, 4 };
+ __builtin_memcpy (&x.v, &high, sizeof (high));
+ __builtin_memcpy ((char *) &x.v + sizeof (high), &high, sizeof (high));
+ }
+
+ *p = x;
+}
+
+#if defined (__BITINT_MAXWIDTH__) && __BITINT_MAXWIDTH__ >= 256
+/* A whole scalar view is not a whole vector or aggregate copy. */
+__attribute__ ((noipa)) static void
+scalar_view (object *p, unsigned n)
+{
+ union { V256 v; unsigned _BitInt (256) bits; } x;
+ x.v = p->v;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ unsigned _BitInt (256) bits = x.bits;
+ x.bits = (bits << 128) | (bits >> 128);
+ }
+
+ p->v = x.v;
+}
+
+#endif
+
+static const V256 expected_volatile[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 5, 6, 7, 8, 5, 6, 7, 8 }
+};
+#if defined (__BITINT_MAXWIDTH__) && __BITINT_MAXWIDTH__ >= 256
+static const V256 expected_scalar[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 5, 6, 7, 8, 1, 2, 3, 4 }
+};
+#endif
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 6; ++n)
+ {
+ object a = { { 1, 2, 3, 4, 5, 6, 7, 8 } }, b = a;
+ V256 expected = { n, 2 * n, 3 * n, 4 * n,
+ n, 2 * n, 3 * n, 4 * n };
+ volatiles (&a, n);
+ initialized (&b, n);
+ if (__builtin_memcmp (&a.v, &expected_volatile[n != 0], sizeof (a.v))
+ || __builtin_memcmp (&b.v, &expected, sizeof (b.v)))
+ __builtin_abort ();
+
+#if defined (__BITINT_MAXWIDTH__) && __BITINT_MAXWIDTH__ >= 256
+ object scalar = { { 1, 2, 3, 4, 5, 6, 7, 8 } };
+ scalar_view (&scalar, n);
+ if (__builtin_memcmp (&scalar.v, &expected_scalar[n % 2],
+ sizeof (scalar.v)))
+ __builtin_abort ();
+
+#endif
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access.c
new file mode 100644
index
0000000000000000000000000000000000000000..50a28af6cc7cf186aa61601a5f19c3f7276172d4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-access.c
@@ -0,0 +1,85 @@
+/* Escaping vector storage, including a PHI of its address. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float V128 __attribute__ ((vector_size (16)));
+typedef float V256 __attribute__ ((vector_size (32)));
+typedef struct { V256 v; } object;
+
+__attribute__ ((noipa)) static void
+change (object *p)
+{
+ p->v[0] += 17;
+}
+
+__attribute__ ((noipa)) static void
+escaped (object *p, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ change (&x);
+ V128 halves[2];
+ __builtin_memcpy (halves, &x.v, sizeof (halves));
+ V128 tmp = halves[0];
+ halves[0] = halves[1];
+ halves[1] = tmp;
+ __builtin_memcpy (&x.v, halves, sizeof (halves));
+ }
+
+ *p = x;
+}
+
+__attribute__ ((noipa)) static void
+phi_escaped (object *p, unsigned n, int which)
+{
+ object x = *p, y = *p;
+ object *selected = which ? &x : &y;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ change (selected);
+ V128 halves[2];
+ __builtin_memcpy (halves, &x.v, sizeof (halves));
+ V128 tmp = halves[0];
+ halves[0] = halves[1];
+ halves[1] = tmp;
+ __builtin_memcpy (&x.v, halves, sizeof (halves));
+ }
+
+ *p = x;
+}
+
+static const V256 expected[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 5, 6, 7, 8, 18, 2, 3, 4 },
+ { 18, 2, 3, 4, 22, 6, 7, 8 },
+ { 22, 6, 7, 8, 35, 2, 3, 4 },
+ { 35, 2, 3, 4, 39, 6, 7, 8 },
+ { 39, 6, 7, 8, 52, 2, 3, 4 }
+};
+static const V256 unchanged[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 5, 6, 7, 8, 1, 2, 3, 4 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 6; ++n)
+ {
+ object a = { { 1, 2, 3, 4, 5, 6, 7, 8 } }, b = a, c = a;
+ escaped (&a, n);
+ phi_escaped (&b, n, 1);
+ phi_escaped (&c, n, 0);
+ if (__builtin_memcmp (&a.v, &expected[n], sizeof (a.v))
+ || __builtin_memcmp (&b.v, &expected[n], sizeof (b.v))
+ || __builtin_memcmp (&c.v, &unchanged[n % 2], sizeof (c.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-boundary.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-boundary.c
new file mode 100644
index
0000000000000000000000000000000000000000..ee04abb9f62b4847832af8611eaff3f247355714
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-boundary.c
@@ -0,0 +1,43 @@
+/* Aggregate copies at a nonzero offset still refer to the complete value. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float V8 __attribute__ ((vector_size (32)));
+typedef struct { V8 v; } object;
+struct wrapper { object before, value, after; };
+
+__attribute__ ((noipa)) void
+process (struct wrapper *p, unsigned n)
+{
+ object x = p->value;
+ for (unsigned i = 0; i < n; ++i)
+ x.v += (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+ p->value = x;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ const struct wrapper initial = {
+ { { 101, 102, 103, 104, 105, 106, 107, 108 } },
+ { { 17, 18, 19, 20, 21, 22, 23, 24 } },
+ { { 201, 202, 203, 204, 205, 206, 207, 208 } }
+ };
+ for (unsigned n = 0; n < 9; ++n)
+ {
+ struct wrapper p = initial;
+ V8 expected = { 17 + n, 18 + 2 * n, 19 + 3 * n, 20 + 4 * n,
+ 21 + 5 * n, 22 + 6 * n, 23 + 7 * n, 24 + 8 * n };
+ process (&p, n);
+ if (__builtin_memcmp (&p.before, &initial.before, sizeof (p.before))
+ || __builtin_memcmp (&p.after, &initial.after, sizeof (p.after))
+ || __builtin_memcmp (&p.value.v, &expected, sizeof (p.value.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-fallback.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-fallback.c
new file mode 100644
index
0000000000000000000000000000000000000000..18fad673550b122f19a28c9e9d520ee6b544b44c
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-fallback.c
@@ -0,0 +1,59 @@
+/* A scalar consumer must still see each update to private vector storage. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-tree-sra -fno-vect-cost-model -Wno-psabi"
} */
+
+#include "tree-vect.h"
+
+typedef float V4 __attribute__ ((vector_size (16)));
+typedef float V8 __attribute__ ((vector_size (32)));
+typedef struct { V8 v; } object;
+
+__attribute__ ((noipa)) V4
+scalar_consumer (V4 x)
+{
+ return x + (V4) { 3, 4, 5, 6 };
+}
+
+__attribute__ ((noipa)) void
+process (object *p, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ x.v += (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+ V4 low;
+ __builtin_memcpy (&low, &x.v, sizeof (low));
+ low = scalar_consumer (low);
+ __builtin_memcpy (&x.v, &low, sizeof (low));
+ }
+
+ *p = x;
+}
+
+static const V8 expected[] = {
+ { 1, 4, 7, 10, 13, 16, 19, 22 },
+ { 5, 10, 15, 20, 18, 22, 26, 30 },
+ { 9, 16, 23, 30, 23, 28, 33, 38 },
+ { 13, 22, 31, 40, 28, 34, 40, 46 },
+ { 17, 28, 39, 50, 33, 40, 47, 54 },
+ { 21, 34, 47, 60, 38, 46, 54, 62 },
+ { 25, 40, 55, 70, 43, 52, 61, 70 },
+ { 29, 46, 63, 80, 48, 58, 68, 78 },
+ { 33, 52, 71, 90, 53, 64, 75, 86 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 9; ++n)
+ {
+ object x = { { 1, 4, 7, 10, 13, 16, 19, 22 } };
+ process (&x, n);
+ if (__builtin_memcmp (&x.v, &expected[n], sizeof (x.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-lanes.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-lanes.c
new file mode 100644
index
0000000000000000000000000000000000000000..f987308ec88757081357d59cf42d4c7594e83245
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-lanes.c
@@ -0,0 +1,67 @@
+/* Storage subaccesses need not have the target's native vector size. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float V2 __attribute__ ((vector_size (8)));
+typedef float V8 __attribute__ ((vector_size (32)));
+typedef struct { V8 v; } object;
+
+__attribute__ ((noipa)) void
+partial (object *p, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V2 pair;
+ __builtin_memcpy (&pair, (char *) &x.v + 8, sizeof (pair));
+ pair += (V2) { 1, 2 };
+ __builtin_memcpy ((char *) &x.v + 8, &pair, sizeof (pair));
+ }
+
+ *p = x;
+}
+
+__attribute__ ((noipa)) void
+reorder (object *p, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V2 pieces[4];
+ __builtin_memcpy (pieces, &x.v, sizeof (pieces));
+ V2 tmp = pieces[0];
+ pieces[0] = pieces[3];
+ pieces[3] = tmp;
+ __builtin_memcpy (&x.v, pieces, sizeof (pieces));
+ }
+
+ *p = x;
+}
+
+static const V8 expected_reorder[] = {
+ { 1, 4, 7, 10, 13, 16, 19, 22 },
+ { 19, 22, 7, 10, 13, 16, 1, 4 }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 9; ++n)
+ {
+ object a = { { 1, 4, 7, 10, 13, 16, 19, 22 } }, b = a;
+ V8 expected = a.v;
+ expected[2] += n;
+ expected[3] += 2 * n;
+ partial (&a, n);
+ reorder (&b, n);
+ if (__builtin_memcmp (&a.v, &expected, sizeof (a.v))
+ || __builtin_memcmp (&b.v, &expected_reorder[n % 2], sizeof (b.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-live.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-live.c
new file mode 100644
index
0000000000000000000000000000000000000000..c602782ffaf67318f873e09aff111ea16850118a
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-live.c
@@ -0,0 +1,41 @@
+/* Write back the last complete private value, including scalar epilogues. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V8 __attribute__ ((vector_size (32)));
+typedef struct { V8 v; } object;
+
+__attribute__ ((noipa)) void
+process (object *p, const object *a, unsigned n)
+{
+ object x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ x.v = a[i].v + (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+ *p = x;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ object a[40];
+ for (unsigned i = 0; i < 40; ++i)
+ a[i].v = (V8) { 0, 3, 6, 9, 12, 15, 18, 21 } + 13 * i;
+
+ for (unsigned n = 0; n < 40; ++n)
+ {
+ object x = { { 101, 102, 103, 104, 105, 106, 107, 108 } };
+ V8 want = x.v;
+ if (n)
+ want = (V8) { 1, 5, 9, 13, 17, 21, 25, 29 } + 13 * (n - 1);
+
+ process (&x, a, n);
+ if (__builtin_memcmp (&x.v, &want, sizeof (x.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-no-sra.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-no-sra.c
new file mode 100644
index
0000000000000000000000000000000000000000..ae695732650b454958b4b38585cb5dbab1f44eb5
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-no-sra.c
@@ -0,0 +1,5 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-tree-sra -fno-vect-cost-model -Wno-psabi"
} */
+
+#include "vect-revectorize-storage.c"
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-order.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-order.c
new file mode 100644
index
0000000000000000000000000000000000000000..86533a8136aa75e37e1b1149a4257a89a8d889b7
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-order.c
@@ -0,0 +1,144 @@
+/* Preserve access order when invariant vector addresses alias stores. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V4 __attribute__ ((vector_size (16)));
+typedef unsigned V8 __attribute__ ((vector_size (32)));
+typedef unsigned V8A __attribute__ ((vector_size (32), aligned (16)));
+
+__attribute__ ((noipa)) void
+reorder (V8 *p, unsigned n)
+{
+ V8 x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ V4 low = __builtin_shufflevector (x, x, 0, 1, 2, 3);
+ V4 high = __builtin_shufflevector (x, x, 4, 5, 6, 7);
+ V4 halves[2];
+ halves[0] = high + (V4) { 1, 2, 3, 4 };
+ halves[1] = low + (V4) { 5, 6, 7, 8 };
+ __builtin_memcpy (&x, halves, sizeof (x));
+ }
+
+ *p = x;
+}
+
+__attribute__ ((noipa)) void
+overlap (V8A *p, const V8A *q, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ *p = *q + (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+}
+
+__attribute__ ((noipa)) void
+last_store (V8 *p, const V8 *q, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ *p = q[i] + (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+}
+
+__attribute__ ((noipa)) void
+invariant_load (V8 *p, const V8 *q, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ p[i] = *q + (V8) { 1, 2, 3, 4, 5, 6, 7, 8 };
+}
+
+static const V8 expected_reorder[] = {
+ { 1, 14, 27, 40, 53, 66, 79, 92 },
+ { 54, 68, 82, 96, 6, 20, 34, 48 },
+ { 7, 22, 37, 52, 59, 74, 89, 104 },
+ { 60, 76, 92, 108, 12, 28, 44, 60 },
+ { 13, 30, 47, 64, 65, 82, 99, 116 },
+ { 66, 84, 102, 120, 18, 36, 54, 72 }
+};
+static const unsigned overlap_input[] = {
+ 3, 10, 17, 24, 31, 38, 45, 52, 59, 66, 73, 80, 87, 94, 101, 108, 115, 122,
+ 129, 136, 143, 150, 157, 164
+};
+
+/* After two iterations, a half-overlapping copy reaches a fixed value. */
+static const unsigned expected_partial[2][8] = {
+ { 4, 12, 20, 28, 9, 18, 27, 36 },
+ { 65, 74, 83, 92, 64, 72, 80, 88 }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ V8 input[6], expected_once[6], expected_twice[6];
+ for (unsigned i = 0; i < 6; ++i)
+ {
+ input[i] = (V8) { 0, 1, 2, 3, 4, 5, 6, 7 } + 17 * i;
+ expected_once[i] = (V8) { 1, 3, 5, 7, 9, 11, 13, 15 } + 17 * i;
+ expected_twice[i] = (V8) { 2, 5, 8, 11, 14, 17, 20, 23 } + 17 * i;
+ }
+
+ for (unsigned n = 0; n < 6; ++n)
+ {
+ V8 x = { 1, 14, 27, 40, 53, 66, 79, 92 };
+ reorder (&x, n);
+ if (__builtin_memcmp (&x, &expected_reorder[n], sizeof (x)))
+ __builtin_abort ();
+
+ for (unsigned offset = 0; offset <= 8; offset += 4)
+ for (unsigned reverse = 0; reverse < 2; ++reverse)
+ {
+ union { V8 v[3]; unsigned lanes[24]; } actual;
+ __builtin_memcpy (&actual, overlap_input, sizeof (actual));
+ unsigned src = reverse ? offset : 0;
+ unsigned dest = reverse ? 0 : offset;
+ unsigned expected[8];
+
+ if (n == 0)
+ __builtin_memcpy (expected, overlap_input + dest,
+ sizeof (expected));
+ else if (offset == 4 && n > 1)
+ __builtin_memcpy (expected, expected_partial[reverse],
+ sizeof (expected));
+ else
+ /* Disjoint copies change once; self-copies add on every trip. */
+ for (unsigned j = 0; j < 8; ++j)
+ expected[j] = overlap_input[src + j]
+ + (j + 1) * (offset == 0 ? n : 1);
+
+ overlap ((V8A *) (actual.lanes + dest),
+ (const V8A *) (actual.lanes + src), n);
+ if (__builtin_memcmp (actual.lanes + dest, expected, sizeof (V8))
+ || __builtin_memcmp (actual.lanes, overlap_input,
+ dest * sizeof (unsigned))
+ || __builtin_memcmp (actual.lanes + dest + 8,
+ overlap_input + dest + 8,
+ (16 - dest) * sizeof (unsigned)))
+ __builtin_abort ();
+ }
+
+ last_store (&x, input, n);
+ if (__builtin_memcmp (&x,
+ n ? &expected_once[n - 1] : &expected_reorder[0],
+ sizeof (x)))
+ __builtin_abort ();
+
+ for (unsigned k = 0; k < 6; ++k)
+ {
+ V8 values[6];
+ __builtin_memcpy (values, input, sizeof (values));
+ invariant_load (values, values + k, n);
+ for (unsigned i = 0; i < 6; ++i)
+ {
+ /* Overwriting Q at iteration K changes subsequent loads. */
+ const V8 *expected
+ = (i >= n ? &input[i]
+ : i > k ? &expected_twice[k] : &expected_once[k]);
+ if (__builtin_memcmp (&values[i], expected, sizeof (V8)))
+ __builtin_abort ();
+ }
+ }
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-reject.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-reject.c
new file mode 100644
index
0000000000000000000000000000000000000000..ad5b16c6c49c5e1d20872d66af31ecc1345e223f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-reject.c
@@ -0,0 +1,6 @@
+/* Rejecting vectorization must leave private storage intact. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-tree-sra -fno-vect-cost-model -Wno-psabi
-fdbg-cnt=vect_loop:0 -fdbg-cnt=vect_slp:0" } */
+
+#include "vect-revectorize-storage.c"
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-slp.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-slp.c
new file mode 100644
index
0000000000000000000000000000000000000000..d49de935410620a730cdc200cb479d0f01ef5318
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage-slp.c
@@ -0,0 +1,6 @@
+/* Exercise vector storage with only basic-block vectorization enabled. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-tree-loop-vectorize -fno-vect-cost-model
-Wno-psabi" } */
+
+#include "vect-revectorize-storage.c"
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage.c
new file mode 100644
index
0000000000000000000000000000000000000000..2b0ad4f0450bb0e5c90be19b8807c0bc45d96ba8
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-storage.c
@@ -0,0 +1,75 @@
+/* Wide vector PHIs and accesses to local vector storage. */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model -Wno-psabi" } */
+
+#include "tree-vect.h"
+
+typedef float __attribute__ ((vector_size (16))) V128;
+typedef float __attribute__ ((vector_size (32))) V256;
+
+typedef struct { V256 v; } TYPE;
+
+__attribute__ ((noipa)) void
+copy_loop (V256 *a, V256 *b, unsigned n)
+{
+ V256 x = *a;
+ V256 y = *b;
+ for (unsigned i = 0; i < n; ++i)
+ {
+ y = x;
+ x = x * 2.0f;
+ }
+
+ *a = x;
+ *b = y;
+}
+
+static inline V256
+reorder (V256 v)
+{
+ V128 d[2], temp;
+ __builtin_memcpy (d, &v, sizeof (v));
+ temp = d[0];
+ d[0] = d[1];
+ d[1] = temp;
+ __builtin_memcpy (&v, d, sizeof (v));
+ return v;
+}
+
+__attribute__ ((noipa)) void
+reorder_loop (TYPE *p, unsigned n)
+{
+ TYPE x = *p;
+ for (unsigned i = 0; i < n; ++i)
+ x.v = reorder (x.v);
+ *p = x;
+}
+
+static const V256 expected_p[] = {
+ { 1, 2, 3, 4, 5, 6, 7, 8 },
+ { 5, 6, 7, 8, 1, 2, 3, 4 }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 5; ++n)
+ {
+ V256 a = { 1, 2, 3, 4, 5, 6, 7, 8 };
+ V256 b = { 9, 10, 11, 12, 13, 14, 15, 16 };
+ TYPE p = { a };
+ float scale = 1U << n;
+ V256 expected_a = a * scale;
+ V256 expected_b = n ? a * (scale / 2) : b;
+ copy_loop (&a, &b, n);
+ reorder_loop (&p, n);
+ if (__builtin_memcmp (&a, &expected_a, sizeof (a))
+ || __builtin_memcmp (&b, &expected_b, sizeof (b))
+ || __builtin_memcmp (&p.v, &expected_p[n % 2], sizeof (p.v)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..fa5139d2d292a85455ccd7d4f40e60d5dcdf3c08
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear-2.c
@@ -0,0 +1,9 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-vect-cost-model -march=armv8-a
-fdump-tree-vect-details" } */
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+#include "vect-revectorize-nonlinear.h"
+
+/* { dg-final { scan-tree-dump "non-consecutive vector access" "vect" } } */
+/* { dg-final { scan-tree-dump-not "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.c
new file mode 100644
index
0000000000000000000000000000000000000000..48b099b7107c6e62128b045d7ff00eb11a59f4d4
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.c
@@ -0,0 +1,9 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -fno-vect-cost-model -march=armv8-a
-fdump-tree-vect-details" } */
+
+typedef unsigned V __attribute__ ((vector_size (8)));
+#include "vect-revectorize-nonlinear.h"
+
+/* { dg-final { scan-tree-dump "non-consecutive vector access" "vect" } } */
+/* { dg-final { scan-tree-dump-not "LOOP VECTORIZED" "vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.h
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.h
new file mode 100644
index
0000000000000000000000000000000000000000..1c019a40a3bc329da6c94281ec04b9831b32246f
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-nonlinear.h
@@ -0,0 +1,106 @@
+/* Non-contiguous accesses must be rejected for native and split vectors. */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+__attribute__ ((noipa)) void
+stride (V *restrict out, const V *restrict in, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = in[2 * i] + (V) { 1 };
+}
+
+__attribute__ ((noipa)) void
+reverse (V *restrict out, const V *restrict in, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = in[-(int) i] + (V) { 1 };
+}
+
+__attribute__ ((noipa)) void
+variable (V *restrict out, const V *restrict in, unsigned n, int step)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = in[(int) i * step] + (V) { 1 };
+}
+
+__attribute__ ((noipa)) void
+scatter (V *restrict out, const V *restrict in, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[2 * i] = in[i] + (V) { 1 };
+}
+
+__attribute__ ((noipa)) void
+gather (V *restrict out, const V *restrict in,
+ const unsigned *restrict indices, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = in[indices[i]] + (V) { 1 };
+}
+
+__attribute__ ((noipa)) void
+gaps (V *restrict out, const V *restrict in)
+{
+ out[0] = in[0] + (V) { 1 };
+ out[1] = in[2] + (V) { 1 };
+}
+
+static const unsigned indices[] = {
+ 7, 20, 33, 46, 59, 7, 20, 33, 46, 59, 7, 20, 33, 46, 59, 7, 20, 33, 46,
+ 59, 7, 20, 33, 46, 59, 7, 20, 33, 46, 59, 7, 20, 33
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ V in[65], expected[65], out[65], strided[33], reversed[33];
+ V forward[33], backward[33], gathered[33];
+ const V guard = { 99 };
+ const unsigned input_lanes[] = { 0, 3, 6, 9, 12, 15, 18, 21 };
+ const unsigned result_lanes[] = { 1, 3, 6, 9, 12, 15, 18, 21 };
+ V input_base, result_base;
+ __builtin_memcpy (&input_base, input_lanes, sizeof (V));
+ __builtin_memcpy (&result_base, result_lanes, sizeof (V));
+ for (unsigned i = 0; i < 65; ++i)
+ {
+ in[i] = input_base + 17 * i;
+ expected[i] = result_base + 17 * i;
+ }
+
+ for (unsigned n = 0; n <= 33; ++n)
+ {
+ stride (strided, in, n);
+ reverse (reversed, in + 64, n);
+ variable (forward, in, n, 2);
+ variable (backward, in + 64, n, -2);
+ gather (gathered, in, indices, n);
+ for (unsigned i = 0; i < n; ++i)
+ if (__builtin_memcmp (&strided[i], &expected[2 * i], sizeof (V))
+ || __builtin_memcmp (&reversed[i], &expected[64 - i], sizeof (V))
+ || __builtin_memcmp (&forward[i], &expected[2 * i], sizeof (V))
+ || __builtin_memcmp (&backward[i], &expected[64 - 2 * i], sizeof
(V))
+ || __builtin_memcmp (&gathered[i], &expected[indices[i]], sizeof
(V)))
+ __builtin_abort ();
+
+ for (unsigned i = 0; i < 65; ++i)
+ out[i] = guard;
+ scatter (out, in, n);
+ for (unsigned i = 0; i < 65; ++i)
+ {
+ const void *want
+ = (i % 2 == 0 && i / 2 < n
+ ? (const void *) &expected[i / 2]
+ : &guard);
+ if (__builtin_memcmp (&out[i], want, sizeof (V)))
+ __builtin_abort ();
+ }
+ }
+
+ gaps (out, in);
+ if (__builtin_memcmp (&out[0], &expected[0], sizeof (V))
+ || __builtin_memcmp (&out[1], &expected[2], sizeof (V)))
+ __builtin_abort ();
+
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..51d68a781e3e92269a1ecb7f8d6aef389f6424d9
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-2.c
@@ -0,0 +1,9 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-options "-O3 -fno-vect-cost-model -march=armv8-a -save-temps" } */
+
+#include "vect-revectorize-register-tuples.c"
+
+/* Disabling the cost model must not widen accesses across incompatible
+ register layouts in either the loop or the basic block. */
+/* { dg-final { scan-assembler-not {\tadd\tv[0-9]+\.4s,} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-3.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-3.c
new file mode 100644
index
0000000000000000000000000000000000000000..68929d0acbd03d20b92f8ee0dd174019058115b8
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples-3.c
@@ -0,0 +1,63 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Extra computation must not hide incompatible tuple register layouts. */
+__attribute__ ((noipa)) uint32x2x4_t
+mix_tuples (uint32x2x4_t a, uint32x2x4_t b)
+{
+ uint32x2x4_t result;
+ for (unsigned i = 0; i < 4; ++i)
+ {
+ uint32x2_t x = a.val[i], y = b.val[i];
+ x = x * y + 1;
+ y = x ^ (y >> 3);
+ x = (x + y) * (x - y);
+ y = (x >> 5) ^ (y << 3);
+ result.val[i] = (x + y) ^ (x * y);
+ }
+
+ return result;
+}
+
+static const uint32x2x4_t input_a[] = {
+ { { { 0U, 4294967295U }, { 1U, 4294967294U }, { 2U, 4294967293U }, { 3U,
+ 4294967292U } } },
+ { { { 1U, 4294967294U }, { 2U, 4294967293U }, { 3U, 4294967292U }, { 4U,
+ 4294967291U } } },
+ { { { 12U, 4294967283U }, { 13U, 4294967282U }, { 14U, 4294967281U }, { 15U,
+ 4294967280U } } }
+};
+static const uint32x2x4_t input_b[] = {
+ { { { 0U, 0U }, { 3U, 5U }, { 6U, 10U }, { 9U, 15U } } },
+ { { { 2U, 1U }, { 5U, 6U }, { 8U, 11U }, { 11U, 16U } } },
+ { { { 24U, 12U }, { 27U, 17U }, { 30U, 22U }, { 33U, 27U } } }
+};
+static const uint32x2x4_t expected[] = {
+ { { { 8U, 8U }, { 32U, 4294967224U }, { 104U, 805292437U }, { 1073728199U,
+ 1342120921U } } },
+ { { { 24U, 4294967288U }, { 88U, 4294967160U }, { 9219U, 1342146659U }, {
+ 31401U, 194161U } } },
+ { { { 4025193843U, 1341793177U }, { 3215063215U, 1206152802U }, {
+ 2949952275U, 3351896838U }, { 2265530307U, 2969541U } } }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 3; ++n)
+ {
+ uint32x2x4_t result = mix_tuples (input_a[n], input_b[n]);
+ if (__builtin_memcmp (&result, &expected[n], sizeof (result)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-assembler-not {\tmul\tv[0-9]+\.4s,} } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples.c
new file mode 100644
index
0000000000000000000000000000000000000000..0526d895af113b26744cba53356fee74b67b9761
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-register-tuples.c
@@ -0,0 +1,116 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-options "-O3" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+#include <arm_neon.h>
+
+/* Tuple components must retain their separate D-register layout. */
+__attribute__ ((noipa)) uint32x2x4_t
+add_tuples (uint32x2x4_t a, uint32x2x4_t b)
+{
+ uint32x2x4_t result;
+ for (unsigned i = 0; i < 4; ++i)
+ result.val[i] = a.val[i] + b.val[i];
+ return result;
+}
+
+__attribute__ ((noipa)) uint32x2x4_t
+add_permuted_tuples (uint32x2x4_t a, uint32x2x4_t b)
+{
+ uint32x2x4_t result;
+ result.val[0] = a.val[3] + b.val[0];
+ result.val[1] = a.val[1] + b.val[1];
+ result.val[2] = a.val[2] + b.val[2];
+ result.val[3] = a.val[0] + b.val[3];
+ return result;
+}
+
+__attribute__ ((noipa)) void
+load_lanes (float32x2x2_t *out, const float *a, const float *b)
+{
+ float32x2x2_t value = vld2_f32 (a);
+ *out = vld2_lane_f32 (b, value, 1);
+}
+
+__attribute__ ((noipa)) void
+multiply_lanes (float *out, const float *a, const float *b)
+{
+ float32x2x2_t av = vld2_f32 (a);
+ float32x2x2_t bv = vld2_f32 (b);
+ float32x2x2_t result;
+ result.val[0] = vmul_lane_f32 (av.val[0], bv.val[0], 0);
+ result.val[1] = vmul_lane_f32 (av.val[1], bv.val[0], 0);
+ vst2_f32 (out, result);
+}
+
+static const uint32x2x4_t input_a[] = {
+ { { { 0U, 4294967295U }, { 1U, 4294967294U }, { 2U, 4294967293U }, { 3U,
+ 4294967292U } } },
+ { { { 1U, 4294967294U }, { 2U, 4294967293U }, { 3U, 4294967292U }, { 4U,
+ 4294967291U } } },
+ { { { 12U, 4294967283U }, { 13U, 4294967282U }, { 14U, 4294967281U }, { 15U,
+ 4294967280U } } }
+};
+static const uint32x2x4_t input_b[] = {
+ { { { 0U, 0U }, { 3U, 5U }, { 6U, 10U }, { 9U, 15U } } },
+ { { { 2U, 1U }, { 5U, 6U }, { 8U, 11U }, { 11U, 16U } } },
+ { { { 24U, 12U }, { 27U, 17U }, { 30U, 22U }, { 33U, 27U } } }
+};
+static const uint32x2x4_t expected_sum[] = {
+ { { { 0U, 4294967295U }, { 4U, 3U }, { 8U, 7U }, { 12U, 11U } } },
+ { { { 3U, 4294967295U }, { 7U, 3U }, { 11U, 7U }, { 15U, 11U } } },
+ { { { 36U, 4294967295U }, { 40U, 3U }, { 44U, 7U }, { 48U, 11U } } }
+};
+static const uint32x2x4_t expected_permuted[] = {
+ { { { 3U, 4294967292U }, { 4U, 3U }, { 8U, 7U }, { 9U, 14U } } },
+ { { { 6U, 4294967292U }, { 7U, 3U }, { 11U, 7U }, { 12U, 14U } } },
+ { { { 39U, 4294967292U }, { 40U, 3U }, { 44U, 7U }, { 45U, 14U } } }
+};
+static const float input_x[][4] = {
+ { 0, -2, -4, -6 },
+ { 1, -1, -3, -5 },
+ { 12, 10, 8, 6 }
+};
+static const float input_y[][4] = {
+ { 1, 2, 3, 4 },
+ { 4, 5, 6, 7 },
+ { 37, 38, 39, 40 }
+};
+static const float expected_product[][4] = {
+ { 0, -2, -4, -6 },
+ { 4, -4, -12, -20 },
+ { 444, 370, 296, 222 }
+};
+
+__attribute__ ((optimize ("no-tree-vectorize"))) int
+main (void)
+{
+ check_vect ();
+ for (unsigned n = 0; n < 3; ++n)
+ {
+ uint32x2x4_t sum = add_tuples (input_a[n], input_b[n]);
+ if (__builtin_memcmp (&sum, &expected_sum[n], sizeof (sum)))
+ __builtin_abort ();
+
+ sum = add_permuted_tuples (input_a[n], input_b[n]);
+ if (__builtin_memcmp (&sum, &expected_permuted[n], sizeof (sum)))
+ __builtin_abort ();
+
+ float out[4];
+ float32x2x2_t loaded;
+ load_lanes (&loaded, input_x[n], input_y[n]);
+ if (vget_lane_f32 (loaded.val[0], 0) != input_x[n][0]
+ || vget_lane_f32 (loaded.val[1], 0) != input_x[n][1]
+ || vget_lane_f32 (loaded.val[0], 1) != input_y[n][0]
+ || vget_lane_f32 (loaded.val[1], 1) != input_y[n][1])
+ __builtin_abort ();
+
+ multiply_lanes (out, input_x[n], input_y[n]);
+ if (__builtin_memcmp (out, expected_product[n], sizeof (out)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-sra-unsupported-vector-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-sra-unsupported-vector-1.c
index
8369eaf68b622982fc1540e3f60fd8284b59d37f..a53324ce940a9d651c5c4e5a1cc7635acae816ae
100644
---
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-sra-unsupported-vector-1.c
+++
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-sra-unsupported-vector-1.c
@@ -19,4 +19,6 @@ foo (TYPE *a, TYPE *b, unsigned n)
*b = y;
}
-/* { dg-final { scan-assembler-times {\tmov\tv[0-9]+\.16b, v[0-9]+\.16b} 2 } }
*/
+/* Each iteration updates both 128-bit pieces. Exit copies can be optimized
+ independently of vectorization. */
+/* { dg-final { scan-assembler-times {\tfadd\tv[0-9]+\.4s, v[0-9]+\.4s,
v[0-9]+\.4s} 2 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..3ffc69394f6f1f37a00c1d4387a106a2b766a73c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-1.c
@@ -0,0 +1,31 @@
+/* Split vector accesses to a temporary left after SRA. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-vect-cost-model
-fdump-tree-vect-details" } */
+
+typedef float __attribute__ ((vector_size (16))) V128;
+typedef float __attribute__ ((vector_size (32))) V256;
+
+typedef struct { V256 v; } TYPE;
+
+inline V256
+reorder (V256 v)
+{
+ V128 d[2], temp;
+ __builtin_memcpy (&d, &v, sizeof (v));
+ temp = d[0];
+ d[0] = d[1];
+ d[1] = temp;
+ __builtin_memcpy (&v, &d, sizeof (v));
+ return v;
+}
+
+void
+process (TYPE *p, unsigned int n)
+{
+ TYPE x = *p;
+ for (unsigned int i = 0; i < n; i++)
+ x.v = reorder (x.v);
+ *p = x;
+}
+
+/* { dg-final { scan-tree-dump {vectorized 1 loops} "vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-2.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..647b819a61e843e2d8e59fb9ec67b8317166cf37
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-2.c
@@ -0,0 +1,7 @@
+/* Basic-block vectorization also splits the remaining vector accesses. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-tree-loop-vectorize
-fno-vect-cost-model -fdump-tree-slp-details" } */
+
+#include "vect-revectorize-storage-1.c"
+
+/* { dg-final { scan-tree-dump {basic block part vectorized} "slp1" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-order-1.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-order-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..be8452d05d30020c7b8960f96a11dbb42ca91426
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-order-1.c
@@ -0,0 +1,7 @@
+/* Split temporary and potentially aliasing vector accesses at VF 1. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-vect-cost-model -Wno-psabi
-fdump-tree-vect-details" } */
+
+#include "../../gcc.dg/vect/vect-revectorize-storage-order.c"
+
+/* { dg-final { scan-tree-dump-times {vectorized 1 loops} 4 "vect" } } */
diff --git
a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject-slp.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject-slp.c
new file mode 100644
index
0000000000000000000000000000000000000000..dfba5bdfcf7d771b4e945bfce1361ef887ecbf85
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject-slp.c
@@ -0,0 +1,7 @@
+/* An analyzed but rejected BB must retain its scalar storage. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -g -march=armv8-a -fno-tree-loop-vectorize
-fno-vect-cost-model -fdbg-cnt=vect_slp:0 -fdump-tree-slp-details" } */
+
+#include "vect-revectorize-storage-1.c"
+
+/* { dg-final { scan-tree-dump-not {basic block part vectorized} "slp1" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject.c
new file mode 100644
index
0000000000000000000000000000000000000000..0e565532e26aa5d73f7608759424d7f58a15fb11
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-storage-reject.c
@@ -0,0 +1,7 @@
+/* An analyzed but rejected loop must retain its scalar storage. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-tree-slp-vectorize
-fno-vect-cost-model -fdbg-cnt=vect_loop:0 -fdump-tree-vect-details" } */
+
+#include "vect-revectorize-storage-1.c"
+
+/* { dg-final { scan-tree-dump-not {LOOP VECTORIZED} "vect" } } */
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index
c343c59ba6078d62f47edafb9f317f2a743625df..2f6e450ad2e4bf571e874db382f42d6591444c96
100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -355,6 +355,24 @@ vect_analyze_data_ref_dependence (struct
data_dependence_relation *ddr,
DR_REF (dra), DR_REF (drb));
}
+ /* Splitting source vector accesses at VF 1 preserves dependencies within
+ and between iterations, provided that grouping preserves their order.
+ In particular, keep temporary stores and reloads left by SRA as memory
+ operations and let later passes forward the resulting vector pieces. */
+ if (!loop->inner
+ && gimple_bb (stmtinfo_a->stmt) == loop->header
+ && gimple_bb (stmtinfo_b->stmt) == loop->header
+ && vect_can_revectorize_type_p (TREE_TYPE (DR_REF (dra)))
+ && vect_can_revectorize_type_p (TREE_TYPE (DR_REF (drb)))
+ && DR_STEP (dra) && DR_STEP (drb)
+ && (integer_zerop (DR_STEP (dra)) || integer_zerop (DR_STEP (drb)))
+ && vect_preserves_scalar_order_p (dr_info_a, dr_info_b))
+ {
+ *max_vf = 1;
+ LOOP_VINFO_NO_DATA_DEPENDENCIES (loop_vinfo) = false;
+ return opt_result::success ();
+ }
+
/* Unknown data dependence. */
if (DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
{
@@ -821,6 +839,13 @@ vect_analyze_data_ref_dependences (loop_vec_info
loop_vinfo,
LOOP_VINFO_NO_DATA_DEPENDENCIES (loop_vinfo) = true;
+ /* A source vector store to an invariant address can be split, but cannot
+ combine iterations. This also covers a store with no other accesses. */
+ for (data_reference_p dr : LOOP_VINFO_DATAREFS (loop_vinfo))
+ if (DR_IS_WRITE (dr) && DR_STEP (dr) && integer_zerop (DR_STEP (dr))
+ && vect_can_revectorize_type_p (TREE_TYPE (DR_REF (dr))))
+ *max_vf = 1;
+
/* For epilogues we either have no aliases or alias versioning
was applied to original loop. Therefore we may just get max_vf
using VF of original loop. */
@@ -3247,6 +3272,10 @@ vect_analyze_group_access_1 (vec_info *vinfo,
dr_vec_info *dr_info)
if (groupsize == 0)
groupsize = count + gaps;
+ /* Source vector groups currently support only contiguous accesses. */
+ if (VECTOR_TYPE_P (scalar_type) && (gaps || groupsize != count))
+ return false;
+
/* This could be UINT_MAX but as we are generating code in a very
inefficient way we have to cap earlier. See PR78699 for example. */
if (groupsize > 4096)
@@ -3355,7 +3384,7 @@ vect_analyze_data_ref_access (vec_info *vinfo,
dr_vec_info *dr_info)
class loop *loop = NULL;
if (STMT_VINFO_GATHER_SCATTER_P (stmt_info))
- return true;
+ return !VECTOR_TYPE_P (scalar_type);
if (loop_vinfo)
loop = LOOP_VINFO_LOOP (loop_vinfo);
@@ -3368,13 +3397,15 @@ vect_analyze_data_ref_access (vec_info *vinfo,
dr_vec_info *dr_info)
return false;
}
- /* Allow loads with zero step in inner-loop vectorization. */
+ /* Source vector references with zero step can be split at VF 1.
+ Dependence analysis limits VF for stores and dependent loads. */
if (loop_vinfo && integer_zerop (step))
{
DR_GROUP_FIRST_ELEMENT (stmt_info) = NULL;
DR_GROUP_NEXT_ELEMENT (stmt_info) = NULL;
if (!nested_in_vect_loop_p (loop, stmt_info))
- return DR_IS_READ (dr);
+ return DR_IS_READ (dr) || vect_can_revectorize_type_p (scalar_type);
+
/* Allow references with zero step for outer loops marked
with pragma omp simd only - it guarantees absence of
loop-carried dependencies between inner loop iterations. */
@@ -3424,7 +3455,7 @@ vect_analyze_data_ref_access (vec_info *vinfo,
dr_vec_info *dr_info)
{
HOST_WIDE_INT dr_step = TREE_INT_CST_LOW (step);
if (!tree_int_cst_compare (step, TYPE_SIZE_UNIT (scalar_type))
- || (dr_step < 0
+ || (!VECTOR_TYPE_P (scalar_type) && dr_step < 0
&& !compare_tree_int (TYPE_SIZE_UNIT (scalar_type), -dr_step)))
{
/* Mark that it is not interleaving. */
@@ -3434,6 +3465,21 @@ vect_analyze_data_ref_access (vec_info *vinfo,
dr_vec_info *dr_info)
}
}
+ /* Revectorization currently supports only forward contiguous or invariant
+ source vector references. A complete group can also be contiguous even
+ when its individual references have a larger step. */
+ if (VECTOR_TYPE_P (scalar_type) && loop_vinfo
+ && (!STMT_VINFO_GROUPED_ACCESS (stmt_info)
+ || TREE_CODE (step) != INTEGER_CST
+ || tree_int_cst_sgn (step) < 0))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "non-consecutive vector access: %G", stmt_info->stmt);
+
+ return false;
+ }
+
if (loop && nested_in_vect_loop_p (loop, stmt_info))
{
if (dump_enabled_p ())
@@ -5940,8 +5986,10 @@ vect_create_data_ref_ptr (vec_info *vinfo, stmt_vec_info
stmt_info,
This is needed when ONLY_INIT is false, and also when AT_LOOP is the
inner-loop nested in LOOP (during outer-loop vectorization). */
- /* No update in loop is required. */
- if (only_init && (!loop_vinfo || at_loop == loop))
+ /* No update in loop is required, including when splitting a source
+ vector access to an invariant address. */
+ if ((only_init && (!loop_vinfo || at_loop == loop))
+ || (!nested_in_vect_loop && iv_step && integer_zerop (iv_step)))
aptr = aggr_ptr_init;
else
{
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index
6c99b29ae6fcce540ede8d440b608f6ff1a7e134..bfb884203aa99f5c25487c853b22f7240c96b304
100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -2235,8 +2235,17 @@ get_load_store_type (vec_info *vinfo, stmt_vec_info
stmt_info,
}
else if (cmp == 0 && loop_vinfo)
{
- gcc_assert (vls_type == VLS_LOAD);
- *memory_access_type = VMAT_INVARIANT;
+ /* At VF 1, access the lanes of a source vector in place. Its
+ address is invariant, but its value can change within the loop. */
+ if (known_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1U)
+ && vect_can_revectorize_type_p
+ (TREE_TYPE (DR_REF (first_dr_info->dr))))
+ *memory_access_type = VMAT_CONTIGUOUS;
+ else
+ {
+ gcc_assert (vls_type == VLS_LOAD);
+ *memory_access_type = VMAT_INVARIANT;
+ }
}
/* Try using LOAD/STORE_LANES. */
else if (slp_node->ldst_lanes
