BB SLP needs a root from which to discover an emulated-vector. A
single store of an unsupported vector can become several native stores, but it
does not provide the grouped scalar-store root that SLP expects.
e.g.
typedef unsigned V8 __attribute__((vector_size(32)));
void bb_store (V8 *out, const V8 *a, const V8 *b)
{
*out = (*a + (V8) {1,3,5,7,9,11,13,15}) ^ *b;
}
We make the standalone store a root with one SLP lane per source element.
Discovery can then follow the XOR and addition using the same lane layout.
The scheduler emits two native pieces and removes the original store once. It
also handles stores to vector declarations and preserves the store layout when
choosing permutations.
The profitability test compares the proposed SLP code with the code GCC
would generate without SLP. The cost model calls the latter the scalar cost,
but that code can already contain vector instructions. In this example each
V8 addition becomes two four-element Adv. SIMD additions. We therefore count
two operations for the original addition. vect_scalar_stmt_count supplies this
count to the loop and BB cost models by finding how many supported vectors
make up the source vector. If there is no supported vector type, it counts one
operation per scalar element. Emulated permutations use a per-element estimate,
since their existing lowering extracts and reinserts the elements.
Constructors introduce another costing issue. For example:
void shuffle (V8 *x)
{
*x = (V8) {(*x)[4], (*x)[0], (*x)[5], (*x)[1],
(*x)[6], (*x)[2], (*x)[7], (*x)[3]};
}
SLP builds one instance for the constructor and another for its store. The
store uses the constructed vector as an external operand, so the old walk of
internal SLP nodes misses their connection. We extend that walk to follow
external operands whose definitions are replaced by SLP and cost the producer
and consumer together. Otherwise each instance can count the constructor's
inputs in its own estimate of the original code. Ordinary external operands
still keep their scalar definitions and do not create this connection.
We cost constructor roots first, then walk the scalar statements feeding the
SLP trees. Each root is marked as costed when its charge is recorded, so a walk
from a consuming store cannot charge it again. For shuffle the dump shows one
costed subgraph, with a vector cost of 8 and an original-code cost of 11. When
joining instance groups, we use both current leaders so that repeated joins
cannot create a cycle.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
Any feedback?
Thanks,
Tamar
gcc/ChangeLog:
PR tree-optimization/119187
* tree-vect-data-refs.cc (vect_slp_analyze_instance_dependence): Handle
non-grouped vector stores.
* tree-vect-slp.cc (vect_build_slp_instance): Allow vector-store roots.
(vect_analyze_slp): Handle roots without scalar root statements.
(vect_optimize_slp_pass::start_choosing_layouts): Preserve store layout.
(vect_slp_check_for_roots): Discover standalone emulated vector stores.
(vect_schedule_slp): Remove repeated vector stores once.
(vect_bb_slp_scalar_cost): Count emulated-vector work.
* tree-vect-stmts.cc (vectorizable_store): Accept direct declaration
stores.
* tree-ssa-strlen.cc (strlen_pass::count_nonzero_bytes): Handle
BIT_FIELD_REF values through the result-type fallback.
* tree-vectorizer.h (vect_scalar_stmt_count): Declare.
* tree-vect-loop.cc (vect_scalar_stmt_count): New. Count scalar
elements when no native vector type is available.
(vect_compute_single_scalar_iteration_cost): Count emulated-vector work.
* tree-vect-slp.cc (vect_bb_slp_mark_stmts_vectorized): Handle roots
without scalar root statements.
(vect_map_to_instance): Merge ultimate leaders.
(vect_bb_partition_graph_r): Follow external operands whose defining
statements are replaced by SLP.
(vect_bb_partition_graph): Include constructor roots in the statement
map.
(vect_bb_vectorization_profitable_p): Cost roots before traversing
their scalar inputs.
gcc/testsuite/ChangeLog:
PR tree-optimization/119187
* gcc.dg/vect/vect-revectorize-bb-store.c: New test.
* gcc.dg/vect/vect-revectorize-wide-array.c: New test.
* gcc.dg/vect/vect-revectorize-partition-cycle.c: New test.
* gcc.target/aarch64/vect-revectorize-bb-load.c: New test.
* g++.target/aarch64/vect-revectorize-ctor.C: New test.
* gcc.dg/Wstringop-overflow-47.c: Accept split vector stores.
* gcc.target/aarch64/vect-revectorize-cost.c: New test.
---
--
diff --git a/gcc/testsuite/g++.target/aarch64/vect-revectorize-ctor.C
b/gcc/testsuite/g++.target/aarch64/vect-revectorize-ctor.C
new file mode 100644
index
0000000000000000000000000000000000000000..3cb3143e385017b04abb7b23e343be33bf4d7910
--- /dev/null
+++ b/gcc/testsuite/g++.target/aarch64/vect-revectorize-ctor.C
@@ -0,0 +1,54 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-options "-O3 -march=armv8-a -fdump-tree-slp-details -save-temps" } */
+
+#include "../../gcc.dg/vect/tree-vect.h"
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+
+__attribute__ ((noipa)) void
+shuffle (V &x)
+{
+ x = (V) { x[4], x[0], x[5], x[1], x[6], x[2], x[7], x[3] };
+}
+
+__attribute__ ((noipa)) unsigned
+shuffle_live (V &x)
+{
+ unsigned lane = x[4];
+ x = (V) { lane, x[0], x[5], x[1], x[6], x[2], x[7], x[3] };
+ return lane;
+}
+
+__attribute__ ((noipa)) void
+shuffle_copy (V &x, V &y)
+{
+ V result = (V) { x[4], x[0], x[5], x[1], x[6], x[2], x[7], x[3] };
+ x = result;
+ y = result;
+}
+
+__attribute__ ((optimize ("no-tree-vectorize")))
+int
+main ()
+{
+ check_vect ();
+ V x = { 0, 101, 202, 303, 404, 505, 606, 707 };
+ V y = x, z = x, copy;
+ const V expected = { 404, 0, 505, 101, 606, 202, 707, 303 };
+ shuffle (x);
+ unsigned lane = shuffle_live (y);
+ shuffle_copy (z, copy);
+ if (lane != 404
+ || __builtin_memcmp (&x, &expected, sizeof (x))
+ || __builtin_memcmp (&y, &expected, sizeof (y))
+ || __builtin_memcmp (&z, &expected, sizeof (z))
+ || __builtin_memcmp (©, &expected, sizeof (copy)))
+ __builtin_abort ();
+ return 0;
+}
+
+/* Keep the cost model enabled to catch unnecessary scalar extractions
+ when a vectorized constructor is an external operand of a store. */
+/* { dg-final { scan-assembler-times {zip1\tv[0-9]+\.4s} 3 } } */
+/* { dg-final { scan-assembler-times {zip2\tv[0-9]+\.4s} 3 } } */
diff --git a/gcc/testsuite/gcc.dg/Wstringop-overflow-47.c
b/gcc/testsuite/gcc.dg/Wstringop-overflow-47.c
index
2e0ab6e63873ebe22bf380c952f3a73b918f87af..eedb75fe5384c5d5f6234c551aa1b4ddba661a36
100644
--- a/gcc/testsuite/gcc.dg/Wstringop-overflow-47.c
+++ b/gcc/testsuite/gcc.dg/Wstringop-overflow-47.c
@@ -60,20 +60,19 @@ void nowarn_i16_64 (int16_t i)
void warn_i16_64 (int16_t i)
{
-/* The IL below that's visible to the warning changes from one target to
- another. On some like aarch64 it's a single vector store, on others
- like x86_64 it's a series of BIT_FIELD_REFs. The overflow by
- the former is detected but the latter is not yet. */
+/* The stores below can be split into narrower stores. Match the final
+ overflowing store as well as the original vector store. Some targets
+ still fail to diagnose these stores (PR 97027). */
- extern char warn_a64[64]; // { dg-message "at offset (1|128) into
destination object 'warn_a64' of size (63|64)" "pr97027 note" { xfail { ! {
aarch64-*-* || { amdgcn-*-* || { arm*-*-* && { arm_neon || { arm_mve ||
arm_thumb1 } } } } } } } }
+ extern char warn_a64[64]; // { dg-message "at offset (1|33|49|128) into
destination object 'warn_a64' of size (63|64)" "pr97027 note" { xfail { ! {
aarch64-*-* || { amdgcn-*-* || { arm*-*-* && { arm_neon || { arm_mve ||
arm_thumb1 } } } } } } } }
void *p = warn_a64 + 1;
I16_64 *q = (I16_64*)p;
- *q = (I16_64){ i }; // { dg-warning "writing (1 byte|64 bytes) into
a region of size (0|63)" "pr97027" { xfail { ! { aarch64-*-* || { amdgcn-*-* ||
{ arm*-*-* && { arm_neon || { arm_mve || arm_thumb1 } } } } } } } }
+ *q = (I16_64){ i }; // { dg-warning "writing (1 byte|16 bytes|32
bytes|64 bytes) into a region of size (0|15|31|63)" "pr97027" { xfail { ! {
aarch64-*-* || { amdgcn-*-* || { arm*-*-* && { arm_neon || { arm_mve ||
arm_thumb1 } } } } } } } }
char a64[64];
p = a64 + 1;
q = (I16_64*)p;
- *q = (I16_64){ i }; // { dg-warning "writing (1 byte|64 bytes) into
a region of size (0|63)" "pr97027" { xfail { ! { aarch64-*-* || { amdgcn-*-* ||
{ arm*-*-* && { arm_neon || { arm_mve || arm_thumb1 } } } } } } } }
+ *q = (I16_64){ i }; // { dg-warning "writing (1 byte|16 bytes|32
bytes|64 bytes) into a region of size (0|15|31|63)" "pr97027" { xfail { ! {
aarch64-*-* || { amdgcn-*-* || { arm*-*-* && { arm_neon || { arm_mve ||
arm_thumb1 } } } } } } } }
sink (p);
}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-store.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-store.c
new file mode 100644
index
0000000000000000000000000000000000000000..3061693b307de8302376a3407d0ee5ffa8d9a020
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-bb-store.c
@@ -0,0 +1,28 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+
+__attribute__ ((noipa)) void
+store (V *out, const V *a, const V *b)
+{
+ *out = (*a + (V) { 1, 3, 5, 7, 9, 11, 13, 15 }) ^ *b;
+}
+
+int
+main (void)
+{
+ check_vect ();
+ V a = { 0, 19, 38, 57, 76, 95, 114, 133 };
+ V b = { 100, 93, 86, 79, 72, 65, 58, 51 };
+ V expected = { 101, 75, 125, 15, 29, 43, 69, 167 };
+ V out;
+ store (&out, &a, &b);
+ if (__builtin_memcmp (&out, &expected, sizeof (out)))
+ __builtin_abort ();
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-partition-cycle.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-partition-cycle.c
new file mode 100644
index
0000000000000000000000000000000000000000..29ed5e4b286fc896d004af208ec95f36754d84b9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-partition-cycle.c
@@ -0,0 +1,91 @@
+/* { dg-do compile } */
+/* { dg-additional-options "-O2" } */
+
+/* Partition merging must not create cycles. From libgfortran arandom_r16. */
+
+typedef __UINT64_TYPE__ u64;
+typedef long index_type;
+
+struct state
+{
+ int init;
+ u64 s[4];
+};
+
+extern struct state *get_state (void);
+extern void init_state (struct state *);
+
+struct dim
+{
+ index_type stride, lower, upper;
+};
+
+struct array
+{
+ long double *base;
+ index_type rank;
+ struct dim dim[15];
+};
+
+static inline u64
+next (struct state *rs)
+{
+ u64 *s = rs->s;
+ u64 result = s[1] * 5;
+ result = ((result << 7) | (result >> 57)) * 9;
+ u64 t = s[1] << 17;
+ s[2] ^= s[0];
+ s[3] ^= s[1];
+ s[1] ^= s[2];
+ s[0] ^= s[3];
+ s[2] ^= t;
+ s[3] = (s[3] << 45) | (s[3] >> 19);
+ return result;
+}
+
+void
+arandom (struct array *x)
+{
+ index_type count[15], extent[15], stride[15];
+ struct state *rs = get_state ();
+ long double *dest = x->base;
+ index_type dim = x->rank;
+ for (index_type n = 0; n < dim; ++n)
+ {
+ count[n] = 0;
+ stride[n] = x->dim[n].stride;
+ extent[n] = x->dim[n].upper - x->dim[n].lower + 1;
+ if (extent[n] <= 0)
+ return;
+ }
+
+ index_type stride0 = stride[0];
+ if (__builtin_expect (!rs->init, 0))
+ init_state (rs);
+
+ while (dest)
+ {
+ u64 a = next (rs), b = next (rs);
+ b &= ~(u64) 0 << 15;
+ *dest = (long double) a * 0x1.p-64L + (long double) b * 0x1.p-128L;
+ dest += stride0;
+ count[0]++;
+ index_type n = 0;
+ while (count[n] == extent[n])
+ {
+ count[n] = 0;
+ dest -= stride[n] * extent[n];
+ n++;
+ if (n == dim)
+ {
+ dest = 0;
+ break;
+ }
+ else
+ {
+ count[n]++;
+ dest += stride[n];
+ }
+ }
+ }
+}
diff --git a/gcc/testsuite/gcc.dg/vect/vect-revectorize-wide-array.c
b/gcc/testsuite/gcc.dg/vect/vect-revectorize-wide-array.c
new file mode 100644
index
0000000000000000000000000000000000000000..d5616c588cf250f6b25f68f063911d965befb8bc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-revectorize-wide-array.c
@@ -0,0 +1,87 @@
+/* { dg-do run } */
+/* { dg-require-effective-target vect_int } */
+/* { dg-additional-options "-O3 -fno-vect-cost-model" } */
+
+#include "tree-vect.h"
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+__attribute__ ((noipa)) void
+f (V *restrict out, const V *restrict a, const V *restrict b, unsigned n)
+{
+ for (unsigned i = 0; i < n; ++i)
+ out[i] = (a[i] + b[i]) * (V) { 1, 2, 3, 4, 5, 6, 7, 8 };
+}
+
+static const V a[] = {
+ { 0, 1, 2, 3, 4, 5, 6, 7 },
+ { 17, 18, 19, 20, 21, 22, 23, 24 },
+ { 34, 35, 36, 37, 38, 39, 40, 41 },
+ { 51, 52, 53, 54, 55, 56, 57, 58 },
+ { 68, 69, 70, 71, 72, 73, 74, 75 },
+ { 85, 86, 87, 88, 89, 90, 91, 92 },
+ { 102, 103, 104, 105, 106, 107, 108, 109 },
+ { 119, 120, 121, 122, 123, 124, 125, 126 },
+ { 136, 137, 138, 139, 140, 141, 142, 143 },
+ { 153, 154, 155, 156, 157, 158, 159, 160 },
+ { 170, 171, 172, 173, 174, 175, 176, 177 },
+ { 187, 188, 189, 190, 191, 192, 193, 194 },
+ { 204, 205, 206, 207, 208, 209, 210, 211 },
+ { 221, 222, 223, 224, 225, 226, 227, 228 },
+ { 238, 239, 240, 241, 242, 243, 244, 245 },
+ { 255, 256, 257, 258, 259, 260, 261, 262 },
+ { 272, 273, 274, 275, 276, 277, 278, 279 }
+};
+static const V b[] = {
+ { 0, 33, 66, 99, 132, 165, 198, 231 },
+ { 1, 34, 67, 100, 133, 166, 199, 232 },
+ { 2, 35, 68, 101, 134, 167, 200, 233 },
+ { 3, 36, 69, 102, 135, 168, 201, 234 },
+ { 4, 37, 70, 103, 136, 169, 202, 235 },
+ { 5, 38, 71, 104, 137, 170, 203, 236 },
+ { 6, 39, 72, 105, 138, 171, 204, 237 },
+ { 7, 40, 73, 106, 139, 172, 205, 238 },
+ { 8, 41, 74, 107, 140, 173, 206, 239 },
+ { 9, 42, 75, 108, 141, 174, 207, 240 },
+ { 10, 43, 76, 109, 142, 175, 208, 241 },
+ { 11, 44, 77, 110, 143, 176, 209, 242 },
+ { 12, 45, 78, 111, 144, 177, 210, 243 },
+ { 13, 46, 79, 112, 145, 178, 211, 244 },
+ { 14, 47, 80, 113, 146, 179, 212, 245 },
+ { 15, 48, 81, 114, 147, 180, 213, 246 },
+ { 16, 49, 82, 115, 148, 181, 214, 247 }
+};
+static const V expected[] = {
+ { 0, 68, 204, 408, 680, 1020, 1428, 1904 },
+ { 18, 104, 258, 480, 770, 1128, 1554, 2048 },
+ { 36, 140, 312, 552, 860, 1236, 1680, 2192 },
+ { 54, 176, 366, 624, 950, 1344, 1806, 2336 },
+ { 72, 212, 420, 696, 1040, 1452, 1932, 2480 },
+ { 90, 248, 474, 768, 1130, 1560, 2058, 2624 },
+ { 108, 284, 528, 840, 1220, 1668, 2184, 2768 },
+ { 126, 320, 582, 912, 1310, 1776, 2310, 2912 },
+ { 144, 356, 636, 984, 1400, 1884, 2436, 3056 },
+ { 162, 392, 690, 1056, 1490, 1992, 2562, 3200 },
+ { 180, 428, 744, 1128, 1580, 2100, 2688, 3344 },
+ { 198, 464, 798, 1200, 1670, 2208, 2814, 3488 },
+ { 216, 500, 852, 1272, 1760, 2316, 2940, 3632 },
+ { 234, 536, 906, 1344, 1850, 2424, 3066, 3776 },
+ { 252, 572, 960, 1416, 1940, 2532, 3192, 3920 },
+ { 270, 608, 1014, 1488, 2030, 2640, 3318, 4064 },
+ { 288, 644, 1068, 1560, 2120, 2748, 3444, 4208 }
+};
+
+int
+main (void)
+{
+ check_vect ();
+ V out[17];
+ __builtin_memset (out, 99, sizeof (out));
+ for (unsigned n = 0; n <= 17; ++n)
+ {
+ f (out, a, b, n);
+ if (__builtin_memcmp (out, expected, n * sizeof (V)))
+ __builtin_abort ();
+ }
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-load.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-load.c
new file mode 100644
index
0000000000000000000000000000000000000000..558b2a98a58012ae6f08d98c24c4ae55f4bfe3dc
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-bb-load.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-tree-loop-vectorize
-fno-vect-cost-model -fdump-tree-slp-details" } */
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+
+void
+f (V *out, const V *a, const V *b)
+{
+ *out = (*a + (V) { 1, 3, 5, 7, 9, 11, 13, 15 }) ^ *b;
+}
+
+/* Check that the loads and arithmetic reach transformation, rather than
+ vectorizing only the store of an externally computed source vector. */
+/* { dg-final { scan-tree-dump-not "not grouped load" "slp2" } } */
+/* { dg-final { scan-tree-dump-not "no vectype for scalar type vector" "slp2"
} } */
+/* { dg-final { scan-tree-dump-times {transform load\.} 2 "slp2" } } */
+/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 1 "slp2" }
} */
diff --git a/gcc/testsuite/gcc.target/aarch64/vect-revectorize-cost.c
b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-cost.c
new file mode 100644
index
0000000000000000000000000000000000000000..70c593330294367e4a0ffebccc7f88617bb33a1a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/vect-revectorize-cost.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv8-a -fno-tree-loop-vectorize
-fdump-tree-slp-details" } */
+
+typedef unsigned V __attribute__ ((vector_size (32)));
+
+void
+f (V *out, const V *a, const V *b)
+{
+ *out = *a + *b;
+}
+
+/* Each original vector takes two native pieces, rather than eight scalar
+ operations. Retain the cost model when checking these estimates. */
+/* { dg-final { scan-tree-dump "2 times scalar_store" "slp2" } } */
+/* { dg-final { scan-tree-dump-not "8 times scalar_" "slp2" } } */
diff --git a/gcc/tree-ssa-strlen.cc b/gcc/tree-ssa-strlen.cc
index
8e2550f553a142e09dfb22110356678581aabf49..781dab1117cff1e6745593249a20d4b25cb14545
100644
--- a/gcc/tree-ssa-strlen.cc
+++ b/gcc/tree-ssa-strlen.cc
@@ -4590,9 +4590,13 @@ strlen_pass::count_nonzero_bytes (tree exp, tree vuse,
gimple *stmt,
exp = gimple_assign_rhs1 (g);
if (!DECL_P (exp)
&& TREE_CODE (exp) != CONSTRUCTOR
- && TREE_CODE (exp) != MEM_REF)
+ && TREE_CODE (exp) != MEM_REF
+ && TREE_CODE (exp) != BIT_FIELD_REF)
return false;
- /* Handle DECLs, CONSTRUCTOR and MEM_REF below. */
+
+ /* A split vector store can supply a BIT_FIELD_REF. Its result type
+ still bounds the number of nonzero bytes, even if none are
+ known. */
stmt = g;
}
else if (gimple_code (g) == GIMPLE_PHI)
diff --git a/gcc/tree-vect-data-refs.cc b/gcc/tree-vect-data-refs.cc
index
2a4c7936a4d5a393419a900d784df38e5118dfa1..d7a4472bb22ddb5cd336217dc4834ea5411953cc
100644
--- a/gcc/tree-vect-data-refs.cc
+++ b/gcc/tree-vect-data-refs.cc
@@ -1147,7 +1147,7 @@ vect_slp_analyze_instance_dependence (vec_info *vinfo,
slp_instance instance)
if (res && store)
{
stmt_vec_info store_info
- = DR_GROUP_FIRST_ELEMENT (SLP_TREE_SCALAR_STMTS (store)[0]);
+ = vect_first_dr_info (SLP_TREE_SCALAR_STMTS (store)[0])->stmt;
class loop *store_loop = gimple_bb (store_info->stmt)->loop_father;
if (! loop_outer (store_loop))
return res;
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index
787ad1a176f266c5b4129efc52ca24fd70952532..fb28596006a16b558ac1779345b6f7457cdf8438
100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -1377,6 +1377,41 @@ vect_verify_loop_lens (loop_vec_info loop_vinfo)
return true;
}
+/* Return the number of operations charged to STMT in the scalar cost model.
+ An emulated vector is lowered to native pieces, or to scalar elements if
+ there is no native vector type. Native vectors and scalars count as one.
*/
+
+unsigned int
+vect_scalar_stmt_count (gimple *stmt)
+{
+ tree lhs = gimple_get_lhs (stmt);
+ unsigned HOST_WIDE_INT nelts;
+ if (lhs
+ && VECTOR_TYPE_P (TREE_TYPE (lhs))
+ && vect_emulated_vector_p (TREE_TYPE (lhs))
+ && TYPE_VECTOR_SUBPARTS (TREE_TYPE (lhs)).is_constant (&nelts))
+ {
+ /* Vector lowering expands an emulated VEC_PERM_EXPR by extracting
+ and reinserting each element, so charge one operation per lane. */
+ if (is_gimple_assign (stmt)
+ && gimple_assign_rhs_code (stmt) == VEC_PERM_EXPR)
+ return nelts;
+
+ /* Cost the original lowering independently of the vector mode that
+ the vectorizer is considering. */
+ tree vectype = vect_native_vector_type (TREE_TYPE (lhs));
+ unsigned int nvectors;
+ if (vectype
+ && constant_multiple_p (poly_uint64 (nelts),
+ TYPE_VECTOR_SUBPARTS (vectype), &nvectors))
+ return nvectors;
+
+ return nelts;
+ }
+
+ return 1;
+}
+
/* Calculate the cost of one scalar iteration of the loop. */
static void
vect_compute_single_scalar_iteration_cost (loop_vec_info loop_vinfo)
@@ -1425,9 +1460,11 @@ vect_compute_single_scalar_iteration_cost (loop_vec_info
loop_vinfo)
else
kind = scalar_stmt;
+ unsigned stmt_factor = vect_scalar_stmt_count (stmt);
+
/* The target cost model applies the inner-loop frequency. */
record_stmt_cost (&LOOP_VINFO_SCALAR_ITERATION_COST (loop_vinfo),
- 1, kind, stmt_info, 0, vect_body);
+ stmt_factor, kind, stmt_info, 0, vect_body);
}
}
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index
fbf13d3faa2b9ea812fabec4407a7e8c966eab1a..4990852cbe6795accb0b6d273bff489938b38447
100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -3816,8 +3816,9 @@ vect_bb_slp_mark_stmts_vectorized (bb_vec_info vinfo)
for (auto instance : BB_VINFO_SLP_INSTANCES (vinfo))
{
for (auto stmt : SLP_INSTANCE_ROOT_STMTS (instance))
- if (!scalar_stmts_in_externs.contains (stmt))
- STMT_SLP_TYPE (stmt) = pure_slp;
+ /* Roots are replaced by their vectorized results even when another
+ instance uses them as external operands. */
+ STMT_SLP_TYPE (stmt) = pure_slp;
auto_vec<stmt_vec_info> worklist;
for (auto stmt : SLP_TREE_SCALAR_STMTS (SLP_INSTANCE_TREE (instance)))
{
@@ -4530,10 +4531,10 @@ vect_build_slp_instance (vec_info *vinfo,
}
/* Failed to SLP. */
- /* While we arrive here even with slp_inst_kind_store we should only
- for group_size == 1. The code to split store groups is only in
- vect_analyze_slp_instance now. */
- gcc_assert (kind != slp_inst_kind_store || group_size == 1);
+ /* Store groups are split by vect_analyze_slp_instance. The only
+ multi-lane store root here is a single repeated vector statement. */
+ gcc_assert (kind != slp_inst_kind_store || group_size == 1
+ || scalar_stmts[0] == scalar_stmts[group_size - 1]);
/* For BB vectorization we get failures only in case of the need of
unrolling, as otherwise we'll simply get operands built from scalars.
@@ -6124,7 +6125,8 @@ vect_analyze_slp (vec_info *vinfo, unsigned max_tree_size,
{
for (unsigned i = 0; i < bb_vinfo->roots.length (); ++i)
{
- vect_location = bb_vinfo->roots[i].roots[0]->stmt;
+ vect_location = bb_vinfo->roots[i].stmts[0]->stmt;
+
/* Apply patterns. */
for (unsigned j = 0; j < bb_vinfo->roots[i].stmts.length (); ++j)
bb_vinfo->roots[i].stmts[j]
@@ -7615,7 +7617,8 @@ vect_optimize_slp_pass::start_choosing_layouts ()
of the others. This can be useful if the address/index input
benefits from permutation. */
if (STMT_VINFO_DATA_REF (rep)
- && STMT_VINFO_GROUPED_ACCESS (rep)
+ && (DR_IS_WRITE (STMT_VINFO_DATA_REF (rep))
+ || STMT_VINFO_GROUPED_ACCESS (rep))
&& !SLP_TREE_LOAD_PERMUTATION (node).exists ())
partition.layout = 0;
@@ -9830,8 +9833,8 @@ get_ultimate_leader (slp_instance instance,
namespace {
/* Subroutine of vect_bb_partition_graph_r. Map KEY to INSTANCE in
- KEY_TO_INSTANCE, making INSTANCE the leader of any previous mapping
- for KEY. Return true if KEY was already in KEY_TO_INSTANCE.
+ KEY_TO_INSTANCE, merging the partitions of INSTANCE and any previous
+ mapping for KEY. Return true if KEY was already in KEY_TO_INSTANCE.
INSTANCE_LEADER is as for get_ultimate_leader. */
@@ -9847,14 +9850,14 @@ vect_map_to_instance (slp_instance instance, T key,
;
else if (key_instance != instance)
{
- /* If we're running into a previously marked key make us the
- leader of the current ultimate leader. This keeps the
- leader chain acyclic and works even when the current instance
- connects two previously independent graph parts. */
+ /* INSTANCE can already belong to another partition after visiting
+ an earlier statement or external operand. Merge the ultimate
+ leaders to keep the leader chain acyclic. */
slp_instance key_leader
= get_ultimate_leader (key_instance, instance_leader);
- if (key_leader != instance)
- instance_leader.put (key_leader, instance);
+ slp_instance leader = get_ultimate_leader (instance, instance_leader);
+ if (key_leader != leader)
+ instance_leader.put (key_leader, leader);
}
key_instance = instance;
return existed_p;
@@ -9870,6 +9873,25 @@ vect_bb_partition_graph_r (bb_vec_info bb_vinfo,
hash_map<slp_tree, slp_instance> &node_to_instance,
hash_map<slp_instance, slp_instance>
&instance_leader)
{
+ if (SLP_TREE_DEF_TYPE (node) != vect_internal_def)
+ {
+ /* A constructor replaced by another instance can still be an
+ external operand here. Cost its producer and consumer together.
+ Other external operands keep their scalar definitions. */
+ for (tree def : SLP_TREE_SCALAR_OPS (node))
+ {
+ if (!def)
+ continue;
+
+ stmt_vec_info def_stmt = bb_vinfo->lookup_def (def);
+ if (def_stmt && PURE_SLP_STMT (def_stmt))
+ vect_map_to_instance (instance, def_stmt, stmt_to_instance,
+ instance_leader);
+ }
+
+ return;
+ }
+
for (auto stmt_info : SLP_TREE_SCALAR_STMTS (node))
if (stmt_info)
vect_map_to_instance (instance, stmt_info, stmt_to_instance,
@@ -9880,7 +9902,7 @@ vect_bb_partition_graph_r (bb_vec_info bb_vinfo,
return;
for (auto child : SLP_TREE_CHILDREN (node))
- if (child && SLP_TREE_DEF_TYPE (child) == vect_internal_def)
+ if (child)
vect_bb_partition_graph_r (bb_vinfo, instance, child, stmt_to_instance,
node_to_instance, instance_leader);
}
@@ -9902,6 +9924,11 @@ vect_bb_partition_graph (bb_vec_info bb_vinfo)
for (unsigned i = 0; bb_vinfo->slp_instances.iterate (i, &instance); ++i)
{
instance_leader.put (instance, instance);
+
+ /* Constructor roots are not in the tree's scalar statement list. */
+ for (auto stmt : SLP_INSTANCE_ROOT_STMTS (instance))
+ vect_map_to_instance (instance, stmt, stmt_to_instance,
+ instance_leader);
vect_bb_partition_graph_r (bb_vinfo,
instance, SLP_INSTANCE_TREE (instance),
stmt_to_instance, node_to_instance,
@@ -10011,9 +10038,12 @@ vect_bb_slp_scalar_cost (bb_vec_info vinfo,
}
else
kind = scalar_stmt;
+ unsigned scalar_stmt_count = vect_scalar_stmt_count (stmt->stmt);
+
/* Cost each scalar stmt only once. */
gimple_set_visited (stmt->stmt, true);
- record_stmt_cost (cost_vec, 1, kind, stmt, NULL_TREE, 0, vect_body);
+ record_stmt_cost (cost_vec, scalar_stmt_count, kind, stmt,
+ NULL_TREE, 0, vect_body);
}
/* Now walk relevant parts of the SSA use-def graph. */
@@ -10078,14 +10108,25 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
stmt_vector_for_cost vector_costs = vNULL;
hash_set<slp_tree> visited;
hash_set<stmt_vec_info> svisited;
- FOR_EACH_VEC_ELT (slp_instances, i, instance)
+
+ /* Cost roots before the DFS, since another instance can use a root as
+ an external operand. Mark them as costed so the DFS cannot charge
+ them again. Their inputs are visited through the producing instance. */
+ for (auto instance : slp_instances)
{
- auto_vec<stmt_vec_info> worklist;
if (SLP_INSTANCE_ROOT_STMTS (instance).exists ())
record_stmt_cost (&scalar_costs,
SLP_INSTANCE_ROOT_STMTS (instance).length (),
scalar_stmt,
SLP_INSTANCE_ROOT_STMTS (instance)[0], 0, vect_body);
+
+ for (auto stmt : SLP_INSTANCE_ROOT_STMTS (instance))
+ svisited.add (stmt);
+ }
+
+ FOR_EACH_VEC_ELT (slp_instances, i, instance)
+ {
+ auto_vec<stmt_vec_info> worklist;
for (auto stmt : SLP_TREE_SCALAR_STMTS (SLP_INSTANCE_TREE (instance)))
{
stmt = vect_orig_stmt (stmt);
@@ -10521,6 +10562,25 @@ vect_slp_check_for_roots (bb_vec_info bb_vinfo)
}
}
}
+ else if (gimple_vdef (assign)
+ /* Native vectors are combined through grouped stores. Only
+ emulated vectors need a standalone store root for
+ splitting. */
+ && vect_can_revectorize_type_p (TREE_TYPE (rhs))
+ && vect_emulated_vector_p (TREE_TYPE (rhs))
+ && !STMT_VINFO_GROUPED_ACCESS (bb_vinfo->lookup_stmt (assign)))
+ {
+ vec<stmt_vec_info> roots = vNULL;
+ auto stmt_vinfo = bb_vinfo->lookup_stmt (assign);
+ vec<stmt_vec_info> stmts;
+ unsigned HOST_WIDE_INT num_entries
+ = TYPE_VECTOR_SUBPARTS (TREE_TYPE (rhs)).to_constant ();
+ stmts.create (num_entries);
+ for (unsigned HOST_WIDE_INT i = 0; i < num_entries; ++i)
+ stmts.quick_push (stmt_vinfo);
+ bb_vinfo->roots.safe_push (slp_root (slp_inst_kind_store,
+ stmts, roots));
+ }
}
}
@@ -13201,6 +13261,7 @@ vect_schedule_slp (vec_info *vinfo, vec<slp_instance>
&slp_instances,
{
slp_tree root = SLP_INSTANCE_TREE (instance);
stmt_vec_info store_info;
+ stmt_vec_info last_store_info = NULL;
unsigned int j;
/* Remove scalar call stmts. Do not do this for basic-block
@@ -13216,11 +13277,17 @@ vect_schedule_slp (vec_info *vinfo, vec<slp_instance>
&slp_instances,
/* Remove vectorized stores original scalar stmts. */
for (j = 0; SLP_TREE_SCALAR_STMTS (root).iterate (j, &store_info); j++)
{
- if (!store_info
- || !STMT_VINFO_DATA_REF (store_info)
+ if (!store_info)
+ break;
+
+ if (store_info == last_store_info)
+ continue;
+
+ if (!STMT_VINFO_DATA_REF (store_info)
|| !DR_IS_WRITE (STMT_VINFO_DATA_REF (store_info)))
break;
+ last_store_info = store_info;
store_info = vect_orig_stmt (store_info);
/* Free the attached stmt_vec_info and remove the stmt. */
vinfo->remove_stmt (store_info);
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index
f0862fc9dd7723dfa4b3cd78abc091bfbe7b5a35..4a459e53ccdb68b0f327309febc35133e680bbd8
100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -8255,7 +8255,8 @@ vectorizable_store (vec_info *vinfo,
&& TREE_CODE (scalar_dest) != COMPONENT_REF
&& TREE_CODE (scalar_dest) != IMAGPART_EXPR
&& TREE_CODE (scalar_dest) != REALPART_EXPR
- && TREE_CODE (scalar_dest) != MEM_REF)
+ && TREE_CODE (scalar_dest) != MEM_REF
+ && !DECL_P (scalar_dest))
return false;
}
else
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index
fcfc14c1a114d4a007d3c52c81b5b5051c34ca7c..7be54e3343daa1ca55b9bcf79bc363902b374445
100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -2764,6 +2764,7 @@ extern bool vectorizable_early_exit (loop_vec_info,
stmt_vec_info,
gimple_stmt_iterator *,
slp_tree, stmt_vector_for_cost *);
extern bool vect_emulated_vector_p (tree);
+extern unsigned int vect_scalar_stmt_count (gimple *);
extern bool vect_can_vectorize_without_simd_p (tree_code);
extern bool vect_can_vectorize_without_simd_p (code_helper);
extern int vect_get_known_peeling_cost (loop_vec_info, int);
