This enables use of a predicate mask or length limit for
vectorization of basic blocks in cases where previously only the
equivalent rolled (i.e. loop) form of some source code would have
been vectorized. Predication is used for groups whose size
is not neatly divisible into vectors of lengths that can be
supported directly by the target.
The initial vector mode for an SLP region is "autodetected" by calling
aarch64_preferred_simd_mode, which prefers SVE modes if supported
unless configured otherwise (e.g. VNx4SI for int). If at least one
profitable subgraph can be scheduled then GCC does not try to vectorize
the region using any other modes, even though their estimated costs
might otherwise have been lower.
For example, if analysis of a 24-byte group succeeds with vector mode
V16QI (using types vector(16) and vector(8) char) then the estimated
cost of the vectorized code is 11+11=22. If analysis of the same group
succeeds with vector mode VNx16QI (using type vector([16,16]) char for
both subtrees) then the estimated cost is 15+15=30. In both cases, the
estimated vectorized cost would beat the estimated scalar cost of
96+48=144, but vector([16,16]) is chosen because VNx16QI is tried
first.
This is mitigated by the fact that a sequence of GIMPLE stmts such as:
vectp.14_86 = x_50(D) + 16;
slp_mask_87 = .WHILE_ULT (0, 8, { 0, ... });
.MASK_STORE (vectp.14_86, 8B, slp_mask_87, vect__34.12_85);
are lowered to a fixed-length vector store (e.g., str d30, [x0, 16]) if
possible, instead of a more literal interpretation such as:
add x0, x0, 16
ptrue p7.b, vl8
st1b z30.b, p7, [x0]
The vect_build_slp_tree_1 function used during building of an SLP
tree is updated to prevent it returning failure for BB SLP if the
group size is not an integral multiple of the number of lanes in the
vector type; it now allows such cases if the group size is known to
be less than or equal to the minimum number of lanes in the vector type.
Update test expectations for gcc.dg/vect/vect-over-widen-*.c,
gcc.target/aarch64/sve/slp_6.c and
gcc.target/aarch64/sve/vec_construct_*.c.
The vec_construct_*.c tests previously expected their output
to use Advanced SIMD instead of SVE despite their use of
vector length agnostic types such as svint16_t and despite
the fact that they are in the aarch64/sve directory. Since
BB SLP can now vectorize these tests using VLA types such
as 'vector([8,8]) char', and because (with one exception) the
resultant code is deemed profitable relative to scalar code,
GCC no longer considers vectorizing using non-VLA types such
as 'vector(8) char' (although the estimated cost with non-VLA
types might have been lower, had it been calculated).
Instruction selection is not the focus of these tests, therefore
I updated them to expect SVE instead (e.g. st1b instead of str)
and added --param=aarch64-autovec-preference=sve-only to reduce
future churn.
Because the cost model takes into account predicate mask
generation for BB SLP with VLA types, the threshold at which
vectorized code wins against scalar code is higher than
before. The number of elements stored by vec_construct_3.c was
increased just enough to allow for that.
Because a V2DI -> V2SI pack becomes a VNx2DI -> VNx4SI -> V2SI
pack when use of predicated tails is enabled, a test that implicitly
requires use of *aarch64_trunc_concat<mode> now fails. This missed
optimization will be restored in a future patch.
gcc/ChangeLog:
* tree-vect-slp.cc (vect_build_slp_tree_1): For BB SLP
vectorization, allow group sizes that are indivisible by the
vector length if they are known not to exceed it. Use the
known minimum number of subparts when choosing a splitting
point for a VLA vector type.
(vect_build_slp_tree_2): Don't call can_duplicate_and_interleave_p
when doing basic block SLP vectorization.
(vect_analyze_slp_instance): For BB SLP vectorization, use the
constant lower bound of a VLA vector type's number of subparts when
choosing a point at which to split an SLP group, provided that the
matching prefix is divisible by that value.
(vectorizable_slp_permutation_1): Instead of asserting that an
SLP tree node's number of lanes is compatible with the chosen
vector width, return a failure indication if incompatible.
(vectorize_slp_instance_root_stmt): Build a BIT_FIELD_REF instead
of a VIEW_CONVERT_EXPR to extract a fixed-length vector from a
larger scalable vector result, when vectorizing a ctor.
* tree-vect-stmts.cc (get_vectype_for_scalar_type): Execute the
old algorithm to find a narrower vector type that can be used to
carve an SLP group into pieces only if partial vector support is
unavailable for either loads or stores, and only if the number of
subparts may be greater than the group size rather than also if
the number of subparts may be equal to the group size.
Otherwise, assume that using a partial vector is more efficient
than carving the group into smaller pieces. If the natural
vector type is not VLA and it has more subparts than the group
size then find the smallest vector type that contains the group.
(vect_get_vector_types_for_stmt): Update the description.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/vect-over-widen-10.c: Update test expectations to
avoid spurious matching of scan-tree-dump-not pattern.
* gcc.dg/vect/vect-over-widen-13.c: As above.
* gcc.dg/vect/vect-over-widen-14.c: As above.
* gcc.dg/vect/vect-over-widen-17.c: As above.
* gcc.dg/vect/vect-over-widen-18.c: As above.
* gcc.dg/vect/vect-over-widen-5.c: As above.
* gcc.dg/vect/vect-over-widen-6.c: As above.
* gcc.dg/vect/vect-over-widen-7.c: As above.
* gcc.dg/vect/vect-over-widen-8.c: As above.
* gcc.dg/vect/vect-over-widen-9.c: As above.
* gcc.target/aarch64/sve/slp_6.c: Require ld1b, ld1h and ld1w
instructions in the output instead of forbidding them because
discovery of reduction groups now succeeds. Also require
the corresponding st1b, st1h and st1w instructions.
* gcc.target/aarch64/sve/vec_construct_1.c:
Expect SVE instead of ASIMD instructions and add
--param=aarch64-autovec-preference=sve-only to stop
flip-flopping.
* gcc.target/aarch64/sve/vec_construct_2.c: Forbid SVE
instructions as well as ASIMD instructions and add
--param=aarch64-autovec-preference=sve-only to stop
flip-flopping.
* gcc.target/aarch64/sve/vec_construct_3.c: Expect SVE instead
of ASIMD instructions and add
--param=aarch64-autovec-preference=sve-only to avoid
flip-flopping. Increase the number of elements stored to
ensure vectorization using SVE is deemed profitable despite
predicate mask costs.
* gcc.target/aarch64/sve/vec_construct_4.c:
Forbid SVE instructions as well as ASIMD instructions and add
--param=aarch64-autovec-preference=sve-only to stop
flip-flopping. Fix a missing end of comment delimiter.
* gcc.target/aarch64/sve/vec_construct_5.c:
Expect SVE instead of ASIMD instructions and add
--param=aarch64-autovec-preference=sve-only to stop
flip-flopping.
* gcc.target/aarch64/sve/truncated_concatenation_1.c:
Mark one scan-assembler-times check as xfail because code
generation regresses (PR 127271).
---
.../gcc.dg/vect/vect-over-widen-10.c | 2 +-
.../gcc.dg/vect/vect-over-widen-13.c | 2 +-
.../gcc.dg/vect/vect-over-widen-14.c | 2 +-
.../gcc.dg/vect/vect-over-widen-17.c | 2 +-
.../gcc.dg/vect/vect-over-widen-18.c | 2 +-
gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c | 2 +-
gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c | 2 +-
gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c | 2 +-
gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c | 2 +-
gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c | 2 +-
gcc/testsuite/gcc.target/aarch64/sve/slp_6.c | 18 +++--
.../aarch64/sve/truncated_concatenation_1.c | 5 +-
.../gcc.target/aarch64/sve/vec_construct_1.c | 11 ++-
.../gcc.target/aarch64/sve/vec_construct_2.c | 4 +-
.../gcc.target/aarch64/sve/vec_construct_3.c | 25 ++++---
.../gcc.target/aarch64/sve/vec_construct_4.c | 5 +-
.../gcc.target/aarch64/sve/vec_construct_5.c | 13 ++--
gcc/tree-vect-slp.cc | 72 +++++++++++++++----
gcc/tree-vect-stmts.cc | 69 ++++++++++++++----
19 files changed, 174 insertions(+), 68 deletions(-)
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c
index f0140e4ef6d..6efcf739db9 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-10.c
@@ -16,5 +16,5 @@
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 1} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 2} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(unsigned char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c
index 08a65ea5518..720353716cf 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-13.c
@@ -48,5 +48,5 @@ main (void)
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* / 2} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* = \(signed char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c
index dfa09f5d2ca..f1d5f95c543 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-14.c
@@ -15,5 +15,5 @@
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 1} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* = \(unsigned char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c
index 53fcfd0c06c..ac1a0f86727 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-17.c
@@ -46,5 +46,5 @@ main (void)
adopts realign_load scheme. It requires rs6000_builtin_mask_for_load to
generate mask whose return type is vector char. */
/* { dg-final { scan-tree-dump-not {vector[^\n]*char} "vect" { target
vect_hw_misalign } } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c
index aa58cd1c957..3ebfaa78270 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-18.c
@@ -47,5 +47,5 @@ main (void)
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* |} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* <<} "vect" } } */
/* { dg-final { scan-tree-dump {vector[^\n]*char} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c
index c2ab11a9d32..1d89789a86d 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-5.c
@@ -49,5 +49,5 @@ main (void)
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+ } "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 1} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(signed char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c
index bda92c965e0..62d5a52587e 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-6.c
@@ -13,5 +13,5 @@
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+ } "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 1} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(unsigned char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c
index 1d55e13fb1f..6e09631009a 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-7.c
@@ -51,5 +51,5 @@ main (void)
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+ } "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 2} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(signed char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c
index 553c0712a79..b6d650beab4 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-8.c
@@ -16,5 +16,5 @@
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* \+ } "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 2} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(unsigned char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c
b/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c
index 36bfc68e053..e82f8a571da 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-over-widen-9.c
@@ -56,5 +56,5 @@ main (void)
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 1} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_over_widening_pattern:
detected:[^\n]* >> 2} "vect" } } */
/* { dg-final { scan-tree-dump {vect_recog_cast_forwprop_pattern:
detected:[^\n]* \(signed char\)} "vect" } } */
-/* { dg-final { scan-tree-dump-not {vector[^ ]* int} "vect" } } */
+/* { dg-final { scan-tree-dump-not {vector[^ ]* int vect__} "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorized 1 loop" 1 "vect" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c
b/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c
index 44d128477d2..0ac3f09cbdb 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/slp_6.c
@@ -36,11 +36,21 @@ vec_slp_##TYPE (TYPE *restrict a, TYPE *restrict b, int n)
\
TEST_ALL (VEC_PERM)
-/* These loops can't use SLP. */
-/* { dg-final { scan-assembler-not {\tld1b\t} } } */
-/* { dg-final { scan-assembler-not {\tld1h\t} } } */
-/* { dg-final { scan-assembler-not {\tld1w\t} } } */
+/* When this test was written, these loops couldn't use SLP. Since changes to
+ enable tail predication, SLP discovery of size 3 reduction groups instead
+ succeeds, producing predicated loads such as ld1b z25.b, p7/z, [x1] for
+ {b[0], b[1], b[2]}. */
+
+/* { dg-final { scan-assembler {\tld1b\t} } } */
+/* { dg-final { scan-assembler {\tld1h\t} } } */
+/* { dg-final { scan-assembler {\tld1w\t} } } */
/* { dg-final { scan-assembler-not {\tld1d\t} } } */
+
+/* { dg-final { scan-assembler {\tst1b\t} } } */
+/* { dg-final { scan-assembler {\tst1h\t} } } */
+/* { dg-final { scan-assembler {\tst1w\t} } } */
+/* { dg-final { scan-assembler-not {\tst1d\t} } } */
+
/* { dg-final { scan-assembler {\tld3b\t} } } */
/* { dg-final { scan-assembler {\tld3h\t} } } */
/* { dg-final { scan-assembler {\tld3w\t} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/truncated_concatenation_1.c
b/gcc/testsuite/gcc.target/aarch64/sve/truncated_concatenation_1.c
index 95577a1a9ef..4cbad506ece 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/truncated_concatenation_1.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/truncated_concatenation_1.c
@@ -29,4 +29,7 @@ int32x2_t f3 (svint64_t a, svint64_t b) {
/* { dg-final { scan-assembler-not {\tfcvtn\t} } }*/
/* { dg-final { scan-assembler-times {\tuzp1\tv[0-9]+\.8b, v[0-9]+\.8b,
v[0-9]+\.8b} 1 } }*/
/* { dg-final { scan-assembler-times {\tuzp1\tv[0-9]+\.4h, v[0-9]+\.4h,
v[0-9]+\.4h} 1 } }*/
-/* { dg-final { scan-assembler-times {\tuzp1\tv[0-9]+\.2s, v[0-9]+\.2s,
v[0-9]+\.2s} 1 } }*/
\ No newline at end of file
+/* PR 127271: xfail because SLP with predicated tails changes a V2DI -> V2SI
+ pack into a VNx2DI -> VNx4SI -> V2SI pack, which prevents use of
+ *aarch64_trunc_concat<mode>. */
+/* { dg-final { scan-assembler-times {\tuzp1\tv[0-9]+\.2s, v[0-9]+\.2s,
v[0-9]+\.2s} 1 { xfail *-*-* } } }*/
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c
b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c
index 2f8ce6808a9..421a990a739 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_1.c
@@ -1,12 +1,11 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -ftree-slp-vectorize" } */
+/* { dg-options "-O2 -ftree-slp-vectorize
--param=aarch64-autovec-preference=sve-only" } */
/* Test that a group of stores of 8 elements derived from a horizontal
reduction is vectorized by constructing a vector and storing it.
Since there are no GPR-to-SIMD register transfers, there is no
- need to charge additional costs for them. Only one STR (Store
- vector reg, unsigned immed, B/H/S/D-form) instruction is required
- instead of 8.
+ need to charge additional costs for them. Only one vector store
+ instruction is required instead of 8.
*/
#include <arm_sve.h>
@@ -30,8 +29,8 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t
src3, svint8_t src4,
s.h = svaddv_s8 (all, src7);
}
-/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\],
v[0-9]+\.b\[[0-9]+\]\n} 7 } } */
-/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */
+/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.h, h[0-9]+\n} 7 } } */
+/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.h, p[0-9]+,
\[x[0-9]+\]\n} 1 } } */
/* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */
/* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c
b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c
index 6715118d7b0..2bf537e13e2 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_2.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -ftree-slp-vectorize" } */
+/* { dg-options "-O2 -ftree-slp-vectorize
--param=aarch64-autovec-preference=sve-only" } */
/* Test that a group of stores of 8 elements derived from the results of calls
to a function that has only vector parameters and returns a scalar result is
@@ -40,3 +40,5 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t
src3, svint8_t src4,
/* { dg-final { scan-assembler-not {\tins\tv[0-9]+\.b\[[0-9]+\], w[0-9]+\n} }
} */
/* { dg-final { scan-assembler-not {\tstr\td[0-9]+, } } } */
+/* { dg-final { scan-assembler-not {\tfmov\th[0-9]+, h[0-9]+\n} } } */
+/* { dg-final { scan-assembler-not {\tinsr\tz[0-9]+\.b, w[0-9]+\n} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c
b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c
index 8143d0050ad..28346d26a73 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_3.c
@@ -1,24 +1,25 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -ftree-slp-vectorize" } */
+/* { dg-options "-O2 -ftree-slp-vectorize
--param=aarch64-autovec-preference=sve-only" } */
-/* Test that a group of stores of 8 elements derived from a horizontal
+/* Test that a group of stores of 14 elements derived from a horizontal
reduction is vectorized by constructing a vector and storing it
even if the results of the reductions are narrowed.
Since there are no GPR-to-SIMD register transfers, there is no
- need to charge additional costs for them. Only one STR (Store
- vector reg, unsigned immed, B/H/S/D-form) instruction is required
- instead of 8.
+ need to charge additional costs for them. Only one vector store
+ instruction is required instead of 14.
*/
#include <arm_sve.h>
struct S
{
- char a, b, c, d, e, f, g, h;
+ char a, b, c, d, e, f, g, h, i, j, k, l, m, n;
} s;
void
foo (svint16_t src0, svint32_t src1, svint16_t src2, svint32_t src3,
- svint32_t src4, svint16_t src5, svint32_t src6, svint16_t src7)
+ svint32_t src4, svint16_t src5, svint32_t src6, svint16_t src7,
+ svint16_t src8, svint32_t src9, svint16_t src10, svint32_t src11,
+ svint32_t src12, svint16_t src13)
{
svbool_t all16 = svptrue_b16 ();
svbool_t all32 = svptrue_b32 ();
@@ -30,10 +31,16 @@ foo (svint16_t src0, svint32_t src1, svint16_t src2,
svint32_t src3,
s.f = svminv_s16 (all16, src5);
s.g = svlastb_s32 (svptrue_pat_b32 (SV_VL1), src6);
s.h = svaddv_s16 (all16, src7);
+ s.i = svmaxv_s16 (all16, src8);
+ s.j = svminv_s32 (all32, src9);
+ s.k = svlastb_s16 (svptrue_pat_b16 (SV_VL1), src10);
+ s.l = svaddv_s32 (all32, src11);
+ s.m = svmaxv_s32 (all32, src12);
+ s.n = svminv_s16 (all16, src13);
}
-/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\],
v[0-9]+\.b\[[0-9]+\]\n} 7 } } */
-/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */
+/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.b, b[0-9]+\n} 13 } } */
+/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.b, p[0-9]+,
\[x[0-9]\]\n} 1 } } */
/* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */
/* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c
b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c
index 49f8114b64c..3d41af684a3 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_4.c
@@ -1,5 +1,5 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -ftree-slp-vectorize" } */
+/* { dg-options "-O2 -ftree-slp-vectorize
--param=aarch64-autovec-preference=sve-only" } */
/* Test that a group of stores of 8 elements derived from a horizontal
reduction is not vectorized by constructing a vector and storing it
@@ -33,5 +33,6 @@ foo (svint16_t src0, svint8_t src1, svint16_t src2, svint8_t
src3,
/* { dg-final { scan-assembler-times {\tstp\tw[0-9]+, w[0-9]+,} 4 } } */
/* { dg-final { scan-assembler-not {\tins\tv[0-9]+\.s\[[0-9]+\], w[0-9]+\n} }
} */
-/* { dg-final { scan-assembler-not {\tfmov\ts[0-9]+, w[0-9]+\n} } }
+/* { dg-final { scan-assembler-not {\tfmov\ts[0-9]+, w[0-9]+\n} } } */
/* { dg-final { scan-assembler-not {\tstp\tq[0-9]+, q[0-9]+,} } } */
+/* { dg-final { scan-assembler-not {\tinsr\tz[0-9]+.s, w[0-9]+\n} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c
b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c
index 983d6c69ebc..28c7e68f303 100644
--- a/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c
+++ b/gcc/testsuite/gcc.target/aarch64/sve/vec_construct_5.c
@@ -1,13 +1,12 @@
/* { dg-do compile } */
-/* { dg-options "-O2 -ftree-slp-vectorize" } */
+/* { dg-options "-O2 -ftree-slp-vectorize
--param=aarch64-autovec-preference=sve-only" } */
/* Test that a group of stores of 8 elements derived from lane extractions is
vectorized by constructing a vector and storing it. Since there are no
GPR-to-SIMD register transfers, there is no need to charge additional costs
- for them. Only one STR (Store vector reg, unsigned immed, B/H/S/D-form)
- instruction is required instead of 8. This is a test for over-restriction
- of an exemption from usual GPR-to-SIMD costs to reductions (in case lane
- extraction is not treated as a reduction).
+ for them. Only one vector store instruction is required instead of 8. This
+ is a test for over-restriction of an exemption from usual GPR-to-SIMD costs
+ to reductions (in case lane extraction is not treated as a reduction).
*/
#include <arm_sve.h>
@@ -30,8 +29,8 @@ foo (svint8_t src0, svint8_t src1, svint8_t src2, svint8_t
src3, svint8_t src4,
s.h = svlastb_s8 (p, src7);
}
-/* { dg-final { scan-assembler-times {\tins\tv[0-9]+\.b\[[0-9]+\],
v[0-9]+\.b\[[0-9]+\]\n} 7 } } */
-/* { dg-final { scan-assembler-times {\tstr\td[0-9]+, } 1 } } */
+/* { dg-final { scan-assembler-times {\tinsr\tz[0-9]+\.h, h[0-9]+\n} 7 } } */
+/* { dg-final { scan-assembler-times {\tst1b\tz[0-9]+\.h, p[0-9]+,
\[x[0-9]+\]\n} 1 } } */
/* { dg-final { scan-assembler-not {\tstr\tb[0-9]+, } } } */
/* { dg-final { scan-assembler-not {\tstrb\tw[0-9]+, } } } */
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index ed3b49a001a..5f0f73c6580 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -1674,14 +1674,19 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char
*swap,
unsigned HOST_WIDE_INT const_nunits = 0;
if (vectype
&& is_a <bb_vec_info> (vinfo)
+ && maybe_gt (group_size, TYPE_VECTOR_SUBPARTS (vectype))
&& !multiple_p (group_size, TYPE_VECTOR_SUBPARTS (vectype)))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
"Build SLP failed: unrolling required "
"in basic block SLP\n");
- if (!TYPE_VECTOR_SUBPARTS (vectype).is_constant (&const_nunits)
- || const_nunits > group_size)
+
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
+ if (!nunits.is_constant (&const_nunits))
+ const_nunits = constant_lower_bound (nunits);
+
+ if (const_nunits > group_size)
{
/* Fatal mismatch. */
matches[0] = false;
@@ -1709,7 +1714,11 @@ vect_build_slp_tree_1 (vec_info *vinfo, unsigned char
*swap,
{
/* With constant vector elements simulate a mismatch at the
point we need to split. */
+ gcc_assert ((const_nunits & (const_nunits - 1)) == 0);
unsigned tail = group_size & (const_nunits - 1);
+ if (tail == 0)
+ tail = const_nunits;
+ gcc_assert (group_size >= tail);
memset (&matches[group_size - tail], 0, sizeof (bool) * tail);
return false;
}
@@ -2442,13 +2451,21 @@ vect_build_slp_tree_2 (vec_info *vinfo, slp_tree node,
/* Check whether we can build the invariant. If we can't
we never will be able to. */
tree type = TREE_TYPE (chains[0][n].op);
- if (!GET_MODE_SIZE (vinfo->vector_mode).is_constant ()
- && (TREE_CODE (type) == BOOLEAN_TYPE
- || !can_duplicate_and_interleave_p (vinfo, group_size,
- type)))
+ if (!GET_MODE_SIZE (vinfo->vector_mode).is_constant ())
{
- matches[0] = false;
- goto out;
+ if (TREE_CODE (type) == BOOLEAN_TYPE)
+ {
+ matches[0] = false;
+ goto out;
+ }
+
+ if (!is_a<bb_vec_info> (vinfo)
+ && !can_duplicate_and_interleave_p (vinfo, group_size,
+ type))
+ {
+ matches[0] = false;
+ goto out;
+ }
}
}
else if (dt != vect_internal_def)
@@ -2879,7 +2896,7 @@ out:
uniform_val = NULL_TREE;
break;
}
- if (!uniform_val
+ if (!uniform_val && !is_a<bb_vec_info> (vinfo)
&& !can_duplicate_and_interleave_p (vinfo,
oprnd_info->ops.length (),
TREE_TYPE (op0)))
@@ -5147,8 +5164,10 @@ vect_analyze_slp_instance (vec_info *vinfo,
tree vectype = get_vectype_for_scalar_type (vinfo, scalar_type,
1 << floor_log2 (i));
unsigned HOST_WIDE_INT const_nunits;
+ poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
if (vectype
- && TYPE_VECTOR_SUBPARTS (vectype).is_constant (&const_nunits))
+ && (const_nunits = constant_lower_bound (nunits)) > 1
+ && (i % const_nunits) == 0)
{
/* Split into two groups at the first vector boundary. */
gcc_assert ((const_nunits & (const_nunits - 1)) == 0);
@@ -12016,7 +12035,21 @@ vectorizable_slp_permutation_1 (vec_info *vinfo,
gimple_stmt_iterator *gsi,
unpack_factor = 1;
}
unsigned olanes = unpack_factor * ncopies * SLP_TREE_LANES (node);
- gcc_assert (repeating_p || multiple_p (olanes, nunits));
+
+ /* With fully-predicated BB-SLP, an external node's number of lanes can be
+ incompatible with the chosen vector width (e.g., lane packs of 3 with a
+ natural 2-lane vector type). */
+ if (!repeating_p && !multiple_p (olanes, nunits))
+ {
+ if (dump_p)
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "unsupported permutation %p: vector type %T,"
+ " nunits=" HOST_WIDE_INT_PRINT_UNSIGNED
+ " ncopies=%" PRIu64 ", lanes=%u and unpack=%u\n",
+ (void *) node, vectype, estimated_poly_value (nunits),
+ ncopies, SLP_TREE_LANES (node), unpack_factor);
+ return -1;
+ }
/* Compute the { { SLP operand, vector index}, lane } permutation sequence
from the { SLP operand, scalar lane } permutation as recorded in the
@@ -12684,8 +12717,21 @@ vectorize_slp_instance_root_stmt (vec_info *vinfo,
slp_tree node, slp_instance i
tree root_lhs = gimple_get_lhs (instance->root_stmts[0]->stmt);
if (!useless_type_conversion_p (TREE_TYPE (root_lhs),
TREE_TYPE (new_def)))
- new_def = build1 (VIEW_CONVERT_EXPR, TREE_TYPE (root_lhs),
- new_def);
+ {
+ tree vectype = TREE_TYPE (root_lhs);
+ poly_uint64 new_def_nunits
+ = TYPE_VECTOR_SUBPARTS (TREE_TYPE (new_def));
+ if (known_lt (TYPE_VECTOR_SUBPARTS (vectype), new_def_nunits))
+ {
+ new_def = build3 (BIT_FIELD_REF, vectype, new_def,
+ TYPE_SIZE (vectype), bitsize_zero_node);
+ }
+ else
+ {
+ new_def
+ = build1 (VIEW_CONVERT_EXPR, TREE_TYPE (root_lhs), new_def);
+ }
+ }
}
else
{
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index edb76b3bf51..cff17c2a149 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -13608,9 +13608,11 @@ get_related_vectype_for_scalar_type (machine_mode
prevailing_mode,
/* Function get_vectype_for_scalar_type.
Returns the vector type corresponding to SCALAR_TYPE as supported
- by the target. If GROUP_SIZE is nonzero and we're performing BB
- vectorization, make sure that the number of elements in the vector
- is no bigger than GROUP_SIZE. */
+ by the target, or NULL_TREE if none. If GROUP_SIZE is nonzero and we're
+ performing BB vectorization, ensure that the number of subparts in the
+ returned type is no bigger than GROUP_SIZE, or no smaller than GROUP_SIZE if
+ a vector mask or length can be used to limit the number of lanes loaded and
+ stored. */
tree
get_vectype_for_scalar_type (vec_info *vinfo, tree scalar_type,
@@ -13635,11 +13637,23 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree
scalar_type,
if (vectype)
vinfo->used_vector_modes.add (TYPE_MODE (vectype));
- /* If the natural choice of vector type doesn't satisfy GROUP_SIZE,
- try again with an explicit number of elements. */
- if (vectype
- && group_size
- && maybe_ge (TYPE_VECTOR_SUBPARTS (vectype), group_size))
+ if (!vectype || !group_size)
+ return vectype;
+
+ /* The natural choice of vector type satisfies GROUP_SIZE if it can be used
to
+ store the whole group without a mask or length limit (known_eq) or if it
+ can be used to carve the group into pieces (known_lt). */
+ const poly_uint64 natural_nelts = TYPE_VECTOR_SUBPARTS (vectype);
+ if (known_le (natural_nelts, group_size))
+ return vectype;
+
+ /* We might be able to use a partial vector to handle a GROUP_SIZE that is
not
+ satisfied by the natural choice of vector type. Assume this is more
+ efficient than carving the group into pieces. */
+ if (vect_get_load_store_partial_vector_style (vectype, true)
+ == vect_load_store_partial_vectors_none
+ || vect_get_load_store_partial_vector_style (vectype, false)
+ == vect_load_store_partial_vectors_none)
{
/* Start with the biggest number of units that fits within
GROUP_SIZE and halve it until we find a valid vector type.
@@ -13661,8 +13675,32 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree
scalar_type,
nunits /= 2;
}
while (nunits > 1 && !vectype);
+
+ return vectype;
+ }
+
+ /* Interoperability between modes requires one to be a constant multiple of
+ the other, so do not try to substitute a narrower fixed-length vector type
+ for a variable-length vector type. Do not try to minimize the number of
+ elements in a variable-length type either because it can increase the size
+ of vector constants (e.g., if QImode values are stored in HImode). */
+ if (!natural_nelts.is_constant ())
+ return vectype;
+
+ /* Try to minimize the number of subparts in the partial vector type. Start
+ with the smallest number of units that contains GROUP_SIZE and double it
+ until we find a valid vector type. */
+ unsigned int nunits = 1 << ceil_log2 (group_size);
+ do
+ {
+ vectype = get_related_vectype_for_scalar_type (vinfo->vector_mode,
+ scalar_type, nunits);
+ nunits *= 2;
}
+ while (known_le (nunits, natural_nelts) && !vectype);
+ /* We should have rediscovered the natural vector type if nothing else. */
+ gcc_assert (vectype != NULL);
return vectype;
}
@@ -13682,10 +13720,10 @@ get_vectype_for_scalar_type (vec_info *vinfo, tree
scalar_type, slp_tree node)
/* Function get_mask_type_for_scalar_type.
Returns the mask type corresponding to a result of comparison
- of vectors of specified SCALAR_TYPE as supported by target.
- If GROUP_SIZE is nonzero and we're performing BB vectorization,
- make sure that the number of elements in the vector is no bigger
- than GROUP_SIZE. */
+ of vectors of specified SCALAR_TYPE as supported by target or NULL_TREE if
+ none. If GROUP_SIZE is nonzero and we're performing BB vectorization,
ensure
+ that the returned mask type is suitable for use with the vector type that
+ would be chosen by get_vectype_for_scalar_type. */
tree
get_mask_type_for_scalar_type (vec_info *vinfo, tree scalar_type,
@@ -14661,9 +14699,10 @@ vect_gen_while_not (gimple_seq *seq, tree mask_type,
tree start_index,
/* Try to compute the vector types required to vectorize STMT_INFO,
returning true on success and false if vectorization isn't possible.
- If GROUP_SIZE is nonzero and we're performing BB vectorization,
- take sure that the number of elements in the vectors is no bigger
- than GROUP_SIZE.
+ If GROUP_SIZE is nonzero and we're performing BB vectorization, ensure that
+ the number of subparts in *STMT_VECTYPE_OUT is no bigger than GROUP_SIZE, or
+ no smaller than GROUP_SIZE if a vector mask or length can be used to limit
+ the number of lanes loaded and stored.
On success:
--
2.43.0
