This patch teaches the vectorizer how to deal with early breaks without needing
a scalar loop.
The previous patch on this split loops into two, loops that have side-effects
and loops that do not. For loops that do not we would then avoid the scalar
loop.
This patch continues that direction but distinguishes in the type of side-effect
and if anything happens that we can't handle we force the use of the scalar
loop.
This patch has a few limitations, mainly because it was getting too big.
- No support for reductions, These need materialization changes.
- Same for stores.
But the code is set up to handle these, but needs more codegen changes.
- No support for ncopies, when a compare or side-effectful statement needs
multiple copies we can't tell which compare the match was found it. Artemiy's
patches are intended to fix this so we can support it.
- Only support for a single early exit. Supporting multiple exits require
additional support in ifcvt, which at the moment does not support early break.
I also have questions about profitability so for now this is GCC 18 material.
The patch attempts to keep as much as posible in analysis. every gcond is
attached to an SSA name representing its mask result and these are filled in by
vectorizable_early_exit.
The bulk of the changes are in vect_update_ivs_after_vectorizer_for_early_breaks
which is rewritten to update the IV PHI nodes without needing the lock step
iteration. If we are performing a reduction based on the IV it is also in
charge of generating the live out IV values. for instance on return i it
handles this such that vectorizable_live_operation doesn't need to get
specific IV handling code.
At the moment it handles on linear ADD based IVs, and the rest, including
non-linear IVs are forced to be vectorized and handled through value extraction
in vectorizable_live_operation.
Peeling is also updated to handle that if we don't need the scalar loop, we
have no merge blocks and so our dominators are different.
There is one akwardness. Because we have to force the IV live, it means we have
to analyze it (as we do today), but it also means that sometims the live out
value seen by vect_update_ivs_after_vectorizer_for_early_breaks is not the same
as that seen by vectorizable_live_operation and the latter then override the
former. This happens specifically when there is a cast in the IV value. i.e.
if return i, but i is an short but the function returns an int.
Since we don't really have SLP instances for the exit blocks, this code
handles it by marking the cast operation as we see them as an early break
dependency such that we can peek through them during codegen.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
-O3, profiled, profiled -O3 and -O3 -LTO bootstrap tested and no issues.
Also SPECCPU 2017 and SPECCPU 2026 tested.
Any feedback?
Thanks,
Tamar
gcc/ChangeLog:
* tree-vect-loop-manip.cc (adjust_debug_stmts): Add an optional
edge argument and update debug uses on that edge.
(iv_phi_p): Expose.
(vect_do_peeling): Move where we set LOOP_VINFO_EARLY_BRK_NITERS_VAR.
* tree-vect-loop.cc (loop_vec_info::_loop_vec_info): Initialize
early-break live-out state has_live_stmts and early_break_dependencies.
(_loop_vec_info::~_loop_vec_info): Release it.
(vect_create_epilog_for_reduction): Record live statements that
require early-break handling.
(vectorizable_induction): Handle early-break live IVs without
forcing a scalar epilogue.
(vectorizable_live_operation): Extract the first active live value
when no scalar epilogue is needed.
(move_early_exit_stmts): Handle inserted early-break live-value
code.
(vect_update_ivs_after_vectorizer_for_early_breaks): Update live
IV PHIs for no-epilogue early breaks.
(vect_transform_loop): Initialize early-break conditions before
updating live IVs.
* tree-vect-stmts.cc (vect_stmt_relevant_p): Mark early-break live
statements.
(vectorizable_operation): Handle early-break live operations.
(vectorizable_condition): Likewise.
(vectorizable_comparison_1): Return the number of comparison
vectors.
(vectorizable_comparison): Use it.
(vectorizable_early_exit): Record the vector condition for live-out
extraction.
* tree-vectorizer.h (LOOP_VINFO_HAS_LIVE_STMT_P): New macro.
(LOOP_VINFO_EARLY_BREAKS_IV_P): Likewise.
(LOOP_VINFO_EARLY_BRK_DEPS): Likewise.
(vect_has_early_break_live_outs): Declare.
(vect_need_early_break_cond_p): Likewise.
(adjust_phi_and_debug_stmts): Likewise.
(iv_phi_p): Likewise.
gcc/testsuite/ChangeLog:
* gcc.dg/vect/vect-early-break-no-epilog_3.c: Update expected
vectorization.
* gcc.dg/vect/vect-early-break-no-epilog_4.c: Likewise.
* gcc.dg/vect/vect-early-break-no-epilog_5.c: Likewise.
* gcc.dg/vect/vect-early-break-no-epilog_6.c: Likewise.
* gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c: New test.
* gcc.dg/vect/vect-early-break-nonlinear-live-1.c: New test.
* gcc.dg/vect/vect-early-break-nonlinear-live-2.c: New test.
* gcc.dg/vect/vect-live-slp-1.c: Update expected vectorization.
* gcc.dg/vect/vect-live-slp-2.c: Likewise.
* gcc.dg/vect/vect-live-slp-3.c: Likewise.
* gcc.dg/vect/vect-uncounted-run_1.c: Likewise.
* gcc.dg/vect/vect-uncounted_1.c: Likewise.
---
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
index
dfb398da9b7881f699b2ce824c8c501c4768a082..633f5339be8612b2c3b30a6fcf3cb55f3b790422
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
@@ -18,4 +18,5 @@ int foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" {
target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
index
99bb66f9fd05afd6145a7085b97d0df4d75d1f68..a2155015b1aff64795aa4d4e645e17593784f47a
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
@@ -18,4 +18,5 @@ int foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" {
target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
index
ec3eec5cfa137f4f99c4320bda525127b7945629..d5585ceca23ee1736132b27306eb897510ac369f
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
@@ -18,4 +18,5 @@ short foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" {
target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
index
46d6a8953cfebc2527a8b420bbec8c4079a4aa5d..c89272d3f4471d024e42a8c9efa467f5d0ed9442
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
@@ -18,4 +18,5 @@ short foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" {
target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog"
"vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c
new file mode 100644
index
0000000000000000000000000000000000000000..3db1a4ce34b578ab9a71799e5468460a7188bce4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c
@@ -0,0 +1,59 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_early_break_hw } */
+/* { dg-require-effective-target vect_int } */
+
+#include <assert.h>
+#include "tree-vect.h"
+
+#define N 16
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ for (int i = 0; i < N; ++i)
+ {
+ if (haystack[i] == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ }
+
+ return 0;
+}
+
+#define CHECK_MATCH(POS) \
+ void \
+ check_match_ ## POS (void) \
+ { \
+ int input[N] __attribute__((aligned (16))) = { \
+ [0 ... N-1] = 0 \
+ }; \
+ input[POS] = VAL; \
+ unsigned long res = foo (input, VAL); \
+ assert (res == (1UL << POS)); \
+ }
+
+CHECK_MATCH (0)
+CHECK_MATCH (3)
+CHECK_MATCH (15)
+
+#undef CHECK_MATCH
+#define CHECK_MATCH(POS) check_match_ ## POS ()
+
+int
+main (void)
+{
+ check_vect ();
+
+ CHECK_MATCH (0);
+ CHECK_MATCH (3);
+ CHECK_MATCH (15);
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c
new file mode 100644
index
0000000000000000000000000000000000000000..45971880c2d0c3ef247a40b3d7ccb7fd03a5febc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c
@@ -0,0 +1,27 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+#define N 16
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ for (int i = 0; i < N; ++i)
+ {
+ if (haystack[i] == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+,
vect_vec_iv_} "vect" { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c
b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c
new file mode 100644
index
0000000000000000000000000000000000000000..71e9bf0e8cc5510dda3cfd675c00b9451d36d8df
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c
@@ -0,0 +1,25 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ while (1)
+ {
+ if (*haystack == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ haystack++;
+ }
+}
+
+/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } }
*/
+/* { dg-final { scan-tree-dump "Peeling for epilogue is not supported for this
nonlinear induction" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
b/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
index
6270c11e025ed6e181c7a607da7b1b4fbe82b325..93af3e37a157221bc35ef2defaf04ed0d8d1f806
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
@@ -73,4 +73,5 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" }
} */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 4 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 12 "vect" { target { i?86-*-* x86_64-*-* } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
b/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
index
c9987018e88b04f5f0ff195baaf528ad86722714..9512565482cea9a75d32cd28e773c09bd5cc081c
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
@@ -67,4 +67,5 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 2 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" }
} */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 2 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 2 "vect" { target { i?86-*-* x86_64-*-* } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
b/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
index
3caeea71ad7f8858e20a1551b8a0eef573366e96..49ed96cdf0e14e7338c91e04f583fe5943076a0e
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
@@ -74,4 +74,6 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" }
} */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 4 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 12 "vect" { target { { i?86-*-* x86_64-*-* } && ilp32 } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 8 "vect" { target { { i?86-*-* x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not
relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
b/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
index
75a8ed1372ff2eaff60ccb6eb1d3242f720452b4..ca3fde85f274b81b89fe1e1bcd34d9489d9eba14
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
@@ -1,5 +1,4 @@
-/* Check we can derive scalar IV return value from vectorized IV value,
- resetting it on entry into the scalar epilogue loop via BIT_FIELD_REF. */
+/* Check we can derive scalar IV return value from the vectorized loop. */
/* { dg-add-options vect_early_break } */
/* { dg-require-effective-target vect_early_break_hw } */
/* { dg-require-effective-target vect_int } */
@@ -51,6 +50,8 @@ main ()
/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } }
*/
/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
-/* Ensure we reset our scalar IV so as to repeat the last vector iteration. */
-/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+,
[0-9]+, 0>} "vect" } } */
-
+/* AArch64 extracts the IV value directly from the first active vector lane.
*/
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+,
vect_i_[0-9_.]+\)} "vect" { target aarch64*-*-* } } } */
+/* Other targets still reset the scalar IV to repeat the last vector
+ iteration in the scalar epilogue loop. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+,
[0-9]+, 0>} "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
b/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
index
e5bfbc64527d49795cd25b42717eb3f7b31a50b8..1aa2f0baa6c3c4c9170f4b654693ad18a4b4bf2a
100644
--- a/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
@@ -1,5 +1,4 @@
-/* Check we can derive scalar IV return value from vectorized IV value,
- resetting it on entry into the scalar epilogue loop via BIT_FIELD_REF. */
+/* Check we can derive scalar IV return value from the vectorized loop. */
/* { dg-add-options vect_early_break } */
/* { dg-do compile } */
/* { dg-require-effective-target vect_early_break } */
@@ -19,6 +18,8 @@ foo (int *haystack, int needle)
/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } }
*/
/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
-/* Ensure we reset our scalar IV so as to repeat the last vector iteration. */
-/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+,
[0-9]+, 0>} "vect" } } */
-
+/* AArch64 extracts the IV value directly from the first active vector lane.
*/
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+,
vect_i_[0-9_.]+\)} "vect" { target aarch64*-*-* } } } */
+/* Other targets still reset the scalar IV to repeat the last vector
+ iteration in the scalar epilogue loop. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+,
[0-9]+, 0>} "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
index
b54353399d9ed6454442c93750b42924513178f5..1d4586052888aec3ed412351ebfe5169c9662f7a
100644
--- a/gcc/tree-vect-loop-manip.cc
+++ b/gcc/tree-vect-loop-manip.cc
@@ -249,7 +249,7 @@ adjust_debug_stmts (tree from, tree to, basic_block bb)
presumably non-loop-closed references left over from other
transformations. */
-static void
+void
adjust_phi_and_debug_stmts (gimple *update_phi, edge e, tree new_def)
{
tree orig_def = PHI_ARG_DEF_FROM_EDGE (update_phi, e);
@@ -2359,7 +2359,7 @@ find_loop_location (class loop *loop)
/* Return true if the phi described by STMT_INFO defines an IV of the
loop to be vectorized. */
-static bool
+bool
iv_phi_p (stmt_vec_info stmt_info)
{
gphi *phi = as_a <gphi *> (stmt_info->stmt);
@@ -3984,6 +3984,7 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters,
tree nitersm1,
gimple_call_set_lhs (tmp_call, tmp_niters_vf);
auto header_gsi = gsi_after_labels (loop->header);
gsi_insert_after (&header_gsi, tmp_call, GSI_SAME_STMT);
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
if (!(LOOP_VINFO_NITERS_UNCOUNTED_P (loop_vinfo)
&& get_loop_exit_edges (loop).length () == 1)
@@ -4006,8 +4007,6 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters,
tree nitersm1,
}
if (LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
vector_iters_vf = tmp_niters_vf;
-
- LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
}
bool recalculate_peel_niters_init
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index
4a04923814224e32bc94dfd3fe6736fe8edaf800..82658c7b13d13c8ec32984a4cac1c6086e14f953
100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -762,6 +762,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared)
partial_load_store_bias (0),
peeling_for_gaps (false),
peeling_for_niter (false),
+ has_live_stmts (false),
early_breaks (false),
loop_iv_cond (NULL),
user_unroll (false),
@@ -777,6 +778,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in,
vec_info_shared *shared)
vec_epilogue_loop_main_exit (NULL),
scalar_loop_main_exit (NULL),
early_break_needs_epilogue (false),
+ early_break_dependencies (NULL),
early_break_niters_var (NULL)
{
/* CHECKME: We want to visit all BBs before their successors (except for
@@ -857,6 +859,7 @@ _loop_vec_info::~_loop_vec_info ()
delete scan_map;
delete scalar_costs;
delete vector_costs;
+ delete early_break_dependencies;
for (auto reduc_info : reduc_infos)
delete reduc_info;
@@ -5524,13 +5527,13 @@ vect_create_epilog_for_reduction (loop_vec_info
loop_vinfo,
def = vect_get_slp_vect_def (slp_node, i);
tree new_def = copy_ssa_name (def);
phi = create_phi_node (new_def, exit_bb);
+ if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ for (unsigned k = 0; k < gimple_phi_num_args (phi); k++)
+ SET_PHI_ARG_DEF (phi, k, def);
+
if (LOOP_VINFO_MAIN_EXIT (loop_vinfo) == loop_exit)
SET_PHI_ARG_DEF (phi, loop_exit->dest_idx, def);
- else
- {
- for (unsigned k = 0; k < gimple_phi_num_args (phi); k++)
- SET_PHI_ARG_DEF (phi, k, def);
- }
+
new_def = gimple_convert (&stmts, vectype, new_def);
reduc_inputs.quick_push (new_def);
gsi_insert_seq_before (&exit_gsi, stmts, GSI_SAME_STMT);
@@ -9679,6 +9682,13 @@ vectorizable_induction (loop_vec_info loop_vinfo,
if (cost_vec) /* transformation not required. */
{
unsigned inside_cost = 0, prologue_cost = 0;
+
+ /* We can't support unpacked IVs yet without a scalar loop. */
+ if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && nvects > 1
+ && !LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, stmt_info))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
/* We eventually need to set a vector type on invariant
arguments. */
unsigned j;
@@ -10185,7 +10195,6 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
tree vectype = SLP_TREE_VECTYPE (slp_node);
poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
gimple *use_stmt;
- use_operand_p use_p;
auto_vec<tree> vec_oprnds;
int vec_entry = 0;
poly_uint64 vec_index = 0;
@@ -10198,6 +10207,11 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
validity so just trigger the transform here. */
if (vect_is_reduction (slp_node))
{
+ /* No support for reductions at the moment as they need some
+ materialization changes. */
+ if (!LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
if (!vec_stmt_p)
{
SLP_TREE_LIVE_LANES (slp_node).safe_push (slp_index);
@@ -10219,8 +10233,11 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
slp_node_instance,
LOOP_VINFO_MAIN_EXIT (loop_vinfo));
- /* If early break we only have to materialize the reduction on the merge
- block, but we have to find an alternate exit first. */
+ /* If early break and we are using a scalar epilog we only have to
+ materialize the reduction on the merge block, but we have to find
+ an alternate exit first. But if early break and we don't need the
+ epilog the reduction needs to be materialized in every exit along with
+ a scoped final reduction. */
if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
{
slp_tree phis_node = slp_node_instance->reduc_phis;
@@ -10231,7 +10248,8 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
vect_create_epilog_for_reduction (loop_vinfo, stmt_info,
phis_node, slp_node_instance,
exit);
- break;
+ if (LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ break;
}
if (LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
vect_create_epilog_for_reduction (loop_vinfo, stmt_info,
@@ -10264,6 +10282,41 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
int num_scalar = SLP_TREE_LANES (slp_node);
int num_vec = vect_get_num_copies (vinfo, slp_node);
poly_uint64 pos = (num_vec * nunits) - num_scalar + slp_index;
+ bool live_early_break_iv_p
+ = LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, stmt_info);
+
+ /* ??? The live value can be a cast of the scalar IV rather than the IV
+ itself.
+ The early-break IV update code below looks through such casts when it
+ rewrites the exit PHIs, but the analysis here still needs to know that the
+ live statement is covered by that repair. Otherwise the cast looks like a
+ normal live value with multiple vector results and forces scalar epilog.
+
+ This is deliberately restricted to fixed-length vectors. For scalable
+ vectors, the regular live-value path can use the target's native
+ extraction sequence, and keeping that path avoids replacing the better
+ variable-length sequence. */
+ if (!live_early_break_iv_p
+ && loop_vinfo
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && is_gimple_assign (STMT_VINFO_STMT (stmt_info))
+ && gimple_assign_cast_p (STMT_VINFO_STMT (stmt_info))
+ && TYPE_VECTOR_SUBPARTS (vectype).is_constant ()
+ && SLP_TREE_CHILDREN (slp_node).length () == 1)
+ {
+ slp_tree child = SLP_TREE_CHILDREN (slp_node)[0];
+ stmt_vec_info child_info = SLP_TREE_REPRESENTATIVE (child);
+ if (child_info
+ && STMT_VINFO_DEF_TYPE (child_info) == vect_induction_def
+ && (STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (child_info)
+ == vect_step_op_add))
+ {
+ if (!LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, child_info))
+ LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).safe_push
+ (child_info);
+ live_early_break_iv_p = true;
+ }
+ }
/* Calculate which vector contains the result, and which lane of
that vector we need. */
@@ -10276,10 +10329,36 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
return false;
}
+ /* For grouped loads without a load permutation, lanes beyond
+ SLP_TREE_LANES are the other elements loaded from the group, rather than
+ further scalar iterations of STMT_INFO. For example, in:
+
+ x = p[2 * i];
+ y = p[2 * i + 1];
+
+ without a permute the vector elements are [x0, y0, x1, y1, ...],
+ so vec_index + 1 does not necessarily refer to the next element of
+ STMT_INFO. */
+ if (STMT_VINFO_GROUPED_ACCESS (stmt_info)
+ && !SLP_TREE_LOAD_PERMUTATION (slp_node).exists ()
+ && known_ge (vec_index, (unsigned) num_scalar))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "Cannot extract grouped-access padding lane for"
+ " live stmt.\n");
+ return false;
+ }
+
if (!vec_stmt_p)
{
/* No transformation required. */
- if (loop_vinfo && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ /* Early-break live IVs are handled by early exit IV handling code which
+ determines how the IV should be extracted. As such even though it's
+ been forced live, do not vectorize it as a normal live operation. */
+ if (loop_vinfo
+ && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
+ && !live_early_break_iv_p)
{
if (SLP_TREE_LANES (slp_node) != 1)
{
@@ -10322,6 +10401,41 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
}
}
}
+
+ /* Check if we have support for the right optabs to not require the
scalar
+ loop. */
+ if (loop_vinfo
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ {
+ if (!live_early_break_iv_p
+ && !direct_internal_fn_supported_p (IFN_EXTRACT_FIRST,
+ vectype, OPTIMIZE_FOR_BOTH))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live value because "
+ "target does not support extract first "
+ "operations for type %T\n", vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+
+ /* Unpacking cannot be supported yet without a scalar loop. */
+ if (num_vec > 1
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && !live_early_break_iv_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live value because "
+ "unpacking live values not currently supported by "
+ "EXTRACT_FIRST operations. %T\n", vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+ }
+
/* ??? Enable for loop costing as well. */
if (!loop_vinfo)
record_stmt_cost (cost_vec, 1, vec_to_scalar, slp_node,
@@ -10378,46 +10492,183 @@ vectorizable_live_operation (vec_info *vinfo,
stmt_vec_info stmt_info,
rather than at the end. */
edge main_e = LOOP_VINFO_MAIN_EXIT (loop_vinfo);
bool all_exits_as_early_p = LOOP_VINFO_EARLY_BREAKS_VECT_PEELED
(loop_vinfo);
+ /* Collect the exit PHIs before building replacement code. The builders
+ below can simplify the new statements, and simplification is allowed
+ to inspect SSA uses. Doing that while FOR_EACH_IMM_USE_STMT has an
+ active iterator for LHS would trip the checking in
+ first_imm_use_stmt. */
+ auto_vec<std::pair<gphi *, unsigned>, 4> exit_phi_args;
FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, lhs)
if (!is_gimple_debug (use_stmt)
&& !flow_bb_inside_loop_p (loop, gimple_bb (use_stmt)))
- FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
+ {
+ use_operand_p use_p;
+ FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
+ {
+ exit_phi_args.safe_push
+ (std::make_pair (as_a <gphi *> (use_stmt),
+ PHI_ARG_INDEX_FROM_USE (use_p)));
+ break;
+ }
+ }
+
+ for (auto phi_arg : exit_phi_args)
+ {
+ gphi *use_phi = phi_arg.first;
+ unsigned int i = phi_arg.second;
+ edge e = gimple_phi_arg_edge (use_phi, i);
+ edge phi_e = e;
+
+ /* A previous live SLP lane can already have split the loop exit
+ edge, so this PHI argument can come from the inserted block
+ rather than directly from the loop. Use the inserted block as
+ the materialization point, but update the PHI argument on PHI_E
+ instead of replacing the whole PHI. */
+ if (!loop_exit_edge_p (loop, e))
+ {
+ if (!single_pred_p (e->src))
+ continue;
+ e = single_pred_edge (e->src);
+ if (!loop_exit_edge_p (loop, e))
+ continue;
+ }
+ bool update_phi_arg_p = phi_e != e;
+ bool main_exit_edge = e == main_e;
+ tree tmp_vec_lhs = vec_lhs;
+ tree tmp_bitstart = bitstart;
+
+ gimple_stmt_iterator exit_gsi;
+ tree new_tree = NULL_TREE;
+
+ /* For early exit where the exit is not in the BB that leads
+ to the latch then we're restarting the iteration in the
+ scalar loop. So get the first live value. */
+ bool early_break_first_element_p
+ = all_exits_as_early_p || !main_exit_edge;
+
+ auto gsi = gsi_for_stmt (use_phi);
+ if (vect_need_early_break_cond_p (loop_vinfo)
+ && early_break_first_element_p)
{
- edge e = gimple_phi_arg_edge (as_a <gphi *> (use_stmt),
- phi_arg_index_from_use (use_p));
- gcc_assert (loop_exit_edge_p (loop, e));
- bool main_exit_edge = e == main_e;
- tree tmp_vec_lhs = vec_lhs;
- tree tmp_bitstart = bitstart;
-
- /* For early exit where the exit is not in the BB that leads
- to the latch then we're restarting the iteration in the
- scalar loop. So get the first live value. */
- bool early_break_first_element_p
- = all_exits_as_early_p || !main_exit_edge;
- if (early_break_first_element_p)
+ auto cond_bb = e->src;
+ tree *slot
+ = LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (cond_bb);
+ if (slot)
{
- tmp_vec_lhs = vec_lhs0;
- tmp_bitstart = build_zero_cst (TREE_TYPE (bitstart));
+ tree cmp_op = *slot;
+
+ gimple_seq stmts = NULL;
+ /* If the live value is a simple vector load, reload the
+ first active scalar element directly on the exit edge.
+ This avoids an EXTRACT_FIRST from a vector register on
+ targets where INDEX_FIRST_ACTIVE is cheap. */
+ data_reference *dr = STMT_VINFO_DATA_REF (stmt_info);
+ gassign *load = NULL;
+ if (single_pred_p (e->dest)
+ && direct_internal_fn_supported_p
+ (IFN_INDEX_FIRST_ACTIVE, TREE_TYPE (cmp_op),
+ OPTIMIZE_FOR_BOTH)
+ && dr
+ && TREE_CODE (tmp_vec_lhs) == SSA_NAME
+ && (load = dyn_cast<gassign *>
+ (SSA_NAME_DEF_STMT (tmp_vec_lhs)))
+ && gimple_bb (load) == single_pred (e->dest)
+ && gimple_assign_rhs_code (load) == MEM_REF)
+ {
+ dr_vec_info *dr_info = STMT_VINFO_DR_INFO (stmt_info);
+ tree vec_ref = gimple_assign_rhs1 (load);
+ tree scalar_type = TREE_TYPE (DR_REF (dr));
+ tree ptr = TREE_OPERAND (vec_ref, 0);
+ tree reload_vuse = gimple_vuse (load);
+ if (gphi *vphi = get_virtual_phi (e->dest))
+ reload_vuse
+ = (gimple_phi_arg_def_from_edge (vphi, e)
+ == reload_vuse) ? gimple_phi_result (vphi)
+ : NULL_TREE;
+
+ if (reload_vuse)
+ {
+ tree first = gimple_build (&stmts,
+ IFN_INDEX_FIRST_ACTIVE,
+ unsigned_intDI_type_node,
+ cmp_op);
+ tree offset = gimple_convert (&stmts, sizetype,
+ first);
+ offset
+ = gimple_build (&stmts, MULT_EXPR, sizetype, offset,
+ size_int (vect_get_scalar_dr_size
(dr_info)));
+
+ tree scalar_ptr_type = build_pointer_type
(scalar_type);
+ tree scalar_ptr = gimple_convert (&stmts,
+ scalar_ptr_type,
+ ptr);
+ tree addr = gimple_build (&stmts, POINTER_PLUS_EXPR,
+ scalar_ptr_type,
+ scalar_ptr, offset);
+ tree scalar_ref
+ = build2 (MEM_REF, scalar_type, addr,
+ unshare_expr (TREE_OPERAND (vec_ref, 1)));
+ vect_copy_ref_info (scalar_ref, DR_REF (dr));
+
+ tree scalar
+ = make_temp_ssa_name (scalar_type, NULL, "first");
+ gassign *reload
+ = gimple_build_assign (scalar, scalar_ref);
+ gimple_set_vuse (reload, reload_vuse);
+ gimple_seq_add_stmt (&stmts, reload);
+
+ new_tree = gimple_convert (&stmts, lhs_type, scalar);
+ }
+ }
+ if (!new_tree)
+ {
+ /* Convert the extracted vector element to the scalar
+ type. */
+ tree tmp = gimple_build (&stmts, CFN_EXTRACT_FIRST,
+ TREE_TYPE (vectype),
+ cmp_op, tmp_vec_lhs);
+ new_tree = gimple_convert (&stmts, lhs_type, tmp);
+ }
+
+ if (!single_pred_p (e->dest))
+ {
+ basic_block new_bb
+ = gsi_insert_seq_on_edge_immediate (e, stmts);
+ edge update_e = new_bb ? single_succ_edge (new_bb) : e;
+ adjust_phi_and_debug_stmts (use_phi, update_e, new_tree);
+ continue;
+ }
+ exit_gsi = gsi_after_labels (e->dest);
+ gsi_insert_seq_before (&exit_gsi, stmts, GSI_SAME_STMT);
}
+ }
+ else if (early_break_first_element_p)
+ {
+ tmp_vec_lhs = vec_lhs0;
+ tmp_bitstart = build_zero_cst (TREE_TYPE (bitstart));
+ }
- gimple_stmt_iterator exit_gsi;
- tree new_tree
- = vectorizable_live_operation_1 (loop_vinfo,
- e->dest, vectype,
- slp_node, bitsize,
- tmp_bitstart, tmp_vec_lhs,
- lhs_type, &exit_gsi);
-
- auto gsi = gsi_for_stmt (use_stmt);
- tree lhs_phi = gimple_phi_result (use_stmt);
- remove_phi_node (&gsi, false);
- gimple *copy = gimple_build_assign (lhs_phi, new_tree);
- gsi_insert_before (&exit_gsi, copy, GSI_SAME_STMT);
- break;
+ if (!new_tree)
+ new_tree
+ = vectorizable_live_operation_1 (loop_vinfo,
+ e->dest, vectype,
+ slp_node, bitsize,
+ tmp_bitstart, tmp_vec_lhs,
+ lhs_type, &exit_gsi);
+
+ if (update_phi_arg_p)
+ {
+ adjust_phi_and_debug_stmts (use_phi, phi_e, new_tree);
+ continue;
}
- /* There a no further out-of-loop uses of lhs by LC-SSA construction. */
+ tree lhs_phi = gimple_phi_result (use_phi);
+ remove_phi_node (&gsi, false);
+ gimple *copy = gimple_build_assign (lhs_phi, new_tree);
+ gsi_insert_before (&exit_gsi, copy, GSI_SAME_STMT);
+ }
+
+ /* There are no further out-of-loop uses of lhs by LC-SSA construction.
*/
FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, lhs)
gcc_assert (is_gimple_debug (use_stmt)
|| flow_bb_inside_loop_p (loop, gimple_bb (use_stmt)));
@@ -11096,30 +11347,72 @@ move_early_exit_stmts (loop_vec_info loop_vinfo)
SET_PHI_ARG_DEF_ON_EDGE (phi, e, last_seen_vuse);
}
-/* Generate adjustment code for early break scalar IVs filling in the value
- we created earlier on for LOOP_VINFO_EARLY_BRK_NITERS_VAR. */
+/* Initialize the SSA names used to communicate the vectorized early-exit
+ conditions to later no-epilogue fixups. Generate adjustment code for early
+ break scalar IVs filling in the value we created earlier on for
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR. */
static void
vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info loop_vinfo)
{
DUMP_VECT_SCOPE ("vect_update_ivs_after_vectorizer_for_early_breaks");
- if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
- /* If no peeling was done then we have no IV to update. */
- || !LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo))
+ if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ return;
+
+ bool needs_live_iv
+ = (!LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ()
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo));
+ bool needs_early_break_conds = vect_need_early_break_cond_p (loop_vinfo);
+
+ if (needs_early_break_conds)
+ {
+ gcc_assert (LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo));
+
+ for (gcond *gcond : LOOP_VINFO_LOOP_CONDS (loop_vinfo))
+ {
+ tree *slot
+ = LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (gimple_bb (gcond));
+ gcc_assert (slot && TYPE_P (*slot));
+
+ tree cmp_op = make_temp_ssa_name (*slot, NULL, "vexit");
+ gcall *varying = gimple_build_call_internal (IFN_VARYING, 0);
+ gimple_call_set_lhs (varying, cmp_op);
+
+ gimple_stmt_iterator gsi = gsi_for_stmt (gcond);
+ gsi_insert_before (&gsi, varying, GSI_SAME_STMT);
+
+ *slot = cmp_op;
+ }
+ }
+
+ if (!LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) && !needs_live_iv)
return;
+ /* Create early break IV variable if it will be needed by either peeling or
+ a return value. */
+ if (!LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo))
+ {
+ tree tmp_niters_vf
+ = make_ssa_name (LOOP_VINFO_EARLY_BRK_IV_TYPE (loop_vinfo));
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
+ }
+ else
+ {
+ /* Remove the existing dummy GIMPLE statement and just keep the def. */
+ tree tmp = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo);
+ gimple *def = SSA_NAME_DEF_STMT (tmp);
+ auto def_gsi = gsi_for_stmt (def);
+ gsi_remove (&def_gsi, true);
+ }
+
tree phi_var = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo);
tree niters_skip = LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo);
tree ty_var = TREE_TYPE (phi_var);
auto loop = LOOP_VINFO_LOOP (loop_vinfo);
+ edge pe = loop_preheader_edge (loop);
tree induc_var = niters_skip ? copy_ssa_name (phi_var) : phi_var;
- /* Remove the existing dummy GIMPLE statement and just keep the def. */
- gimple *def = SSA_NAME_DEF_STMT (phi_var);
- auto def_gsi = gsi_for_stmt (def);
- gsi_remove (&def_gsi, true);
-
auto induction_phi = create_phi_node (induc_var, loop->header);
tree induc_def = PHI_RESULT (induction_phi);
@@ -11146,11 +11439,188 @@ vect_update_ivs_after_vectorizer_for_early_breaks
(loop_vec_info loop_vinfo)
add_phi_arg (induction_phi, iter_var,
loop_latch_edge (loop), UNKNOWN_LOCATION);
- add_phi_arg (induction_phi, init_var,
- loop_preheader_edge (loop), UNKNOWN_LOCATION);
+ add_phi_arg (induction_phi, init_var, pe, UNKNOWN_LOCATION);
- /* Find the first insertion point in the BB. */
- auto pe = loop_preheader_edge (loop);
+ /* Write the init_stmts in the loop-preheader block. */
+ auto psi = gsi_last_nondebug_bb (pe->src);
+ gsi_insert_seq_after (&psi, init_stmts, GSI_LAST_NEW_STMT);
+
+ /* Write the adjustments at the end of the iv increment. */
+ bool insert_after;
+ gimple_stmt_iterator incr_gsi;
+ vect_iv_increment_position (LOOP_VINFO_MAIN_EXIT (loop_vinfo), &incr_gsi,
+ &insert_after);
+
+ if (insert_after)
+ gsi_insert_seq_after (&incr_gsi, stmts, GSI_NEW_STMT);
+ else
+ gsi_insert_seq_before (&incr_gsi, stmts, GSI_NEW_STMT);
+
+ /* Check if we're staying within the vector loop, and if so rewrite the
+ conditions. This is only needed when live scalar IVs use the early-exit
+ edge: loops without such IVs can still be vectorized as early breaks, but
+ there is no per-exit IV value to repair. */
+ if (needs_live_iv)
+ {
+ for (auto gcond : LOOP_VINFO_LOOP_CONDS (loop_vinfo))
+ {
+ if (LOOP_VINFO_LOOP_IV_COND (loop_vinfo) == gcond
+ && !LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
+ continue;
+ auto bb = gcond->bb;
+ tree cmp_op = *LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (bb);
+ tree mask_type = TREE_TYPE (cmp_op);
+ /* Now rewrite the code to use the new IV for linear IVs. */
+ edge exit_e = EDGE_SUCC (bb, 0);
+ if (flow_bb_inside_loop_p (loop, exit_e->dest))
+ exit_e = EDGE_SUCC (bb, 1);
+
+ /* Split the edge so when we add the new LCSSA PHIs on the exit we
+ don't confuse things if the destination is shared with a versioning
+ block. */
+ basic_block update_bb = split_loop_exit_edge (exit_e);
+ gimple_stmt_iterator last_gsi = gsi_after_labels (update_bb);
+
+ /* Create the mask partitioning if the target cannot count the
+ position of the first active element directly. CMP_OP is the
+ already-masked exit predicate produced by vectorizable_early_exit,
+ so do not use the COND or COND_LEN forms here; applying loop
+ controls again would double-mask the value. The block is reached
+ only after taking the early-exit edge, so CMP_OP is known to have
at
+ least one active element. Targets can use that to avoid handling
+ the empty-predicate case.
+
+ This distinction lets targets choose either a direct first-active
+ operation or a partition-and-count sequence. */
+ internal_fn active_fn = IFN_INDEX_FIRST_ACTIVE;
+ tree active_mask = cmp_op;
+ if (!direct_internal_fn_supported_p (active_fn, mask_type,
+ OPTIMIZE_FOR_BOTH))
+ {
+ active_mask = make_temp_ssa_name (mask_type, NULL, "pred");
+ gcall *p_call
+ = gimple_build_call_internal (IFN_TAKE_EXCL_ACTIVE, 1,
+ cmp_op);
+ gimple_call_set_lhs (p_call, active_mask);
+ gsi_insert_before (&last_gsi, p_call, GSI_SAME_STMT);
+ active_fn = IFN_COUNT_ACTIVE;
+ }
+
+ gphi_iterator gsi1;
+ for (gsi1 = gsi_start_phis (update_bb); !gsi_end_p (gsi1);)
+ {
+ gphi *phi1 = gsi1.phi ();
+ tree arg = PHI_ARG_DEF_FROM_EDGE (phi1, exit_e);
+ if (TREE_CODE (arg) != SSA_NAME)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ /* Look through any promotion to find the original IV. This must
+ match the casted-live-IV recognition in
+ vectorizable_live_operation; otherwise the loop could be
+ accepted as no-epilogue during analysis but would not have
+ its exit PHI repaired here. */
+ gimple *def_stmt = NULL;
+ while (TREE_CODE (arg) == SSA_NAME
+ && (def_stmt = SSA_NAME_DEF_STMT (arg))
+ && is_gimple_assign (def_stmt)
+ && gimple_assign_cast_p (def_stmt))
+ arg = gimple_assign_rhs1 (def_stmt);
+
+ if (!def_stmt
+ || gimple_code (def_stmt) != GIMPLE_PHI
+ || gimple_bb (def_stmt) != loop->header)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ gphi *phi = as_a <gphi *> (def_stmt);
+ stmt_vec_info phi_info = loop_vinfo->lookup_stmt (phi);
+
+ /* Skip reduction and virtual phis. */
+ if (!phi_info || !iv_phi_p (phi_info))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "reduc or virtual phi. skip.\n");
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ tree type = TREE_TYPE (gimple_phi_result (phi));
+ tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (phi_info);
+ step_expr = unshare_expr (step_expr);
+
+ /* FORNOW: We do not support IVs whose evolution function is a
+ polynomial of degree >= 2 or exponential. */
+ gcc_assert (!tree_is_chrec (step_expr));
+
+ gimple_seq stmts = NULL;
+ enum vect_induction_op_type induction_type
+ = STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (phi_info);
+
+ /* We can only optimize linear inductions because the repair is
+ computed as INIT + (VECTOR_ITERS + FIRST_ACTIVE_LANE) * STEP.
+ Non-linear inductions must keep using the existing vector live
+ paths, which can still fall back to a scalar epilogue. */
+ if (induction_type != vect_step_op_add)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "rewriting live value for : phi: %G",
+ (gimple *) phi);
+
+ tree res = PHI_RESULT (phi1);
+ tree reduc_iv = gimple_build (&stmts, active_fn,
+ unsigned_intDI_type_node,
+ active_mask);
+
+ tree stype = TREE_TYPE (step_expr);
+ /* Use the raw vector-iteration count here rather than
+ PHI_VAR. With masked alignment peeling, the first vector
+ iteration starts at -skip so that FIRST_ACTIVE_LANE can
+ compensate for the inactive peeled elements. PHI_VAR is
+ the value after the ordinary peeling adjustment below has
+ clamped the negative count back to zero, which is correct
+ for the final iteration count but too late for repairing an
+ early-exit IV. */
+ tree niters
+ = fold_build2 (PLUS_EXPR, stype,
+ fold_convert (stype, induc_var),
+ fold_convert (stype, reduc_iv));
+ tree off = fold_build2 (MULT_EXPR, stype, niters, step_expr);
+
+ tree start = PHI_ARG_DEF_FROM_EDGE (phi, pe);
+ tree ni;
+ if (POINTER_TYPE_P (type))
+ ni = fold_build_pointer_plus (start, off);
+ else
+ ni = fold_convert
+ (stype, fold_build2 (PLUS_EXPR, stype,
+ fold_convert (stype, start), off));
+
+ gimple_seq new_stmts = NULL;
+ tree ni_name = force_gimple_operand
+ (ni, &new_stmts, false, create_tmp_var (type, "tmp_iv"));
+
+ gsi_insert_seq_before (&last_gsi, stmts, GSI_SAME_STMT);
+ gsi_insert_seq_before (&last_gsi, new_stmts, GSI_SAME_STMT);
+
+ auto res_g = gimple_build_assign
+ (res, fold_convert (TREE_TYPE (res), ni_name));
+ gsi_insert_before (&last_gsi, res_g, GSI_SAME_STMT);
+ remove_phi_node (&gsi1, false);
+ continue;
+ }
+ }
+ }
/* If we've done any peeling, calculate the peeling adjustment needed to the
final IV. */
@@ -11166,33 +11636,22 @@ vect_update_ivs_after_vectorizer_for_early_breaks
(loop_vec_info loop_vinfo)
gimple_convert (&iv_stmts, induc_type,
induc_def));
gimple_seq_add_stmt_without_update (&iv_stmts, stmt);
- basic_block exit_bb = NULL;
+ edge exit_e = NULL;
/* Identify the early exit merge block. I wish we had stored this. */
for (auto e : get_loop_exit_edges (loop))
if (e != LOOP_VINFO_MAIN_EXIT (loop_vinfo))
{
- exit_bb = e->dest;
+ exit_e = e;
break;
}
- gcc_assert (exit_bb);
+ gcc_assert (exit_e);
+ basic_block exit_bb = exit_e->dest;
+ if (!dominated_by_p (CDI_DOMINATORS, exit_bb, loop->header))
+ exit_bb = split_loop_exit_edge (exit_e);
auto exit_gsi = gsi_after_labels (exit_bb);
gsi_insert_seq_before (&exit_gsi, iv_stmts, GSI_SAME_STMT);
}
- /* Write the init_stmts in the loop-preheader block. */
- auto psi = gsi_last_nondebug_bb (pe->src);
- gsi_insert_seq_after (&psi, init_stmts, GSI_LAST_NEW_STMT);
-
- /* Write the adjustments at the end of the iv increment. */
- bool insert_after;
- gimple_stmt_iterator incr_gsi;
- vect_iv_increment_position (LOOP_VINFO_MAIN_EXIT (loop_vinfo), &incr_gsi,
- &insert_after);
-
- if (insert_after)
- gsi_insert_seq_after (&incr_gsi, stmts, GSI_NEW_STMT);
- else
- gsi_insert_seq_before (&incr_gsi, stmts, GSI_NEW_STMT);
}
/* Function vect_transform_loop.
@@ -11244,7 +11703,9 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple
*loop_vectorized_call)
/* Make sure there exists a single-predecessor exit bb. Do this before
versioning. */
edge e = LOOP_VINFO_MAIN_EXIT (loop_vinfo);
- if (! single_pred_p (e->dest) && !LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ if (! single_pred_p (e->dest)
+ && (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ || LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo)))
{
split_loop_exit_edge (e, true);
if (dump_enabled_p ())
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index
65d82d51310b48178af61ce62862ad7ffb814a5a..be170a298ae30ed90b34770781da7c428da9e611
100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -413,7 +413,7 @@ vect_stmt_relevant_p (stmt_vec_info stmt_info,
loop_vec_info loop_vinfo,
gcc_assert (gimple_code (USE_STMT (use_p)) == GIMPLE_PHI);
*live_p = true;
- LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo) = true;
}
}
}
@@ -427,6 +427,12 @@ vect_stmt_relevant_p (stmt_vec_info stmt_info,
loop_vec_info loop_vinfo,
*relevant = vect_used_only_live;
}
+ if (*live_p
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def
+ && !LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).contains (stmt_info))
+ LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).safe_push (stmt_info);
+
return (*live_p || *relevant);
}
@@ -6786,7 +6792,7 @@ vectorizable_operation (vec_info *vinfo,
vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
internal_fn cond_fn = get_conditional_internal_fn (code);
- internal_fn cond_len_fn = get_conditional_len_internal_fn (code);
+ internal_fn cond_len_fn = get_len_internal_fn (cond_fn);
/* If operating on inactive elements could generate spurious traps,
we need to restrict the operation to active lanes. Note that this
@@ -12385,13 +12391,16 @@ vectorizable_condition (vec_info *vinfo,
otherwise, vectorize STMT_INFO: create a vectorized comparison, and insert
it at GSI.
- Return true if STMT_INFO is vectorizable in this way. */
+ Return true if STMT_INFO is vectorizable in this way. If
+ COMPARISON_VECTYPE_OUT is nonnull, store the vector type of the comparison
+ inputs there. */
static bool
vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
stmt_vec_info stmt_info, tree_code code,
gimple_stmt_iterator *gsi,
- slp_tree slp_node, stmt_vector_for_cost *cost_vec)
+ slp_tree slp_node, stmt_vector_for_cost *cost_vec,
+ tree *comparison_vectype_out)
{
tree lhs, rhs1, rhs2;
tree vectype1 = NULL_TREE, vectype2 = NULL_TREE;
@@ -12450,6 +12459,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree
vectype,
&& (VECTOR_BOOLEAN_TYPE_P (vectype1) ^ VECTOR_BOOLEAN_TYPE_P (vectype2)))
return false;
+ if (comparison_vectype_out)
+ *comparison_vectype_out = vectype;
+
/* Boolean values may have another representation in vectors
and therefore we prefer bit operations over comparison for
them (which also works for scalar masks). We store opcodes
@@ -12613,7 +12625,7 @@ vectorizable_comparison (vec_info *vinfo,
enum tree_code code = gimple_assign_rhs_code (stmt);
tree vectype = SLP_TREE_VECTYPE (slp_node);
if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
- slp_node, cost_vec))
+ slp_node, cost_vec, NULL))
return false;
if (cost_vec)
@@ -12776,16 +12788,22 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
exit_true_edge->dest);
- /* See if we support ADDHN and use that for the reduction. */
+ /* See if we support ADDHN and use that for the branch reduction. Some
+ targets produce all-ones/all-zeroes lanes for boolean comparisons. ADDHN
+ can compress a 128-bit predicate into a 64-bit scalar cheaply enough for
+ an any-active zero test. This is only a branch check; if the exit is
+ taken and we need the exact lane index, the target-specific
+ INDEX_FIRST_ACTIVE or COUNT_ACTIVE path can still apply a lane mask later
+ in the exit block. */
internal_fn ifn = IFN_VEC_TRUNC_ADD_HIGH;
bool addhn_supported_p
- = direct_internal_fn_supported_p (ifn, vectype, OPTIMIZE_FOR_BOTH);
+ = (vec_num > 1
+ && direct_internal_fn_supported_p (ifn, vectype, OPTIMIZE_FOR_BOTH));
tree narrow_type = NULL_TREE;
if (addhn_supported_p)
{
- /* Calculate the narrowing type for the result. */
- auto halfprec = TYPE_PRECISION (TREE_TYPE (vectype)) / 2;
- auto unsignedp = TYPE_UNSIGNED (TREE_TYPE (vectype));
+ unsigned int halfprec = TYPE_PRECISION (TREE_TYPE (vectype)) / 2;
+ bool unsignedp = TYPE_UNSIGNED (TREE_TYPE (vectype));
tree itype = build_nonstandard_integer_type (halfprec, unsignedp);
tree tmp_type = build_vector_type (itype, TYPE_VECTOR_SUBPARTS
(vectype));
narrow_type = truth_type_for (tmp_type);
@@ -12794,10 +12812,10 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
TYPE_MODE (narrow_type)))
{
if (dump_enabled_p ())
- dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
- "can't use ADDHN reduction because cbranch for "
- "the narrowed type is not supported by the "
- "target.\n");
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't use ADDHN reduction because cbranch for "
+ "the narrowed type is not supported by the "
+ "target.\n");
addhn_supported_p = false;
}
}
@@ -12816,9 +12834,12 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
return false;
}
+ tree comparison_vectype = NULL_TREE;
if (!vectorizable_comparison_1 (loop_vinfo, vectype, stmt_info, code,
gsi,
- slp_node, cost_vec))
+ slp_node, cost_vec,
+ &comparison_vectype))
return false;
+ gcc_assert (comparison_vectype);
if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
{
@@ -12832,6 +12853,68 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
if (!vect_compute_type_for_early_break_scalar_iv (loop_vinfo))
return false;
+ tree mask_vectype = (VECTOR_BOOLEAN_TYPE_P (vectype)
+ ? vectype : truth_type_for (vectype));
+
+ if (!LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ()
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ {
+ /* The no-epilogue live-IV path needs the number of scalar
+ iterations executed by the vector loop before the early exit.
+ Targets can provide that directly as INDEX_FIRST_ACTIVE, or
+ provide a predicate for the lanes before the first active lane and
+ then count those lanes. This lets the vector loop update a live IV
+ without replaying the iteration in a scalar epilogue. */
+ if (!direct_internal_fn_supported_p (IFN_INDEX_FIRST_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)
+ && (!direct_internal_fn_supported_p (IFN_TAKE_EXCL_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)
+ || !direct_internal_fn_supported_p (IFN_COUNT_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live induction because "
+ "target does not support predicate position operations "
+ "for type %T\n", mask_vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+ else if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ LOOP_VINFO_MUST_USE_PARTIAL_VECTORS_P (loop_vinfo) = true;
+ }
+
+ /* The no-epilogue path uses the vector predicate to compute the exact
+ scalar iteration at which a live value is needed. When the early exit
+ produces more than one vector predicate, the branch condition is
+ currently a reduction of those predicates. That is enough to know that
+ some lane exited, but it loses which vector contained the first active
+ lane. Keep the scalar epilogue for both live statements and live IVs
in
+ that case until the no-epilogue path can preserve that information.
+
+ Use the number of data-lane vectors rather than the number of copies
+ of the boolean result. */
+ poly_uint64 cmp_units
+ = LOOP_VINFO_VECT_FACTOR (loop_vinfo) * SLP_TREE_LANES (slp_node);
+ unsigned int cmp_vec_num = vect_get_num_vectors (cmp_units,
+ comparison_vectype);
+ if (cmp_vec_num > 1
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
+ if (!LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo))
+ {
+ if (!LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)
+ = new hash_map<basic_block, tree>;
+ LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->put (bb, vectype);
+ }
+
return true;
}
@@ -12953,13 +13036,31 @@ vectorizable_early_exit (loop_vec_info loop_vinfo,
stmt_vec_info stmt_info,
gcc_assert (new_temp);
+ /* Emit the condition if required. */
+ if (vect_need_early_break_cond_p (loop_vinfo))
+ {
+ tree mask_type = TREE_TYPE (new_temp);
+ tree cmp_op = *LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (bb);
+ gcc_assert (cmp_op);
+ gcc_checking_assert (types_compatible_p (TREE_TYPE (cmp_op), mask_type));
+ gimple *def = SSA_NAME_DEF_STMT (cmp_op);
+ if (gimple_call_internal_p (def, IFN_VARYING))
+ {
+ gimple_stmt_iterator varying_gsi = gsi_for_stmt (def);
+ gsi_remove (&varying_gsi, true);
+ }
+ new_stmt = gimple_build_assign (cmp_op, new_temp);
+ vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt,
+ &cond_gsi);
+ new_temp = cmp_op;
+ }
+
tree cst = build_zero_cst (vectype_out);
gimple_cond_set_condition (cond_stmt, NE_EXPR, new_temp, cst);
update_stmt (orig_stmt);
/* ??? */
SLP_TREE_VEC_DEFS (slp_node).truncate (0);
-
return true;
}
@@ -14623,4 +14724,3 @@ vect_gen_len (tree len, tree start_index, tree
end_index, tree len_limit)
return stmts;
}
-
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index
7ce13828c0a2e5b8429d939832ae7a4ca7f33151..d989ac92e68a4521143231e396a13f844b089e1b
100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -1183,6 +1183,10 @@ public:
we need to peel off iterations at the end to form an epilogue loop. */
bool peeling_for_niter;
+ /* True if the loop has any live statements that are used outside of the
+ loop. */
+ bool has_live_stmts;
+
/* When the loop has early breaks that we can vectorize we need to peel
the loop for the break finding loop. */
bool early_breaks;
@@ -1272,6 +1276,11 @@ public:
happen. */
auto_vec<gimple*> early_break_vuses;
+ /* Mapping from an early-exit BB to the vectorized condition associated with
+ that block. During analysis the value is the required vector type; during
+ transformation it is the SSA name that holds the condition. */
+ hash_map <basic_block, tree> *early_break_dependencies;
+
/* The IV adjustment value for inductions that needs to be materialized
inside the relevant exit blocks in order to adjust for early break. */
tree early_break_niters_var;
@@ -1344,6 +1353,7 @@ public:
#define LOOP_VINFO_REDUCTIONS(L) (L)->reductions
#define LOOP_VINFO_PEELING_FOR_GAPS(L) (L)->peeling_for_gaps
#define LOOP_VINFO_PEELING_FOR_NITER(L) (L)->peeling_for_niter
+#define LOOP_VINFO_HAS_LIVE_STMT_P(L) (L)->has_live_stmts
#define LOOP_VINFO_EARLY_BREAKS(L) (L)->early_breaks
#define LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG(L) (L)->early_break_needs_epilogue
#define LOOP_VINFO_EARLY_BRK_STORES(L) (L)->early_break_stores
@@ -1352,8 +1362,12 @@ public:
|| LOOP_VINFO_NITERS_UNCOUNTED_P (L))
#define LOOP_VINFO_EARLY_BREAKS_LIVE_IVS(L) \
(L)->early_break_live_ivs
+#define LOOP_VINFO_EARLY_BREAKS_IV_P(L, S) \
+ ((L) && LOOP_VINFO_EARLY_BREAKS (L) \
+ && LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (L).contains (S))
#define LOOP_VINFO_EARLY_BRK_DEST_BB(L) (L)->early_break_dest_bb
#define LOOP_VINFO_EARLY_BRK_VUSES(L) (L)->early_break_vuses
+#define LOOP_VINFO_EARLY_BRK_DEPS(L) (L)->early_break_dependencies
#define LOOP_VINFO_EARLY_BRK_NITERS_VAR(L) (L)->early_break_niters_var
#define LOOP_VINFO_EARLY_BRK_IV_TYPE(L) (L)->early_break_iv_type
#define LOOP_VINFO_LOOP_CONDS(L) (L)->conds
@@ -1371,6 +1385,26 @@ public:
#define LOOP_VINFO_DRS_ADVANCED_BY(L) (L)->drs_advanced_by
#define LOOP_VINFO_ALTERNATE_DEFS(L) (L)->alternate_defs
+inline bool
+vect_has_early_break_live_outs (loop_vec_info loop_vinfo)
+{
+ return (LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo)
+ || !LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ());
+}
+
+/* True if no-epilogue early-break transform needs exact vector predicates for
+ each early exit in order to materialize live-out values. The dependency map
+ is populated by analysis and later rewritten to hold the corresponding SSA
+ names before statement transform. */
+
+inline bool
+vect_need_early_break_cond_p (loop_vec_info loop_vinfo)
+{
+ return (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo));
+}
+
#define LOOP_VINFO_FULLY_MASKED_P(L) \
(LOOP_VINFO_USING_PARTIAL_VECTORS_P (L) \
&& !LOOP_VINFO_MASKS (L).is_empty ())
@@ -2498,6 +2532,8 @@ extern bool vect_can_advance_ivs_p (loop_vec_info);
extern void vect_update_inits_of_drs (loop_vec_info, tree, tree_code);
extern edge vec_init_loop_exit_info (class loop *);
extern void vect_iv_increment_position (edge, gimple_stmt_iterator *, bool *);
+extern void adjust_phi_and_debug_stmts (gimple *, edge, tree);
+extern bool iv_phi_p (stmt_vec_info);
/* In tree-vect-stmts.cc. */
extern tree get_related_vectype_for_scalar_type (machine_mode, tree,
--
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
index dfb398da9b7881f699b2ce824c8c501c4768a082..633f5339be8612b2c3b30a6fcf3cb55f3b790422 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_3.c
@@ -18,4 +18,5 @@ int foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" { target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
index 99bb66f9fd05afd6145a7085b97d0df4d75d1f68..a2155015b1aff64795aa4d4e645e17593784f47a 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_4.c
@@ -18,4 +18,5 @@ int foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" { target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
index ec3eec5cfa137f4f99c4320bda525127b7945629..d5585ceca23ee1736132b27306eb897510ac369f 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_5.c
@@ -18,4 +18,5 @@ short foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" { target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
index 46d6a8953cfebc2527a8b420bbec8c4079a4aa5d..c89272d3f4471d024e42a8c9efa467f5d0ed9442 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-no-epilog_6.c
@@ -18,4 +18,5 @@ short foo (void)
}
/* { dg-final { scan-tree-dump "LOOP VECTORIZED" "vect" } } */
-/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" } } */
+/* { dg-final { scan-tree-dump "early break does not require epilog" "vect" { target aarch64*-*-* } } } */
+/* { dg-final { scan-tree-dump-not "early break does not require epilog" "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c
new file mode 100644
index 0000000000000000000000000000000000000000..3db1a4ce34b578ab9a71799e5468460a7188bce4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1-run.c
@@ -0,0 +1,59 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do run } */
+/* { dg-require-effective-target vect_early_break_hw } */
+/* { dg-require-effective-target vect_int } */
+
+#include <assert.h>
+#include "tree-vect.h"
+
+#define N 16
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ for (int i = 0; i < N; ++i)
+ {
+ if (haystack[i] == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ }
+
+ return 0;
+}
+
+#define CHECK_MATCH(POS) \
+ void \
+ check_match_ ## POS (void) \
+ { \
+ int input[N] __attribute__((aligned (16))) = { \
+ [0 ... N-1] = 0 \
+ }; \
+ input[POS] = VAL; \
+ unsigned long res = foo (input, VAL); \
+ assert (res == (1UL << POS)); \
+ }
+
+CHECK_MATCH (0)
+CHECK_MATCH (3)
+CHECK_MATCH (15)
+
+#undef CHECK_MATCH
+#define CHECK_MATCH(POS) check_match_ ## POS ()
+
+int
+main (void)
+{
+ check_vect ();
+
+ CHECK_MATCH (0);
+ CHECK_MATCH (3);
+ CHECK_MATCH (15);
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c
new file mode 100644
index 0000000000000000000000000000000000000000..45971880c2d0c3ef247a40b3d7ccb7fd03a5febc
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-1.c
@@ -0,0 +1,27 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+#define N 16
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ for (int i = 0; i < N; ++i)
+ {
+ if (haystack[i] == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ }
+
+ return 0;
+}
+
+/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+, vect_vec_iv_} "vect" { target aarch64*-*-* } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c
new file mode 100644
index 0000000000000000000000000000000000000000..71e9bf0e8cc5510dda3cfd675c00b9451d36d8df
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-early-break-nonlinear-live-2.c
@@ -0,0 +1,25 @@
+/* { dg-add-options vect_early_break } */
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_early_break } */
+/* { dg-require-effective-target vect_int } */
+
+#define VAL 7
+
+__attribute__((noipa, noinline))
+unsigned long
+foo (int *haystack, int needle)
+{
+ haystack = __builtin_assume_aligned (haystack, 16);
+ unsigned int x = 1;
+
+ while (1)
+ {
+ if (*haystack == needle)
+ return (unsigned long) x;
+ x <<= 1;
+ haystack++;
+ }
+}
+
+/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } } */
+/* { dg-final { scan-tree-dump "Peeling for epilogue is not supported for this nonlinear induction" "vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c b/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
index 6270c11e025ed6e181c7a607da7b1b4fbe82b325..93af3e37a157221bc35ef2defaf04ed0d8d1f806 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-1.c
@@ -73,4 +73,5 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" } } */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 4 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 12 "vect" { target { i?86-*-* x86_64-*-* } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c b/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
index c9987018e88b04f5f0ff195baaf528ad86722714..9512565482cea9a75d32cd28e773c09bd5cc081c 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-2.c
@@ -67,4 +67,5 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 2 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 2 "vect" } } */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 2 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 2 "vect" { target { i?86-*-* x86_64-*-* } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c b/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
index 3caeea71ad7f8858e20a1551b8a0eef573366e96..49ed96cdf0e14e7338c91e04f583fe5943076a0e 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-live-slp-3.c
@@ -74,4 +74,6 @@ main (void)
/* { dg-final { scan-tree-dump-times "vectorized 1 loops" 4 "vect" } } */
/* { dg-final { scan-tree-dump-times "vectorizing stmts using SLP" 4 "vect" } } */
-/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 4 "vect" } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 12 "vect" { target { { i?86-*-* x86_64-*-* } && ilp32 } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 8 "vect" { target { { i?86-*-* x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump-times "vec_stmt_relevant_p: stmt live but not relevant" 4 "vect" { target { ! { i?86-*-* x86_64-*-* } } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c b/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
index 75a8ed1372ff2eaff60ccb6eb1d3242f720452b4..ca3fde85f274b81b89fe1e1bcd34d9489d9eba14 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-uncounted-run_1.c
@@ -1,5 +1,4 @@
-/* Check we can derive scalar IV return value from vectorized IV value,
- resetting it on entry into the scalar epilogue loop via BIT_FIELD_REF. */
+/* Check we can derive scalar IV return value from the vectorized loop. */
/* { dg-add-options vect_early_break } */
/* { dg-require-effective-target vect_early_break_hw } */
/* { dg-require-effective-target vect_int } */
@@ -51,6 +50,8 @@ main ()
/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } } */
/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
-/* Ensure we reset our scalar IV so as to repeat the last vector iteration. */
-/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+, [0-9]+, 0>} "vect" } } */
-
+/* AArch64 extracts the IV value directly from the first active vector lane. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+, vect_i_[0-9_.]+\)} "vect" { target aarch64*-*-* } } } */
+/* Other targets still reset the scalar IV to repeat the last vector
+ iteration in the scalar epilogue loop. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+, [0-9]+, 0>} "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c b/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
index e5bfbc64527d49795cd25b42717eb3f7b31a50b8..1aa2f0baa6c3c4c9170f4b654693ad18a4b4bf2a 100644
--- a/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
+++ b/gcc/testsuite/gcc.dg/vect/vect-uncounted_1.c
@@ -1,5 +1,4 @@
-/* Check we can derive scalar IV return value from vectorized IV value,
- resetting it on entry into the scalar epilogue loop via BIT_FIELD_REF. */
+/* Check we can derive scalar IV return value from the vectorized loop. */
/* { dg-add-options vect_early_break } */
/* { dg-do compile } */
/* { dg-require-effective-target vect_early_break } */
@@ -19,6 +18,8 @@ foo (int *haystack, int needle)
/* { dg-final { scan-tree-dump "Loop being analyzed as uncounted." "vect" } } */
/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
-/* Ensure we reset our scalar IV so as to repeat the last vector iteration. */
-/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+, [0-9]+, 0>} "vect" } } */
-
+/* AArch64 extracts the IV value directly from the first active vector lane. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = \.EXTRACT_FIRST \(vexit_[0-9_]+, vect_i_[0-9_.]+\)} "vect" { target aarch64*-*-* } } } */
+/* Other targets still reset the scalar IV to repeat the last vector
+ iteration in the scalar epilogue loop. */
+/* { dg-final { scan-tree-dump {_[0-9_]+ = BIT_FIELD_REF <vect_i_[0-9_.]+, [0-9]+, 0>} "vect" { target { ! aarch64*-*-* } } } } */
diff --git a/gcc/tree-vect-loop-manip.cc b/gcc/tree-vect-loop-manip.cc
index b54353399d9ed6454442c93750b42924513178f5..1d4586052888aec3ed412351ebfe5169c9662f7a 100644
--- a/gcc/tree-vect-loop-manip.cc
+++ b/gcc/tree-vect-loop-manip.cc
@@ -249,7 +249,7 @@ adjust_debug_stmts (tree from, tree to, basic_block bb)
presumably non-loop-closed references left over from other
transformations. */
-static void
+void
adjust_phi_and_debug_stmts (gimple *update_phi, edge e, tree new_def)
{
tree orig_def = PHI_ARG_DEF_FROM_EDGE (update_phi, e);
@@ -2359,7 +2359,7 @@ find_loop_location (class loop *loop)
/* Return true if the phi described by STMT_INFO defines an IV of the
loop to be vectorized. */
-static bool
+bool
iv_phi_p (stmt_vec_info stmt_info)
{
gphi *phi = as_a <gphi *> (stmt_info->stmt);
@@ -3984,6 +3984,7 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1,
gimple_call_set_lhs (tmp_call, tmp_niters_vf);
auto header_gsi = gsi_after_labels (loop->header);
gsi_insert_after (&header_gsi, tmp_call, GSI_SAME_STMT);
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
if (!(LOOP_VINFO_NITERS_UNCOUNTED_P (loop_vinfo)
&& get_loop_exit_edges (loop).length () == 1)
@@ -4006,8 +4007,6 @@ vect_do_peeling (loop_vec_info loop_vinfo, tree niters, tree nitersm1,
}
if (LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
vector_iters_vf = tmp_niters_vf;
-
- LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
}
bool recalculate_peel_niters_init
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 4a04923814224e32bc94dfd3fe6736fe8edaf800..82658c7b13d13c8ec32984a4cac1c6086e14f953 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -762,6 +762,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared)
partial_load_store_bias (0),
peeling_for_gaps (false),
peeling_for_niter (false),
+ has_live_stmts (false),
early_breaks (false),
loop_iv_cond (NULL),
user_unroll (false),
@@ -777,6 +778,7 @@ _loop_vec_info::_loop_vec_info (class loop *loop_in, vec_info_shared *shared)
vec_epilogue_loop_main_exit (NULL),
scalar_loop_main_exit (NULL),
early_break_needs_epilogue (false),
+ early_break_dependencies (NULL),
early_break_niters_var (NULL)
{
/* CHECKME: We want to visit all BBs before their successors (except for
@@ -857,6 +859,7 @@ _loop_vec_info::~_loop_vec_info ()
delete scan_map;
delete scalar_costs;
delete vector_costs;
+ delete early_break_dependencies;
for (auto reduc_info : reduc_infos)
delete reduc_info;
@@ -5524,13 +5527,13 @@ vect_create_epilog_for_reduction (loop_vec_info loop_vinfo,
def = vect_get_slp_vect_def (slp_node, i);
tree new_def = copy_ssa_name (def);
phi = create_phi_node (new_def, exit_bb);
+ if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ for (unsigned k = 0; k < gimple_phi_num_args (phi); k++)
+ SET_PHI_ARG_DEF (phi, k, def);
+
if (LOOP_VINFO_MAIN_EXIT (loop_vinfo) == loop_exit)
SET_PHI_ARG_DEF (phi, loop_exit->dest_idx, def);
- else
- {
- for (unsigned k = 0; k < gimple_phi_num_args (phi); k++)
- SET_PHI_ARG_DEF (phi, k, def);
- }
+
new_def = gimple_convert (&stmts, vectype, new_def);
reduc_inputs.quick_push (new_def);
gsi_insert_seq_before (&exit_gsi, stmts, GSI_SAME_STMT);
@@ -9679,6 +9682,13 @@ vectorizable_induction (loop_vec_info loop_vinfo,
if (cost_vec) /* transformation not required. */
{
unsigned inside_cost = 0, prologue_cost = 0;
+
+ /* We can't support unpacked IVs yet without a scalar loop. */
+ if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && nvects > 1
+ && !LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, stmt_info))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
/* We eventually need to set a vector type on invariant
arguments. */
unsigned j;
@@ -10185,7 +10195,6 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
tree vectype = SLP_TREE_VECTYPE (slp_node);
poly_uint64 nunits = TYPE_VECTOR_SUBPARTS (vectype);
gimple *use_stmt;
- use_operand_p use_p;
auto_vec<tree> vec_oprnds;
int vec_entry = 0;
poly_uint64 vec_index = 0;
@@ -10198,6 +10207,11 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
validity so just trigger the transform here. */
if (vect_is_reduction (slp_node))
{
+ /* No support for reductions at the moment as they need some
+ materialization changes. */
+ if (!LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
if (!vec_stmt_p)
{
SLP_TREE_LIVE_LANES (slp_node).safe_push (slp_index);
@@ -10219,8 +10233,11 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
slp_node_instance,
LOOP_VINFO_MAIN_EXIT (loop_vinfo));
- /* If early break we only have to materialize the reduction on the merge
- block, but we have to find an alternate exit first. */
+ /* If early break and we are using a scalar epilog we only have to
+ materialize the reduction on the merge block, but we have to find
+ an alternate exit first. But if early break and we don't need the
+ epilog the reduction needs to be materialized in every exit along with
+ a scoped final reduction. */
if (LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
{
slp_tree phis_node = slp_node_instance->reduc_phis;
@@ -10231,7 +10248,8 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
vect_create_epilog_for_reduction (loop_vinfo, stmt_info,
phis_node, slp_node_instance,
exit);
- break;
+ if (LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ break;
}
if (LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
vect_create_epilog_for_reduction (loop_vinfo, stmt_info,
@@ -10264,6 +10282,41 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
int num_scalar = SLP_TREE_LANES (slp_node);
int num_vec = vect_get_num_copies (vinfo, slp_node);
poly_uint64 pos = (num_vec * nunits) - num_scalar + slp_index;
+ bool live_early_break_iv_p
+ = LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, stmt_info);
+
+ /* ??? The live value can be a cast of the scalar IV rather than the IV
+ itself.
+ The early-break IV update code below looks through such casts when it
+ rewrites the exit PHIs, but the analysis here still needs to know that the
+ live statement is covered by that repair. Otherwise the cast looks like a
+ normal live value with multiple vector results and forces scalar epilog.
+
+ This is deliberately restricted to fixed-length vectors. For scalable
+ vectors, the regular live-value path can use the target's native
+ extraction sequence, and keeping that path avoids replacing the better
+ variable-length sequence. */
+ if (!live_early_break_iv_p
+ && loop_vinfo
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && is_gimple_assign (STMT_VINFO_STMT (stmt_info))
+ && gimple_assign_cast_p (STMT_VINFO_STMT (stmt_info))
+ && TYPE_VECTOR_SUBPARTS (vectype).is_constant ()
+ && SLP_TREE_CHILDREN (slp_node).length () == 1)
+ {
+ slp_tree child = SLP_TREE_CHILDREN (slp_node)[0];
+ stmt_vec_info child_info = SLP_TREE_REPRESENTATIVE (child);
+ if (child_info
+ && STMT_VINFO_DEF_TYPE (child_info) == vect_induction_def
+ && (STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (child_info)
+ == vect_step_op_add))
+ {
+ if (!LOOP_VINFO_EARLY_BREAKS_IV_P (loop_vinfo, child_info))
+ LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).safe_push
+ (child_info);
+ live_early_break_iv_p = true;
+ }
+ }
/* Calculate which vector contains the result, and which lane of
that vector we need. */
@@ -10276,10 +10329,36 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
return false;
}
+ /* For grouped loads without a load permutation, lanes beyond
+ SLP_TREE_LANES are the other elements loaded from the group, rather than
+ further scalar iterations of STMT_INFO. For example, in:
+
+ x = p[2 * i];
+ y = p[2 * i + 1];
+
+ without a permute the vector elements are [x0, y0, x1, y1, ...],
+ so vec_index + 1 does not necessarily refer to the next element of
+ STMT_INFO. */
+ if (STMT_VINFO_GROUPED_ACCESS (stmt_info)
+ && !SLP_TREE_LOAD_PERMUTATION (slp_node).exists ()
+ && known_ge (vec_index, (unsigned) num_scalar))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "Cannot extract grouped-access padding lane for"
+ " live stmt.\n");
+ return false;
+ }
+
if (!vec_stmt_p)
{
/* No transformation required. */
- if (loop_vinfo && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ /* Early-break live IVs are handled by early exit IV handling code which
+ determines how the IV should be extracted. As such even though it's
+ been forced live, do not vectorize it as a normal live operation. */
+ if (loop_vinfo
+ && LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo)
+ && !live_early_break_iv_p)
{
if (SLP_TREE_LANES (slp_node) != 1)
{
@@ -10322,6 +10401,41 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
}
}
}
+
+ /* Check if we have support for the right optabs to not require the scalar
+ loop. */
+ if (loop_vinfo
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ {
+ if (!live_early_break_iv_p
+ && !direct_internal_fn_supported_p (IFN_EXTRACT_FIRST,
+ vectype, OPTIMIZE_FOR_BOTH))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live value because "
+ "target does not support extract first "
+ "operations for type %T\n", vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+
+ /* Unpacking cannot be supported yet without a scalar loop. */
+ if (num_vec > 1
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && !live_early_break_iv_p)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live value because "
+ "unpacking live values not currently supported by "
+ "EXTRACT_FIRST operations. %T\n", vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+ }
+
/* ??? Enable for loop costing as well. */
if (!loop_vinfo)
record_stmt_cost (cost_vec, 1, vec_to_scalar, slp_node,
@@ -10378,46 +10492,183 @@ vectorizable_live_operation (vec_info *vinfo, stmt_vec_info stmt_info,
rather than at the end. */
edge main_e = LOOP_VINFO_MAIN_EXIT (loop_vinfo);
bool all_exits_as_early_p = LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo);
+ /* Collect the exit PHIs before building replacement code. The builders
+ below can simplify the new statements, and simplification is allowed
+ to inspect SSA uses. Doing that while FOR_EACH_IMM_USE_STMT has an
+ active iterator for LHS would trip the checking in
+ first_imm_use_stmt. */
+ auto_vec<std::pair<gphi *, unsigned>, 4> exit_phi_args;
FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, lhs)
if (!is_gimple_debug (use_stmt)
&& !flow_bb_inside_loop_p (loop, gimple_bb (use_stmt)))
- FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
+ {
+ use_operand_p use_p;
+ FOR_EACH_IMM_USE_ON_STMT (use_p, imm_iter)
+ {
+ exit_phi_args.safe_push
+ (std::make_pair (as_a <gphi *> (use_stmt),
+ PHI_ARG_INDEX_FROM_USE (use_p)));
+ break;
+ }
+ }
+
+ for (auto phi_arg : exit_phi_args)
+ {
+ gphi *use_phi = phi_arg.first;
+ unsigned int i = phi_arg.second;
+ edge e = gimple_phi_arg_edge (use_phi, i);
+ edge phi_e = e;
+
+ /* A previous live SLP lane can already have split the loop exit
+ edge, so this PHI argument can come from the inserted block
+ rather than directly from the loop. Use the inserted block as
+ the materialization point, but update the PHI argument on PHI_E
+ instead of replacing the whole PHI. */
+ if (!loop_exit_edge_p (loop, e))
+ {
+ if (!single_pred_p (e->src))
+ continue;
+ e = single_pred_edge (e->src);
+ if (!loop_exit_edge_p (loop, e))
+ continue;
+ }
+ bool update_phi_arg_p = phi_e != e;
+ bool main_exit_edge = e == main_e;
+ tree tmp_vec_lhs = vec_lhs;
+ tree tmp_bitstart = bitstart;
+
+ gimple_stmt_iterator exit_gsi;
+ tree new_tree = NULL_TREE;
+
+ /* For early exit where the exit is not in the BB that leads
+ to the latch then we're restarting the iteration in the
+ scalar loop. So get the first live value. */
+ bool early_break_first_element_p
+ = all_exits_as_early_p || !main_exit_edge;
+
+ auto gsi = gsi_for_stmt (use_phi);
+ if (vect_need_early_break_cond_p (loop_vinfo)
+ && early_break_first_element_p)
{
- edge e = gimple_phi_arg_edge (as_a <gphi *> (use_stmt),
- phi_arg_index_from_use (use_p));
- gcc_assert (loop_exit_edge_p (loop, e));
- bool main_exit_edge = e == main_e;
- tree tmp_vec_lhs = vec_lhs;
- tree tmp_bitstart = bitstart;
-
- /* For early exit where the exit is not in the BB that leads
- to the latch then we're restarting the iteration in the
- scalar loop. So get the first live value. */
- bool early_break_first_element_p
- = all_exits_as_early_p || !main_exit_edge;
- if (early_break_first_element_p)
+ auto cond_bb = e->src;
+ tree *slot
+ = LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (cond_bb);
+ if (slot)
{
- tmp_vec_lhs = vec_lhs0;
- tmp_bitstart = build_zero_cst (TREE_TYPE (bitstart));
+ tree cmp_op = *slot;
+
+ gimple_seq stmts = NULL;
+ /* If the live value is a simple vector load, reload the
+ first active scalar element directly on the exit edge.
+ This avoids an EXTRACT_FIRST from a vector register on
+ targets where INDEX_FIRST_ACTIVE is cheap. */
+ data_reference *dr = STMT_VINFO_DATA_REF (stmt_info);
+ gassign *load = NULL;
+ if (single_pred_p (e->dest)
+ && direct_internal_fn_supported_p
+ (IFN_INDEX_FIRST_ACTIVE, TREE_TYPE (cmp_op),
+ OPTIMIZE_FOR_BOTH)
+ && dr
+ && TREE_CODE (tmp_vec_lhs) == SSA_NAME
+ && (load = dyn_cast<gassign *>
+ (SSA_NAME_DEF_STMT (tmp_vec_lhs)))
+ && gimple_bb (load) == single_pred (e->dest)
+ && gimple_assign_rhs_code (load) == MEM_REF)
+ {
+ dr_vec_info *dr_info = STMT_VINFO_DR_INFO (stmt_info);
+ tree vec_ref = gimple_assign_rhs1 (load);
+ tree scalar_type = TREE_TYPE (DR_REF (dr));
+ tree ptr = TREE_OPERAND (vec_ref, 0);
+ tree reload_vuse = gimple_vuse (load);
+ if (gphi *vphi = get_virtual_phi (e->dest))
+ reload_vuse
+ = (gimple_phi_arg_def_from_edge (vphi, e)
+ == reload_vuse) ? gimple_phi_result (vphi)
+ : NULL_TREE;
+
+ if (reload_vuse)
+ {
+ tree first = gimple_build (&stmts,
+ IFN_INDEX_FIRST_ACTIVE,
+ unsigned_intDI_type_node,
+ cmp_op);
+ tree offset = gimple_convert (&stmts, sizetype,
+ first);
+ offset
+ = gimple_build (&stmts, MULT_EXPR, sizetype, offset,
+ size_int (vect_get_scalar_dr_size (dr_info)));
+
+ tree scalar_ptr_type = build_pointer_type (scalar_type);
+ tree scalar_ptr = gimple_convert (&stmts,
+ scalar_ptr_type,
+ ptr);
+ tree addr = gimple_build (&stmts, POINTER_PLUS_EXPR,
+ scalar_ptr_type,
+ scalar_ptr, offset);
+ tree scalar_ref
+ = build2 (MEM_REF, scalar_type, addr,
+ unshare_expr (TREE_OPERAND (vec_ref, 1)));
+ vect_copy_ref_info (scalar_ref, DR_REF (dr));
+
+ tree scalar
+ = make_temp_ssa_name (scalar_type, NULL, "first");
+ gassign *reload
+ = gimple_build_assign (scalar, scalar_ref);
+ gimple_set_vuse (reload, reload_vuse);
+ gimple_seq_add_stmt (&stmts, reload);
+
+ new_tree = gimple_convert (&stmts, lhs_type, scalar);
+ }
+ }
+ if (!new_tree)
+ {
+ /* Convert the extracted vector element to the scalar
+ type. */
+ tree tmp = gimple_build (&stmts, CFN_EXTRACT_FIRST,
+ TREE_TYPE (vectype),
+ cmp_op, tmp_vec_lhs);
+ new_tree = gimple_convert (&stmts, lhs_type, tmp);
+ }
+
+ if (!single_pred_p (e->dest))
+ {
+ basic_block new_bb
+ = gsi_insert_seq_on_edge_immediate (e, stmts);
+ edge update_e = new_bb ? single_succ_edge (new_bb) : e;
+ adjust_phi_and_debug_stmts (use_phi, update_e, new_tree);
+ continue;
+ }
+ exit_gsi = gsi_after_labels (e->dest);
+ gsi_insert_seq_before (&exit_gsi, stmts, GSI_SAME_STMT);
}
+ }
+ else if (early_break_first_element_p)
+ {
+ tmp_vec_lhs = vec_lhs0;
+ tmp_bitstart = build_zero_cst (TREE_TYPE (bitstart));
+ }
- gimple_stmt_iterator exit_gsi;
- tree new_tree
- = vectorizable_live_operation_1 (loop_vinfo,
- e->dest, vectype,
- slp_node, bitsize,
- tmp_bitstart, tmp_vec_lhs,
- lhs_type, &exit_gsi);
-
- auto gsi = gsi_for_stmt (use_stmt);
- tree lhs_phi = gimple_phi_result (use_stmt);
- remove_phi_node (&gsi, false);
- gimple *copy = gimple_build_assign (lhs_phi, new_tree);
- gsi_insert_before (&exit_gsi, copy, GSI_SAME_STMT);
- break;
+ if (!new_tree)
+ new_tree
+ = vectorizable_live_operation_1 (loop_vinfo,
+ e->dest, vectype,
+ slp_node, bitsize,
+ tmp_bitstart, tmp_vec_lhs,
+ lhs_type, &exit_gsi);
+
+ if (update_phi_arg_p)
+ {
+ adjust_phi_and_debug_stmts (use_phi, phi_e, new_tree);
+ continue;
}
- /* There a no further out-of-loop uses of lhs by LC-SSA construction. */
+ tree lhs_phi = gimple_phi_result (use_phi);
+ remove_phi_node (&gsi, false);
+ gimple *copy = gimple_build_assign (lhs_phi, new_tree);
+ gsi_insert_before (&exit_gsi, copy, GSI_SAME_STMT);
+ }
+
+ /* There are no further out-of-loop uses of lhs by LC-SSA construction. */
FOR_EACH_IMM_USE_STMT (use_stmt, imm_iter, lhs)
gcc_assert (is_gimple_debug (use_stmt)
|| flow_bb_inside_loop_p (loop, gimple_bb (use_stmt)));
@@ -11096,30 +11347,72 @@ move_early_exit_stmts (loop_vec_info loop_vinfo)
SET_PHI_ARG_DEF_ON_EDGE (phi, e, last_seen_vuse);
}
-/* Generate adjustment code for early break scalar IVs filling in the value
- we created earlier on for LOOP_VINFO_EARLY_BRK_NITERS_VAR. */
+/* Initialize the SSA names used to communicate the vectorized early-exit
+ conditions to later no-epilogue fixups. Generate adjustment code for early
+ break scalar IVs filling in the value we created earlier on for
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR. */
static void
vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info loop_vinfo)
{
DUMP_VECT_SCOPE ("vect_update_ivs_after_vectorizer_for_early_breaks");
- if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
- /* If no peeling was done then we have no IV to update. */
- || !LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo))
+ if (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ return;
+
+ bool needs_live_iv
+ = (!LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ()
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo));
+ bool needs_early_break_conds = vect_need_early_break_cond_p (loop_vinfo);
+
+ if (needs_early_break_conds)
+ {
+ gcc_assert (LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo));
+
+ for (gcond *gcond : LOOP_VINFO_LOOP_CONDS (loop_vinfo))
+ {
+ tree *slot
+ = LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (gimple_bb (gcond));
+ gcc_assert (slot && TYPE_P (*slot));
+
+ tree cmp_op = make_temp_ssa_name (*slot, NULL, "vexit");
+ gcall *varying = gimple_build_call_internal (IFN_VARYING, 0);
+ gimple_call_set_lhs (varying, cmp_op);
+
+ gimple_stmt_iterator gsi = gsi_for_stmt (gcond);
+ gsi_insert_before (&gsi, varying, GSI_SAME_STMT);
+
+ *slot = cmp_op;
+ }
+ }
+
+ if (!LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) && !needs_live_iv)
return;
+ /* Create early break IV variable if it will be needed by either peeling or
+ a return value. */
+ if (!LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo))
+ {
+ tree tmp_niters_vf
+ = make_ssa_name (LOOP_VINFO_EARLY_BRK_IV_TYPE (loop_vinfo));
+ LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo) = tmp_niters_vf;
+ }
+ else
+ {
+ /* Remove the existing dummy GIMPLE statement and just keep the def. */
+ tree tmp = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo);
+ gimple *def = SSA_NAME_DEF_STMT (tmp);
+ auto def_gsi = gsi_for_stmt (def);
+ gsi_remove (&def_gsi, true);
+ }
+
tree phi_var = LOOP_VINFO_EARLY_BRK_NITERS_VAR (loop_vinfo);
tree niters_skip = LOOP_VINFO_MASK_SKIP_NITERS (loop_vinfo);
tree ty_var = TREE_TYPE (phi_var);
auto loop = LOOP_VINFO_LOOP (loop_vinfo);
+ edge pe = loop_preheader_edge (loop);
tree induc_var = niters_skip ? copy_ssa_name (phi_var) : phi_var;
- /* Remove the existing dummy GIMPLE statement and just keep the def. */
- gimple *def = SSA_NAME_DEF_STMT (phi_var);
- auto def_gsi = gsi_for_stmt (def);
- gsi_remove (&def_gsi, true);
-
auto induction_phi = create_phi_node (induc_var, loop->header);
tree induc_def = PHI_RESULT (induction_phi);
@@ -11146,11 +11439,188 @@ vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info loop_vinfo)
add_phi_arg (induction_phi, iter_var,
loop_latch_edge (loop), UNKNOWN_LOCATION);
- add_phi_arg (induction_phi, init_var,
- loop_preheader_edge (loop), UNKNOWN_LOCATION);
+ add_phi_arg (induction_phi, init_var, pe, UNKNOWN_LOCATION);
- /* Find the first insertion point in the BB. */
- auto pe = loop_preheader_edge (loop);
+ /* Write the init_stmts in the loop-preheader block. */
+ auto psi = gsi_last_nondebug_bb (pe->src);
+ gsi_insert_seq_after (&psi, init_stmts, GSI_LAST_NEW_STMT);
+
+ /* Write the adjustments at the end of the iv increment. */
+ bool insert_after;
+ gimple_stmt_iterator incr_gsi;
+ vect_iv_increment_position (LOOP_VINFO_MAIN_EXIT (loop_vinfo), &incr_gsi,
+ &insert_after);
+
+ if (insert_after)
+ gsi_insert_seq_after (&incr_gsi, stmts, GSI_NEW_STMT);
+ else
+ gsi_insert_seq_before (&incr_gsi, stmts, GSI_NEW_STMT);
+
+ /* Check if we're staying within the vector loop, and if so rewrite the
+ conditions. This is only needed when live scalar IVs use the early-exit
+ edge: loops without such IVs can still be vectorized as early breaks, but
+ there is no per-exit IV value to repair. */
+ if (needs_live_iv)
+ {
+ for (auto gcond : LOOP_VINFO_LOOP_CONDS (loop_vinfo))
+ {
+ if (LOOP_VINFO_LOOP_IV_COND (loop_vinfo) == gcond
+ && !LOOP_VINFO_EARLY_BREAKS_VECT_PEELED (loop_vinfo))
+ continue;
+ auto bb = gcond->bb;
+ tree cmp_op = *LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (bb);
+ tree mask_type = TREE_TYPE (cmp_op);
+ /* Now rewrite the code to use the new IV for linear IVs. */
+ edge exit_e = EDGE_SUCC (bb, 0);
+ if (flow_bb_inside_loop_p (loop, exit_e->dest))
+ exit_e = EDGE_SUCC (bb, 1);
+
+ /* Split the edge so when we add the new LCSSA PHIs on the exit we
+ don't confuse things if the destination is shared with a versioning
+ block. */
+ basic_block update_bb = split_loop_exit_edge (exit_e);
+ gimple_stmt_iterator last_gsi = gsi_after_labels (update_bb);
+
+ /* Create the mask partitioning if the target cannot count the
+ position of the first active element directly. CMP_OP is the
+ already-masked exit predicate produced by vectorizable_early_exit,
+ so do not use the COND or COND_LEN forms here; applying loop
+ controls again would double-mask the value. The block is reached
+ only after taking the early-exit edge, so CMP_OP is known to have at
+ least one active element. Targets can use that to avoid handling
+ the empty-predicate case.
+
+ This distinction lets targets choose either a direct first-active
+ operation or a partition-and-count sequence. */
+ internal_fn active_fn = IFN_INDEX_FIRST_ACTIVE;
+ tree active_mask = cmp_op;
+ if (!direct_internal_fn_supported_p (active_fn, mask_type,
+ OPTIMIZE_FOR_BOTH))
+ {
+ active_mask = make_temp_ssa_name (mask_type, NULL, "pred");
+ gcall *p_call
+ = gimple_build_call_internal (IFN_TAKE_EXCL_ACTIVE, 1,
+ cmp_op);
+ gimple_call_set_lhs (p_call, active_mask);
+ gsi_insert_before (&last_gsi, p_call, GSI_SAME_STMT);
+ active_fn = IFN_COUNT_ACTIVE;
+ }
+
+ gphi_iterator gsi1;
+ for (gsi1 = gsi_start_phis (update_bb); !gsi_end_p (gsi1);)
+ {
+ gphi *phi1 = gsi1.phi ();
+ tree arg = PHI_ARG_DEF_FROM_EDGE (phi1, exit_e);
+ if (TREE_CODE (arg) != SSA_NAME)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ /* Look through any promotion to find the original IV. This must
+ match the casted-live-IV recognition in
+ vectorizable_live_operation; otherwise the loop could be
+ accepted as no-epilogue during analysis but would not have
+ its exit PHI repaired here. */
+ gimple *def_stmt = NULL;
+ while (TREE_CODE (arg) == SSA_NAME
+ && (def_stmt = SSA_NAME_DEF_STMT (arg))
+ && is_gimple_assign (def_stmt)
+ && gimple_assign_cast_p (def_stmt))
+ arg = gimple_assign_rhs1 (def_stmt);
+
+ if (!def_stmt
+ || gimple_code (def_stmt) != GIMPLE_PHI
+ || gimple_bb (def_stmt) != loop->header)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ gphi *phi = as_a <gphi *> (def_stmt);
+ stmt_vec_info phi_info = loop_vinfo->lookup_stmt (phi);
+
+ /* Skip reduction and virtual phis. */
+ if (!phi_info || !iv_phi_p (phi_info))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "reduc or virtual phi. skip.\n");
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ tree type = TREE_TYPE (gimple_phi_result (phi));
+ tree step_expr = STMT_VINFO_LOOP_PHI_EVOLUTION_PART (phi_info);
+ step_expr = unshare_expr (step_expr);
+
+ /* FORNOW: We do not support IVs whose evolution function is a
+ polynomial of degree >= 2 or exponential. */
+ gcc_assert (!tree_is_chrec (step_expr));
+
+ gimple_seq stmts = NULL;
+ enum vect_induction_op_type induction_type
+ = STMT_VINFO_LOOP_PHI_EVOLUTION_TYPE (phi_info);
+
+ /* We can only optimize linear inductions because the repair is
+ computed as INIT + (VECTOR_ITERS + FIRST_ACTIVE_LANE) * STEP.
+ Non-linear inductions must keep using the existing vector live
+ paths, which can still fall back to a scalar epilogue. */
+ if (induction_type != vect_step_op_add)
+ {
+ gsi_next (&gsi1);
+ continue;
+ }
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "rewriting live value for : phi: %G",
+ (gimple *) phi);
+
+ tree res = PHI_RESULT (phi1);
+ tree reduc_iv = gimple_build (&stmts, active_fn,
+ unsigned_intDI_type_node,
+ active_mask);
+
+ tree stype = TREE_TYPE (step_expr);
+ /* Use the raw vector-iteration count here rather than
+ PHI_VAR. With masked alignment peeling, the first vector
+ iteration starts at -skip so that FIRST_ACTIVE_LANE can
+ compensate for the inactive peeled elements. PHI_VAR is
+ the value after the ordinary peeling adjustment below has
+ clamped the negative count back to zero, which is correct
+ for the final iteration count but too late for repairing an
+ early-exit IV. */
+ tree niters
+ = fold_build2 (PLUS_EXPR, stype,
+ fold_convert (stype, induc_var),
+ fold_convert (stype, reduc_iv));
+ tree off = fold_build2 (MULT_EXPR, stype, niters, step_expr);
+
+ tree start = PHI_ARG_DEF_FROM_EDGE (phi, pe);
+ tree ni;
+ if (POINTER_TYPE_P (type))
+ ni = fold_build_pointer_plus (start, off);
+ else
+ ni = fold_convert
+ (stype, fold_build2 (PLUS_EXPR, stype,
+ fold_convert (stype, start), off));
+
+ gimple_seq new_stmts = NULL;
+ tree ni_name = force_gimple_operand
+ (ni, &new_stmts, false, create_tmp_var (type, "tmp_iv"));
+
+ gsi_insert_seq_before (&last_gsi, stmts, GSI_SAME_STMT);
+ gsi_insert_seq_before (&last_gsi, new_stmts, GSI_SAME_STMT);
+
+ auto res_g = gimple_build_assign
+ (res, fold_convert (TREE_TYPE (res), ni_name));
+ gsi_insert_before (&last_gsi, res_g, GSI_SAME_STMT);
+ remove_phi_node (&gsi1, false);
+ continue;
+ }
+ }
+ }
/* If we've done any peeling, calculate the peeling adjustment needed to the
final IV. */
@@ -11166,33 +11636,22 @@ vect_update_ivs_after_vectorizer_for_early_breaks (loop_vec_info loop_vinfo)
gimple_convert (&iv_stmts, induc_type,
induc_def));
gimple_seq_add_stmt_without_update (&iv_stmts, stmt);
- basic_block exit_bb = NULL;
+ edge exit_e = NULL;
/* Identify the early exit merge block. I wish we had stored this. */
for (auto e : get_loop_exit_edges (loop))
if (e != LOOP_VINFO_MAIN_EXIT (loop_vinfo))
{
- exit_bb = e->dest;
+ exit_e = e;
break;
}
- gcc_assert (exit_bb);
+ gcc_assert (exit_e);
+ basic_block exit_bb = exit_e->dest;
+ if (!dominated_by_p (CDI_DOMINATORS, exit_bb, loop->header))
+ exit_bb = split_loop_exit_edge (exit_e);
auto exit_gsi = gsi_after_labels (exit_bb);
gsi_insert_seq_before (&exit_gsi, iv_stmts, GSI_SAME_STMT);
}
- /* Write the init_stmts in the loop-preheader block. */
- auto psi = gsi_last_nondebug_bb (pe->src);
- gsi_insert_seq_after (&psi, init_stmts, GSI_LAST_NEW_STMT);
-
- /* Write the adjustments at the end of the iv increment. */
- bool insert_after;
- gimple_stmt_iterator incr_gsi;
- vect_iv_increment_position (LOOP_VINFO_MAIN_EXIT (loop_vinfo), &incr_gsi,
- &insert_after);
-
- if (insert_after)
- gsi_insert_seq_after (&incr_gsi, stmts, GSI_NEW_STMT);
- else
- gsi_insert_seq_before (&incr_gsi, stmts, GSI_NEW_STMT);
}
/* Function vect_transform_loop.
@@ -11244,7 +11703,9 @@ vect_transform_loop (loop_vec_info loop_vinfo, gimple *loop_vectorized_call)
/* Make sure there exists a single-predecessor exit bb. Do this before
versioning. */
edge e = LOOP_VINFO_MAIN_EXIT (loop_vinfo);
- if (! single_pred_p (e->dest) && !LOOP_VINFO_EARLY_BREAKS (loop_vinfo))
+ if (! single_pred_p (e->dest)
+ && (!LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ || LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo)))
{
split_loop_exit_edge (e, true);
if (dump_enabled_p ())
diff --git a/gcc/tree-vect-stmts.cc b/gcc/tree-vect-stmts.cc
index 65d82d51310b48178af61ce62862ad7ffb814a5a..be170a298ae30ed90b34770781da7c428da9e611 100644
--- a/gcc/tree-vect-stmts.cc
+++ b/gcc/tree-vect-stmts.cc
@@ -413,7 +413,7 @@ vect_stmt_relevant_p (stmt_vec_info stmt_info, loop_vec_info loop_vinfo,
gcc_assert (gimple_code (USE_STMT (use_p)) == GIMPLE_PHI);
*live_p = true;
- LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo) = true;
}
}
}
@@ -427,6 +427,12 @@ vect_stmt_relevant_p (stmt_vec_info stmt_info, loop_vec_info loop_vinfo,
*relevant = vect_used_only_live;
}
+ if (*live_p
+ && LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def
+ && !LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).contains (stmt_info))
+ LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).safe_push (stmt_info);
+
return (*live_p || *relevant);
}
@@ -6786,7 +6792,7 @@ vectorizable_operation (vec_info *vinfo,
vec_loop_masks *masks = (loop_vinfo ? &LOOP_VINFO_MASKS (loop_vinfo) : NULL);
vec_loop_lens *lens = (loop_vinfo ? &LOOP_VINFO_LENS (loop_vinfo) : NULL);
internal_fn cond_fn = get_conditional_internal_fn (code);
- internal_fn cond_len_fn = get_conditional_len_internal_fn (code);
+ internal_fn cond_len_fn = get_len_internal_fn (cond_fn);
/* If operating on inactive elements could generate spurious traps,
we need to restrict the operation to active lanes. Note that this
@@ -12385,13 +12391,16 @@ vectorizable_condition (vec_info *vinfo,
otherwise, vectorize STMT_INFO: create a vectorized comparison, and insert
it at GSI.
- Return true if STMT_INFO is vectorizable in this way. */
+ Return true if STMT_INFO is vectorizable in this way. If
+ COMPARISON_VECTYPE_OUT is nonnull, store the vector type of the comparison
+ inputs there. */
static bool
vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
stmt_vec_info stmt_info, tree_code code,
gimple_stmt_iterator *gsi,
- slp_tree slp_node, stmt_vector_for_cost *cost_vec)
+ slp_tree slp_node, stmt_vector_for_cost *cost_vec,
+ tree *comparison_vectype_out)
{
tree lhs, rhs1, rhs2;
tree vectype1 = NULL_TREE, vectype2 = NULL_TREE;
@@ -12450,6 +12459,9 @@ vectorizable_comparison_1 (vec_info *vinfo, tree vectype,
&& (VECTOR_BOOLEAN_TYPE_P (vectype1) ^ VECTOR_BOOLEAN_TYPE_P (vectype2)))
return false;
+ if (comparison_vectype_out)
+ *comparison_vectype_out = vectype;
+
/* Boolean values may have another representation in vectors
and therefore we prefer bit operations over comparison for
them (which also works for scalar masks). We store opcodes
@@ -12613,7 +12625,7 @@ vectorizable_comparison (vec_info *vinfo,
enum tree_code code = gimple_assign_rhs_code (stmt);
tree vectype = SLP_TREE_VECTYPE (slp_node);
if (!vectorizable_comparison_1 (vinfo, vectype, stmt_info, code, gsi,
- slp_node, cost_vec))
+ slp_node, cost_vec, NULL))
return false;
if (cost_vec)
@@ -12776,16 +12788,22 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
bool flipped = flow_bb_inside_loop_p (LOOP_VINFO_LOOP (loop_vinfo),
exit_true_edge->dest);
- /* See if we support ADDHN and use that for the reduction. */
+ /* See if we support ADDHN and use that for the branch reduction. Some
+ targets produce all-ones/all-zeroes lanes for boolean comparisons. ADDHN
+ can compress a 128-bit predicate into a 64-bit scalar cheaply enough for
+ an any-active zero test. This is only a branch check; if the exit is
+ taken and we need the exact lane index, the target-specific
+ INDEX_FIRST_ACTIVE or COUNT_ACTIVE path can still apply a lane mask later
+ in the exit block. */
internal_fn ifn = IFN_VEC_TRUNC_ADD_HIGH;
bool addhn_supported_p
- = direct_internal_fn_supported_p (ifn, vectype, OPTIMIZE_FOR_BOTH);
+ = (vec_num > 1
+ && direct_internal_fn_supported_p (ifn, vectype, OPTIMIZE_FOR_BOTH));
tree narrow_type = NULL_TREE;
if (addhn_supported_p)
{
- /* Calculate the narrowing type for the result. */
- auto halfprec = TYPE_PRECISION (TREE_TYPE (vectype)) / 2;
- auto unsignedp = TYPE_UNSIGNED (TREE_TYPE (vectype));
+ unsigned int halfprec = TYPE_PRECISION (TREE_TYPE (vectype)) / 2;
+ bool unsignedp = TYPE_UNSIGNED (TREE_TYPE (vectype));
tree itype = build_nonstandard_integer_type (halfprec, unsignedp);
tree tmp_type = build_vector_type (itype, TYPE_VECTOR_SUBPARTS (vectype));
narrow_type = truth_type_for (tmp_type);
@@ -12794,10 +12812,10 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
TYPE_MODE (narrow_type)))
{
if (dump_enabled_p ())
- dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
- "can't use ADDHN reduction because cbranch for "
- "the narrowed type is not supported by the "
- "target.\n");
+ dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
+ "can't use ADDHN reduction because cbranch for "
+ "the narrowed type is not supported by the "
+ "target.\n");
addhn_supported_p = false;
}
}
@@ -12816,9 +12834,12 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
return false;
}
+ tree comparison_vectype = NULL_TREE;
if (!vectorizable_comparison_1 (loop_vinfo, vectype, stmt_info, code, gsi,
- slp_node, cost_vec))
+ slp_node, cost_vec,
+ &comparison_vectype))
return false;
+ gcc_assert (comparison_vectype);
if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
{
@@ -12832,6 +12853,68 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
if (!vect_compute_type_for_early_break_scalar_iv (loop_vinfo))
return false;
+ tree mask_vectype = (VECTOR_BOOLEAN_TYPE_P (vectype)
+ ? vectype : truth_type_for (vectype));
+
+ if (!LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ()
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo))
+ {
+ /* The no-epilogue live-IV path needs the number of scalar
+ iterations executed by the vector loop before the early exit.
+ Targets can provide that directly as INDEX_FIRST_ACTIVE, or
+ provide a predicate for the lanes before the first active lane and
+ then count those lanes. This lets the vector loop update a live IV
+ without replaying the iteration in a scalar epilogue. */
+ if (!direct_internal_fn_supported_p (IFN_INDEX_FIRST_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)
+ && (!direct_internal_fn_supported_p (IFN_TAKE_EXCL_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)
+ || !direct_internal_fn_supported_p (IFN_COUNT_ACTIVE,
+ mask_vectype,
+ OPTIMIZE_FOR_BOTH)))
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (
+ MSG_MISSED_OPTIMIZATION, vect_location,
+ "epilogue loop is required for live induction because "
+ "target does not support predicate position operations "
+ "for type %T\n", mask_vectype);
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+ }
+ else if (LOOP_VINFO_CAN_USE_PARTIAL_VECTORS_P (loop_vinfo))
+ LOOP_VINFO_MUST_USE_PARTIAL_VECTORS_P (loop_vinfo) = true;
+ }
+
+ /* The no-epilogue path uses the vector predicate to compute the exact
+ scalar iteration at which a live value is needed. When the early exit
+ produces more than one vector predicate, the branch condition is
+ currently a reduction of those predicates. That is enough to know that
+ some lane exited, but it loses which vector contained the first active
+ lane. Keep the scalar epilogue for both live statements and live IVs in
+ that case until the no-epilogue path can preserve that information.
+
+ Use the number of data-lane vectors rather than the number of copies
+ of the boolean result. */
+ poly_uint64 cmp_units
+ = LOOP_VINFO_VECT_FACTOR (loop_vinfo) * SLP_TREE_LANES (slp_node);
+ unsigned int cmp_vec_num = vect_get_num_vectors (cmp_units,
+ comparison_vectype);
+ if (cmp_vec_num > 1
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo) = true;
+
+ if (!LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo))
+ {
+ if (!LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo))
+ LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)
+ = new hash_map<basic_block, tree>;
+ LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->put (bb, vectype);
+ }
+
return true;
}
@@ -12953,13 +13036,31 @@ vectorizable_early_exit (loop_vec_info loop_vinfo, stmt_vec_info stmt_info,
gcc_assert (new_temp);
+ /* Emit the condition if required. */
+ if (vect_need_early_break_cond_p (loop_vinfo))
+ {
+ tree mask_type = TREE_TYPE (new_temp);
+ tree cmp_op = *LOOP_VINFO_EARLY_BRK_DEPS (loop_vinfo)->get (bb);
+ gcc_assert (cmp_op);
+ gcc_checking_assert (types_compatible_p (TREE_TYPE (cmp_op), mask_type));
+ gimple *def = SSA_NAME_DEF_STMT (cmp_op);
+ if (gimple_call_internal_p (def, IFN_VARYING))
+ {
+ gimple_stmt_iterator varying_gsi = gsi_for_stmt (def);
+ gsi_remove (&varying_gsi, true);
+ }
+ new_stmt = gimple_build_assign (cmp_op, new_temp);
+ vect_finish_stmt_generation (loop_vinfo, stmt_info, new_stmt,
+ &cond_gsi);
+ new_temp = cmp_op;
+ }
+
tree cst = build_zero_cst (vectype_out);
gimple_cond_set_condition (cond_stmt, NE_EXPR, new_temp, cst);
update_stmt (orig_stmt);
/* ??? */
SLP_TREE_VEC_DEFS (slp_node).truncate (0);
-
return true;
}
@@ -14623,4 +14724,3 @@ vect_gen_len (tree len, tree start_index, tree end_index, tree len_limit)
return stmts;
}
-
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index 7ce13828c0a2e5b8429d939832ae7a4ca7f33151..d989ac92e68a4521143231e396a13f844b089e1b 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -1183,6 +1183,10 @@ public:
we need to peel off iterations at the end to form an epilogue loop. */
bool peeling_for_niter;
+ /* True if the loop has any live statements that are used outside of the
+ loop. */
+ bool has_live_stmts;
+
/* When the loop has early breaks that we can vectorize we need to peel
the loop for the break finding loop. */
bool early_breaks;
@@ -1272,6 +1276,11 @@ public:
happen. */
auto_vec<gimple*> early_break_vuses;
+ /* Mapping from an early-exit BB to the vectorized condition associated with
+ that block. During analysis the value is the required vector type; during
+ transformation it is the SSA name that holds the condition. */
+ hash_map <basic_block, tree> *early_break_dependencies;
+
/* The IV adjustment value for inductions that needs to be materialized
inside the relevant exit blocks in order to adjust for early break. */
tree early_break_niters_var;
@@ -1344,6 +1353,7 @@ public:
#define LOOP_VINFO_REDUCTIONS(L) (L)->reductions
#define LOOP_VINFO_PEELING_FOR_GAPS(L) (L)->peeling_for_gaps
#define LOOP_VINFO_PEELING_FOR_NITER(L) (L)->peeling_for_niter
+#define LOOP_VINFO_HAS_LIVE_STMT_P(L) (L)->has_live_stmts
#define LOOP_VINFO_EARLY_BREAKS(L) (L)->early_breaks
#define LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG(L) (L)->early_break_needs_epilogue
#define LOOP_VINFO_EARLY_BRK_STORES(L) (L)->early_break_stores
@@ -1352,8 +1362,12 @@ public:
|| LOOP_VINFO_NITERS_UNCOUNTED_P (L))
#define LOOP_VINFO_EARLY_BREAKS_LIVE_IVS(L) \
(L)->early_break_live_ivs
+#define LOOP_VINFO_EARLY_BREAKS_IV_P(L, S) \
+ ((L) && LOOP_VINFO_EARLY_BREAKS (L) \
+ && LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (L).contains (S))
#define LOOP_VINFO_EARLY_BRK_DEST_BB(L) (L)->early_break_dest_bb
#define LOOP_VINFO_EARLY_BRK_VUSES(L) (L)->early_break_vuses
+#define LOOP_VINFO_EARLY_BRK_DEPS(L) (L)->early_break_dependencies
#define LOOP_VINFO_EARLY_BRK_NITERS_VAR(L) (L)->early_break_niters_var
#define LOOP_VINFO_EARLY_BRK_IV_TYPE(L) (L)->early_break_iv_type
#define LOOP_VINFO_LOOP_CONDS(L) (L)->conds
@@ -1371,6 +1385,26 @@ public:
#define LOOP_VINFO_DRS_ADVANCED_BY(L) (L)->drs_advanced_by
#define LOOP_VINFO_ALTERNATE_DEFS(L) (L)->alternate_defs
+inline bool
+vect_has_early_break_live_outs (loop_vec_info loop_vinfo)
+{
+ return (LOOP_VINFO_HAS_LIVE_STMT_P (loop_vinfo)
+ || !LOOP_VINFO_EARLY_BREAKS_LIVE_IVS (loop_vinfo).is_empty ());
+}
+
+/* True if no-epilogue early-break transform needs exact vector predicates for
+ each early exit in order to materialize live-out values. The dependency map
+ is populated by analysis and later rewritten to hold the corresponding SSA
+ names before statement transform. */
+
+inline bool
+vect_need_early_break_cond_p (loop_vec_info loop_vinfo)
+{
+ return (LOOP_VINFO_EARLY_BREAKS (loop_vinfo)
+ && !LOOP_VINFO_EARLY_BRK_NEEDS_EPILOG (loop_vinfo)
+ && vect_has_early_break_live_outs (loop_vinfo));
+}
+
#define LOOP_VINFO_FULLY_MASKED_P(L) \
(LOOP_VINFO_USING_PARTIAL_VECTORS_P (L) \
&& !LOOP_VINFO_MASKS (L).is_empty ())
@@ -2498,6 +2532,8 @@ extern bool vect_can_advance_ivs_p (loop_vec_info);
extern void vect_update_inits_of_drs (loop_vec_info, tree, tree_code);
extern edge vec_init_loop_exit_info (class loop *);
extern void vect_iv_increment_position (edge, gimple_stmt_iterator *, bool *);
+extern void adjust_phi_and_debug_stmts (gimple *, edge, tree);
+extern bool iv_phi_p (stmt_vec_info);
/* In tree-vect-stmts.cc. */
extern tree get_related_vectype_for_scalar_type (machine_mode, tree,
