From: Kyrylo Tkachov <[email protected]>
vect_bb_vectorization_profitable_p requires the code of each loop of a BB
SLP region to be profitable on its own. A loop can run any number of
times. The loop of the region entry is the exception. Within a loop the
function adds up the costs as if all of its code ran on every iteration.
In the threaded interpreters of the PR, BB SLP keeps next_instr and
stack_pointer in one vector. So every handler stores both with one vector
store (slp1 dump, simplified):
latch:
_229 = {_228, _227};
dispatch:
# vect_next_instr_234 = PHI <_229(4), vect_next_instr_231(2)>
_237 = BIT_FIELD_REF <vect_next_instr_234, 64, 64>;
_235 = BIT_FIELD_REF <vect_next_instr_234, 64, 0>;
goto gotovar.0_89;
handler:
MEM <vector(2) long unsigned int> [frame] = vect_next_instr_234;
Each iteration runs one handler. But the costing uses the savings of
the stores of all handlers to pay for the vector code that runs on every
iteration. This code is the construction in the latch and the lane
extracts in the dispatch block. For the testcase of the PR the loop
costs 24 for the vector and 42 for the scalar code on aarch64 with
generic tuning. As a result, CPython's evaluator moves its state
between general and vector registers on every bytecode dispatch.
The code of a block runs together with the code of the blocks dominating
it. So also require each block with costs in a loop to be profitable
together with its dominators in that loop. The check adds up the
unweighted costs that the target returns for their code.
In the example on aarch64 the dispatch block saves 2. This is because
aarch64 does not cost the vector PHI and its lane extracts for BB SLP (a
separate patch changes that). A handler saves 1 with its vector store.
The latch adds the construction, 4, so the latch fails the check with 2
more (8 more with -mcpu=neoverse-v2). On x86_64 the dispatch block costs
28 more, a handler saves 12 and the latch costs 28 more, so every block
fails.
The check also rejects a loop whose first exit comes before the code that
pays for the vector code. A block whose vector code costs more than its
scalar code still passes when its dominators save more, as in
bb-slp-pr127246-13.c.
Like the existing check, the new check leaves out the loop of the region
entry. It also does not apply to loops whose costs are all in one block.
This is because the costs that the target returns for single stmts do not
include its final adjustments.
The previous patch places vector code in the blocks of the scalar code
that it replaces. So the check does not charge the vector code of one
arm to the block before the branch.
With trunk, CPython 3.14.5 built with PGO and LTO for Neoverse V2 has a
vector dispatch. Its evaluator has 2 indirect branches and 153 moves
between general and vector registers. With this series the evaluator has
331 indirect dispatch branches and no such moves. The geometric mean time
on 17 pyperformance benchmarks is 48.2% lower, the same as with
-fno-tree-slp-vectorize.
On SPEC CPU2026 (aarch64, -O3 -mcpu=olympus -flto) 138 fewer basic block
parts are vectorized in 12 of the 27 benchmarks.
The performance impact is within the noise though.
Bootstrapped and regression tested on aarch64-unknown-linux-gnu and
x86_64-pc-linux-gnu.
Ok for trunk?
Thanks,
Kyrill
gcc/ChangeLog:
PR tree-optimization/127246
* tree-vectorizer.h (vector_costs::add_slp_cost): Document the
return value.
* tree-vect-slp.cc (add_slp_costs): Add BBS and BB_COSTS parameters.
When BBS is set, also split the groups at changes of the block and
add the cost of each group to its block in BB_COSTS.
(li_cost): New struct.
(li_cost_vec_cmp): Use li_cost.
(vect_bb_slp_dominator_cost): New function.
(vect_bb_vectorization_profitable_p): Record the vector minus the
scalar cost of each block of a loop part and require each block
with costs to be profitable together with its dominators in the
loop.
gcc/testsuite/ChangeLog:
PR tree-optimization/127246
* gcc.dg/vect/bb-slp-pr127246-1.c: New test.
* gcc.dg/vect/bb-slp-pr127246-2.c: New test.
* gcc.dg/vect/bb-slp-pr127246-9.c: New test.
* gcc.dg/vect/bb-slp-pr127246-13.c: New test.
* gcc.target/aarch64/pr127246.c: New test.
* gcc.target/aarch64/pr127246-6.c: New test.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-1.c | 55 +++++++
.../gcc.dg/vect/bb-slp-pr127246-13.c | 31 ++++
gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-2.c | 50 ++++++
gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-9.c | 25 +++
gcc/testsuite/gcc.target/aarch64/pr127246-6.c | 51 ++++++
gcc/testsuite/gcc.target/aarch64/pr127246.c | 16 ++
gcc/tree-vect-slp.cc | 153 ++++++++++++++----
gcc/tree-vectorizer.h | 3 +-
8 files changed, 351 insertions(+), 33 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-1.c
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-13.c
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-2.c
create mode 100644 gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-9.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/pr127246-6.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/pr127246.c
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-1.c
b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-1.c
new file mode 100644
index 00000000000..b8f8b3f1d26
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-1.c
@@ -0,0 +1,55 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-require-effective-target label_values } */
+/* { dg-require-effective-target indirect_jumps } */
+/* { dg-additional-options "-fvect-cost-model=dynamic" } */
+/* { dg-additional-options "-mtune=generic" { target aarch64*-*-* } } */
+
+/* Each dispatch of the threaded interpreter runs one handler. Keeping
+ next_instr and stack_pointer in one vector, so that a handler can store
+ both with one vector store, costs a vector construction and two lane
+ extracts on every dispatch, more than the stores of one handler save. */
+
+struct frame
+{
+ const unsigned short *instr_ptr;
+ long *stackpointer;
+};
+
+/* Opcode number and stack delta. */
+#define OPS(X) \
+ X (0, -1) X (1, 0) X (2, -1) X (3, 0) X (4, -1) X (5, 0) \
+ X (6, -1) X (7, 0) X (8, -1) X (9, 0) X (10, -1) X (11, 0) \
+ X (12, -1) X (13, 0) X (14, -1) X (15, 0) X (16, -1) X (17, 0) \
+ X (18, -1) X (19, 0)
+
+extern long op (long);
+
+#define DISPATCH() goto *targets[*next_instr & 0xff]
+
+#define HANDLER(N, SD) \
+ L_##N: \
+ frame->instr_ptr = next_instr; \
+ next_instr += 1; \
+ frame->stackpointer = stack_pointer; \
+ stack_pointer[-1] = op (N); \
+ stack_pointer += (SD); \
+ DISPATCH ();
+
+#define ENTRY(N, SD) &&L_##N,
+
+void
+interp (struct frame *frame)
+{
+ static const void *const targets[] = { OPS (ENTRY) };
+ const unsigned short *next_instr = frame->instr_ptr;
+ long *stack_pointer = frame->stackpointer;
+
+ DISPATCH ();
+
+ OPS (HANDLER)
+}
+
+/* { dg-final { scan-tree-dump "vectorization is not profitable" "slp1" {
target { { { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump-not "optimized: basic block" "slp1" { target {
{ { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump "and its dominators costs" "slp1" { target { {
{ aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-13.c
b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-13.c
new file mode 100644
index 00000000000..de0dbb2b041
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-13.c
@@ -0,0 +1,31 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-additional-options "-fvect-cost-model=dynamic
-fno-tree-loop-vectorize" } */
+/* { dg-additional-options "-mtune=generic" { target aarch64*-*-* } } */
+
+/* The vector code of the conditional block costs more than the scalar
+ code it replaces, but the vector code of the block that dominates it
+ saves more, so the loop body stays vectorized. */
+
+extern void use (long, long);
+
+void
+f (long *restrict p, long *restrict q, long *restrict r, long *restrict s,
+ long *restrict t, int *restrict c, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ long a = q[2 * i] + s[2 * i], b = q[2 * i + 1] + s[2 * i + 1];
+ p[2 * i] = a;
+ p[2 * i + 1] = b;
+ if (c[i])
+ {
+ long x = a ^ t[2 * i], y = b ^ t[2 * i + 1];
+ r[2 * i] = x;
+ r[2 * i + 1] = y;
+ use (x, y);
+ }
+ }
+}
+
+/* { dg-final { scan-tree-dump-times "optimized: basic block part vectorized"
2 "slp1" { target { { { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } }
} } */
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-2.c
b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-2.c
new file mode 100644
index 00000000000..a10b2895d12
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-2.c
@@ -0,0 +1,50 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-additional-options "-fvect-cost-model=dynamic" } */
+/* { dg-additional-options "-mtune=generic" { target aarch64*-*-* } } */
+
+/* The switch version of bb-slp-pr127246-1.c. Each iteration runs one case,
+ and the vector construction and the lane extracts of every iteration cost
+ more than the stores of one case save. */
+
+struct frame
+{
+ const unsigned short *instr_ptr;
+ long *stackpointer;
+};
+
+/* Opcode number and stack delta. */
+#define OPS(X) \
+ X (0, -1) X (1, 0) X (2, -1) X (3, 0) X (4, -1) X (5, 0) \
+ X (6, -1) X (7, 0) X (8, -1) X (9, 0) X (10, -1) X (11, 0) \
+ X (12, -1) X (13, 0) X (14, -1) X (15, 0) X (16, -1) X (17, 0) \
+ X (18, -1) X (19, 0)
+
+extern long op (long);
+
+#define HANDLER(N, SD) \
+ case N: \
+ frame->instr_ptr = next_instr; \
+ next_instr += 1; \
+ frame->stackpointer = stack_pointer; \
+ stack_pointer[-1] = op (N); \
+ stack_pointer += (SD); \
+ break;
+
+long
+interp (struct frame *frame)
+{
+ const unsigned short *next_instr = frame->instr_ptr;
+ long *stack_pointer = frame->stackpointer;
+ for (;;)
+ switch (*next_instr & 0xff)
+ {
+ OPS (HANDLER)
+ default:
+ return stack_pointer[0];
+ }
+}
+
+/* { dg-final { scan-tree-dump "vectorization is not profitable" "slp1" {
target { { { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump-not "optimized: basic block" "slp1" { target {
{ { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump "and its dominators costs" "slp1" { target { {
{ aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
diff --git a/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-9.c
b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-9.c
new file mode 100644
index 00000000000..0b1126e2810
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/bb-slp-pr127246-9.c
@@ -0,0 +1,25 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-additional-options "-fvect-cost-model=dynamic
-fno-tree-loop-vectorize" } */
+/* { dg-additional-options "-mtune=generic" { target aarch64*-*-* } } */
+
+/* The loop can leave through its first exit before it reaches the stores.
+ On that path the vector load of a and b and the extraction of both
+ lanes for the exit test are not paid for by the code after it. */
+
+void
+f (long *restrict p, long *restrict q, long *restrict r, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ long a = q[2 * i], b = q[2 * i + 1];
+ if (a == b)
+ break;
+ long c = r[2 * i], d = r[2 * i + 1];
+ p[2 * i] = ((a + c) ^ (a - c)) + ((a | c) & (a ^ c));
+ p[2 * i + 1] = ((b + d) ^ (b - d)) + ((b | d) & (b ^ d));
+ }
+}
+
+/* { dg-final { scan-tree-dump "and its dominators costs" "slp1" { target { {
{ aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
+/* { dg-final { scan-tree-dump-not "optimized: basic block" "slp1" { target {
{ { aarch64*-*-* || i?86-*-* } || x86_64-*-* } && lp64 } } } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/pr127246-6.c
b/gcc/testsuite/gcc.target/aarch64/pr127246-6.c
new file mode 100644
index 00000000000..adc9d745641
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/pr127246-6.c
@@ -0,0 +1,51 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-options "-O2 -mtune=generic -fdump-tree-slp-details" } */
+
+/* The construction of x and y and the additions are placed in the
+ conditional block, where the scalar code they replace runs. In g the
+ check of each block of the loop with its dominators therefore does not
+ see them in the loop header. */
+
+extern unsigned short foo (unsigned short);
+
+void
+f (unsigned short *out, unsigned short a, unsigned short b, int c)
+{
+ unsigned short x = foo (a), y = foo (b);
+ if (__builtin_expect (c, 0))
+ {
+ out[0] = x + 3;
+ out[1] = y + 5;
+ out[2] = x + 7;
+ out[3] = y + 9;
+ out[4] = x + 11;
+ out[5] = y + 13;
+ out[6] = x + 15;
+ out[7] = y + 17;
+ }
+}
+
+void
+g (unsigned short *restrict out, unsigned short *a, int *c, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ unsigned short x = foo (a[i]), y = foo (a[i + 1]);
+ if (c[i] == 42)
+ {
+ out[8 * i + 0] = x + 3;
+ out[8 * i + 1] = y + 5;
+ out[8 * i + 2] = x + 7;
+ out[8 * i + 3] = y + 9;
+ out[8 * i + 4] = x + 11;
+ out[8 * i + 5] = y + 13;
+ out[8 * i + 6] = x + 15;
+ out[8 * i + 7] = y + 17;
+ }
+ }
+}
+
+/* f has no loop and is vectorized by slp2, g by slp1. */
+/* { dg-final { scan-tree-dump "optimized: basic block part vectorized" "slp2"
} } */
+/* { dg-final { scan-tree-dump "optimized: basic block part vectorized" "slp1"
} } */
diff --git a/gcc/testsuite/gcc.target/aarch64/pr127246.c
b/gcc/testsuite/gcc.target/aarch64/pr127246.c
new file mode 100644
index 00000000000..cd6628ae1f0
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/pr127246.c
@@ -0,0 +1,16 @@
+/* PR tree-optimization/127246 */
+/* { dg-do compile } */
+/* { dg-require-effective-target lp64 } */
+/* { dg-options "-O2 -mtune=generic" } */
+
+#pragma GCC target "+nosve"
+
+/* Do not move the interpreter state between general and vector registers
+ on every dispatch. */
+
+#include "../../gcc.dg/vect/bb-slp-pr127246-1.c"
+
+/* { dg-final { scan-assembler-not {\tins\t} } } */
+/* { dg-final { scan-assembler-not {\tumov\t} } } */
+/* { dg-final { scan-assembler-not {\tfmov\t} } } */
+/* { dg-final { scan-assembler-not {\tdup\t} } } */
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index 5b00c5b472c..ab0072b8191 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -9491,25 +9491,34 @@ vect_slp_prune_covered_roots (slp_tree node,
hash_set<stmt_vec_info> &roots,
vect_slp_prune_covered_roots (child, roots, visited);
}
-/* Hand over COST_VEC to the target COSTS grouped by SLP node. */
+/* Hand over COST_VEC to the target COSTS grouped by SLP node. When BBS is
+ not NULL it holds the block of the code each cost describes. Then also
+ split groups at changes of the block and add the cost the target records
+ for each group to the entry of its block in BB_COSTS. */
static void
-add_slp_costs (vector_costs *costs, stmt_vector_for_cost& cost_vec)
+add_slp_costs (vector_costs *costs, stmt_vector_for_cost& cost_vec,
+ vec<basic_block> *bbs = nullptr,
+ hash_map<basic_block, int> *bb_costs = nullptr)
{
for (unsigned start = 0; start < cost_vec.length ();)
{
unsigned end = start + 1;
while (end < cost_vec.length ()
- && cost_vec[start].node == cost_vec[end].node)
+ && cost_vec[start].node == cost_vec[end].node
+ && (!bbs || (*bbs)[start] == (*bbs)[end]))
end++;
+ unsigned cost;
if (cost_vec[start].node)
- costs->add_slp_cost (cost_vec[start].node,
- array_slice<stmt_info_for_cost>
- (cost_vec.begin () + start, end - start));
+ cost = costs->add_slp_cost (cost_vec[start].node,
+ array_slice<stmt_info_for_cost>
+ (cost_vec.begin () + start, end - start));
else
- costs->vector_costs::add_slp_cost (cost_vec[start].node,
- array_slice<stmt_info_for_cost>
- (cost_vec.begin () + start, end - start));
+ cost = costs->vector_costs::add_slp_cost (cost_vec[start].node,
+ array_slice<stmt_info_for_cost>
+ (cost_vec.begin () + start, end - start));
+ if (bbs)
+ bb_costs->get_or_insert ((*bbs)[start]) += (int) cost;
start = end;
}
}
@@ -9872,20 +9881,61 @@ vect_bb_slp_node_bb (bb_vec_info bb_vinfo, slp_tree
node)
return node->si ? gimple_bb (node->si) : bb_vinfo->bbs[0];
}
+/* A cost entry together with the index of the loop and the block containing
+ the code it describes. */
+
+struct li_cost
+{
+ unsigned loop;
+ basic_block bb;
+ stmt_info_for_cost *cost;
+};
+
/* Comparator for the loop-index sorted cost vectors. */
static int
li_cost_vec_cmp (const void *a_, const void *b_, void *)
{
- auto *a = (const std::pair<unsigned, stmt_info_for_cost *> *)a_;
- auto *b = (const std::pair<unsigned, stmt_info_for_cost *> *)b_;
- if (a->first < b->first)
+ auto *a = (const li_cost *)a_;
+ auto *b = (const li_cost *)b_;
+ if (a->loop < b->loop)
return -1;
- else if (a->first == b->first)
+ else if (a->loop == b->loop)
return 0;
return 1;
}
+/* Return the vector minus the scalar cost of the code of BB and of the
+ blocks dominating it in LOOP. BB_COSTS holds that difference for each
+ block with costs. ACC memoizes the result for the blocks visited. */
+
+static int
+vect_bb_slp_dominator_cost (basic_block bb, class loop *loop,
+ hash_map<basic_block, int> &bb_costs,
+ hash_map<basic_block, int> &acc)
+{
+ auto_vec<basic_block, 16> chain;
+ int cost = 0;
+ for (basic_block b = bb; b && flow_bb_inside_loop_p (loop, b);
+ b = get_immediate_dominator (CDI_DOMINATORS, b))
+ {
+ if (int *a = acc.get (b))
+ {
+ cost = *a;
+ break;
+ }
+ chain.safe_push (b);
+ }
+ while (!chain.is_empty ())
+ {
+ basic_block b = chain.pop ();
+ if (int *c = bb_costs.get (b))
+ cost += *c;
+ acc.put (b, cost);
+ }
+ return cost;
+}
+
/* Check if vectorization of the basic block is profitable for the
subgraph denoted by SLP_INSTANCES. */
@@ -9948,30 +9998,30 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
loops stmts separately. */
/* First produce cost vectors sorted by loop index. */
- auto_vec<std::pair<unsigned, stmt_info_for_cost *> >
- li_scalar_costs (scalar_costs.length ());
- auto_vec<std::pair<unsigned, stmt_info_for_cost *> >
- li_vector_costs (vector_costs.length ());
+ auto_vec<li_cost> li_scalar_costs (scalar_costs.length ());
+ auto_vec<li_cost> li_vector_costs (vector_costs.length ());
stmt_info_for_cost *cost;
FOR_EACH_VEC_ELT (scalar_costs, i, cost)
{
- unsigned l = gimple_bb (cost->stmt_info->stmt)->loop_father->num;
- li_scalar_costs.quick_push (std::make_pair (l, cost));
+ basic_block bb = gimple_bb (cost->stmt_info->stmt);
+ li_scalar_costs.quick_push ({ (unsigned) bb->loop_father->num, bb,
+ cost });
}
- /* Use a random used loop as fallback in case the first vector_costs
+ /* Use a random used block as fallback in case the first vector_costs
entry does not have a location associated with it. */
- unsigned l = li_scalar_costs[0].first;
+ basic_block bb = li_scalar_costs[0].bb;
FOR_EACH_VEC_ELT (vector_costs, i, cost)
{
/* Use SLP node placement according to the computed schedule. */
if (cost->node)
- l = vect_bb_slp_node_bb (bb_vinfo, cost->node)->loop_father->num;
+ bb = vect_bb_slp_node_bb (bb_vinfo, cost->node);
/* SLP instance root stmts do not have an associated SLP node. */
else if (cost->stmt_info)
- l = gimple_bb (cost->stmt_info->stmt)->loop_father->num;
+ bb = gimple_bb (cost->stmt_info->stmt);
/* And since vect_prologue_cost_for_slp can end up costing with
neither, inherit from the previous node. */
- li_vector_costs.quick_push (std::make_pair (l, cost));
+ li_vector_costs.quick_push ({ (unsigned) bb->loop_father->num, bb,
+ cost });
}
li_scalar_costs.stablesort (li_cost_vec_cmp, NULL);
li_vector_costs.stablesort (li_cost_vec_cmp, NULL);
@@ -9988,9 +10038,12 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
|| vi < li_vector_costs.length ())
{
unsigned sl
- = si < li_scalar_costs.length () ? li_scalar_costs[si].first : -1U;
+ = si < li_scalar_costs.length () ? li_scalar_costs[si].loop : -1U;
unsigned vl
- = vi < li_vector_costs.length () ? li_vector_costs[vi].first : -1U;
+ = vi < li_vector_costs.length () ? li_vector_costs[vi].loop : -1U;
+ /* The vector minus the scalar cost of the code of each block of the
+ part. */
+ hash_map<basic_block, int> bb_costs;
class vector_costs *scalar_target_cost_data = nullptr;
scalar_cost = 0;
@@ -10002,12 +10055,13 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
scalar_target_cost_data = init_cost (bb_vinfo, true);
do
{
- add_stmt_cost (scalar_target_cost_data,
- li_scalar_costs[si].second);
+ bb_costs.get_or_insert (li_scalar_costs[si].bb)
+ -= (int) add_stmt_cost (scalar_target_cost_data,
+ li_scalar_costs[si].cost);
si++;
}
while (si < li_scalar_costs.length ()
- && li_scalar_costs[si].first == sl);
+ && li_scalar_costs[si].loop == sl);
scalar_target_cost_data->finish_cost (nullptr);
scalar_cost = scalar_target_cost_data->body_cost ();
total_scalar_cost += scalar_cost;
@@ -10028,14 +10082,16 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
"Vector cost for part in loop %d\n", vl);
class vector_costs *vect_target_cost_data = init_cost (bb_vinfo, false);
auto_vec<stmt_info_for_cost> tem;
+ auto_vec<basic_block> tem_bbs;
do
{
- tem.safe_push (*li_vector_costs[vi].second);
+ tem.safe_push (*li_vector_costs[vi].cost);
+ tem_bbs.safe_push (li_vector_costs[vi].bb);
vi++;
}
while (vi < li_vector_costs.length ()
- && li_vector_costs[vi].first == vl);
- add_slp_costs (vect_target_cost_data, tem);
+ && li_vector_costs[vi].loop == vl);
+ add_slp_costs (vect_target_cost_data, tem, &tem_bbs, &bb_costs);
vect_target_cost_data->finish_cost (scalar_target_cost_data);
vec_prologue_cost = vect_target_cost_data->prologue_cost ();
vec_inside_cost = vect_target_cost_data->body_cost ();
@@ -10076,6 +10132,39 @@ vect_bb_vectorization_profitable_p (bb_vec_info
bb_vinfo,
example). */
if (vec_outside_cost + vec_inside_cost > scalar_cost)
profitable = false;
+ /* The code of a block in the loop runs together with the code of the
+ blocks dominating it in the loop, but not necessarily with the code
+ of other blocks of the loop. So require each block with costs to be
+ profitable together with its dominators in the loop. The costs the
+ target records for single stmts do not include its final
+ adjustments, so leave parts with costs in one block to the check
+ above. */
+ else if (bb_costs.elements () > 1)
+ {
+ class loop *loop = get_loop (cfun, vl);
+ hash_map<basic_block, int> acc;
+ basic_block fail_bb = NULL;
+ int fail_cost = 0;
+ for (auto p : bb_costs)
+ {
+ int excess = vect_bb_slp_dominator_cost (p.first, loop, bb_costs,
+ acc);
+ if (excess > 0 && (!fail_bb || p.first->index < fail_bb->index))
+ {
+ fail_bb = p.first;
+ fail_cost = excess;
+ }
+ }
+ if (fail_bb)
+ {
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ " The vector code of bb%d and its dominators "
+ "costs %d more than the scalar code\n",
+ fail_bb->index, fail_cost);
+ profitable = false;
+ }
+ }
}
if (dump_enabled_p ())
diff --git a/gcc/tree-vectorizer.h b/gcc/tree-vectorizer.h
index ef04b8057aa..4deca8fccb5 100644
--- a/gcc/tree-vectorizer.h
+++ b/gcc/tree-vectorizer.h
@@ -1815,7 +1815,8 @@ public:
vect_cost_model_location where);
/* Update the costs in response to adding costs in V which are all from
- vectorizing NODE to the respective part. */
+ vectorizing NODE to the respective part. Return the sum of the costs
+ recorded for V. */
virtual unsigned int add_slp_cost (slp_tree node,
const array_slice<stmt_info_for_cost> &v);
--
2.50.1 (Apple Git-155)
