From: Kyrylo Tkachov <[email protected]>
branch_prob () cannot place a counter on an abnormal edge. It
represents the flow of a computed goto with fake edges. One fake edge
goes from the entry block to each target. Another goes from the
dispatch block to the exit block. branch_prob () also marks the
abnormal edges as ignored. An ignored edge that is not on the spanning
tree is not instrumented. The graph solver does not infer its count.
Its count therefore remains at the default zero, whatever the program
did.
compute_branch_probabilities () then turns that zero into a precise
never. A precise never is proof of deadness.
propagate_unlikely_bbs_forward () cannot reach any dispatch target from
the entry block, and determine_unlikely_bbs () zeroes the measured
counts of the whole region. For a threaded interpreter that region is
the entire hot loop. Thus, -fprofile-use reads measured block counts
and then discards them.
For
static const void *tab[] = { &&L_ADD, ... };
goto *tab[*pc++];
-fdump-ipa-profile shows 5000 executions for the dispatch block and 1000
for each handler. The optimized dump then reads
;; basic block 5, loop depth 1, count 0 (precise), probably never executed
for every one of them.
Record the probability of an ignored edge outside the spanning tree as a
guess. A guessed never is still a zero probability. It does not invent
a split between the dispatch targets. A target with a measured zero
count also stays at zero. The guessed probability stops the unmeasured
edge count from being evidence that the target is dead. The same dump
now reads
;; basic block 5, loop depth 1, count 5000 (precise), maybe hot
The rule is applied per edge and attributes no flow. It does not depend
on how many dispatch blocks a function has. It also needs no notion of
which dispatch a target belongs to.
Evaluated on aarch64 (NVIDIA Vera) with CPython 3.14.5 at -O3. CPython
was configured with --enable-optimizations and --with-lto. In
Python/ceval.c, the share of the basic blocks of
_PyEval_EvalFrameDefault left with a precise zero count falls from 98.3%
to 23.1%. During the full-program LTO link, the function grows by 0.7%
and the interpreter text grows by 0.1%. A fixed-work
pyperformance-derived workload takes 2.5% fewer cycles.
Bootstrapped and tested on aarch64-none-linux-gnu.
gcc/ChangeLog:
* profile.cc (compute_branch_probabilities): Mark probabilities of
ignored edges outside the spanning tree as guessed.
gcc/testsuite/ChangeLog:
* gcc.dg/tree-prof/comp-goto-2.c: New test.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
gcc/profile.cc | 5 ++
gcc/testsuite/gcc.dg/tree-prof/comp-goto-2.c | 61 ++++++++++++++++++++
2 files changed, 66 insertions(+)
create mode 100644 gcc/testsuite/gcc.dg/tree-prof/comp-goto-2.c
diff --git a/gcc/profile.cc b/gcc/profile.cc
index 5ae64d4043f..1683794e5e3 100644
--- a/gcc/profile.cc
+++ b/gcc/profile.cc
@@ -719,6 +719,11 @@ compute_branch_probabilities (unsigned cfg_checksum,
unsigned lineno_checksum)
bool prev_never = e->probability == profile_probability::never ();
e->probability = profile_probability::probability_in_gcov_type
(edge_gcov_count (e), bb_gcov_count (bb));
+ /* Ignored edges outside the spanning tree have neither a counter
+ nor an inferred count. Do not treat their zero count as a
+ precise never. */
+ if (EDGE_INFO (e)->ignore && !EDGE_INFO (e)->on_tree)
+ e->probability = e->probability.guessed ();
if (e->probability == profile_probability::never ()
&& !prev_never
&& flag_profile_partial_training)
diff --git a/gcc/testsuite/gcc.dg/tree-prof/comp-goto-2.c
b/gcc/testsuite/gcc.dg/tree-prof/comp-goto-2.c
new file mode 100644
index 00000000000..d005b52bc60
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/tree-prof/comp-goto-2.c
@@ -0,0 +1,61 @@
+/* The ignored edges out of a computed goto are outside the spanning tree,
+ so their zero count used to become a precise "never" and mark every opcode
+ handler as dead. Check that the measured counts survive -fprofile-use. */
+/* { dg-options "-O2 -fdump-tree-optimized-blocks-details" } */
+/* { dg-require-effective-target indirect_jumps } */
+/* { dg-require-effective-target label_values } */
+
+enum { OP_ADD, OP_SUB, OP_MUL, OP_STORE, OP_HALT, NOPS };
+
+unsigned char code[8];
+long mem[4] = { 1, 2, 3, 4 };
+
+__attribute__((noinline)) long
+interp (const unsigned char *pc)
+{
+ static const void *const tab[NOPS]
+ = { &&L_ADD, &&L_SUB, &&L_MUL, &&L_STORE, &&L_HALT };
+ long acc = 0;
+
+#define DISPATCH() goto *tab[*pc++]
+
+ DISPATCH ();
+L_ADD:
+ acc += mem[*pc++];
+ DISPATCH ();
+L_SUB:
+ acc -= mem[*pc++];
+ DISPATCH ();
+L_MUL:
+ acc *= 3;
+ DISPATCH ();
+L_STORE:
+ mem[*pc++] = acc;
+ DISPATCH ();
+L_HALT:
+ return acc;
+}
+
+int
+main (void)
+{
+ long r = 0;
+ int i;
+
+ code[0] = OP_ADD; code[1] = 1;
+ code[2] = OP_SUB; code[3] = 2;
+ code[4] = OP_MUL;
+ code[5] = OP_STORE; code[6] = 3;
+ code[7] = OP_HALT;
+
+ for (i = 0; i < 1000; i++)
+ r += interp (code);
+
+ return r == 0;
+}
+
+/* No block of the dispatch loop may be left with a precise zero count. */
+/* { dg-final-use-not-autofdo { scan-tree-dump-not "count 0 \\(precise"
"optimized" } } */
+/* The ignored non-spanning-tree edges carry no measurement, so they must be
+ a guess. */
+/* { dg-final-use-not-autofdo { scan-tree-dump "never \\(guessed\\)"
"optimized" } } */
--
2.50.1 (Apple Git-155)
