From: Kyrylo Tkachov <[email protected]> This is an alternative to https://gcc.gnu.org/pipermail/gcc-patches/2026-August/728947.html, which answered Honza's comments at https://gcc.gnu.org/pipermail/gcc-patches/2026-August/728898.html. That patch gives each target of a computed goto an even, guessed share of the remaining probability. This patch recovers the measured count of each target instead. This is better: the targets keep their measured distribution, so bb-reorder knows the likely target, which was a point of the review. The entry count and the consistency of the profile also become correct, and 728947 does not fix these. Ignored edges that are not computed goto edges, for example those of setjmp receivers, keep their current probabilities.
branch_prob cannot instrument abnormal edges. It replaces each of them by a fake edge from the entry block to its destination and a fake edge from its source to the exit block, and ignores the abnormal edge. The ignored edge keeps a zero count, so compute_branch_probabilities gives every edge of a factored computed goto a precise never probability, although the blocks it reaches have measured counts. Since bae9c5e7c6e, rebuild_frequencies calls determine_unlikely_bbs for read profiles. Its forward propagation cannot pass the dispatch, so it sets the count of the whole dispatch loop to zero. In CPython 3.14 built with profile feedback, 2882 of the 2932 blocks of _PyEval_EvalFrameDefault end up as never executed. The fake flow also inflates the entry count, so the node count of that function counts its dispatches as calls. When the only abnormal predecessor of a block is a computed goto, the fake edge from the entry block carries exactly the flow of that edge. Move the flow back to the abnormal edge and remove it from the fake edge of the computed goto and from the entry and exit counts. The profile stays flow-consistent, the dispatch keeps its measured targets and the entry count becomes the number of calls again. 677 of the 3157 blocks of _PyEval_EvalFrameDefault keep a zero count, the handlers that the training run did not execute. In gcc.dg/tree-prof/pr127246-1.c, run is called 10 times and dispatches 20020 times. The edges from its dispatch change as follows, and the node count of run changes from 20030 to 10: target before after op_inc [never] count:0 [0.0%] count:10 op_dec [never] count:0 [50.0%] count:10000 op_jnz [never] count:0 [50.0%] count:10000 op_halt [never] count:0 [0.0%] count:10 On CPython 3.14.5 built with PGO and LTO on aarch64 (Neoverse V2), this reduces the geometric mean run time of 17 pyperformance benchmarks by 29.8%. With -fno-tree-slp-vectorize, which avoids the vectorization problem of the PR, it reduces it by 8.6% compared to the unpatched compiler with the same option. Bootstrapped and regression tested on aarch64-unknown-linux-gnu and x86_64-pc-linux-gnu. Ok for trunk? Thanks, Kyrill Cc: Jan Hubicka <[email protected]> gcc/ChangeLog: PR gcov-profile/127246 * profile.cc (compute_branch_probabilities): Move the flow of the fake edges of a computed goto back to its abnormal edges. gcc/testsuite/ChangeLog: PR gcov-profile/127246 * gcc.dg/tree-prof/pr127246-1.c: New test. * gcc.dg/tree-prof/pr127246-2.c: New test. Signed-off-by: Kyrylo Tkachov <[email protected]> --- gcc/profile.cc | 55 +++++++++++++++++++++ gcc/testsuite/gcc.dg/tree-prof/pr127246-1.c | 53 ++++++++++++++++++++ gcc/testsuite/gcc.dg/tree-prof/pr127246-2.c | 12 +++++ 3 files changed, 120 insertions(+) create mode 100644 gcc/testsuite/gcc.dg/tree-prof/pr127246-1.c create mode 100644 gcc/testsuite/gcc.dg/tree-prof/pr127246-2.c diff --git a/gcc/profile.cc b/gcc/profile.cc index 52704e93866..82b21db6e58 100644 --- a/gcc/profile.cc +++ b/gcc/profile.cc @@ -668,6 +668,61 @@ compute_branch_probabilities (unsigned cfg_checksum, unsigned lineno_checksum) error ("corrupted profile info: profile data is not flow-consistent"); } + /* branch_prob replaced each abnormal edge by fake edges from the entry + block to its destination and from its source to the exit block. When + the only abnormal predecessor of a block is a computed goto, the fake + edge from the entry block carries exactly the flow of that edge. Move + it back to the edge so that the dispatch keeps its measured targets. */ + if (!inconsistent || flag_profile_correction) + { + for (edge fake : ENTRY_BLOCK_PTR_FOR_FN (cfun)->succs) + { + if (!(fake->flags & EDGE_FAKE) || EDGE_INFO (fake)->ignore) + continue; + + edge abnormal = NULL; + unsigned num_ignored = 0; + for (edge e : fake->dest->preds) + if (EDGE_INFO (e)->ignore) + { + abnormal = e; + num_ignored++; + } + if (num_ignored != 1) + continue; + gimple *last = last_nondebug_stmt (abnormal->src); + if (!last || !computed_goto_p (last)) + continue; + + /* The computed goto has no other way out than its abnormal edges, + so its fake edge to the exit block is on the spanning tree. */ + edge out = find_edge (abnormal->src, EXIT_BLOCK_PTR_FOR_FN (cfun)); + gcc_checking_assert (out && (out->flags & EDGE_FAKE) + && !EDGE_INFO (out)->ignore); + /* Counters of a program with threads need not be consistent, so do + not move more than the computed goto has left. */ + gcov_type flow = MIN (edge_gcov_count (fake), edge_gcov_count (out)); + if (flow <= 0) + continue; + + edge_gcov_count (abnormal) = flow; + edge_gcov_count (fake) -= flow; + edge_gcov_count (out) -= flow; + bb_gcov_count (ENTRY_BLOCK_PTR_FOR_FN (cfun)) -= flow; + bb_gcov_count (EXIT_BLOCK_PTR_FOR_FN (cfun)) -= flow; + /* An emptied fake edge was not measured to be dead, so keep it from + turning a partial profile into a guess below. */ + if (!edge_gcov_count (fake)) + fake->probability = profile_probability::never (); + if (!edge_gcov_count (out)) + out->probability = profile_probability::never (); + if (dump_file) + fprintf (dump_file, "Computed goto edge from %i to %i, count: %" + PRId64 "\n", abnormal->src->index, abnormal->dest->index, + (int64_t) flow); + } + } + /* For every edge, calculate its branch probability and add a reg_note to the branch insn to indicate this. */ diff --git a/gcc/testsuite/gcc.dg/tree-prof/pr127246-1.c b/gcc/testsuite/gcc.dg/tree-prof/pr127246-1.c new file mode 100644 index 00000000000..81897fe9b56 --- /dev/null +++ b/gcc/testsuite/gcc.dg/tree-prof/pr127246-1.c @@ -0,0 +1,53 @@ +/* PR gcov-profile/127246 */ +/* { dg-require-effective-target label_values } */ +/* { dg-require-effective-target indirect_jumps } */ +/* { dg-options "-O2 -fdump-ipa-profile -fdump-tree-rebuild_frequencies1-details -fdump-rtl-compgotos" } */ + +/* The edges of a factored computed goto cannot be instrumented. Their + counts must be recovered from the profile, or the measured counts of the + whole dispatch loop are lost. op_inc is also reached by a normal edge. */ + +enum { OP_INC, OP_DEC, OP_JNZ, OP_HALT }; + +__attribute__ ((noipa)) long +run (const unsigned char *pc, long n) +{ + static void *const labels[] = { &&op_inc, &&op_dec, &&op_jnz, &&op_halt }; + long acc = 0; + goto *labels[*pc]; + op_inc: + acc++; + pc++; + goto *labels[*pc]; + op_dec: + n--; + pc++; + goto *labels[*pc]; + op_jnz: + if (n) + { + pc -= 2; + goto op_inc; + } + pc++; + goto *labels[*pc]; + op_halt: + return acc; +} + +static const unsigned char prog[] = { OP_INC, OP_DEC, OP_JNZ, OP_HALT }; + +int +main (void) +{ + for (int i = 0; i < 10; i++) + if (run (prog, 1000) != 1000) + __builtin_abort (); + return 0; +} + +/* { dg-final-use-not-autofdo { scan-ipa-dump-times "Computed goto edge from \[0-9\]+ to \[0-9\]+, count: \[1-9\]" 4 "profile" } } */ +/* { dg-final-use-not-autofdo { scan-tree-dump-not "marked unlikely by (forward|backward) prop" "rebuild_frequencies1" } } */ +/* { dg-final-use-not-autofdo { scan-tree-dump-not "Profile is inconsistent" "rebuild_frequencies1" } } */ +/* { dg-final-use-not-autofdo { scan-ipa-dump "Function flags: count:10 \\(precise\\)" "profile" } } */ +/* { dg-final-use-not-autofdo { scan-rtl-dump "Duplicating computed goto" "compgotos" { target { aarch64*-*-* || x86_64-*-* } } } } */ diff --git a/gcc/testsuite/gcc.dg/tree-prof/pr127246-2.c b/gcc/testsuite/gcc.dg/tree-prof/pr127246-2.c new file mode 100644 index 00000000000..7bfce713d17 --- /dev/null +++ b/gcc/testsuite/gcc.dg/tree-prof/pr127246-2.c @@ -0,0 +1,12 @@ +/* PR gcov-profile/127246 */ +/* { dg-require-effective-target label_values } */ +/* { dg-require-effective-target indirect_jumps } */ +/* { dg-options "-O2 -fprofile-partial-training -fdump-ipa-profile -fdump-tree-optimized-blocks-details" } */ + +/* With -fprofile-partial-training the recovered counts of the computed goto + edges must stay measured rather than become guesses. */ + +#include "pr127246-1.c" + +/* { dg-final-use-not-autofdo { scan-ipa-dump-times "Computed goto edge from \[0-9\]+ to \[0-9\]+, count: \[1-9\]" 4 "profile" } } */ +/* { dg-final-use-not-autofdo { scan-tree-dump-not "\\(guessed\\)\\\]\[^\\n\\r\]*\\(ABNORMAL" "optimized" } } */ -- 2.50.1 (Apple Git-155)
