https://gcc.gnu.org/bugzilla/show_bug.cgi?id=118407
--- Comment #28 from GCC Commits <cvs-commit at gcc dot gnu.org> --- The master branch has been updated by Tamar Christina <[email protected]>: https://gcc.gnu.org/g:cf99490e7ebcf72e3478454448dc265a32723240 commit r17-4704-gcf99490e7ebcf72e3478454448dc265a32723240 Author: Tamar Christina <[email protected]> Date: Sun Sep 27 23:00:01 2026 +0100 vect: fix loop exit profiles with early break [PR118407] It's been a few years since the patches for PR118407 but have been stuck on a review. Those patches attempted to make an framework for supporting early break profiles in not just the vectorizer but in other passes as well. This patch takes a different approach. We know that the vectorizer is the one introducing the inaccurate profile information, so have it fix it up. There are other places in the vectorizer where this happens for non-early break too. As an example this loop unsigned vect_a[N]; unsigned vect_b[N]; unsigned test4(unsigned x) { unsigned ret = 0; for (int i = 0; i < N; i++) { vect_b[i] = x + i; if (vect_a[i]*2 != x) break; vect_a[i] = x; } return ret; } generates with -O3 -march=armv9-a the following loop: test4: cntw x1 cmp w1, 799 bhi .L7 adrp x4, .LANCHOR0 add x3, x4, :lo12:.LANCHOR0 ptrue p7.b, all add x5, x3, 3200 mov w1, 0 mov z30.s, w0 mov w6, 800 index z29.s, #0, #1 b .L4 .L3: st1w z28.s, p7, [x5, x1, lsl 2] st1w z30.s, p7, [x3, x1, lsl 2] incw z29.s incw x1 whilelo p7.s, w1, w6 b.none .L5 .L4: ld1w z31.s, p7/z, [x3, x1, lsl 2] add z28.s, z29.s, z30.s add z31.s, z31.s, z31.s cmpne p15.s, p7/z, z31.s, z30.s b.none .L3 mov w5, 800 mov w2, w1 sub w5, w5, w1 .L2: notice the pointless indirect branch to L4, and the commonly taken branch of .L3 has to do a branch back to the latch instead of a fall through. The CFG becomes increasingly branchy as the number of exits increase. With this patch we generate: test4: cntw x1 cmp w1, 799 bhi .L7 adrp x3, .LANCHOR0 add x3, x3, :lo12:.LANCHOR0 ptrue p7.b, all add x4, x3, 3200 mov w1, 0 mov z30.s, w0 mov w5, 800 index z29.s, #0, #1 .p2align 5,,15 .L4: ld1w z31.s, p7/z, [x3, x1, lsl 2] add z28.s, z29.s, z30.s add z31.s, z31.s, z31.s cmpne p15.s, p7/z, z31.s, z30.s b.any .L12 st1w z28.s, p7, [x4, x1, lsl 2] st1w z30.s, p7, [x3, x1, lsl 2] incw z29.s incw x1 whilelo p7.s, w1, w5 b.any .L4 .L5: mov w0, 0 ret which is a more natural layout because it now knows the exit is unlikely since they have been scaled correctly. The scaling is done by: let P be the scalar probability that one iteration takes an early exit edge. 1 - P is then the probabiliy that the exit is not taken. Since every vector iteration handles VF scalar iterations, the early exit branch can be taken when any of the lanes are true. We invert this and first calculate the probablity that you reach the latch, i.e. the probability that the vector code does not exit is the probability Q that none of the lanes are true. Q = (1 - P) * (1 - P) * ... * (1 - P) = (1 - P) ^ VF and so the probability that at least one lane breaks is 1 - Q, or rather 1 - (i - P) ^ VF. For example, if the scalar break probability is 10% and VF is 4: P = 0.1 1 - P = 0.9 (1 - P) ^ VF = 0.9 ^ 4 = 0.6561 1 - (1 - p) ^ VF = 1 - 0.6561 = 0.3439 So a scalar early-exit edge that is taken 10% of the time becomes a vector early-exit edge that is taken about 34.4% of the time, because each vector iteration gives the break condition four chances to trigger. For counted loops we need to exclude the main exit from the calculations as for those the main exit is counted and so does not have this per-lane probability. For uncounted loops, all exits are based on vector compare, i.e. none of the exits are counted and we have a different loop control. That means for uncounted loops all exits should be using the per-lane probability and we should skip the counted exit adjustments in scale_profile_for_vect_loop. NOTE: This does not fully fix PR118407 but only fixes part of the issue. there are other adjustments needed but this is the most glaring one. gcc/ChangeLog: PR tree-optimization/118407 * tree-vect-loop.cc (vect_update_early_break_profiles): New. (scale_profile_for_vect_loop): Use it and pass loop_vinfo. (vect_transform_loop): Pass loop_vinfo to scale_profile_for_vect_loop. gcc/testsuite/ChangeLog: PR tree-optimization/118407 * gcc.target/aarch64/noeffect2.c: Update tests. * gcc.target/aarch64/sve/noeffect2.c: Update test. * gcc.target/aarch64/sve/noeffect3.c: Likewise.
