https://gcc.gnu.org/bugzilla/show_bug.cgi?id=128211
Bug ID: 128211
Summary: [17 Regression] 538.imagick_r 9% slower on AArch64
since r17-3330
Product: gcc
Version: 17.0
Status: UNCONFIRMED
Severity: normal
Priority: P3
Component: target
Assignee: unassigned at gcc dot gnu.org
Reporter: dhruvc at gcc dot gnu.org
CC: rguenth at gcc dot gnu.org
Target Milestone: ---
Target: aarch64
This one was quite tricky to bisect, had to use an LLM for help (also for
phrasing the bug text).
------
538.imagick_r is ~9% slower on Neoverse-V2 since r17-3330-g4fe2072d5d16ce
("tree-optimization/126887 - allow backward threading to loop exit"), and still
is on trunk (r17-5571-gccd612d2b9f).
The threading change itself is not at fault: it changes register allocation in
MeanShiftImage, and the new assignment runs into a store-to-load forwarding
behaviour of the core for STP/LDP pairs.
This does get fixed by reverting r17-3330-g4fe2072d5d16ce.
Flags: -Ofast -flto=auto -march=native -mcpu=grace
What happens
------------
All of the time is in MeanShiftImage's innermost loop, which calls
GetVirtualPixelsFromNexus once per sample. That callee saves and restores
callee-saved registers in pairs:
stp x21, x22, [sp, #32] // entry
... (uses x21, x22 as scratch)
ldp x21, x22, [sp, #32] // exit
With r17-3330 the caller keeps two values across the call in that pair:
x21 cache_view loop-invariant: base of the next call's address chain
(cache_view->image->cache->type, nexus_info, ...)
x22 count loop-carried: cinc after the floating-point distance
compare on the pixel just returned
The LDP does not hand back x21 until the STP's x22 data is available. So every
call's address chain waits for the previous iteration's load + FP compare
chain, and iterations that used to overlap now run back to back. Without
r17-3330, count is in x24 (paired with x23, which only feeds a predicted
branch) and cache_view is in x22 (paired with a constant), so nothing late is
tied to the pointer.
Min-repro
---------
The generated min-repro is a little strange as it relies on forcing register
saving in the callee, but it seemed to be the easiest way to do this without
copying the source from imagick itself:
====
/* AArch64: a pointer restored by a callee's LDP waits for the late value in
the other half of the pair.
callee saves and restores x21..x24 as stp/ldp x21, x22 and
stp/ldp x23, x24. loop keeps a loop-invariant pointer in x21 and a
counter that depends on the value returned by the call in COUNT_REG:
gcc -O2 -DCOUNT_REG='"x22"' pairdep.c counter shares the pointer's pair
gcc -O2 -DCOUNT_REG='"x24"' pairdep.c counter in the other pair */
#include <stdio.h>
#include <time.h>
#ifndef COUNT_REG
#define COUNT_REG "x22"
#endif
struct inner { long pad[2]; int idx; float data[64]; };
struct outer { long pad; struct inner *in; };
/* Two dependent loads, and x21..x24 used as scratch so the prologue and
epilogue save and restore them in pairs. */
__attribute__ ((noipa)) const float *
callee (const struct outer *o)
{
const struct inner *in = o->in;
__asm__ volatile ("" ::: "x21", "x22", "x23", "x24");
return in->data + in->idx;
}
__attribute__ ((noipa)) long
loop (long n, struct outer **ptr_init, float ref, float limit)
{
register struct outer **ptr asm ("x21") = ptr_init;
register long count asm (COUNT_REG) = 0;
for (long i = 0; i < n; i++)
{
/* Keep PTR and COUNT in their registers across the call. */
__asm__ ("" : "+r" (ptr), "+r" (count));
const float *p = callee (*ptr);
float d = (*p - ref) * (*p - ref);
count += (d * d + d <= limit); /* branchless: cinc */
}
return count;
}
int
main (void)
{
static struct inner in = { { 0, 0 }, 0, { 1.5f } };
static struct outer o = { 0, &in };
static struct outer *o_p = &o;
long n = 200000000;
struct timespec t0, t1;
clock_gettime (CLOCK_MONOTONIC, &t0);
long count = loop (n, &o_p, 1.0f, 2.0f);
clock_gettime (CLOCK_MONOTONIC, &t1);
double s = (t1.tv_sec - t0.tv_sec) + 1e-9 * (t1.tv_nsec - t0.tv_nsec);
printf ("count in %s: %.3f s %.2f ns/iter count=%ld\n", COUNT_REG, s,
s * 1e9 / n, count);
return 0;
}
====
------
Compiler explorer: https://godbolt.org/z/n3orzsxzd
Note: I am not entirely sure about the accuracy of the mechanism reported by
the LLM, but it does make sense because the backend stalls seem to go up 3x as
measured using perf. Cycles also go up ~14%. The other counters seem to stay
roughly the same, both in the SPEC benchmark and in the min-repro (those being
instructions retired, ops speculated, branch mispredictions etc.)
