From: Kyrylo Tkachov <[email protected]>
LSE has been around for more than a decade and most (all?) datacenter AArch64
machines have it these days. However, distros need to support Armv8.0-A
machines as well sometimes, which is why we have -moutline-atomics.
These patches attempt to maintain that v8.0 compatibility while shifting the
performance benefits in LSE's favour as much as possible.
The current -moutline-atomics scheme has a number of drawbacks.
It follows the base PCS, so the register allocator has to assume that
x0 to x18, v0 to v7, v16 to v31, every SVE and predicate register and the
condition flags are dead across an operation whose real footprint is a
handful of registers. The function stops being a leaf and gets a frame.
A value that has to survive the call is
parked in a callee-saved register that the frame then has to save. Sibling
calls are lost. And the availability test is repeated at every call site,
where it could have been hoisted out of the loop.
This series adds -moutline-atomics=inline, which performs the test inline,
uses an LSE instruction directly, and expands the ARMv8.0 sequence on a
branch that is marked as unlikely not taken. No call is generated on either
path.
-moutline-atomics and -mno-outline-atomics become spellings of =call and
=none, so nothing changes for anyone who does not ask for it.
```
int incr (int *p) { return __atomic_fetch_add (p, 1, __ATOMIC_RELAXED); }
=call =inline
stp x29, x30, [sp, -16]! mov x1, x0
mov x1, x0 adrp x0, __aarch64_have_lse_atomics
mov w0, 1 ldrb w0, [x0, #:lo12:...]
mov x29, sp cbz w0, .L2
bl __aarch64_ldadd4_relax mov w0, 1
ldp x29, x30, [sp], 16 ldadd w0, w0, [x1]
ret ret
.L2:
ldxr w0, [x1]
add w2, w0, 1
stxr w3, w2, [x1]
cbnz w3, .L2
ret
```
This is not a new idea. Richard Henderson raised it in the cover letter of
the original outline atomics submission, saying that a custom calling
convention for the helpers "can come later as an optimization", and Wilco
also supported it.
[1/2] aarch64: add -moutline-atomics=inline
[2/2] aarch64: enable -moutline-atomics=inline by default
Patch 1 is the main one that implements the scheme.
Patch 2 proposes setting -moutline-atomics=inline as default, which I think is
the right way forward as it is backwards compatible and has a number of
benefits.
Evaluation
----------
Measured on Grace (Neoverse V2) and Vera (Olympus), against =call.
```
case Grace Vera
RocksDB HyperClockCache lookup -13.2% -9.6%
RocksDB LRUCache lookup (control) +0.4% -0.4%
refcount with live FP state -26.3% -18.4%
six live SIMD values across an atomic -20.3% -15.8%
tight uncontended loop -5.1% -3.2%
```
Rebuilding only glibc, leaving the application binary untouched so that the
C library is the only variable, libc.so goes from 926 out-of-line atomic
call sites to none, and glibc's own bench-malloc-simple gives:
```
row Grace Vera
main_arena_mt_allocs_0025 -11.2% -0.3%
thread_arena__allocs_0100 -10.8% -1.7%
main_arena_st_allocs_0100 (control) +0.1% -0.1%
```
Even for the LL/SC Armv8.0 path the code shouldn't be slower, though I haven't
put effort in measuring it, just the correctness. It should still benefit from
less stack frame setup, and from hoisting availability checks.
Code size grows: 0.86 percent of .text on a RocksDB binary with 1928 atomic
sites, 1.75 percent on glibc. -Os keeps the calls.
What do folks think?
Thanks,
Kyrill
