https://gcc.gnu.org/bugzilla/show_bug.cgi?id=128288
Bug ID: 128288
Summary: Struct filled by a 4-iteration loop is copied through
the stack with 16-byte loads after 8-byte stores (loop
unrolled only after SRA)
Product: gcc
Version: 16.2.0
Status: UNCONFIRMED
Keywords: missed-optimization
Severity: normal
Priority: P3
Component: tree-optimization
Assignee: unassigned at gcc dot gnu.org
Reporter: chfast at gmail dot com
CC: chfast at gmail dot com
Target Milestone: ---
Target: x86_64-pc-linux-gnu
struct u256 { unsigned long long w[4]; };
void add(struct u256 *r, const struct u256 *x, const struct u256 *y)
{
struct u256 s;
unsigned long long k = 0;
for (int i = 0; i < 4; ++i)
s.w[i] = __builtin_addcll(x->w[i], y->w[i], k, &k);
*r = s;
}
GCC 16.2 -O3 (the same with GCC 14, 15 and trunk):
add:
movq (%rsi), %rax
addq (%rdx), %rax
movq %rax, -40(%rsp)
movq 8(%rsi), %rax
adcq 8(%rdx), %rax
movq %rax, -32(%rsp)
movq 16(%rsi), %rax
movdqa -40(%rsp), %xmm0
adcq 16(%rdx), %rax
movq %rax, -24(%rsp)
movq 24(%rdx), %rax
adcq 24(%rsi), %rax
movups %xmm0, (%rdi)
movq %rax, -16(%rsp)
movdqa -24(%rsp), %xmm0
movups %xmm0, 16(%rdi)
ret
The words of s are stored with 8-byte stores and read back with 16-byte loads,
so store-to-load forwarding fails. cunrolli does not unroll the loop ("Not
unrolling loop 1: size would grow."), so SRA sees s.w[i] with a variable index
("Disqualifying s - No scalar replacements to be created."). cunroll unrolls
the loop later, after SRA.
With #pragma GCC unroll 4 the loop is unrolled by cunrolli, s is scalarized and
the words are stored to *r directly:
add:
movq (%rdx), %rax
addq (%rsi), %rax
movq 8(%rsi), %rcx
movq %rax, %r8
adcq 8(%rdx), %rcx
movq 16(%rsi), %rax
movq 24(%rsi), %rsi
adcq 16(%rdx), %rax
adcq 24(%rdx), %rsi
movq %r8, (%rdi)
movq %rcx, 8(%rdi)
movq %rax, 16(%rdi)
movq %rsi, 24(%rdi)
ret
Calling add in a loop takes 9.2 ns per call at -O3, 1.8 ns with the pragma and
1.2 ns with clang -O3 (Ryzen 9 5950X). We see the same store and copy pattern
in intx (a multi-precision integer library), in the results of its division and
addmod code.
Related: PR80689 (struct copy with 128-bit loads after scalar stores), PR128248
(the same loop is not unrolled at -O2).
