From: Kyrylo Tkachov <[email protected]>
For a loop such as
void copyReverseGeneric (int *dst, int *src)
{
for (int i = 0; i < 10000; ++i)
dst[i] = __builtin_bswap32 (src[i]);
}
vectorised and unrolled by two, ivopts can address the memory references
either with a single shared "index" IV and register-offset addressing
(reg + reg), or with one pointer IV per stream and immediate-offset
addressing (reg + imm). It prices the two forms equally, so it falls
back on its existing preference for fewer IVs and picks the shared index.
On targets without a "base + index + offset" addressing mode that is
unfortunate. Since the constant offset cannot be folded in, the shared
index needs a separate base register for every consecutive access (e.g.
src and src + 16), whereas a single per-base pointer IV reaches all of a
stream's accesses through immediate offsets. So even without pairing the
register-offset form wastes registers held live across the loop, and
where the target has load/store-pair instructions it additionally
prevents the accesses from being combined into a pair. The
immediate-offset form avoids both.
The distinguishing signal is whether the target can fold a constant
displacement into a register-offset address. A new helper,
addr_reg_offset_imm_valid_p, probes this with valid_mem_ref_p. When the
target cannot, and the use has a pairing sibling (from the preceding
patch), get_address_cost charges one extra unit to the register-offset
form when optimising for speed, so selection prefers the immediate-offset
form once enough accesses would pair to pay for the extra IV.
AFAICT ivopts already preferences fewer candidates, chiefly through the
"return cost + n_cands" term in ivopts_estimate_reg_pressure.
The cost added here stays out of that
machinery: it is not a function of the candidate count, but is charged
per address use and only to the register-offset form, entering the set
cost solely through cand_use_cost, never through cand_cost or the n_cands
term. It therefore adds no per-candidate bias and does not make larger
candidate sets unattractive; it only raises the price of register-offset
addressing for pairable accesses. Like every address cost it is scaled
by block frequency (one unit per use in the common single-loop case), and
accumulated over a group it is of the same order as the one extra IV the
immediate-offset form needs, so an isolated access is left alone.
The heuristic is implemented in target-independent code and adds no new
target hooks; it uses the existing valid_mem_ref_p query to test whether
the target can fold a constant offset into a register-offset address. It
is motivated by aarch64, which cannot fold the offset and which has
LDP/STP. On targets that can fold it, such as x86, the change is a
no-op: the penalty fires zero times and a bootstrap there is bit-identical
to an unpatched one. For the example above the loop now compiles to a
post-increment LDP/STP on aarch64:
.L3:
ldp q31, q30, [x2], 32
rev32 v31.16b, v31.16b
rev32 v30.16b, v30.16b
stp q31, q30, [x3], 32
cmp x2, x0
bne .L3
Bootstrapped and regression-tested on aarch64-unknown-linux-gnu with no
regressions; also bootstrapped on x86_64-pc-linux-gnu, where it is a
no-op. This triggers ~5k times in SPEC2026 where it produces more
LDP/STPs and reduces the overall text size.
Ok for trunk?
Thanks,
Kyrill
gcc/ChangeLog:
PR target/121315
* tree-ssa-loop-ivopts.cc (addr_reg_offset_imm_valid_p): New function.
(get_address_cost): Charge a small extra cost for register-offset
addressing of a use that has a pairing sibling when the target cannot
fold a constant offset into the address.
gcc/testsuite/ChangeLog:
PR target/121315
* gcc.target/aarch64/pr121315.c: New test.
* gcc.target/aarch64/pr121315-2.c: New test.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
gcc/testsuite/gcc.target/aarch64/pr121315-2.c | 31 +++++++++++
gcc/testsuite/gcc.target/aarch64/pr121315.c | 15 ++++++
gcc/tree-ssa-loop-ivopts.cc | 51 +++++++++++++++++++
3 files changed, 97 insertions(+)
create mode 100644 gcc/testsuite/gcc.target/aarch64/pr121315-2.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/pr121315.c
diff --git a/gcc/testsuite/gcc.target/aarch64/pr121315-2.c
b/gcc/testsuite/gcc.target/aarch64/pr121315-2.c
new file mode 100644
index 00000000000..c76b4e0480c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/pr121315-2.c
@@ -0,0 +1,31 @@
+/* PR target/121315: consecutive accesses that share a base should be
+ addressed with an immediate offset (a per-base pointer IV) rather than a
+ register offset, so the load/store-pair pass can fuse them. Check this
+ holds for several element types. */
+/* { dg-do compile } */
+/* { dg-options "-O3 -mcpu=neoverse-v2" } */
+
+void
+copy_add_l (long *d, long *s)
+{
+ for (int i = 0; i < 1000; ++i)
+ d[i] = s[i] + 1;
+}
+
+void
+copy_mul_f (float *d, float *s)
+{
+ for (int i = 0; i < 1000; ++i)
+ d[i] = s[i] * 2.0f;
+}
+
+void
+copy_add_d (double *d, double *s)
+{
+ for (int i = 0; i < 1000; ++i)
+ d[i] = s[i] + s[i];
+}
+
+/* Each loop should use an LDP and an STP of q-registers. */
+/* { dg-final { scan-assembler-times {ldp\tq[0-9]+, q[0-9]+} 3 } } */
+/* { dg-final { scan-assembler-times {stp\tq[0-9]+, q[0-9]+} 3 } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/pr121315.c
b/gcc/testsuite/gcc.target/aarch64/pr121315.c
new file mode 100644
index 00000000000..fa71f9e0acc
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/pr121315.c
@@ -0,0 +1,15 @@
+/* PR target/121315: ivopts should prefer immediate-offset addressing for
+ consecutive accesses that share a base, so the accesses can be combined
+ into LDP/STP rather than using register-offset addressing that blocks it.
*/
+/* { dg-do compile } */
+/* { dg-options "-O3 -mcpu=neoverse-v2" } */
+
+void
+copyReverseGeneric (int *dst, int *src)
+{
+ for (int i = 0; i < 10000; ++i)
+ dst[i] = __builtin_bswap32 (src[i]);
+}
+
+/* { dg-final { scan-assembler {ldp\tq[0-9]+, q[0-9]+} } } */
+/* { dg-final { scan-assembler {stp\tq[0-9]+, q[0-9]+} } } */
diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc
index 557b35ed2a1..dc86cedaf12 100644
--- a/gcc/tree-ssa-loop-ivopts.cc
+++ b/gcc/tree-ssa-loop-ivopts.cc
@@ -4699,6 +4699,34 @@ get_address_cost_ainc (poly_int64 ainc_step, poly_int64
ainc_offset,
return infinite_cost;
}
+/* Return true if the target supports a register-offset address that also
+ has a constant displacement, i.e. "base + index + offset", for memory of
+ MEM_MODE in address space AS. On such targets a single register-offset
+ induction variable can serve several accesses that share a base but differ
+ by a constant offset, by folding the offset into each address. When it is
+ false, each such access needs its own base register if register-offset
+ addressed, so immediate-offset addressing (a per-base pointer IV) is
+ preferable. */
+
+static bool
+addr_reg_offset_imm_valid_p (machine_mode mem_mode, addr_space_t as,
+ code_helper code = ERROR_MARK)
+{
+ HOST_WIDE_INT size;
+ /* Use the access size as a representative consecutive offset; bail out for
+ modes without a compile-time constant size, treating them as if the
+ offset were foldable so that no penalty is applied. This is a
+ deliberately conservative probe (unscaled index, a single representative
+ displacement): it is exact for targets that have no register-offset plus
+ immediate form at all, and on other targets it only affects costs. */
+ if (!GET_MODE_SIZE (mem_mode).is_constant (&size) || size == 0)
+ return true;
+
+ struct mem_address parts = {NULL_TREE, integer_one_node, integer_one_node,
+ NULL_TREE, build_int_cst (sizetype, size)};
+ return valid_mem_ref_p (mem_mode, as, &parts, code);
+}
+
/* Return cost of computing USE's address expression by using CAND.
AFF_INV and AFF_VAR represent invariant and variant parts of the
address expression, respectively. If AFF_INV is simple, store
@@ -4859,6 +4887,29 @@ get_address_cost (struct ivopts_data *data, struct
iv_use *use,
if (parts.offset != NULL_TREE && !integer_zerop (parts.offset))
cost.complexity += 1;
+ /* This is a register-offset address form: a loop-invariant base register
+ (COMP_INV, possibly with a symbol folded in) plus the candidate used as an
+ index register. If the use shares its base with another access in the
+ loop, and the target cannot fold a constant displacement into such an
+ address, then each access needs a distinct base register and the accesses
+ cannot be combined into a load/store pair. Charge one address-cost unit
+ per such use: accumulated over the group's uses this offsets roughly the
+ cost of the extra induction variable that the immediate-offset form needs,
+ so per-base pointer IVs are preferred once enough accesses would pair to
+ make that worthwhile, while a single isolated access is left alone. */
+ if (speed
+ && comp_inv != NULL_TREE
+ && parts.index != NULL_TREE
+ && use->has_pairing_sibling
+ && !addr_reg_offset_imm_valid_p (mem_mode, as, code))
+ {
+ cost.cost += 1;
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "Penalizing register-offset address for use %d.%d:"
+ " base is shared with a pairable sibling access.\n",
+ use->group_id, use->id);
+ }
+
return cost;
}
--
2.50.1 (Apple Git-155)
