This patch starts an initial tuning for early break code. It's primary intended
to recover losses seen in SPECCPU 2017 and SPECCPU 2026 due to how it picks
between SVE and Adv. SIMD and how it decided on unrolling.
The issue-rate model treats a scalar loop body as steady-line code. That is a
poor model for early-break loops, because the scalar loop makes a serial branch
decision for each element. The vector loop replaces those serial decisions
with vector compares and a predicate reduction, so the scalar issue-rate
estimate can make the scalar loop look cheaper than it really is.
This patch avoids using that scalar issue-rate estimate to disparage
early-break vector loops. It also gives Adv.SIMD early-break loops a preferred
two-vector unroll shape. One vector serializes the compare through the
predicate reduction and branch; two vectors let independent compares run before
the reduction. The cap keeps the loop body from growing beyond the useful
shape.
For example, consider:
void *a[640];
void f1 (void *x, unsigned long n)
{
void **end = a + n;
for (void **p = a; p != end; ++p)
if (*p == x)
return;
}
Before this patch, the early-break vector form could lose against the scalar
loop during cost comparison. The generated loop was then the ordinary scalar
pointer walk:
ldr x2, [x0], 8
cmp x2, x1
b.eq .Lexit
cmp x0, x3
b.ne .Lloop
With this patch, aarch64-none-elf-gcc keeps the Adv.SIMD early-break form and
uses two independent vector compares before reducing the predicate to a scalar
branch decision:
ldp q31, q30, [x0]
cmeq v31.2d, v31.2d, v29.2d
cmeq v30.2d, v30.2d, v29.2d
addhn v31.2s, v31.2d, v30.2d
fmov x2, d31
cbz x2, .Lloop
The same cost-model issue applies to SVE early-break loops. The exact code
generation is different, but the costing rule is the same: don't penalize the
vector loop using a scalar issue-rate estimate that ignores the scalar branch
chain.
Bootstrapped Regtested on aarch64-none-linux-gnu,
arm-none-linux-gnueabihf, x86_64-pc-linux-gnu
-m32, -m64 and no issues.
-O3, profiled, profiled -O3 and -O3 -LTO bootstrap tested and no issues.
Also SPECCPU 2017 and SPECCPU 2026 tested.
OK for master?
Thanks,
Tamar
gcc/ChangeLog:
* config/aarch64/aarch64.cc
(aarch64_vector_costs::adjust_body_cost_sve): Take an early_break
argument and avoid the scalar issue-rate cliff for early breaks.
(aarch64_vector_costs::determine_suggested_unroll_factor): Prefer
two-vector Adv.SIMD early-break loops and cap them there.
(aarch64_vector_costs::adjust_body_cost): Pass the early-break state
and avoid disparaging early-break vector loops using scalar issue-rate
estimates.
(aarch64_vector_costs::better_main_loop_than_p): Use the generic
total-cost comparison for early-break loops.
---
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index
a9271ac9ad9850116415d2e65ca4596c2dad3ae1..a1142e16fcaf5b1d9bbcfa879a05f14395faa5f9
100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -17771,6 +17771,11 @@ private:
iterate, otherwise it is zero. */
uint64_t m_num_vector_iterations = 0;
+ /* The number of scalar conditions that can break out of the loop before
+ the latch condition. These form a serial branch-decision chain in the
+ scalar loop. */
+ unsigned int m_num_early_breaks = 0;
+
/* Used only when vectorizing loops. Estimates the number and kind of
operations that would be needed by one iteration of the scalar
or vector loop. There is one entry for each tuning option of
@@ -17908,6 +17913,8 @@ record_potential_advsimd_unrolling (loop_vec_info
loop_vinfo)
void
aarch64_vector_costs::analyze_loop_vinfo (loop_vec_info loop_vinfo)
{
+ m_num_early_breaks = LOOP_VINFO_LOOP_CONDS (loop_vinfo).length ();
+
/* Record the number of times that the vector loop would execute,
if known. */
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
@@ -19278,8 +19285,13 @@ determine_suggested_unroll_factor (loop_vec_info
loop_vinfo)
/* Limit unroll factor to a value adjustable by the user, the default
value is 4. */
unsigned int unroll_factor = aarch64_vect_unroll_limit;
- unsigned int factor
- = vec_ops.reduction_latency > 1 ? vec_ops.reduction_latency : 1;
+ unsigned int factor = MAX (vec_ops.reduction_latency, 1);
+ /* Adv.SIMD early-break loops reduce the vector condition to a scalar
+ branch decision. A single-vector loop serializes one compare through
+ the reduction and the scalar branch. Two vectors let independent
+ compares run before the reduction. */
+ if (!sve && m_num_early_breaks)
+ factor = MAX (factor, 2);
unsigned int temp;
/* Sanity check, this should never happen. */
@@ -19308,10 +19320,13 @@ determine_suggested_unroll_factor (loop_vec_info
loop_vinfo)
temp = CEIL (factor * vec_issue->general_ops_per_cycle,
vec_ops.general_ops);
unroll_factor = MIN (unroll_factor, temp);
- }
+ }
max_unroll_factor = MAX (max_unroll_factor, unroll_factor);
}
+ if (!sve && m_num_early_breaks)
+ max_unroll_factor = MIN (max_unroll_factor, 2);
+
/* For known iteration loops, cap suggested unroll factor to avoid redundant
unrolled chunks. Use CEIL rather than truncating division to make sure
the completely unrolled vector loop covers all scalar iterations. */
@@ -19362,6 +19377,23 @@ adjust_body_cost (loop_vec_info loop_vinfo,
fractional_cost scalar_cycles_per_iter
= scalar_ops.min_cycles_per_iter () * estimated_vf;
+ if (m_num_early_breaks)
+ {
+ fractional_cost scalar_control_cycles_per_iter
+ = estimated_vf * m_num_early_breaks;
+ if (scalar_cycles_per_iter < scalar_control_cycles_per_iter)
+ {
+ scalar_cycles_per_iter = scalar_control_cycles_per_iter;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "Increasing scalar issue estimate to %f cycles"
+ " per vector iteration to account for %u serial"
+ " early-break gcond%s per scalar iteration\n",
+ scalar_cycles_per_iter.as_double (),
+ m_num_early_breaks,
+ m_num_early_breaks == 1 ? "" : "s");
+ }
+ }
fractional_cost vector_cycles_per_iter = vector_ops.min_cycles_per_iter ();
--
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index a9271ac9ad9850116415d2e65ca4596c2dad3ae1..a1142e16fcaf5b1d9bbcfa879a05f14395faa5f9 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -17771,6 +17771,11 @@ private:
iterate, otherwise it is zero. */
uint64_t m_num_vector_iterations = 0;
+ /* The number of scalar conditions that can break out of the loop before
+ the latch condition. These form a serial branch-decision chain in the
+ scalar loop. */
+ unsigned int m_num_early_breaks = 0;
+
/* Used only when vectorizing loops. Estimates the number and kind of
operations that would be needed by one iteration of the scalar
or vector loop. There is one entry for each tuning option of
@@ -17908,6 +17913,8 @@ record_potential_advsimd_unrolling (loop_vec_info loop_vinfo)
void
aarch64_vector_costs::analyze_loop_vinfo (loop_vec_info loop_vinfo)
{
+ m_num_early_breaks = LOOP_VINFO_LOOP_CONDS (loop_vinfo).length ();
+
/* Record the number of times that the vector loop would execute,
if known. */
class loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
@@ -19278,8 +19285,13 @@ determine_suggested_unroll_factor (loop_vec_info loop_vinfo)
/* Limit unroll factor to a value adjustable by the user, the default
value is 4. */
unsigned int unroll_factor = aarch64_vect_unroll_limit;
- unsigned int factor
- = vec_ops.reduction_latency > 1 ? vec_ops.reduction_latency : 1;
+ unsigned int factor = MAX (vec_ops.reduction_latency, 1);
+ /* Adv.SIMD early-break loops reduce the vector condition to a scalar
+ branch decision. A single-vector loop serializes one compare through
+ the reduction and the scalar branch. Two vectors let independent
+ compares run before the reduction. */
+ if (!sve && m_num_early_breaks)
+ factor = MAX (factor, 2);
unsigned int temp;
/* Sanity check, this should never happen. */
@@ -19308,10 +19320,13 @@ determine_suggested_unroll_factor (loop_vec_info loop_vinfo)
temp = CEIL (factor * vec_issue->general_ops_per_cycle,
vec_ops.general_ops);
unroll_factor = MIN (unroll_factor, temp);
- }
+ }
max_unroll_factor = MAX (max_unroll_factor, unroll_factor);
}
+ if (!sve && m_num_early_breaks)
+ max_unroll_factor = MIN (max_unroll_factor, 2);
+
/* For known iteration loops, cap suggested unroll factor to avoid redundant
unrolled chunks. Use CEIL rather than truncating division to make sure
the completely unrolled vector loop covers all scalar iterations. */
@@ -19362,6 +19377,23 @@ adjust_body_cost (loop_vec_info loop_vinfo,
fractional_cost scalar_cycles_per_iter
= scalar_ops.min_cycles_per_iter () * estimated_vf;
+ if (m_num_early_breaks)
+ {
+ fractional_cost scalar_control_cycles_per_iter
+ = estimated_vf * m_num_early_breaks;
+ if (scalar_cycles_per_iter < scalar_control_cycles_per_iter)
+ {
+ scalar_cycles_per_iter = scalar_control_cycles_per_iter;
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "Increasing scalar issue estimate to %f cycles"
+ " per vector iteration to account for %u serial"
+ " early-break gcond%s per scalar iteration\n",
+ scalar_cycles_per_iter.as_double (),
+ m_num_early_breaks,
+ m_num_early_breaks == 1 ? "" : "s");
+ }
+ }
fractional_cost vector_cycles_per_iter = vector_ops.min_cycles_per_iter ();
