https://gcc.gnu.org/g:7a9cb231ec361918c927e7526dd3a2f84e7459bd
commit r17-6043-g7a9cb231ec361918c927e7526dd3a2f84e7459bd Author: Tamar Christina <[email protected]> Date: Mon Nov 9 08:01:11 2026 +0000 AArch64: classify left fold reductions as single cycle In the following testcase float conditional_sum (const float *restrict a, const unsigned char *restrict c, int n) { float sum = 0.0f; for (int i = 0; i < n; ++i) if (c[i]) sum += a[i]; return sum; } we can vectorize the loads and condition while keeping the additions in their original order using FADDA. However, the choice of vector mode matters here. A full byte vector has four times as many elements as the corresponding float vector. To process all those elements we need several float vectors and several FADDA instructions. Each FADDA takes the scalar accumulator produced by the previous one, so these instructions form a serial chain. The AArch64 cost model already accounts for this when the reduction is marked as having a single def-use cycle. In that case it multiplies the reduction latency by the number of copies. But fold-left reductions aren't necessarily marked that way, even though they also share one accumulator across the copies. This patch makes aarch64_force_single_cycle include FOLD_LEFT_REDUCTION, so that the existing latency calculation accounts for the whole chain through the accumulator. For the example above, at -O3 -mcpu=neoverse-v3, this lets us choose VNx4QI. The byte vector then has the same number of elements as the float vector, avoiding the extra reduction copies and predicate unpacks. The testcase checks that we still vectorize using FADDA, choose VNx4QI and have no unpack instructions. gcc/ChangeLog: * config/aarch64/aarch64.cc (aarch64_force_single_cycle): Include fold-left reductions so that the reduction latency accounts for all copies sharing the same accumulator. gcc/testsuite/ChangeLog: * gcc.target/aarch64/sve/reduc_strict_10.c: New test. Diff: --- gcc/config/aarch64/aarch64.cc | 3 ++- .../gcc.target/aarch64/sve/reduc_strict_10.c | 20 ++++++++++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc index 6c1229d0a959..63926a62a983 100644 --- a/gcc/config/aarch64/aarch64.cc +++ b/gcc/config/aarch64/aarch64.cc @@ -18690,7 +18690,8 @@ aarch64_force_single_cycle (vec_info *vinfo, slp_tree node) auto reduc_info = info_for_reduction (as_a <loop_vec_info> (vinfo), node); if (!reduc_info) return false; - return VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info); + return VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info) + || VECT_REDUC_INFO_TYPE (reduc_info) == FOLD_LEFT_REDUCTION; } /* COUNT, KIND and STMT_INFO are the same as for vector_costs::add_stmt_cost diff --git a/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c b/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c new file mode 100644 index 000000000000..99ce8171bff5 --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c @@ -0,0 +1,20 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -mcpu=neoverse-v3 -msve-vector-bits=scalable -fdump-tree-vect-details" } */ + +/* Account for the serial FADDA operations when comparing vector modes. + Prefer a partial byte vector matching the float vector's lane count, + avoiding predicate unpacks. */ +float +conditional_sum (const float *restrict a, const unsigned char *restrict c, int n) +{ + float sum = 0.0f; + for (int i = 0; i < n; ++i) + if (c[i]) + sum += a[i]; + return sum; +} + +/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */ +/* { dg-final { scan-tree-dump "Choosing vector mode VNx4QI" "vect" } } */ +/* { dg-final { scan-assembler {\tfadda\ts[0-9]+,} } } */ +/* { dg-final { scan-assembler-not {\t[psu]unpk(lo|hi)\t} } } */
