https://gcc.gnu.org/g:7a9cb231ec361918c927e7526dd3a2f84e7459bd

commit r17-6043-g7a9cb231ec361918c927e7526dd3a2f84e7459bd
Author: Tamar Christina <[email protected]>
Date:   Mon Nov 9 08:01:11 2026 +0000

    AArch64: classify left fold reductions as single cycle
    
    In the following testcase
    
    float
    conditional_sum (const float *restrict a,
                     const unsigned char *restrict c, int n)
    {
      float sum = 0.0f;
      for (int i = 0; i < n; ++i)
        if (c[i])
          sum += a[i];
      return sum;
    }
    
    we can vectorize the loads and condition while keeping the additions in 
their
    original order using FADDA.
    
    However, the choice of vector mode matters here. A full byte vector has four
    times as many elements as the corresponding float vector.  To process all 
those
    elements we need several float vectors and several FADDA instructions.  Each
    FADDA takes the scalar accumulator produced by the previous one, so these
    instructions form a serial chain.
    
    The AArch64 cost model already accounts for this when the reduction is 
marked as
    having a single def-use cycle.  In that case it multiplies the reduction 
latency
    by the number of copies.  But fold-left reductions aren't necessarily marked
    that way, even though they also share one accumulator across the copies.
    
    This patch makes aarch64_force_single_cycle include FOLD_LEFT_REDUCTION, so 
that
    the existing latency calculation accounts for the whole chain through the
    accumulator.
    
    For the example above, at -O3 -mcpu=neoverse-v3, this lets us choose VNx4QI.
    The byte vector then has the same number of elements as the float vector,
    avoiding the extra reduction copies and predicate unpacks.  The testcase 
checks
    that we still vectorize using FADDA, choose VNx4QI and have no unpack
    instructions.
    
    gcc/ChangeLog:
    
            * config/aarch64/aarch64.cc (aarch64_force_single_cycle): Include
            fold-left reductions so that the reduction latency accounts for
            all copies sharing the same accumulator.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.target/aarch64/sve/reduc_strict_10.c: New test.

Diff:
---
 gcc/config/aarch64/aarch64.cc                        |  3 ++-
 .../gcc.target/aarch64/sve/reduc_strict_10.c         | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index 6c1229d0a959..63926a62a983 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -18690,7 +18690,8 @@ aarch64_force_single_cycle (vec_info *vinfo, slp_tree 
node)
   auto reduc_info = info_for_reduction (as_a <loop_vec_info> (vinfo), node);
   if (!reduc_info)
     return false;
-  return VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info);
+  return VECT_REDUC_INFO_FORCE_SINGLE_CYCLE (reduc_info)
+        || VECT_REDUC_INFO_TYPE (reduc_info) == FOLD_LEFT_REDUCTION;
 }
 
 /* COUNT, KIND and STMT_INFO are the same as for vector_costs::add_stmt_cost
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c 
b/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c
new file mode 100644
index 000000000000..99ce8171bff5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/reduc_strict_10.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -mcpu=neoverse-v3 -msve-vector-bits=scalable 
-fdump-tree-vect-details" } */
+
+/* Account for the serial FADDA operations when comparing vector modes.
+   Prefer a partial byte vector matching the float vector's lane count,
+   avoiding predicate unpacks.  */
+float
+conditional_sum (const float *restrict a, const unsigned char *restrict c, int 
n)
+{
+  float sum = 0.0f;
+  for (int i = 0; i < n; ++i)
+    if (c[i])
+      sum += a[i];
+  return sum;
+}
+
+/* { dg-final { scan-tree-dump "vectorized 1 loops in function" "vect" } } */
+/* { dg-final { scan-tree-dump "Choosing vector mode VNx4QI" "vect" } } */
+/* { dg-final { scan-assembler {\tfadda\ts[0-9]+,} } } */
+/* { dg-final { scan-assembler-not {\t[psu]unpk(lo|hi)\t} } } */

Reply via email to