https://gcc.gnu.org/g:309061fef508cab7e137b06cf93f6b55bb9e1470

commit r17-6059-g309061fef508cab7e137b06cf93f6b55bb9e1470
Author: Tamar Christina <[email protected]>
Date:   Mon Nov 9 13:51:58 2026 +0000

    middle-end: implement floating point dotprod
    
    In the following testcase
    
    float
    dot (const _Float16 *restrict a,
         const _Float16 *restrict b, int n)
    {
      float sum = 0.0f;
      for (int i = 0; i < n; ++i)
        sum += (float) a[i] * (float) b[i];
      return sum;
    }
    
    we have a widening dot product.  Each input is converted to float before the
    multiplication and the products are accumulated into a float result.
    
    The vectorizer already recognizes the equivalent integer pattern, but the
    recognition and lane reducing code currently only handle integer types. So 
even
    when the target has a floating-point dot product instruction we don't make 
use
    of it here.
    
    This patch extends the existing DOT_PROD_EXPR pattern to floating-point
    reductions.  It reuses sdot_prod for floating-point inputs and removes the
    integer-only restriction on its mode names.
    
    The important detail is where the conversions happen.  For example,
    
      sum += (float) a[i] * (float) b[i];
    
    can use a widening dot product, whereas
    
      _Float16 product = a[i] * b[i];
      sum += (float) product;
    
    cannot.  In the second case the product has already been rounded to _Float16
    before we add it to sum. Moving the conversion before the multiplication 
would
    change the result.
    
    For this reason the floating-point recognition looks for two direct 
conversions
    from the same narrower floating-point type, followed by a multiplication 
and a
    reduction in the wider type. It doesn't look through a conversion of the 
product
    as the integer pattern does.  Integer-to-float conversions are also left 
alone.
    
    The same recognition handles float inputs with a double accumulator.  
Whether
    either form is used depends on the target providing the corresponding 
optab.  As
    with the existing dot-product pattern, the summation must allow 
reassociation,
    so this isn't applied to an ordered floating-point reduction.
    
    The tests cover both widening forms and check that we reject products 
rounded
    before widening, integer-to-float conversions and reductions that don't 
permit
    reassociation.  The next patch wires up the AArch64 instructions.
    
    gcc/ChangeLog:
    
            * doc/md.texi (Standard Pattern Names for Vectorization): Document
            floating-point inputs and widening semantics for sdot_prod.  Require
            matching mode classes and at least twice the input element width.
            Describe product rounding and reassociation.
            * optabs.def (sdot_prod_optab): Allow floating-point modes.
            * tree-vect-loop.cc (vectorizable_lane_reducing): Allow
            floating-point types.
            * tree-vect-patterns.cc (vect_recog_dot_prod_pattern): Recognize
            floating-point dot products with directly widened inputs.  Require
            matching input types of at most half the accumulator precision.  
Keep
            product promotions and extension sign checks specific to integer
            dot products.  Update the pattern description.
            * tree.def (DOT_PROD_EXPR): Document floating-point conversions and
            product rounding.  Require all element types to be integral or all
            floating-point.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.dg/vect/vect-reduc-dot-float-1.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-2.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-3.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-4.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-5.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-6.c: New test.
            * gcc.dg/vect/vect-reduc-dot-float-7.c: New test.

Diff:
---
 gcc/doc/md.texi                                    | 21 ++++--
 gcc/optabs.def                                     |  2 +-
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-1.c | 16 +++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-2.c | 17 +++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-3.c | 15 +++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-4.c | 22 ++++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-5.c | 19 ++++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-6.c | 16 +++++
 gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-7.c | 18 +++++
 gcc/tree-vect-loop.cc                              |  2 +-
 gcc/tree-vect-patterns.cc                          | 78 ++++++++++++++++------
 gcc/tree.def                                       |  7 +-
 12 files changed, 202 insertions(+), 31 deletions(-)

diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index b99040f56368..921f7623be41 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7977,13 +7977,20 @@ for (i = 0; i < LEN + BIAS; i++)
 @mdindex sdot_prod@var{m}@var{n}
 @item @samp{sdot_prod@var{m}@var{n}}
 
-Multiply operand 1 by operand 2 without loss of precision, given that
-both operands contain signed elements.  Add each product to the overlapping
-element of operand 3 and store the result in operand 0.  Operands 0 and 3
-have mode @var{m} and operands 1 and 2 have mode @var{n}, with @var{n}
-having narrower elements than @var{m}.
-
-Semantically the expressions perform the multiplication in the following signs
+Multiply operand 1 by operand 2.  Both operands contain signed integer or
+floating-point elements.  Add each product to the overlapping element of
+operand 3 and store the result in operand 0.
+Operands 0 and 3 have mode @var{m} and operands 1 and 2 have mode @var{n},
+with the elements of @var{m} at least twice as wide as those of @var{n}.
+Both modes must have the same mode class.
+
+For floating-point operands, convert each input element to the element type
+of mode @var{m} before multiplication.  Round each product to that type
+before summation.  The additions may be reassociated, so the order of
+summation is unspecified.
+
+For integer operands, multiplication is performed without loss of precision,
+and the expressions have the following semantics:
 
 @smallexample
 sdot<signed op0, signed op1, signed op2, signed op3> ==
diff --git a/gcc/optabs.def b/gcc/optabs.def
index 243ef810f4be..d6806a6156aa 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -111,7 +111,7 @@ OPTAB_CD(mask_scatter_store_optab, "mask_scatter_store$a$b")
 OPTAB_CD(mask_len_scatter_store_optab, "mask_len_scatter_store$a$b")
 OPTAB_CD(vec_extract_optab, "vec_extract$a$b")
 OPTAB_CD(vec_init_optab, "vec_init$a$b")
-OPTAB_CD (sdot_prod_optab, "sdot_prod$I$a$b")
+OPTAB_CD (sdot_prod_optab, "sdot_prod$a$b")
 OPTAB_CD (udot_prod_optab, "udot_prod$I$a$b")
 OPTAB_CD (usdot_prod_optab, "usdot_prod$I$a$b")
 OPTAB_CD (usad_optab, "usad$I$a$b")
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-1.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-1.c
new file mode 100644
index 000000000000..df19bd4d3d54
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-1.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -ffast-math -ftree-vectorize 
-fdump-tree-vect-details" } */
+
+double
+dot_float_to_double (const float *restrict a, const float *restrict b, int n)
+{
+  double sum = 0.0;
+  for (int i = 0; i < n; ++i)
+    sum += (double) a[i] * (double) b[i];
+  return sum;
+}
+
+/* The pattern is recognized even on targets without a floating-point
+   dot-product instruction.  */
+/* { dg-final { scan-tree-dump "vect_recog_dot_prod_pattern: detected" "vect" 
} } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-2.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-2.c
new file mode 100644
index 000000000000..d25938148f93
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-2.c
@@ -0,0 +1,17 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-require-effective-target float16 } */
+/* { dg-add-options float16 } */
+/* { dg-additional-options "-O3 -ffast-math -ftree-vectorize 
-fdump-tree-vect-details" } */
+
+float
+dot_half_to_float (const _Float16 *restrict a,
+                  const _Float16 *restrict b, int n)
+{
+  float sum = 0.0f;
+  for (int i = 0; i < n; ++i)
+    sum += (float) a[i] * (float) b[i];
+  return sum;
+}
+
+/* { dg-final { scan-tree-dump "vect_recog_dot_prod_pattern: detected" "vect" 
} } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-3.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-3.c
new file mode 100644
index 000000000000..af24392ca612
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-3.c
@@ -0,0 +1,15 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -fno-associative-math -ftree-vectorize 
-fdump-tree-vect-details" } */
+
+double
+dot_ordered (const float *restrict a, const float *restrict b, int n)
+{
+  double sum = 0.0;
+  for (int i = 0; i < n; ++i)
+    sum += (double) a[i] * (double) b[i];
+  return sum;
+}
+
+/* Reassociation is required for a dot-product reduction.  */
+/* { dg-final { scan-tree-dump-not "vect_recog_dot_prod_pattern: detected" 
"vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-4.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-4.c
new file mode 100644
index 000000000000..f0b62de3bb9d
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-4.c
@@ -0,0 +1,22 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-require-effective-target float16 } */
+/* { dg-add-options float16 } */
+/* { dg-additional-options "-O3 -ffast-math -ftree-vectorize 
-fdump-tree-vect-details" } */
+
+/* Multiplying before the conversion rounds each product to _Float16 and
+   must not be represented as a widening dot product.  */
+float
+dot_rounded_half (const _Float16 *restrict a,
+                 const _Float16 *restrict b, int n)
+{
+  float sum = 0.0f;
+  for (int i = 0; i < n; ++i)
+    {
+      _Float16 product = a[i] * b[i];
+      sum += (float) product;
+    }
+  return sum;
+}
+
+/* { dg-final { scan-tree-dump-not "vect_recog_dot_prod_pattern: detected" 
"vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-5.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-5.c
new file mode 100644
index 000000000000..fb9e8c604bc3
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-5.c
@@ -0,0 +1,19 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -ffast-math -fdump-tree-vect-details" } */
+
+double
+dot_rounded_float (const float *restrict a, const float *restrict b, int n)
+{
+  double sum = 0.0;
+  for (int i = 0; i < n; ++i)
+    {
+      float product = a[i] * b[i];
+      sum += (double) product;
+    }
+  return sum;
+}
+
+/* Rounding the product to float before widening is not a widening dot
+   product, even when the summation can be reassociated.  */
+/* { dg-final { scan-tree-dump-not "vect_recog_dot_prod_pattern: detected" 
"vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-6.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-6.c
new file mode 100644
index 000000000000..a60c51bbe7e2
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-6.c
@@ -0,0 +1,16 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-additional-options "-O3 -ffast-math -fdump-tree-vect-details" } */
+
+float
+dot_int_to_float (const short *restrict a, const short *restrict b, int n)
+{
+  float sum = 0.0f;
+  for (int i = 0; i < n; ++i)
+    sum += (float) a[i] * (float) b[i];
+  return sum;
+}
+
+/* Integer-to-float conversions do not match a floating-point dot product.
+   The operands must be widened from a narrower floating-point type.  */
+/* { dg-final { scan-tree-dump-not "vect_recog_dot_prod_pattern: detected" 
"vect" } } */
diff --git a/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-7.c 
b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-7.c
new file mode 100644
index 000000000000..3ded284a28ec
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/vect/vect-reduc-dot-float-7.c
@@ -0,0 +1,18 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target vect_float } */
+/* { dg-require-effective-target float16 } */
+/* { dg-add-options float16 } */
+/* { dg-additional-options "-O3 -ffast-math -fdump-tree-vect-details" } */
+
+/* The accumulator can be more than twice as wide as the inputs.  */
+double
+dot_half_to_double (const _Float16 *restrict a,
+                   const _Float16 *restrict b, int n)
+{
+  double sum = 0.0;
+  for (int i = 0; i < n; ++i)
+    sum += (double) a[i] * (double) b[i];
+  return sum;
+}
+
+/* { dg-final { scan-tree-dump "vect_recog_dot_prod_pattern: detected" "vect" 
} } */
diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 26563e61dc1a..78bfd607c5d7 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -6899,7 +6899,7 @@ vectorizable_lane_reducing (loop_vec_info loop_vinfo, 
stmt_vec_info stmt_info,
 
   tree type = TREE_TYPE (gimple_assign_lhs (stmt));
 
-  if (!INTEGRAL_TYPE_P (type))
+  if (!INTEGRAL_TYPE_P (type) && !SCALAR_FLOAT_TYPE_P (type))
     return false;
 
   /* Do not try to vectorize bit-precision reductions.  */
diff --git a/gcc/tree-vect-patterns.cc b/gcc/tree-vect-patterns.cc
index 4a458d833cb5..cbe834e2aafb 100644
--- a/gcc/tree-vect-patterns.cc
+++ b/gcc/tree-vect-patterns.cc
@@ -1235,9 +1235,10 @@ vect_recog_cond_expr_convert_pattern (vec_info *vinfo,
      [S6  prod = (TYPE2) prod;  #optional]
      S7  sum_1 = prod + sum_0;
 
-   where 'TYPE1' is exactly double the size of type 'type1a' and 'type1b',
-   the sign of 'TYPE1' must be one of 'type1a' or 'type1b' but the sign of
-   'type1a' and 'type1b' can differ.
+   For integer reductions, 'TYPE1' is exactly double the size of 'type1a'
+   and 'type1b', and its sign must match one of them.  Floating-point
+   reductions require matching input types at most half the width of TYPE1,
+   with the inputs converted before multiplication, and TYPE2 the same as 
TYPE1.
 
    Input:
 
@@ -1278,8 +1279,8 @@ vect_recog_dot_prod_pattern (vec_info *vinfo,
           DDPROD = (TYPE2) DPROD;
           sum_1 = DDPROD + sum_0;
      In which
-     - DX is double the size of X
-     - DY is double the size of Y
+     - DX and DY are at least twice as wide as X and Y (exactly twice for
+       integers).
      - DX, DY, DPROD all have the same type but the sign
        between X, Y and DPROD can differ.
      - sum is the same size of DPROD or bigger
@@ -1302,32 +1303,67 @@ vect_recog_dot_prod_pattern (vec_info *vinfo,
 
   type = TREE_TYPE (gimple_get_lhs (last_stmt));
 
+  /* So far so good.  Since LAST_STMT is a summation reduction, OPRND1 is
+     the reduction variable defined by a loop-header phi.  OPRND0 is the
+     value added to it, which must come from a multiplication.  For integer
+     reductions, first look through a possible promotion of OPRND0.  */
+  bool float_p = SCALAR_FLOAT_TYPE_P (type);
+  enum optab_subtype subtype = optab_vector;
   vect_unpromoted_value unprom_mult;
-  oprnd0 = vect_look_through_possible_promotion (vinfo, oprnd0, &unprom_mult);
-
-  /* So far so good.  Since last_stmt was detected as a (summation) reduction,
-     we know that oprnd1 is the reduction variable (defined by a loop-header
-     phi), and oprnd0 is an ssa-name defined by a stmt in the loop body.
-     Left to check that oprnd0 is defined by a (widen_)mult_expr  */
-  if (!oprnd0)
+  if (!float_p)
+    oprnd0 = vect_look_through_possible_promotion (vinfo, oprnd0, 
&unprom_mult);
+  if (!oprnd0 || TREE_CODE (oprnd0) != SSA_NAME)
     return NULL;
 
   stmt_vec_info mult_vinfo = vect_get_internal_def (vinfo, oprnd0);
   if (!mult_vinfo)
     return NULL;
 
-  /* FORNOW.  Can continue analyzing the def-use chain when this stmt in a phi
-     inside the loop (in case we are analyzing an outer-loop).  */
+  /* FORNOW.  We could continue following the def-use chain if this statement
+     is a phi inside the loop, as when analyzing an outer loop.  */
   vect_unpromoted_value unprom0[2];
-  enum optab_subtype subtype = optab_vector;
-  if (!vect_widened_op_tree (vinfo, mult_vinfo, MULT_EXPR, WIDEN_MULT_EXPR,
-                            false, 2, unprom0, &half_type, &subtype))
+  if (float_p)
+    {
+      /* Require multiplication of two directly promoted floating-point
+        inputs.  A product rounded before promotion is not equivalent.  */
+      gassign *mult = dyn_cast <gassign *> (mult_vinfo->stmt);
+      if (!mult || gimple_assign_rhs_code (mult) != MULT_EXPR)
+       return NULL;
+
+      for (unsigned int i = 0; i < 2; ++i)
+       {
+         tree op = gimple_op (mult, i + 1);
+         stmt_vec_info cast_info
+           = (TREE_CODE (op) == SSA_NAME
+              ? vect_get_internal_def (vinfo, op) : NULL);
+         gassign *cast
+           = cast_info ? dyn_cast <gassign *> (cast_info->stmt) : NULL;
+         if (!cast || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (cast)))
+           return NULL;
+
+         op = gimple_assign_rhs1 (cast);
+         tree op_type = TREE_TYPE (op);
+         vect_def_type dt;
+         if (TREE_CODE (op) != SSA_NAME
+             || !SCALAR_FLOAT_TYPE_P (op_type)
+             || TYPE_PRECISION (op_type) * 2 > TYPE_PRECISION (type)
+             || !vect_is_simple_use (op, vinfo, &dt))
+           return NULL;
+         unprom0[i].set_op (op, dt, cast_info);
+       }
+      half_type = unprom0[0].type;
+      if (!types_compatible_p (half_type, unprom0[1].type))
+       return NULL;
+    }
+  else if (!vect_widened_op_tree (vinfo, mult_vinfo, MULT_EXPR, 
WIDEN_MULT_EXPR,
+                               false, 2, unprom0, &half_type, &subtype))
     return NULL;
 
-  /* If there are two widening operations, make sure they agree on the sign
-     of the extension.  The result of an optab_vector_mixed_sign operation
-     is signed; otherwise, the result has the same sign as the operands.  */
-  if (TYPE_PRECISION (unprom_mult.type) != TYPE_PRECISION (type)
+  /* If there are two integer widening operations, make sure they agree on
+     the sign of the extension.  The result of an optab_vector_mixed_sign
+     operation is signed; otherwise, it has the same sign as the operands.  */
+  if (!float_p
+      && TYPE_PRECISION (unprom_mult.type) != TYPE_PRECISION (type)
       && (subtype == optab_vector_mixed_sign
          ? TYPE_UNSIGNED (unprom_mult.type)
          : TYPE_SIGN (unprom_mult.type) != TYPE_SIGN (half_type)))
diff --git a/gcc/tree.def b/gcc/tree.def
index c11ad1abf24b..6424425c2c49 100644
--- a/gcc/tree.def
+++ b/gcc/tree.def
@@ -1461,12 +1461,17 @@ DEFTREECODE (TRANSACTION_EXPR, "transaction_expr", 
tcc_expression, 1)
 /* Widening dot-product.
    The first two arguments are of type t1.
    The third argument and the result are of type t2, such that t2 is at least
-   twice the size of t1. DOT_PROD_EXPR(arg1,arg2,arg3) is equivalent to:
+   twice the size of t1.
+   All element types must be integral or all floating-point.
+   For integers, DOT_PROD_EXPR(arg1,arg2,arg3) is equivalent to:
        tmp = WIDEN_MULT_EXPR(arg1, arg2);
        arg3 = PLUS_EXPR (tmp, arg3);
    or:
        tmp = WIDEN_MULT_EXPR(arg1, arg2);
         arg3 = WIDEN_SUM_EXPR (tmp, arg3);
+   For floating-point dot products, convert the inputs to t2, multiply in
+   t2, and add the products to arg3.  Each product is rounded to t2 before
+   summation.
    Like WIDEN_SUM_EXPR, this code is only created for a reduction whose
    summation may be reassociated, so which products are added into which
    element of arg3 is not observable.  */

Reply via email to