param_avoid_fma_max_bits gates two independent transforms: the
widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a
loop-carried multiply-add as fmul + fadd, and the reassoc
loop-carried-FMA reorder added in r14-5779-g746344dd538
(PR tree-optimization/110279), which parallelises 3+ operand chains.
Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given
type they switch on at the same threshold and a target cannot keep one
while dropping the other.
This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1
family): a 2-operand reduction such as an sgemm inner K-loop is left as
fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet
setting avoid-fma-max-bits to 0 to avoid it also disables the reorder.
Add a generic -fwidening-mul-defer-fma (default on) that gates only the
widening_mul deferring; avoid-fma-max-bits keeps gating the reorder
alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other
target and no default behaviour changes.
gcc/ChangeLog:
* common.opt (fwidening-mul-defer-fma): New option.
* tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children):
Gate fma_deferring_state's enable predicate on
flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits.
* config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA):
Update the comment for the -fwidening-mul-defer-fma effect.
* config/aarch64/aarch64.cc (aarch64_override_options_internal):
Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear
flag_widening_mul_defer_fma.
* doc/invoke.texi (-fwidening-mul-defer-fma): Document.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/widening-mul-defer-fma-1.c: New test.
* gcc.target/aarch64/widening-mul-defer-fma-2.c: New test.
* gcc.target/aarch64/widening-mul-defer-fma-3.c: New test.
Signed-off-by: Philipp Tomsich <[email protected]>
---
gcc/common.opt | 4 ++++
gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++
gcc/config/aarch64/aarch64.cc | 12 ++++++++---
gcc/doc/invoke.texi | 17 ++++++++++++++-
.../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++
.../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++
.../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++
gcc/tree-ssa-math-opts.cc | 3 ++-
8 files changed, 95 insertions(+), 5 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
diff --git a/gcc/common.opt b/gcc/common.opt
index 218dddf5dfe9..9be7dd6b2315 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -3639,6 +3639,10 @@ fweb
Common Var(flag_web) Optimization EnabledBy(funroll-loops)
Construct webs and split unrelated uses of single variable.
+fwidening-mul-defer-fma
+Common Var(flag_widening_mul_defer_fma) Init(1) Optimization
+Defer forming an FMA whose result feeds a loop-header PHI, leaving a separate
multiply and add, when FMA avoidance is requested via
--param=avoid-fma-max-bits.
+
ftree-builtin-call-dce
Common Var(flag_tree_builtin_call_dce) Init(0) Optimization
Enable conditional dead code elimination for builtin calls.
diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def
b/gcc/config/aarch64/aarch64-tuning-flags.def
index 058dadecccaa..5b087bfc9f03 100644
--- a/gcc/config/aarch64/aarch64-tuning-flags.def
+++ b/gcc/config/aarch64/aarch64-tuning-flags.def
@@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants",
CSE_SVE_VL_CONSTANTS)
AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput",
MATCHED_VECTOR_THROUGHPUT)
+/* For cores whose pipeline disfavours loop-carried serial FMAs: set
+ avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand
+ chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul
+ pass's FMA deferring. */
AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA)
AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA)
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index 78f1eae8336c..fdb5bbbdfbce 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct gcc_options
*opts,
&& opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level)
opts->x_flag_prefetch_loop_arrays = 1;
- /* Avoid loop-dependant FMA chains. */
+ /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper
+ keeps using --param=avoid-fma-max-bits; the widening-mul-side
+ deferring is gated separately by -fwidening-mul-defer-fma, so we
+ suppress only the deferring on these cores while leaving the reassoc
+ reorder active. */
if (aarch64_tune_params.extra_tuning_flags
& AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA)
- SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits,
- 512);
+ {
+ SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512);
+ SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0);
+ }
/* Consider fully pipelined FMA in reassociation. */
if (aarch64_tune_params.extra_tuning_flags
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 208625cafc55..f39902f77e9b 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}.
-funconstrained-commons -funit-at-a-time -funroll-all-loops
-funroll-loops -funsafe-math-optimizations -funswitch-loops
-fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt
--fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs
+-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa
+-fuse-linker-plugin -fzero-call-used-regs
-O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize}
@item Program Instrumentation Options
@@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables no
longer stay in a
Enabled by default with @option{-funroll-loops}.
+@opindex fwidening-mul-defer-fma
+@opindex fno-widening-mul-defer-fma
+@item -fwidening-mul-defer-fma
+When FMA avoidance is requested for loop-carried reductions (that is, when
+@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass
+refuses to contract a multiply-add whose result feeds a loop-header PHI,
+leaving a separate multiply and add. Use @option{-fno-widening-mul-defer-fma}
+to suppress this deferring and contract such reductions into an FMA, while
+still leaving @option{--param=avoid-fma-max-bits} in effect for the
+@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains.
+This is useful on targets whose pipeline disfavours the extra floating-point
+operation more than the loop-carried FMA latency. The option is enabled by
+default.
+
@opindex fwhole-program
@opindex fno-whole-program
@item -fwhole-program
diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
new file mode 100644
index 000000000000..b542274eb763
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c
@@ -0,0 +1,19 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */
+
+/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which
+ clears -fwidening-mul-defer-fma so that a 2-operand loop-carried
+ reduction is contracted to fmadd rather than deferred to fmul + fadd.
+ (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmul\t} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
new file mode 100644
index 000000000000..764dd2457435
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c
@@ -0,0 +1,21 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize
--param=avoid-fma-max-bits=512" } */
+
+/* With FMA deferring active (avoid-fma-max-bits > 0) and the default
+ -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA
+ whose result feeds the loop-header phi: the reduction stays as a
+ separate fmul + fadd. This is the behaviour the new flag decouples
+ from the reassoc reorder (also gated by avoid-fma-max-bits). */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmul\t} } } */
+/* { dg-final { scan-assembler {\tfadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmadd\t} } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
new file mode 100644
index 000000000000..5e041afc18fc
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c
@@ -0,0 +1,20 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -ffast-math -fno-tree-vectorize
--param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */
+
+/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA
+ avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512)
+ but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring,
+ so the loop-carried multiply-add is contracted to a single fmadd. This
+ proves the new flag decouples the deferring from avoid-fma-max-bits. */
+
+double
+dot (const double *a, const double *b, int n)
+{
+ double s = 0.0;
+ for (int i = 0; i < n; i++)
+ s += a[i] * b[i];
+ return s;
+}
+
+/* { dg-final { scan-assembler {\tfmadd\t} } } */
+/* { dg-final { scan-assembler-not {\tfmul\t} } } */
diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc
index f0ede668d95e..a00146922713 100644
--- a/gcc/tree-ssa-math-opts.cc
+++ b/gcc/tree-ssa-math-opts.cc
@@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block bb)
{
gimple_stmt_iterator gsi;
- fma_deferring_state fma_state (param_avoid_fma_max_bits > 0);
+ fma_deferring_state fma_state (param_avoid_fma_max_bits > 0
+ && flag_widening_mul_defer_fma);
for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi);
psi = psi_next)
--
2.34.1
base-commit: 3b443e764144526748d2b51d017d745f1956377e
branch: ptomsich/475-widening-mul-defer-fma
