On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich <[email protected]> wrote: > > param_avoid_fma_max_bits gates two independent transforms: the > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a > loop-carried multiply-add as fmul + fadd, and the reassoc > loop-carried-FMA reorder added in r14-5779-g746344dd538 > (PR tree-optimization/110279), which parallelises 3+ operand chains. > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given > type they switch on at the same threshold and a target cannot keep one > while dropping the other. > > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1 > family): a 2-operand reduction such as an sgemm inner K-loop is left as > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder. > > Add a generic -fwidening-mul-defer-fma (default on) that gates only the > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other > target and no default behaviour changes.
Note this is just a review on the documentation and nothing else. > > gcc/ChangeLog: > > * common.opt (fwidening-mul-defer-fma): New option. > * tree-ssa-math-opts.cc (math_opts_dom_walker::after_dom_children): > Gate fma_deferring_state's enable predicate on > flag_widening_mul_defer_fma in addition to param_avoid_fma_max_bits. > * config/aarch64/aarch64-tuning-flags.def (AVOID_CROSS_LOOP_FMA): > Update the comment for the -fwidening-mul-defer-fma effect. > * config/aarch64/aarch64.cc (aarch64_override_options_internal): > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, clear > flag_widening_mul_defer_fma. > * doc/invoke.texi (-fwidening-mul-defer-fma): Document. > > gcc/testsuite/ChangeLog: > > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test. > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test. > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test. > > Signed-off-by: Philipp Tomsich <[email protected]> > --- > > gcc/common.opt | 4 ++++ > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++ > gcc/config/aarch64/aarch64.cc | 12 ++++++++--- > gcc/doc/invoke.texi | 17 ++++++++++++++- > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++ > .../aarch64/widening-mul-defer-fma-2.c | 21 +++++++++++++++++++ > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++ > gcc/tree-ssa-math-opts.cc | 3 ++- > 8 files changed, 95 insertions(+), 5 deletions(-) > create mode 100644 > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > > diff --git a/gcc/common.opt b/gcc/common.opt > index 218dddf5dfe9..9be7dd6b2315 100644 > --- a/gcc/common.opt > +++ b/gcc/common.opt > @@ -3639,6 +3639,10 @@ fweb > Common Var(flag_web) Optimization EnabledBy(funroll-loops) > Construct webs and split unrelated uses of single variable. > > +fwidening-mul-defer-fma > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a > separate multiply and add, when FMA avoidance is requested via > --param=avoid-fma-max-bits. Don't mention --param in user documentation directly. > + > ftree-builtin-call-dce > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization > Enable conditional dead code elimination for builtin calls. > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def > b/gcc/config/aarch64/aarch64-tuning-flags.def > index 058dadecccaa..5b087bfc9f03 100644 > --- a/gcc/config/aarch64/aarch64-tuning-flags.def > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION ("cse_sve_vl_constants", > CSE_SVE_VL_CONSTANTS) > > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", > MATCHED_VECTOR_THROUGHPUT) > > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ operand > + chains, and clear -fwidening-mul-defer-fma to suppress the widening-mul > + pass's FMA deferring. */ > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", AVOID_CROSS_LOOP_FMA) > > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", FULLY_PIPELINED_FMA) > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc > index 78f1eae8336c..fdb5bbbdfbce 100644 > --- a/gcc/config/aarch64/aarch64.cc > +++ b/gcc/config/aarch64/aarch64.cc > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct > gcc_options *opts, > && opts->x_optimize >= aarch64_tune_params.prefetch->default_opt_level) > opts->x_flag_prefetch_loop_arrays = 1; > > - /* Avoid loop-dependant FMA chains. */ > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder helper > + keeps using --param=avoid-fma-max-bits; the widening-mul-side > + deferring is gated separately by -fwidening-mul-defer-fma, so we > + suppress only the deferring on these cores while leaving the reassoc > + reorder active. */ > if (aarch64_tune_params.extra_tuning_flags > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA) > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, > - 512); > + { > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, 512); > + SET_OPTION_IF_UNSET (opts, opts_set, flag_widening_mul_defer_fma, 0); > + } > > /* Consider fully pipelined FMA in reassociation. */ > if (aarch64_tune_params.extra_tuning_flags > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi > index 208625cafc55..f39902f77e9b 100644 > --- a/gcc/doc/invoke.texi > +++ b/gcc/doc/invoke.texi > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}. > -funconstrained-commons -funit-at-a-time -funroll-all-loops > -funroll-loops -funsafe-math-optimizations -funswitch-loops > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt > --fweb -fwhole-program -fwpa -fuse-linker-plugin -fzero-call-used-regs > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa > +-fuse-linker-plugin -fzero-call-used-regs > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize} > > @item Program Instrumentation Options > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since variables > no longer stay in a > > Enabled by default with @option{-funroll-loops}. > > +@opindex fwidening-mul-defer-fma > +@opindex fno-widening-mul-defer-fma > +@item -fwidening-mul-defer-fma > +When FMA avoidance is requested for loop-carried reductions (that is, when > +@option{--param=avoid-fma-max-bits} is non-zero), the widening-multiply pass > +refuses to contract a multiply-add whose result feeds a loop-header PHI, > +leaving a separate multiply and add. Use > @option{-fno-widening-mul-defer-fma} > +to suppress this deferring and contract such reductions into an FMA, while > +still leaving @option{--param=avoid-fma-max-bits} in effect for the > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer chains. > +This is useful on targets whose pipeline disfavours the extra floating-point > +operation more than the loop-carried FMA latency. The option is enabled by > +default. Here either. > + > @opindex fwhole-program > @opindex fno-whole-program > @item -fwhole-program > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > new file mode 100644 > index 000000000000..b542274eb763 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > @@ -0,0 +1,19 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" } */ > + > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, which > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried > + reduction is contracted to fmadd rather than deferred to fmul + fadd. > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder enabled.) */ > + > +double > +dot (const double *a, const double *b, int n) > +{ > + double s = 0.0; > + for (int i = 0; i < n; i++) > + s += a[i] * b[i]; > + return s; > +} > + > +/* { dg-final { scan-assembler {\tfmadd\t} } } */ > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > new file mode 100644 > index 000000000000..764dd2457435 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > @@ -0,0 +1,21 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize > --param=avoid-fma-max-bits=512" } */ > + > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form an FMA > + whose result feeds the loop-header phi: the reduction stays as a > + separate fmul + fadd. This is the behaviour the new flag decouples > + from the reassoc reorder (also gated by avoid-fma-max-bits). */ > + > +double > +dot (const double *a, const double *b, int n) > +{ > + double s = 0.0; > + for (int i = 0; i < n; i++) > + s += a[i] * b[i]; > + return s; > +} > + > +/* { dg-final { scan-assembler {\tfmul\t} } } */ > +/* { dg-final { scan-assembler {\tfadd\t} } } */ > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > new file mode 100644 > index 000000000000..5e041afc18fc > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > @@ -0,0 +1,20 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize > --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */ > + > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with FMA > + avoidance still requested for the reassoc reorder (avoid-fma-max-bits=512) > + but -fno-widening-mul-defer-fma now suppresses the widening_mul deferring, > + so the loop-carried multiply-add is contracted to a single fmadd. This > + proves the new flag decouples the deferring from avoid-fma-max-bits. */ > + > +double > +dot (const double *a, const double *b, int n) > +{ > + double s = 0.0; > + for (int i = 0; i < n; i++) > + s += a[i] * b[i]; > + return s; > +} > + > +/* { dg-final { scan-assembler {\tfmadd\t} } } */ > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */ > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc > index f0ede668d95e..a00146922713 100644 > --- a/gcc/tree-ssa-math-opts.cc > +++ b/gcc/tree-ssa-math-opts.cc > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children (basic_block > bb) > { > gimple_stmt_iterator gsi; > > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0); > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0 > + && flag_widening_mul_defer_fma); > > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p (psi); > psi = psi_next) > -- > 2.34.1 > > base-commit: 3b443e764144526748d2b51d017d745f1956377e > branch: ptomsich/475-widening-mul-defer-fma
