Superseded by v2 at https://gcc.gnu.org/pipermail/gcc-patches/2026-July/723165.html
On Tue, 7 Jul 2026 at 23:53, Philipp Tomsich <[email protected]> wrote: > > On Tue, 7 Jul 2026 at 23:51, Andrea Pinski > <[email protected]> wrote: > > > > On Tue, Jul 7, 2026 at 2:46 PM Philipp Tomsich <[email protected]> > > wrote: > > > > > > On Tue, 7 Jul 2026 at 23:40, Andrea Pinski > > > <[email protected]> wrote: > > > > > > > > On Tue, Jul 7, 2026 at 2:35 PM Philipp Tomsich > > > > <[email protected]> wrote: > > > > > > > > > > param_avoid_fma_max_bits gates two independent transforms: the > > > > > widening_mul FMA-deferring in tree-ssa-math-opts.cc, which leaves a > > > > > loop-carried multiply-add as fmul + fadd, and the reassoc > > > > > loop-carried-FMA reorder added in r14-5779-g746344dd538 > > > > > (PR tree-optimization/110279), which parallelises 3+ operand chains. > > > > > Both fire when TYPE_SIZE (elt) <= avoid-fma-max-bits, so for a given > > > > > type they switch on at the same threshold and a target cannot keep one > > > > > while dropping the other. > > > > > > > > > > This hurts the AArch64 AVOID_CROSS_LOOP_FMA cores (the Ampere-1 > > > > > family): a 2-operand reduction such as an sgemm inner K-loop is left > > > > > as > > > > > fmul + fadd, slower than fmadd on their dispatch-bound pipeline, yet > > > > > setting avoid-fma-max-bits to 0 to avoid it also disables the reorder. > > > > > > > > > > Add a generic -fwidening-mul-defer-fma (default on) that gates only > > > > > the > > > > > widening_mul deferring; avoid-fma-max-bits keeps gating the reorder > > > > > alone. The AVOID_CROSS_LOOP_FMA callback clears the flag. No other > > > > > target and no default behaviour changes. > > > > > > > > Note this is just a review on the documentation and nothing else. > > > > > > > > > > > > > > gcc/ChangeLog: > > > > > > > > > > * common.opt (fwidening-mul-defer-fma): New option. > > > > > * tree-ssa-math-opts.cc > > > > > (math_opts_dom_walker::after_dom_children): > > > > > Gate fma_deferring_state's enable predicate on > > > > > flag_widening_mul_defer_fma in addition to > > > > > param_avoid_fma_max_bits. > > > > > * config/aarch64/aarch64-tuning-flags.def > > > > > (AVOID_CROSS_LOOP_FMA): > > > > > Update the comment for the -fwidening-mul-defer-fma effect. > > > > > * config/aarch64/aarch64.cc > > > > > (aarch64_override_options_internal): > > > > > Inside the AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA block, > > > > > clear > > > > > flag_widening_mul_defer_fma. > > > > > * doc/invoke.texi (-fwidening-mul-defer-fma): Document. > > > > > > > > > > gcc/testsuite/ChangeLog: > > > > > > > > > > * gcc.target/aarch64/widening-mul-defer-fma-1.c: New test. > > > > > * gcc.target/aarch64/widening-mul-defer-fma-2.c: New test. > > > > > * gcc.target/aarch64/widening-mul-defer-fma-3.c: New test. > > > > > > > > > > Signed-off-by: Philipp Tomsich <[email protected]> > > > > > --- > > > > > > > > > > gcc/common.opt | 4 ++++ > > > > > gcc/config/aarch64/aarch64-tuning-flags.def | 4 ++++ > > > > > gcc/config/aarch64/aarch64.cc | 12 ++++++++--- > > > > > gcc/doc/invoke.texi | 17 ++++++++++++++- > > > > > .../aarch64/widening-mul-defer-fma-1.c | 19 +++++++++++++++++ > > > > > .../aarch64/widening-mul-defer-fma-2.c | 21 > > > > > +++++++++++++++++++ > > > > > .../aarch64/widening-mul-defer-fma-3.c | 20 ++++++++++++++++++ > > > > > gcc/tree-ssa-math-opts.cc | 3 ++- > > > > > 8 files changed, 95 insertions(+), 5 deletions(-) > > > > > create mode 100644 > > > > > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > > > > > create mode 100644 > > > > > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > > > > > create mode 100644 > > > > > gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > > > > > > > > > > diff --git a/gcc/common.opt b/gcc/common.opt > > > > > index 218dddf5dfe9..9be7dd6b2315 100644 > > > > > --- a/gcc/common.opt > > > > > +++ b/gcc/common.opt > > > > > @@ -3639,6 +3639,10 @@ fweb > > > > > Common Var(flag_web) Optimization EnabledBy(funroll-loops) > > > > > Construct webs and split unrelated uses of single variable. > > > > > > > > > > +fwidening-mul-defer-fma > > > > > +Common Var(flag_widening_mul_defer_fma) Init(1) Optimization > > > > > +Defer forming an FMA whose result feeds a loop-header PHI, leaving a > > > > > separate multiply and add, when FMA avoidance is requested via > > > > > --param=avoid-fma-max-bits. > > > > > > > > Don't mention --param in user documentation directly. > > > > > > That brings up a question we discussed in internal reviews: should > > > this even be a -fwidening-mul-defer-fma flag then — or just another > > > --param? > > > After all, this is the companion for a --param, so the alternate > > > opinion was to have a second --param that would be set to either 0 or > > > 1. However, such a param (that will always be set either to 0 or 1) > > > seems odd. > > > > > > There are plenty of params that are already only support 0 or 1. > > Just after the line 34 of params.opt 4 in a row: > > -param=asan-globals= > > Common Joined UInteger Var(param_asan_globals) Init(1) IntegerRange(0, 1) > > Param > > Enable asan globals protection. > > > > And fully-pipelined-fma is one too: > > -param=fully-pipelined-fma= > > Common Joined UInteger Var(param_fully_pipelined_fma) Init(0) > > IntegerRange(0, 1) Param Optimization > > Whether the target fully pipelines FMA instructions. If non-zero, > > reassociation considers the benefit of parallelizing FMA's multiplication > > part and addition part, assuming FMUL and FMA use the same units that can > > also do FADD. > > I'll take this as "follow the established precedent, even if you don't > find it pretty". > ACK. > > > > > > > > > The third option discussed was a target-hook, but that doesn't allow a > > > commandline-override. > > > > > > Any thoughts? > > > > > > > > > > > > + > > > > > ftree-builtin-call-dce > > > > > Common Var(flag_tree_builtin_call_dce) Init(0) Optimization > > > > > Enable conditional dead code elimination for builtin calls. > > > > > diff --git a/gcc/config/aarch64/aarch64-tuning-flags.def > > > > > b/gcc/config/aarch64/aarch64-tuning-flags.def > > > > > index 058dadecccaa..5b087bfc9f03 100644 > > > > > --- a/gcc/config/aarch64/aarch64-tuning-flags.def > > > > > +++ b/gcc/config/aarch64/aarch64-tuning-flags.def > > > > > @@ -40,6 +40,10 @@ AARCH64_EXTRA_TUNING_OPTION > > > > > ("cse_sve_vl_constants", CSE_SVE_VL_CONSTANTS) > > > > > > > > > > AARCH64_EXTRA_TUNING_OPTION ("matched_vector_throughput", > > > > > MATCHED_VECTOR_THROUGHPUT) > > > > > > > > > > +/* For cores whose pipeline disfavours loop-carried serial FMAs: set > > > > > + avoid-fma-max-bits to 512 to enable the reassoc reorder for 3+ > > > > > operand > > > > > + chains, and clear -fwidening-mul-defer-fma to suppress the > > > > > widening-mul > > > > > + pass's FMA deferring. */ > > > > > AARCH64_EXTRA_TUNING_OPTION ("avoid_cross_loop_fma", > > > > > AVOID_CROSS_LOOP_FMA) > > > > > > > > > > AARCH64_EXTRA_TUNING_OPTION ("fully_pipelined_fma", > > > > > FULLY_PIPELINED_FMA) > > > > > diff --git a/gcc/config/aarch64/aarch64.cc > > > > > b/gcc/config/aarch64/aarch64.cc > > > > > index 78f1eae8336c..fdb5bbbdfbce 100644 > > > > > --- a/gcc/config/aarch64/aarch64.cc > > > > > +++ b/gcc/config/aarch64/aarch64.cc > > > > > @@ -20087,11 +20087,17 @@ aarch64_override_options_internal (struct > > > > > gcc_options *opts, > > > > > && opts->x_optimize >= > > > > > aarch64_tune_params.prefetch->default_opt_level) > > > > > opts->x_flag_prefetch_loop_arrays = 1; > > > > > > > > > > - /* Avoid loop-dependant FMA chains. */ > > > > > + /* Avoid loop-dependant FMA chains. The reassoc-side reorder > > > > > helper > > > > > + keeps using --param=avoid-fma-max-bits; the widening-mul-side > > > > > + deferring is gated separately by -fwidening-mul-defer-fma, so we > > > > > + suppress only the deferring on these cores while leaving the > > > > > reassoc > > > > > + reorder active. */ > > > > > if (aarch64_tune_params.extra_tuning_flags > > > > > & AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA) > > > > > - SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, > > > > > - 512); > > > > > + { > > > > > + SET_OPTION_IF_UNSET (opts, opts_set, param_avoid_fma_max_bits, > > > > > 512); > > > > > + SET_OPTION_IF_UNSET (opts, opts_set, > > > > > flag_widening_mul_defer_fma, 0); > > > > > + } > > > > > > > > > > /* Consider fully pipelined FMA in reassociation. */ > > > > > if (aarch64_tune_params.extra_tuning_flags > > > > > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi > > > > > index 208625cafc55..f39902f77e9b 100644 > > > > > --- a/gcc/doc/invoke.texi > > > > > +++ b/gcc/doc/invoke.texi > > > > > @@ -694,7 +694,8 @@ Objective-C and Objective-C++ Dialects}. > > > > > -funconstrained-commons -funit-at-a-time -funroll-all-loops > > > > > -funroll-loops -funsafe-math-optimizations -funswitch-loops > > > > > -fipa-ra -fvariable-expansion-in-unroller -fvect-cost-model -fvpt > > > > > --fweb -fwhole-program -fwpa -fuse-linker-plugin > > > > > -fzero-call-used-regs > > > > > +-fweb -fwhole-program -fwidening-mul-defer-fma -fwpa > > > > > +-fuse-linker-plugin -fzero-call-used-regs > > > > > -O -O0 -O1 -O2 -O3 -Os -Ofast -Og -Oz --optimize} > > > > > > > > > > @item Program Instrumentation Options > > > > > @@ -15917,6 +15918,20 @@ however, make debugging impossible, since > > > > > variables no longer stay in a > > > > > > > > > > Enabled by default with @option{-funroll-loops}. > > > > > > > > > > +@opindex fwidening-mul-defer-fma > > > > > +@opindex fno-widening-mul-defer-fma > > > > > +@item -fwidening-mul-defer-fma > > > > > +When FMA avoidance is requested for loop-carried reductions (that > > > > > is, when > > > > > +@option{--param=avoid-fma-max-bits} is non-zero), the > > > > > widening-multiply pass > > > > > +refuses to contract a multiply-add whose result feeds a loop-header > > > > > PHI, > > > > > +leaving a separate multiply and add. Use > > > > > @option{-fno-widening-mul-defer-fma} > > > > > +to suppress this deferring and contract such reductions into an FMA, > > > > > while > > > > > +still leaving @option{--param=avoid-fma-max-bits} in effect for the > > > > > +@file{tree-ssa-reassoc.cc} loop-carried-FMA reordering of longer > > > > > chains. > > > > > +This is useful on targets whose pipeline disfavours the extra > > > > > floating-point > > > > > +operation more than the loop-carried FMA latency. The option is > > > > > enabled by > > > > > +default. > > > > Here either. > > > > > > > > > > > > > + > > > > > @opindex fwhole-program > > > > > @opindex fno-whole-program > > > > > @item -fwhole-program > > > > > diff --git > > > > > a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > > > > > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > > > > > new file mode 100644 > > > > > index 000000000000..b542274eb763 > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-1.c > > > > > @@ -0,0 +1,19 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize -mcpu=ampere1" > > > > > } */ > > > > > + > > > > > +/* The ampere1 family sets AARCH64_EXTRA_TUNE_AVOID_CROSS_LOOP_FMA, > > > > > which > > > > > + clears -fwidening-mul-defer-fma so that a 2-operand loop-carried > > > > > + reduction is contracted to fmadd rather than deferred to fmul + > > > > > fadd. > > > > > + (avoid-fma-max-bits stays 512 to keep the reassoc reorder > > > > > enabled.) */ > > > > > + > > > > > +double > > > > > +dot (const double *a, const double *b, int n) > > > > > +{ > > > > > + double s = 0.0; > > > > > + for (int i = 0; i < n; i++) > > > > > + s += a[i] * b[i]; > > > > > + return s; > > > > > +} > > > > > + > > > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */ > > > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */ > > > > > diff --git > > > > > a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > > > > > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > > > > > new file mode 100644 > > > > > index 000000000000..764dd2457435 > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-2.c > > > > > @@ -0,0 +1,21 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize > > > > > --param=avoid-fma-max-bits=512" } */ > > > > > + > > > > > +/* With FMA deferring active (avoid-fma-max-bits > 0) and the default > > > > > + -fwidening-mul-defer-fma, the widening_mul pass refuses to form > > > > > an FMA > > > > > + whose result feeds the loop-header phi: the reduction stays as a > > > > > + separate fmul + fadd. This is the behaviour the new flag > > > > > decouples > > > > > + from the reassoc reorder (also gated by avoid-fma-max-bits). */ > > > > > + > > > > > +double > > > > > +dot (const double *a, const double *b, int n) > > > > > +{ > > > > > + double s = 0.0; > > > > > + for (int i = 0; i < n; i++) > > > > > + s += a[i] * b[i]; > > > > > + return s; > > > > > +} > > > > > + > > > > > +/* { dg-final { scan-assembler {\tfmul\t} } } */ > > > > > +/* { dg-final { scan-assembler {\tfadd\t} } } */ > > > > > +/* { dg-final { scan-assembler-not {\tfmadd\t} } } */ > > > > > diff --git > > > > > a/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > > > > > b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > > > > > new file mode 100644 > > > > > index 000000000000..5e041afc18fc > > > > > --- /dev/null > > > > > +++ b/gcc/testsuite/gcc.target/aarch64/widening-mul-defer-fma-3.c > > > > > @@ -0,0 +1,20 @@ > > > > > +/* { dg-do compile } */ > > > > > +/* { dg-options "-O2 -ffast-math -fno-tree-vectorize > > > > > --param=avoid-fma-max-bits=512 -fno-widening-mul-defer-fma" } */ > > > > > + > > > > > +/* Companion to widening-mul-defer-fma-2.c: the same reduction with > > > > > FMA > > > > > + avoidance still requested for the reassoc reorder > > > > > (avoid-fma-max-bits=512) > > > > > + but -fno-widening-mul-defer-fma now suppresses the widening_mul > > > > > deferring, > > > > > + so the loop-carried multiply-add is contracted to a single fmadd. > > > > > This > > > > > + proves the new flag decouples the deferring from > > > > > avoid-fma-max-bits. */ > > > > > + > > > > > +double > > > > > +dot (const double *a, const double *b, int n) > > > > > +{ > > > > > + double s = 0.0; > > > > > + for (int i = 0; i < n; i++) > > > > > + s += a[i] * b[i]; > > > > > + return s; > > > > > +} > > > > > + > > > > > +/* { dg-final { scan-assembler {\tfmadd\t} } } */ > > > > > +/* { dg-final { scan-assembler-not {\tfmul\t} } } */ > > > > > diff --git a/gcc/tree-ssa-math-opts.cc b/gcc/tree-ssa-math-opts.cc > > > > > index f0ede668d95e..a00146922713 100644 > > > > > --- a/gcc/tree-ssa-math-opts.cc > > > > > +++ b/gcc/tree-ssa-math-opts.cc > > > > > @@ -6607,7 +6607,8 @@ math_opts_dom_walker::after_dom_children > > > > > (basic_block bb) > > > > > { > > > > > gimple_stmt_iterator gsi; > > > > > > > > > > - fma_deferring_state fma_state (param_avoid_fma_max_bits > 0); > > > > > + fma_deferring_state fma_state (param_avoid_fma_max_bits > 0 > > > > > + && flag_widening_mul_defer_fma); > > > > > > > > > > for (gphi_iterator psi_next, psi = gsi_start_phis (bb); !gsi_end_p > > > > > (psi); > > > > > psi = psi_next) > > > > > -- > > > > > 2.34.1 > > > > > > > > > > base-commit: 3b443e764144526748d2b51d017d745f1956377e > > > > > branch: ptomsich/475-widening-mul-defer-fma
