On Mon, Sep 7, 2026 at 4:35 PM Konstantinos Eleftheriou <[email protected]> wrote: > > Extends pass_loop_jam with a parallel-reduction branch for the > GEMV / NNUE inference shape > > for (i) { > s = bias[i]; > for (j) s += w[i][j] * x[j]; > out[i] = s; > } > > Stock unroll_jam_possible_p rejects this nest because `s' is a > reduction PHI, not a simple induction; even if it were accepted, > stock fuse_loops would collapse the post-unroll per-sibling reduction > PHIs into one serial chain, multiplying the inner loop's serial > dependency height by N and defeating the goal of unroll-and-jam.
So given we do not unroll-and-jam this, wouldn't it be better to "simply" support outer loop vectorizing for it because outer loop vectorization is exactly unroll-and-jam under the hood? > unroll_jam_reduction_possible_p detects the candidate, the existing > tree_unroll_loop strip-mines the outer loop by N, and > unroll_jam_reduction_fuse_sibling fuses the N inner siblings into one > body that retains N parallel reduction PHIs in its header and hoists > each sibling's preamble (s_k = bias[i+k]) to the fused preheader. The > post-reduction stores stay on the linear sibling-exit chain; > cleanup_tree_cfg coalesces it and rewrite_into_loop_closed_ssa restores > LC-SSA. The vectorizer then sees one inner loop with N independent > chains and emits N parallel dot-product / FMA accumulators, cutting the > serial dependency height across the reduction by N. > > N is the min of TARGET_LOOP_JAM_REDUCTION_FACTOR (opt-in, default 0 > = "no benefit measured"), --param=unroll-jam-reduction-streams and > adjust_unroll_factor's data-dep safety bound, narrowed to a divisor of > the outer trip count. The hook receives the accumulator mode plus the > narrowest scalar-int mode found at a widening multiply in the update > chain; op_mode == acc_mode signals "no widening multiply", letting a > target opt in for MAC shapes only. Inner loops whose trip count is > below --param=unroll-jam-reduction-min-inner-iters are left alone: > their per-stream serial chain is too short to profit from parallelism. > > AArch64 opts in for QI/HI -> SI (DOT_PROD class) and SI -> DI > (SMLAL_DI); all other shapes (plain reductions, max/min, FP) return 0. > The per-CPU factor is sourced from a new tune_params field > loop_jam_reduction_factor rather than a hardcoded return value, so the > profitability decision lives in the tuning structure next to the other > latency-driven knobs. Cores with measured wins ship factor 4; every > other tuning leaves the field at 0 until measured. The new tests pin > -mtune to a factor-4 core so they exercise that path independently of > the configured default tuning. > > FP reductions are deferred to outer-loop vectorization's lane-broadcast > FMLA path. The chained-reduction case (single accumulator carried > across outer iterations) is left as a follow-up. > > ~8% speedup on SPEC CPU 2026 706.stockfish_r. > > gcc/ChangeLog: > > * common.opt (floop-unroll-and-jam-reductions): New option, > EnabledBy(floop-unroll-and-jam). > * config/aarch64/aarch64-protos.h (struct tune_params): > Add loop_jam_reduction_factor. > * config/aarch64/aarch64.cc (aarch64_loop_jam_reduction_factor): > New function; source factor from aarch64_tune_params and opt > in for QI/HI->SI and SI->DI MAC shapes only. > (TARGET_LOOP_JAM_REDUCTION_FACTOR): Define. > * config/aarch64/tuning_models/a64fx.h: Default > loop_jam_reduction_factor to 0 (opt-out). > * config/aarch64/tuning_models/ampere1.h: Set > loop_jam_reduction_factor to 4. > * config/aarch64/tuning_models/ampere1a.h: Likewise. > * config/aarch64/tuning_models/ampere1b.h: Likewise. > * config/aarch64/tuning_models/cortexa320.h: Default > loop_jam_reduction_factor to 0 (opt-out). > * config/aarch64/tuning_models/cortexa35.h: Likewise. > * config/aarch64/tuning_models/cortexa53.h: Likewise. > * config/aarch64/tuning_models/cortexa57.h: Likewise. > * config/aarch64/tuning_models/cortexa72.h: Likewise. > * config/aarch64/tuning_models/cortexa73.h: Likewise. > * config/aarch64/tuning_models/cortexx925.h: Likewise. > * config/aarch64/tuning_models/emag.h: Likewise. > * config/aarch64/tuning_models/exynosm1.h: Likewise. > * config/aarch64/tuning_models/fujitsu_monaka.h: Likewise. > * config/aarch64/tuning_models/generic.h: Likewise. > * config/aarch64/tuning_models/generic_armv8_a.h: Likewise. > * config/aarch64/tuning_models/generic_armv9_a.h: Likewise. > * config/aarch64/tuning_models/hip12.h: Likewise. > * config/aarch64/tuning_models/neoverse512tvb.h: Likewise. > * config/aarch64/tuning_models/neoversen1.h: Set > loop_jam_reduction_factor to 4. > * config/aarch64/tuning_models/neoversen2.h: Default > loop_jam_reduction_factor to 0 (opt-out). > * config/aarch64/tuning_models/neoversen3.h: Likewise. > * config/aarch64/tuning_models/neoversev1.h: Likewise. > * config/aarch64/tuning_models/neoversev2.h: Likewise. > * config/aarch64/tuning_models/neoversev3.h: Likewise. > * config/aarch64/tuning_models/neoversev3ae.h: Likewise. > * config/aarch64/tuning_models/olympus.h: Likewise. > * config/aarch64/tuning_models/qdf24xx.h: Likewise. > * config/aarch64/tuning_models/saphira.h: Likewise. > * config/aarch64/tuning_models/thunderx.h: Likewise. > * config/aarch64/tuning_models/thunderx2t99.h: Likewise. > * config/aarch64/tuning_models/thunderx3t110.h: Likewise. > * config/aarch64/tuning_models/thunderxt88.h: Likewise. > * config/aarch64/tuning_models/tsv110.h: Likewise. > * config/aarch64/tuning_models/xgene1.h: Likewise. > * doc/invoke.texi (-floop-unroll-and-jam-reductions): Document. > * doc/tm.texi: Regenerate. > * doc/tm.texi.in (TARGET_LOOP_JAM_REDUCTION_FACTOR): Add hook > reference. > * gimple-loop-jam.cc: Include cfganal.h and tree-into-ssa.h. > (inner_reduction_phi): New helper. > (narrowest_reduc_input_mode): New helper that walks the > reduction update chain for widening multiplies. > (gsi_move_to_bb_end_before_terminator): New helper. > (collapse_single_arg_phi): New helper. > (loop_is_dot_product_candidate_p): New static helper. > (unroll_jam_reduction_possible_p): New candidate detector; > calls loop_is_dot_product_candidate_p and layers outer-loop > guards on top. > (migrate_loop_invariant_defs_to_preheader): New helper that > hoists the SSA-use closure of a sibling's reduction init and > body operands to the destination preheader in RPO. > (match_sibling_header_phis): New helper pairing sibling non- > reduction PHIs positionally. > (unroll_jam_reduction_fuse_sibling): New per-sibling fuser. > (unroll_jam_reduction_bypass_sibling): New CFG bypass for the > drained source sibling. > (compute_jam_dependence_factors): New helper. Walks OUTER's > data dependences and accumulates the unroll-factor safety > bound, ref-removal profit signal, removed-ref count, and total > dataref count. > (select_reduction_jam_factor): New helper combining target > factor, param cap and data-dep safety bound, narrowed to a > divisor of the outer trip count. Emit a TDF_DETAILS note > when the outer trip has no divisor in [2, factor cap]. > (do_unroll_jam_reductions): New driver helper. > (tree_loop_unroll_and_jam): Try the reduction-jam path on > nests the IV-only path skips; cleanup CFG, rebuild LC-SSA and > reset SCEV per transformed nest. > (pass_loop_jam::gate): Also enable for > flag_unroll_jam_reductions. > * hooks.cc (hook_uint_mode_mode_0): New default hook. > * hooks.h (hook_uint_mode_mode_0): Declare. > * params.opt (unroll-jam-reduction-streams): New param. > (unroll-jam-reduction-min-inner-iters): New param. > * target.def (loop_jam_reduction_factor): New target hook. > > gcc/testsuite/ChangeLog: > > * gcc.target/aarch64/unroll-jam-reductions-1.c: New test. > * gcc.target/aarch64/unroll-jam-reductions-2.c: New test. > * gcc.target/aarch64/unroll-jam-reductions-3.c: New test. > * gcc.target/aarch64/unroll-jam-reductions-4.c: New test. > * gcc.target/aarch64/unroll-jam-reductions-5.c: New test. > > Co-Authored-By: Philipp Tomsich <[email protected]> > Signed-off-by: Konstantinos Eleftheriou <[email protected]> > --- > > gcc/common.opt | 5 + > gcc/config/aarch64/aarch64-protos.h | 8 + > gcc/config/aarch64/aarch64.cc | 29 + > gcc/config/aarch64/tuning_models/a64fx.h | 3 +- > gcc/config/aarch64/tuning_models/ampere1.h | 3 +- > gcc/config/aarch64/tuning_models/ampere1a.h | 3 +- > gcc/config/aarch64/tuning_models/ampere1b.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa320.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa35.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa53.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa57.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa72.h | 3 +- > gcc/config/aarch64/tuning_models/cortexa73.h | 3 +- > gcc/config/aarch64/tuning_models/cortexx925.h | 3 +- > gcc/config/aarch64/tuning_models/emag.h | 3 +- > gcc/config/aarch64/tuning_models/exynosm1.h | 3 +- > .../aarch64/tuning_models/fujitsu_monaka.h | 3 +- > gcc/config/aarch64/tuning_models/generic.h | 3 +- > .../aarch64/tuning_models/generic_armv8_a.h | 3 +- > .../aarch64/tuning_models/generic_armv9_a.h | 3 +- > gcc/config/aarch64/tuning_models/hip12.h | 3 +- > .../aarch64/tuning_models/neoverse512tvb.h | 3 +- > gcc/config/aarch64/tuning_models/neoversen1.h | 3 +- > gcc/config/aarch64/tuning_models/neoversen2.h | 3 +- > gcc/config/aarch64/tuning_models/neoversen3.h | 3 +- > gcc/config/aarch64/tuning_models/neoversev1.h | 3 +- > gcc/config/aarch64/tuning_models/neoversev2.h | 3 +- > gcc/config/aarch64/tuning_models/neoversev3.h | 3 +- > .../aarch64/tuning_models/neoversev3ae.h | 3 +- > gcc/config/aarch64/tuning_models/olympus.h | 3 +- > gcc/config/aarch64/tuning_models/qdf24xx.h | 3 +- > gcc/config/aarch64/tuning_models/saphira.h | 3 +- > gcc/config/aarch64/tuning_models/thunderx.h | 3 +- > .../aarch64/tuning_models/thunderx2t99.h | 3 +- > .../aarch64/tuning_models/thunderx3t110.h | 3 +- > .../aarch64/tuning_models/thunderxt88.h | 3 +- > gcc/config/aarch64/tuning_models/tsv110.h | 3 +- > gcc/config/aarch64/tuning_models/xgene1.h | 3 +- > gcc/doc/invoke.texi | 17 +- > gcc/doc/tm.texi | 11 + > gcc/doc/tm.texi.in | 2 + > gcc/gimple-loop-jam.cc | 932 +++++++++++++++++- > gcc/hooks.cc | 7 + > gcc/hooks.h | 1 + > gcc/params.opt | 8 + > gcc/target.def | 13 + > .../aarch64/unroll-jam-reductions-1.c | 78 ++ > .../aarch64/unroll-jam-reductions-2.c | 89 ++ > .../aarch64/unroll-jam-reductions-3.c | 189 ++++ > .../aarch64/unroll-jam-reductions-4.c | 30 + > .../aarch64/unroll-jam-reductions-5.c | 72 ++ > 51 files changed, 1557 insertions(+), 39 deletions(-) > create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c > create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c > create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c > create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c > create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c > > diff --git a/gcc/common.opt b/gcc/common.opt > index 69a55b8d40d6..634b63569c67 100644 > --- a/gcc/common.opt > +++ b/gcc/common.opt > @@ -1928,6 +1928,11 @@ floop-unroll-and-jam > Common Var(flag_unroll_jam) Optimization > Perform unroll-and-jam on loops. > > +floop-unroll-and-jam-reductions > +Common Var(flag_unroll_jam_reductions) Optimization > EnabledBy(floop-unroll-and-jam) > +Perform reduction-aware unroll-and-jam on loops with inner reductions > +to enable multi-stream parallelism during vectorization. > + > fgnu-tm > Common Var(flag_tm) > Enable support for GNU transactional memory. > diff --git a/gcc/config/aarch64/aarch64-protos.h > b/gcc/config/aarch64/aarch64-protos.h > index e8ae3d427949..88237e243738 100644 > --- a/gcc/config/aarch64/aarch64-protos.h > +++ b/gcc/config/aarch64/aarch64-protos.h > @@ -590,6 +590,14 @@ struct tune_params > > /* Dispatch constraints for instruction scheduling. */ > const struct dispatch_constraint_info *dispatch_constraints; > + > + /* Preferred number of parallel reduction streams for > + -floop-unroll-and-jam-reductions on this core. Zero opts out; > + two or more enables the transform on reduction nests whose > + (acc_mode, op_mode) pair the target hook accepts. Cores with a > + two-wide int/FP pipe and several-cycle MAC latency typically > + want 4. */ > + unsigned int loop_jam_reduction_factor; > }; > > /* Classifies an address. > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc > index ccb4f5fbb10d..546912534499 100644 > --- a/gcc/config/aarch64/aarch64.cc > +++ b/gcc/config/aarch64/aarch64.cc > @@ -17402,6 +17402,32 @@ aarch64_sched_issue_rate (void) > return aarch64_tune_params.issue_rate; > } > > +/* Implement TARGET_LOOP_JAM_REDUCTION_FACTOR. Return the preferred > + number of parallel reduction streams for > + -floop-unroll-and-jam-reductions on a reduction accumulating into > + ACC_MODE from operands of OP_MODE. We opt in only for the widening > + MAC shapes the vectorizer lowers to a latency-bound > + DOT_PROD / SMLAL chain: > + * QImode / HImode -> SImode (DOT_PROD-class, USDOT / SDOT / UDOT) > + * SImode -> DImode (SMLAL_DI) > + The stream count comes from the per-core tuning structure; cores > + that have not been measured leave loop_jam_reduction_factor at 0, > + which suppresses the transform. */ > + > +static unsigned int > +aarch64_loop_jam_reduction_factor (machine_mode acc_mode, > + machine_mode op_mode) > +{ > + unsigned int factor = aarch64_tune_params.loop_jam_reduction_factor; > + if (factor < 2 || !SCALAR_INT_MODE_P (acc_mode)) > + return 0; > + > + if ((acc_mode == SImode && (op_mode == QImode || op_mode == HImode)) > + || (acc_mode == DImode && op_mode == SImode)) > + return factor; > + return 0; > +} > + > /* Implement TARGET_SCHED_VARIABLE_ISSUE. */ > static int > aarch64_sched_variable_issue (FILE *, int, rtx_insn *insn, int more) > @@ -34491,6 +34517,9 @@ aarch64_libgcc_floating_mode_supported_p > #undef TARGET_SCHED_ISSUE_RATE > #define TARGET_SCHED_ISSUE_RATE aarch64_sched_issue_rate > > +#undef TARGET_LOOP_JAM_REDUCTION_FACTOR > +#define TARGET_LOOP_JAM_REDUCTION_FACTOR aarch64_loop_jam_reduction_factor > + > #undef TARGET_SCHED_VARIABLE_ISSUE > #define TARGET_SCHED_VARIABLE_ISSUE aarch64_sched_variable_issue > > diff --git a/gcc/config/aarch64/tuning_models/a64fx.h > b/gcc/config/aarch64/tuning_models/a64fx.h > index c3222960f175..171e64ce23ad 100644 > --- a/gcc/config/aarch64/tuning_models/a64fx.h > +++ b/gcc/config/aarch64/tuning_models/a64fx.h > @@ -166,7 +166,8 @@ static const struct tune_params a64fx_tunings = > &a64fx_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_A64FX. */ > diff --git a/gcc/config/aarch64/tuning_models/ampere1.h > b/gcc/config/aarch64/tuning_models/ampere1.h > index e2a42f585dd9..0057db3765b4 100644 > --- a/gcc/config/aarch64/tuning_models/ampere1.h > +++ b/gcc/config/aarch64/tuning_models/ampere1.h > @@ -136,7 +136,8 @@ static const struct tune_params ampere1_tunings = > &ere1_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 4 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_AMPERE1. */ > diff --git a/gcc/config/aarch64/tuning_models/ampere1a.h > b/gcc/config/aarch64/tuning_models/ampere1a.h > index 0954ac551a0b..1350c0f3b5c5 100644 > --- a/gcc/config/aarch64/tuning_models/ampere1a.h > +++ b/gcc/config/aarch64/tuning_models/ampere1a.h > @@ -58,7 +58,8 @@ static const struct tune_params ampere1a_tunings = > &ere1_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 4 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_AMPERE1A. */ > diff --git a/gcc/config/aarch64/tuning_models/ampere1b.h > b/gcc/config/aarch64/tuning_models/ampere1b.h > index 8ba434d27bbf..0fc6eb36bd2a 100644 > --- a/gcc/config/aarch64/tuning_models/ampere1b.h > +++ b/gcc/config/aarch64/tuning_models/ampere1b.h > @@ -137,7 +137,8 @@ static const struct tune_params ampere1b_tunings = > &ere1b_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 4 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_AMPERE1B */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa320.h > b/gcc/config/aarch64/tuning_models/cortexa320.h > index d5e1f2b29938..89ed24360008 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa320.h > +++ b/gcc/config/aarch64/tuning_models/cortexa320.h > @@ -189,7 +189,8 @@ static const struct tune_params cortexa320_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXA320. */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa35.h > b/gcc/config/aarch64/tuning_models/cortexa35.h > index fd2fc47f5e23..ff9eeb406799 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa35.h > +++ b/gcc/config/aarch64/tuning_models/cortexa35.h > @@ -57,7 +57,8 @@ static const struct tune_params cortexa35_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXA35. */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa53.h > b/gcc/config/aarch64/tuning_models/cortexa53.h > index e71cc1b26afb..13dbb152c6e9 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa53.h > +++ b/gcc/config/aarch64/tuning_models/cortexa53.h > @@ -66,7 +66,8 @@ static const struct tune_params cortexa53_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXA53. */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa57.h > b/gcc/config/aarch64/tuning_models/cortexa57.h > index 4f3ec60ba275..c78c15198158 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa57.h > +++ b/gcc/config/aarch64/tuning_models/cortexa57.h > @@ -103,7 +103,8 @@ static const struct tune_params cortexa57_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXA57. */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa72.h > b/gcc/config/aarch64/tuning_models/cortexa72.h > index df0fcf110867..8a4e5e9d8a36 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa72.h > +++ b/gcc/config/aarch64/tuning_models/cortexa72.h > @@ -55,7 +55,8 @@ static const struct tune_params cortexa72_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXA72. */ > diff --git a/gcc/config/aarch64/tuning_models/cortexa73.h > b/gcc/config/aarch64/tuning_models/cortexa73.h > index c871b454d40f..40d46f23ae82 100644 > --- a/gcc/config/aarch64/tuning_models/cortexa73.h > +++ b/gcc/config/aarch64/tuning_models/cortexa73.h > @@ -56,7 +56,8 @@ static const struct tune_params cortexa73_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > > diff --git a/gcc/config/aarch64/tuning_models/cortexx925.h > b/gcc/config/aarch64/tuning_models/cortexx925.h > index 2a73b3697065..b99d6b4300e5 100644 > --- a/gcc/config/aarch64/tuning_models/cortexx925.h > +++ b/gcc/config/aarch64/tuning_models/cortexx925.h > @@ -227,7 +227,8 @@ static const struct tune_params cortexx925_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_CORTEXX925. */ > diff --git a/gcc/config/aarch64/tuning_models/emag.h > b/gcc/config/aarch64/tuning_models/emag.h > index 82bbc554f41b..6e87d28173b2 100644 > --- a/gcc/config/aarch64/tuning_models/emag.h > +++ b/gcc/config/aarch64/tuning_models/emag.h > @@ -55,7 +55,8 @@ static const struct tune_params emag_tunings = > &xgene1_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_EMAG. */ > diff --git a/gcc/config/aarch64/tuning_models/exynosm1.h > b/gcc/config/aarch64/tuning_models/exynosm1.h > index a561ccf6fa71..928f3d4f3014 100644 > --- a/gcc/config/aarch64/tuning_models/exynosm1.h > +++ b/gcc/config/aarch64/tuning_models/exynosm1.h > @@ -139,7 +139,8 @@ static const struct tune_params exynosm1_tunings = > &exynosm1_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_EXYNOSM1. */ > diff --git a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h > b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h > index 71549bd0ea70..70d71bf253de 100644 > --- a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h > +++ b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h > @@ -59,7 +59,8 @@ static const struct tune_params fujitsu_monaka_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_FUJITSU_MONAKA. */ > diff --git a/gcc/config/aarch64/tuning_models/generic.h > b/gcc/config/aarch64/tuning_models/generic.h > index af013e478d45..71dbebeb47c0 100644 > --- a/gcc/config/aarch64/tuning_models/generic.h > +++ b/gcc/config/aarch64/tuning_models/generic.h > @@ -188,7 +188,8 @@ static const struct tune_params generic_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_GENERIC. */ > diff --git a/gcc/config/aarch64/tuning_models/generic_armv8_a.h > b/gcc/config/aarch64/tuning_models/generic_armv8_a.h > index 29cf3a5a7c7a..15900a7ba733 100644 > --- a/gcc/config/aarch64/tuning_models/generic_armv8_a.h > +++ b/gcc/config/aarch64/tuning_models/generic_armv8_a.h > @@ -180,7 +180,8 @@ static const struct tune_params generic_armv8_a_tunings = > &generic_armv8_a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_GENERIC_ARMV8_A. */ > diff --git a/gcc/config/aarch64/tuning_models/generic_armv9_a.h > b/gcc/config/aarch64/tuning_models/generic_armv9_a.h > index 05e777d20efb..7c1b78f88a53 100644 > --- a/gcc/config/aarch64/tuning_models/generic_armv9_a.h > +++ b/gcc/config/aarch64/tuning_models/generic_armv9_a.h > @@ -255,7 +255,8 @@ static const struct tune_params generic_armv9_a_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_GENERIC_ARMV9_A. */ > diff --git a/gcc/config/aarch64/tuning_models/hip12.h > b/gcc/config/aarch64/tuning_models/hip12.h > index cfb204b2e533..2d5c107d71d7 100644 > --- a/gcc/config/aarch64/tuning_models/hip12.h > +++ b/gcc/config/aarch64/tuning_models/hip12.h > @@ -221,7 +221,8 @@ static const struct tune_params hip12_tunings = > &generic_armv8_a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_HIP12. */ > diff --git a/gcc/config/aarch64/tuning_models/neoverse512tvb.h > b/gcc/config/aarch64/tuning_models/neoverse512tvb.h > index d6e2c2e8120d..a128d5ed7b3a 100644 > --- a/gcc/config/aarch64/tuning_models/neoverse512tvb.h > +++ b/gcc/config/aarch64/tuning_models/neoverse512tvb.h > @@ -162,7 +162,8 @@ static const struct tune_params neoverse512tvb_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSE512TVB. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversen1.h > b/gcc/config/aarch64/tuning_models/neoversen1.h > index 87c7fbcb7f22..9b2ec9322bba 100644 > --- a/gcc/config/aarch64/tuning_models/neoversen1.h > +++ b/gcc/config/aarch64/tuning_models/neoversen1.h > @@ -55,7 +55,8 @@ static const struct tune_params neoversen1_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 4 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEN1. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversen2.h > b/gcc/config/aarch64/tuning_models/neoversen2.h > index 21597e6b50f8..e1648c88eb78 100644 > --- a/gcc/config/aarch64/tuning_models/neoversen2.h > +++ b/gcc/config/aarch64/tuning_models/neoversen2.h > @@ -224,7 +224,8 @@ static const struct tune_params neoversen2_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEN2. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversen3.h > b/gcc/config/aarch64/tuning_models/neoversen3.h > index 37f9819148f6..8796914772b4 100644 > --- a/gcc/config/aarch64/tuning_models/neoversen3.h > +++ b/gcc/config/aarch64/tuning_models/neoversen3.h > @@ -223,7 +223,8 @@ static const struct tune_params neoversen3_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEN3. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversev1.h > b/gcc/config/aarch64/tuning_models/neoversev1.h > index 253f11e87a68..7f93e4092550 100644 > --- a/gcc/config/aarch64/tuning_models/neoversev1.h > +++ b/gcc/config/aarch64/tuning_models/neoversev1.h > @@ -233,7 +233,8 @@ static const struct tune_params neoversev1_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > > diff --git a/gcc/config/aarch64/tuning_models/neoversev2.h > b/gcc/config/aarch64/tuning_models/neoversev2.h > index 6df0cc444b80..8ceb7d5538fb 100644 > --- a/gcc/config/aarch64/tuning_models/neoversev2.h > +++ b/gcc/config/aarch64/tuning_models/neoversev2.h > @@ -364,7 +364,8 @@ static const struct tune_params neoversev2_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - &neoversev2_dispatch_constraint_info /* dispatch_constraints. */ > + &neoversev2_dispatch_constraint_info, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEV2. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversev3.h > b/gcc/config/aarch64/tuning_models/neoversev3.h > index 7e0144032dc6..ea840d2cbb7c 100644 > --- a/gcc/config/aarch64/tuning_models/neoversev3.h > +++ b/gcc/config/aarch64/tuning_models/neoversev3.h > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEV3. */ > diff --git a/gcc/config/aarch64/tuning_models/neoversev3ae.h > b/gcc/config/aarch64/tuning_models/neoversev3ae.h > index bbe4dc547015..fe39c7b67c9d 100644 > --- a/gcc/config/aarch64/tuning_models/neoversev3ae.h > +++ b/gcc/config/aarch64/tuning_models/neoversev3ae.h > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3ae_tunings = > &generic_armv9a_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_NEOVERSEV3AE. */ > diff --git a/gcc/config/aarch64/tuning_models/olympus.h > b/gcc/config/aarch64/tuning_models/olympus.h > index 1e701722e224..ab1ba2464d23 100644 > --- a/gcc/config/aarch64/tuning_models/olympus.h > +++ b/gcc/config/aarch64/tuning_models/olympus.h > @@ -565,7 +565,8 @@ static struct tune_params olympus_tunings = > &olympus_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - &olympus_dispatch_constraint_info /* dispatch_constraints. */ > + &olympus_dispatch_constraint_info, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_OLYMPUS. */ > diff --git a/gcc/config/aarch64/tuning_models/qdf24xx.h > b/gcc/config/aarch64/tuning_models/qdf24xx.h > index a15a4fc56897..73eb0b8c48cf 100644 > --- a/gcc/config/aarch64/tuning_models/qdf24xx.h > +++ b/gcc/config/aarch64/tuning_models/qdf24xx.h > @@ -131,7 +131,8 @@ static const struct tune_params qdf24xx_tunings = > &qdf24xx_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_QDF24XX. */ > diff --git a/gcc/config/aarch64/tuning_models/saphira.h > b/gcc/config/aarch64/tuning_models/saphira.h > index 369d3618f7f1..582c4517fcdc 100644 > --- a/gcc/config/aarch64/tuning_models/saphira.h > +++ b/gcc/config/aarch64/tuning_models/saphira.h > @@ -57,7 +57,8 @@ static const struct tune_params saphira_tunings = > &generic_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_SAPHIRA. */ > diff --git a/gcc/config/aarch64/tuning_models/thunderx.h > b/gcc/config/aarch64/tuning_models/thunderx.h > index ed8de307276a..a275659c0b9e 100644 > --- a/gcc/config/aarch64/tuning_models/thunderx.h > +++ b/gcc/config/aarch64/tuning_models/thunderx.h > @@ -112,7 +112,8 @@ static const struct tune_params thunderx_tunings = > &thunderx_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_THUNDERX. */ > diff --git a/gcc/config/aarch64/tuning_models/thunderx2t99.h > b/gcc/config/aarch64/tuning_models/thunderx2t99.h > index f6be657ac9c3..5afbc21ef40c 100644 > --- a/gcc/config/aarch64/tuning_models/thunderx2t99.h > +++ b/gcc/config/aarch64/tuning_models/thunderx2t99.h > @@ -132,7 +132,8 @@ static const struct tune_params thunderx2t99_tunings = > &thunderx2t99_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_THUNDERX2T99. */ > diff --git a/gcc/config/aarch64/tuning_models/thunderx3t110.h > b/gcc/config/aarch64/tuning_models/thunderx3t110.h > index cdcbec680945..0d7f29bee209 100644 > --- a/gcc/config/aarch64/tuning_models/thunderx3t110.h > +++ b/gcc/config/aarch64/tuning_models/thunderx3t110.h > @@ -131,7 +131,8 @@ static const struct tune_params thunderx3t110_tunings = > &thunderx3t110_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_THUNDERX3T110. */ > diff --git a/gcc/config/aarch64/tuning_models/thunderxt88.h > b/gcc/config/aarch64/tuning_models/thunderxt88.h > index 44a1eac59a96..d0c39c07c88a 100644 > --- a/gcc/config/aarch64/tuning_models/thunderxt88.h > +++ b/gcc/config/aarch64/tuning_models/thunderxt88.h > @@ -67,7 +67,8 @@ static const struct tune_params thunderxt88_tunings = > &thunderxt88_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_THUNDERXT88. */ > diff --git a/gcc/config/aarch64/tuning_models/tsv110.h > b/gcc/config/aarch64/tuning_models/tsv110.h > index 366f2b3e2323..0800bdc59ced 100644 > --- a/gcc/config/aarch64/tuning_models/tsv110.h > +++ b/gcc/config/aarch64/tuning_models/tsv110.h > @@ -132,7 +132,8 @@ static const struct tune_params tsv110_tunings = > &tsv110_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_TSV110. */ > diff --git a/gcc/config/aarch64/tuning_models/xgene1.h > b/gcc/config/aarch64/tuning_models/xgene1.h > index 0d4aaf80b61a..5fd5e06932c3 100644 > --- a/gcc/config/aarch64/tuning_models/xgene1.h > +++ b/gcc/config/aarch64/tuning_models/xgene1.h > @@ -140,7 +140,8 @@ static const struct tune_params xgene1_tunings = > &xgene1_prefetch_tune, > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ > - nullptr /* dispatch_constraints. */ > + nullptr, /* dispatch_constraints. */ > + 0 /* loop_jam_reduction_factor. */ > }; > > #endif /* GCC_AARCH64_H_XGENE1. */ > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi > index a74e99081d35..9f3215deb942 100644 > --- a/gcc/doc/invoke.texi > +++ b/gcc/doc/invoke.texi > @@ -641,7 +641,8 @@ Objective-C and Objective-C++ Dialects}. > -fivopts -fkeep-inline-functions -fkeep-static-functions > -fkeep-static-consts -flimit-function-alignment -flive-range-shrinkage > -floop-block -floop-interchange -floop-strip-mine > --floop-unroll-and-jam -floop-nest-optimize > +-floop-unroll-and-jam -floop-unroll-and-jam-reductions > +-floop-nest-optimize > -floop-parallelize-all -flra-remat -flto -flto-compression-level=@var{n} > -flto-toplevel-asm-heuristics > -flto-partition=@var{alg} -flto-incremental=@var{path} > @@ -15307,6 +15308,20 @@ nest this unrolls the outer loop by some factor and > fuses the resulting > multiple inner loops. This flag is enabled by default at @option{-O3}. > It is also enabled by @option{-fprofile-use} and @option{-fauto-profile}. > > +@opindex floop-unroll-and-jam-reductions > +@opindex fno-loop-unroll-and-jam-reductions > +@item -floop-unroll-and-jam-reductions > +Apply a reduction-aware variant of unroll and jam to loop nests whose > +inner loop carries a reduction accumulator reset every outer iteration > +(the GEMV / dot-product shape), producing N parallel reduction streams. > +N is taken from @code{TARGET_LOOP_JAM_REDUCTION_FACTOR} (no-op on > +targets that do not opt in) and capped by > +@option{--param=unroll-jam-reduction-streams}. Inner loops whose > +trip count is below @option{--param=unroll-jam-reduction-min-inner-iters} > +are left alone, because the per-stream serial chain is too short to > +profit from parallelism. The outer loop trip count must be a > +compile-time constant. Enabled whenever @option{-floop-unroll-and-jam} is. > + > @opindex ftree-loop-im > @opindex fno-tree-loop-im > @item -ftree-loop-im > diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi > index ca68bdc3e615..be0771e5348a 100644 > --- a/gcc/doc/tm.texi > +++ b/gcc/doc/tm.texi > @@ -12653,6 +12653,17 @@ is required only when the target has special > constraints like maximum > number of memory accesses. > @end deftypefn > > +@deftypefn {Target Hook} unsigned TARGET_LOOP_JAM_REDUCTION_FACTOR > (machine_mode @var{acc_mode}, machine_mode @var{op_mode}) > +Return the preferred number of parallel reduction streams > +@option{-floop-unroll-and-jam-reductions} should produce for an inner > +reduction that accumulates into mode @var{acc_mode} from operands of > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source > +of the widening multiply, and is always strictly narrower than > +@var{acc_mode}: plain reductions with no widening multiply are > +filtered out before the hook is consulted. Returning less than two > +disables the transform for that mode pair. > +@end deftypefn > + > @defmac POWI_MAX_MULTS > If defined, this macro is interpreted as a signed integer C expression > that specifies the maximum number of floating point multiplications > diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in > index 532e95ef216f..027a311f310e 100644 > --- a/gcc/doc/tm.texi.in > +++ b/gcc/doc/tm.texi.in > @@ -8011,6 +8011,8 @@ lists. > > @hook TARGET_LOOP_UNROLL_ADJUST > > +@hook TARGET_LOOP_JAM_REDUCTION_FACTOR > + > @defmac POWI_MAX_MULTS > If defined, this macro is interpreted as a signed integer C expression > that specifies the maximum number of floating point multiplications > diff --git a/gcc/gimple-loop-jam.cc b/gcc/gimple-loop-jam.cc > index 2c4162d17a2a..527f7c6cce35 100644 > --- a/gcc/gimple-loop-jam.cc > +++ b/gcc/gimple-loop-jam.cc > @@ -34,12 +34,14 @@ along with GCC; see the file COPYING3. If not see > #include "cfgloop.h" > #include "tree-scalar-evolution.h" > #include "gimple-iterator.h" > +#include "cfganal.h" > #include "cfghooks.h" > #include "tree-data-ref.h" > #include "tree-ssa-loop-ivopts.h" > #include "tree-vectorizer.h" > #include "tree-ssa-sccvn.h" > #include "tree-cfgcleanup.h" > +#include "tree-into-ssa.h" > > /* Unroll and Jam transformation > > @@ -376,6 +378,684 @@ fuse_loops (class loop *loop) > } > } > > +/* Reduction-jam: a parallel-reduction extension of unroll-and-jam. > + > + The transformation above accepts only inner PHIs that are simple > + inductions. A reduction PHI cannot be fused the same way: merging > + the streams would chain them into a single serial accumulation, > + multiplying the serial dependency height instead of cutting it. > + This handles the GEMV / NNUE shape > + > + for (i = 0; i < M; i++) { > + s = bias[i]; > + for (j = 0; j < K; j++) s += w[i][j] * x[j]; > + out[i] = s; > + } > + > + where s starts afresh every outer iteration. Unrolling the outer > + loop by N leaves N sibling inner loops, each with its own reduction > + PHI and its own init (bias[i], bias[i+1], ...). Instead of merging > + them we keep all N PHIs in the fused header and hoist each sibling's > + init ahead of the body; the post-reduction stores stay on the > + sibling-exit chain, coalesced afterward. The vectorizer then emits > + N independent DOT_PROD / FMA chains, so the serial dependency height > + across the reduction drops by a factor of N. > + > + The strip factor N is the smallest of the target's > + TARGET_LOOP_JAM_REDUCTION_FACTOR, param_unroll_jam_reduction_streams > + and the data-dep safety bound, narrowed to a divisor of the outer > + trip count. We ignore adjust_unroll_factor's profit signal on > + purpose: it scores the memory refs that fusion removes, which is > + not what this transform is after. Targets that have not measured a > + win return 0 from the hook and we skip the loop. See params.opt > + for the tunables. */ > + > +/* Return the single non-virtual, non-simple-IV PHI in LOOP's header > + (the reduction accumulator), or NULL if there is none or more than > + one. */ > + > +static gphi * > +inner_reduction_phi (class loop *loop) > +{ > + gphi *result = NULL; > + for (gphi_iterator psi = gsi_start_phis (loop->header); > + !gsi_end_p (psi); gsi_next (&psi)) > + { > + gphi *phi = psi.phi (); > + tree op = gimple_phi_result (phi); > + if (virtual_operand_p (op)) > + continue; > + affine_iv iv; > + if (simple_iv (loop, loop, op, &iv, true)) > + continue; > + if (result) > + return NULL; > + result = phi; > + } > + return result; > +} > + > +/* Walk the SSA def chain from RED_PHI's latch update and return the > + narrowest scalar-int mode feeding a multiply inside INNER, or > + ACC_MODE if nothing narrower turns up. The walk follows only > + PLUS / MINUS / CONVERT chains so that a multiply buried in an array > + index (the "w * j" in "s += a[w * j]") is not mistaken for the MAC > + source. */ > + > +static machine_mode > +narrowest_reduc_input_mode (gphi *red_phi, class loop *inner, > + machine_mode acc_mode) > +{ > + tree update = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner)); > + if (TREE_CODE (update) != SSA_NAME) > + return acc_mode; > + > + machine_mode narrow = VOIDmode; > + auto_vec<tree, 8> worklist; > + hash_set<tree> seen; > + worklist.safe_push (update); > + > + while (!worklist.is_empty ()) > + { > + tree op = worklist.pop (); > + if (seen.add (op)) > + continue; > + gimple *def = SSA_NAME_DEF_STMT (op); > + if (gimple_code (def) == GIMPLE_PHI > + || !gimple_bb (def) > + || !flow_bb_inside_loop_p (inner, gimple_bb (def)) > + || !is_gimple_assign (def)) > + continue; > + > + /* Stay in the reduction value chain. Loads and pointer > + arithmetic feed address computation, not the accumulator. */ > + if (gimple_assign_load_p (def) > + || POINTER_TYPE_P (TREE_TYPE (gimple_assign_lhs (def)))) > + continue; > + > + use_operand_p use_p; > + ssa_op_iter oi; > + tree_code code = gimple_assign_rhs_code (def); > + if (code == MULT_EXPR || code == WIDEN_MULT_EXPR) > + { > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE) > + { > + tree mop = USE_FROM_PTR (use_p); > + while (TREE_CODE (mop) == SSA_NAME) > + { > + gimple *mdef = SSA_NAME_DEF_STMT (mop); > + if (!is_gimple_assign (mdef) > + || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (mdef))) > + break; > + mop = gimple_assign_rhs1 (mdef); > + } > + machine_mode m = TYPE_MODE (TREE_TYPE (mop)); > + if (SCALAR_INT_MODE_P (m) > + && (narrow == VOIDmode > + || known_lt (GET_MODE_PRECISION (m), > + GET_MODE_PRECISION (narrow)))) > + narrow = m; > + } > + continue; > + } > + > + /* Only scalar-reduction operations continue the walk. */ > + if (code != PLUS_EXPR > + && code != MINUS_EXPR > + && !CONVERT_EXPR_CODE_P (code)) > + continue; > + > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE) > + worklist.safe_push (USE_FROM_PTR (use_p)); > + } > + > + if (narrow == VOIDmode > + || known_ge (GET_MODE_PRECISION (narrow), > + GET_MODE_PRECISION (acc_mode))) > + return acc_mode; > + return narrow; > +} > + > +/* Move *FROM to BB, immediately before BB's terminator branch (or to > + the end of BB if there is none). */ > + > +static void > +gsi_move_to_bb_end_before_terminator (gimple_stmt_iterator *from, > + basic_block bb) > +{ > + gimple_stmt_iterator to = gsi_last_bb (bb); > + if (!gsi_end_p (to) && is_ctrl_stmt (gsi_stmt (to))) > + gsi_move_before (from, &to); > + else > + gsi_move_after (from, &to); > +} > + > +/* Propagate the single argument of the PHI at PSI into its uses and > + remove the PHI, leaving PSI at the next one. */ > + > +static void > +collapse_single_arg_phi (gphi_iterator *psi) > +{ > + gphi *phi = psi->phi (); > + gcc_checking_assert (gimple_phi_num_args (phi) == 1); > + tree result = gimple_phi_result (phi); > + if (!has_zero_uses (result)) > + replace_uses_by (result, gimple_phi_arg_def (phi, 0)); > + remove_phi_node (psi, true); > +} > + > +/* True if LOOP is a widening MAC reduction the vectorizer can turn > + into a DOT_PROD_EXPR: a single-BB body with one exit, loads only, a > + constant trip count of at least MIN_INNER_ITERS, one non-IV > + reduction PHI alongside a unit-step counter IV, and a widening > + multiply in the update chain. If OUTER is given, every IV's base > + and step must be invariant in it. On success *OUT_RED_PHI, > + *OUT_ACC_MODE and *OUT_OP_MODE receive the reduction PHI, the > + accumulator mode and the narrowest operand mode. */ > + > +static bool > +loop_is_dot_product_candidate_p (class loop *loop, > + unsigned HOST_WIDE_INT min_inner_iters, > + class loop *outer, > + gphi **out_red_phi, > + machine_mode *out_acc_mode, > + machine_mode *out_op_mode) > +{ > + if (loop->num_nodes != 2 || !empty_block_p (loop->latch)) > + return false; > + edge exit = single_exit (loop); > + if (!exit) > + return false; > + > + /* Header is the body; loads only. */ > + bool has_vuse = false; > + for (gimple_stmt_iterator gsi = gsi_start_bb (loop->header); > + !gsi_end_p (gsi); gsi_next (&gsi)) > + { > + gimple *s = gsi_stmt (gsi); > + if (gimple_vdef (s)) > + return false; > + if (gimple_vuse (s)) > + has_vuse = true; > + } > + if (!has_vuse) > + return false; > + > + /* Single reduction PHI with init defined outside the loop. */ > + gphi *red_phi = inner_reduction_phi (loop); > + if (!red_phi) > + return false; > + tree init = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_preheader_edge (loop)); > + if (!expr_invariant_in_loop_p (loop, init)) > + return false; > + > + /* Trip count is a known integer >= MIN_INNER_ITERS. niter.niter is > + the latch count (trip - 1); reject HOST_WIDE_INT_M1U separately > + so the +1 conversion below does not wrap. */ > + tree_niter_desc niter; > + if (!number_of_iterations_exit (loop, exit, &niter, false, true) > + || niter.cmp == ERROR_MARK > + || !integer_zerop (niter.may_be_zero) > + || !tree_fits_uhwi_p (niter.niter)) > + return false; > + > + unsigned HOST_WIDE_INT latch_count = tree_to_uhwi (niter.niter); > + if (latch_count == HOST_WIDE_INT_M1U > + || latch_count + 1 < min_inner_iters) > + return false; > + > + /* Plus a unit-step counter IV in the same header. When OUTER is > + given, every IV's step and base must also be OUTER-invariant. */ > + bool found_unit_iv = false; > + for (gphi_iterator psi = gsi_start_phis (loop->header); > + !gsi_end_p (psi); gsi_next (&psi)) > + { > + gphi *phi = psi.phi (); > + tree op = gimple_phi_result (phi); > + if (virtual_operand_p (op)) > + continue; > + affine_iv iv; > + if (!simple_iv (loop, loop, op, &iv, true)) > + continue; > + if (outer > + && (!expr_invariant_in_loop_p (outer, iv.step) > + || !expr_invariant_in_loop_p (outer, iv.base))) > + return false; > + if (iv.step && integer_onep (iv.step)) > + found_unit_iv = true; > + } > + if (!found_unit_iv) > + return false; > + > + /* Widening multiply somewhere in the reduction update chain. */ > + machine_mode acc_mode = TYPE_MODE (TREE_TYPE (gimple_phi_result > (red_phi))); > + machine_mode op_mode = narrowest_reduc_input_mode (red_phi, loop, > acc_mode); > + if (op_mode == acc_mode) > + return false; > + > + if (out_red_phi) > + *out_red_phi = red_phi; > + if (out_acc_mode) > + *out_acc_mode = acc_mode; > + if (out_op_mode) > + *out_op_mode = op_mode; > + return true; > +} > + > +/* Detect whether the OUTER / INNER nest has the parallel-reduction > + shape that unroll_jam_reduction_fuse_sibling can transform. The > + rejection messages below name each requirement. The only outer-body > + VDEF must be the post-reduction store, so that after strip-mining > + every replicated store sits downstream of every sibling's preheader. > + And the outer trip count must be a multiple of N, which keeps us off > + tree_unroll_loop's remainder path. */ > + > +static bool > +unroll_jam_reduction_possible_p (class loop *outer, class loop *inner, > + unsigned HOST_WIDE_INT *out_outer_trip, > + unsigned *out_target_factor) > +{ > + auto reject = [&] (const char *reason) -> bool > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, "reduction-jam: rejecting outer loop %d: %s\n", > + outer->num, reason); > + return false; > + }; > + > + /* Loop-tree relationships. The driver iterates LI_ONLY_INNERMOST, > + so INNER has no children; check only that OUTER has a single > + inner sibling (INNER) and that INNER's parent is OUTER. */ > + if (!inner || inner != outer->inner || inner->next > + || loop_outer (inner) != outer) > + return reject ("not a perfect 2-level nest"); > + > + if (!dominated_by_p (CDI_DOMINATORS, outer->latch, inner->header)) > + return reject ("inner has unsuitable dominance"); > + > + /* Outer loop must carry only simple IVs. */ > + for (gphi_iterator psi = gsi_start_phis (outer->header); > + !gsi_end_p (psi); gsi_next (&psi)) > + { > + gphi *phi = psi.phi (); > + tree op = gimple_phi_result (phi); > + if (virtual_operand_p (op)) > + continue; > + affine_iv iv; > + if (!simple_iv (outer, outer, op, &iv, true)) > + return reject ("outer carries a non-simple-IV PHI"); > + } > + > + /* Loop-local dot-product checks; on success red_phi and the modes > + feed the target-hook gate below. */ > + gphi *red_phi = NULL; > + machine_mode acc_mode = VOIDmode; > + machine_mode op_mode = VOIDmode; > + if (!loop_is_dot_product_candidate_p > + (inner, > + (unsigned HOST_WIDE_INT) param_unroll_jam_reduction_min_inner_iters, > + outer, &red_phi, &acc_mode, &op_mode)) > + return reject ("inner not a dot-product candidate"); > + > + /* Opt-in gate: ask the target whether the (accumulator, operand) > + mode pair benefits from N parallel reduction streams. Targets > + that have not measured a benefit return 0, in which case we skip > + the transform regardless of -floop-unroll-and-jam-reductions. > + A return of 1 is also opt-out (single-stream = no parallelism). */ > + unsigned target_factor > + = targetm.loop_jam_reduction_factor (acc_mode, op_mode); > + if (target_factor < 2) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, > + "reduction-jam: target opted out for acc=%s op=%s\n", > + GET_MODE_NAME (acc_mode), GET_MODE_NAME (op_mode)); > + return false; > + } > + *out_target_factor = target_factor; > + > + /* The fuser drains stmts from inner->header alone, so a non-empty > + latch would be silently mishandled; loop_is_dot_product_candidate_p > + already rejects one. */ > + gcc_checking_assert (empty_block_p (inner->latch)); > + > + /* Require the chain from red_phi back to itself via the latch arg to > + be a PLUS_EXPR reduction. This limits the transform to plain > + summation; the N streams are independent, so each is still summed > + in its original order and no reassociation takes place. */ > + tree latch_arg = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner)); > + if (!check_reduction_path (dump_user_location_t (), inner, red_phi, > + latch_arg, PLUS_EXPR)) > + return reject ("inner reduction chain not associative-PLUS"); > + > + /* loop_is_dot_product_candidate_p confirmed inner has a single exit. */ > + edge inner_exit = single_exit (inner); > + > + /* The outer body may carry at most one VDEF, only in the inner > + exit's destination BB (the post-reduction store). Also reject any > + non-VDEF side effect (volatile load, asm volatile, side-effecting > + builtin), matching bb_prevents_fusion_p on the IV-only path. > + > + The store BB must be dominated by inner->header so that after > + tree_unroll_loop strip-mines outer by N every replicated store > + sits downstream of its corresponding inner copy, and therefore > + downstream of every sibling's preheader. The fuser relies on > + this when rewiring VUSEs. */ > + basic_block store_bb = inner_exit->dest; > + if (!dominated_by_p (CDI_DOMINATORS, store_bb, inner->header)) > + return reject ("outer-body store not downstream of inner header"); > + > + /* The outer body must be straight-line apart from the exit test. A > + control-flow merge there leaves a PHI that would not dominate the > + fused preheader after strip-mining, and the migrator leaves body > + PHIs in place, so the SSA fails verification. */ > + edge outer_exit = single_exit (outer); > + if (!outer_exit) > + return reject ("outer loop has no single exit"); > + > + basic_block *bbs = get_loop_body (outer); > + const char *fail = NULL; > + for (unsigned i = 0; i < outer->num_nodes && !fail; ++i) > + { > + basic_block bb = bbs[i]; > + if (flow_bb_inside_loop_p (inner, bb)) > + continue; > + if (EDGE_COUNT (bb->succs) > 1 && bb != outer_exit->src) > + { > + fail = "outer body has control flow"; > + break; > + } > + unsigned vdef_count = 0; > + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); > + !gsi_end_p (gsi); gsi_next (&gsi)) > + { > + gimple *s = gsi_stmt (gsi); > + if (gimple_has_side_effects (s)) > + { > + fail = "outer body has side effects"; > + break; > + } > + if (gimple_vdef (s)) > + vdef_count++; > + } > + if (!fail && (bb == store_bb ? vdef_count > 1 : vdef_count > 0)) > + fail = "outer body has extra stores"; > + } > + free (bbs); > + if (fail) > + return reject (fail); > + > + /* Outer trip count must be a known constant (N is later narrowed to > + a divisor of it; the remainder codepath of tree_unroll_loop is > + correct in principle but currently untested for this pattern). > + niter from number_of_iterations_exit is the latch execution count, > + so the trip count is niter + 1. */ > + tree_niter_desc outer_niter; > + if (!number_of_iterations_exit (outer, outer_exit, &outer_niter, > + false, true) > + || outer_niter.cmp == ERROR_MARK > + || !integer_zerop (outer_niter.may_be_zero) > + || !tree_fits_uhwi_p (outer_niter.niter)) > + return reject ("outer trip count unknown"); > + > + unsigned HOST_WIDE_INT outer_latch_count = tree_to_uhwi > (outer_niter.niter); > + if (outer_latch_count == HOST_WIDE_INT_M1U) > + return reject ("outer trip count overflows HOST_WIDE_INT"); > + > + *out_outer_trip = outer_latch_count + 1; > + > + return true; > +} > + > +/* Move the transitive SSA-use closure of SRC_INIT, and of every > + operand SRC's header body references, into DST_PREHEADER so the > + defs stay live once SRC's body moves. Defs that already dominate > + DST_PREHEADER, and defs inside SRC, are left to the body move. > + > + RPO is a reverse-postorder over CFUN's BBs, computed once by the > + caller and shared across the nest's siblings; walking it keeps > + def-use order among the migrated stmts intact. DST_LIVE_VUSE is > + the vop live at DST_PREHEADER, and migrated stmts are rewired to it > + so none of them reference a vop defined downstream. */ > + > +static void > +migrate_loop_invariant_defs_to_preheader (class loop *src, tree src_init, > + basic_block dst_preheader, > + const auto_vec<int> &rpo, > + tree dst_live_vuse) > +{ > + hash_set<gimple *> needed; > + auto_vec<gimple *, 16> worklist; > + > + auto consider = [&] (tree op) > + { > + if (TREE_CODE (op) != SSA_NAME) > + return; > + gimple *def = SSA_NAME_DEF_STMT (op); > + if (!def || is_a <gphi *> (def)) > + return; > + basic_block def_bb = gimple_bb (def); > + if (!def_bb > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, def_bb) > + || flow_bb_inside_loop_p (src, def_bb)) > + return; > + if (needed.add (def)) > + return; > + worklist.safe_push (def); > + }; > + > + auto consider_uses = [&] (gimple *s) > + { > + use_operand_p use_p; > + ssa_op_iter oi; > + FOR_EACH_SSA_USE_OPERAND (use_p, s, oi, SSA_OP_USE) > + consider (USE_FROM_PTR (use_p)); > + }; > + > + consider (src_init); > + for (gimple_stmt_iterator si = gsi_start_bb (src->header); > + !gsi_end_p (si); gsi_next (&si)) > + { > + gimple *s = gsi_stmt (si); > + if (!is_a <gphi *> (s) && !is_ctrl_stmt (s)) > + consider_uses (s); > + } > + while (!worklist.is_empty ()) > + consider_uses (worklist.pop ()); > + > + if (needed.is_empty ()) > + return; > + > + /* Walk BBs in reverse-postorder so that, for any def-use edge among > + the migrated stmts, the def is moved into DST_PREHEADER ahead of > + the use. */ > + int n = rpo.length (); > + for (int i = 0; i < n; ++i) > + { > + basic_block bb = BASIC_BLOCK_FOR_FN (cfun, rpo[i]); > + if (bb == dst_preheader > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb)) > + continue; > + for (gimple_stmt_iterator si = gsi_start_bb (bb); > + !gsi_end_p (si);) > + { > + gimple *stmt = gsi_stmt (si); > + if (!needed.contains (stmt)) > + { > + gsi_next (&si); > + continue; > + } > + if (dst_live_vuse && gimple_vuse (stmt)) > + { > + gimple_set_vuse (stmt, dst_live_vuse); > + update_stmt (stmt); > + } > + gsi_move_to_bb_end_before_terminator (&si, dst_preheader); > + } > + } > +} > + > +/* Pair SRC's non-reduction header PHIs with DST's by position, > + recording each pair in PHI_RENAME. Both are inner siblings of the > + same strip-mined outer loop, so their header PHI lists start with > + the duplicated originals in the same order. DST may have picked up > + extra reduction PHIs at the tail from earlier sibling fuses, so > + walk SRC to completion against DST's prefix. SRC_REDUC_PHI is > + renamed separately and virtual PHIs are skipped. */ > + > +static void > +match_sibling_header_phis (class loop *src, class loop *dst, > + gphi *src_reduc_phi, > + hash_map<tree, tree> &phi_rename) > +{ > + gphi_iterator spsi = gsi_start_phis (src->header); > + gphi_iterator dpsi = gsi_start_phis (dst->header); > + for (; !gsi_end_p (spsi); gsi_next (&spsi), gsi_next (&dpsi)) > + { > + gcc_checking_assert (!gsi_end_p (dpsi)); > + gphi *src_phi = spsi.phi (); > + gphi *dst_phi = dpsi.phi (); > + tree src_res = gimple_phi_result (src_phi); > + tree dst_res = gimple_phi_result (dst_phi); > + gcc_checking_assert (virtual_operand_p (src_res) > + == virtual_operand_p (dst_res)); > + if (src_phi != src_reduc_phi && !virtual_operand_p (src_res)) > + phi_rename.put (src_res, dst_res); > + } > +} > + > +/* Fuse the SRC sibling inner loop into DST, keeping SRC's reduction > + PHI as a parallel chain in DST's header. Both are inner siblings > + of an outer loop that passed unroll_jam_reduction_possible_p and > + was then strip-mined by tree_unroll_loop. > + > + The caller has already flattened LC-SSA across the nest (Phase 1a). > + Here SRC's reduction init and body operands move up to DST's > + preheader (Phase 1b), a fresh reduction PHI for SRC's stream goes > + into DST's header (Phase 2), and SRC's body follows with its PHI > + uses renamed (Phase 3). The caller then bypasses the drained SRC > + (Phase 4); its cleanup_tree_cfg folds the leftover chain of exit > + BBs holding the post-reduction stores, and > + rewrite_into_loop_closed_ssa puts LC-SSA back. */ > + > +static void > +unroll_jam_reduction_fuse_sibling (class loop *src, class loop *dst, > + const auto_vec<int> &rpo) > +{ > + gphi *src_reduc_phi = inner_reduction_phi (src); > + tree src_reduc_res = gimple_phi_result (src_reduc_phi); > + tree src_init = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi, > + loop_preheader_edge (src)); > + tree src_update = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi, > + loop_latch_edge (src)); > + > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, "reduction-jam: fusing src loop %d into dst %d\n", > + src->num, dst->num); > + > + basic_block dst_preheader = loop_preheader_edge (dst)->src; > + > + /* The migrated stmts must reference the vop live at DST's preheader. > + The matcher's only outer-body VDEF is the post-reduction store, > + which sits below DST after strip-mining, so nothing redefines the > + vop before DST: it is outer->header's virtual PHI result. A > + load-only outer has no such PHI; its loads already share one > + loop-invariant vop, so leave their VUSE alone (NULL_TREE). */ > + gphi *outer_vphi = get_virtual_phi (loop_outer (dst)->header); > + tree dst_live_vuse = outer_vphi ? gimple_phi_result (outer_vphi) : > NULL_TREE; > + > + /* Phase 1b. */ > + migrate_loop_invariant_defs_to_preheader (src, src_init, dst_preheader, > + rpo, dst_live_vuse); > + > + /* Phase 2: a new reduction PHI in DST's header for SRC's stream. */ > + tree new_name = make_ssa_name (TREE_TYPE (src_reduc_res)); > + gphi *new_phi = create_phi_node (new_name, dst->header); > + add_phi_arg (new_phi, src_init, loop_preheader_edge (dst), > + UNKNOWN_LOCATION); > + add_phi_arg (new_phi, src_update, loop_latch_edge (dst), > + UNKNOWN_LOCATION); > + > + /* Phase 3: rewrite uses, then move SRC's body into DST's header. */ > + hash_map<tree, tree> phi_rename; > + phi_rename.put (src_reduc_res, new_name); > + match_sibling_header_phis (src, dst, src_reduc_phi, phi_rename); > + > + auto_vec<gimple *, 32> src_body; > + for (gimple_stmt_iterator si = gsi_start_bb (src->header); > + !gsi_end_p (si); gsi_next (&si)) > + { > + gimple *s = gsi_stmt (si); > + if (is_a <gphi *> (s) || is_ctrl_stmt (s)) > + continue; > + src_body.safe_push (s); > + } > + for (gimple *stmt : src_body) > + { > + use_operand_p use_p; > + ssa_op_iter oi; > + FOR_EACH_SSA_USE_OPERAND (use_p, stmt, oi, SSA_OP_USE) > + { > + tree *map = phi_rename.get (USE_FROM_PTR (use_p)); > + if (map) > + SET_USE (use_p, *map); > + } > + if (dst_live_vuse && gimple_vuse (stmt)) > + { > + gimple_set_vuse (stmt, dst_live_vuse); > + update_stmt (stmt); > + } > + gimple_stmt_iterator from = gsi_for_stmt (stmt); > + gsi_move_to_bb_end_before_terminator (&from, dst->header); > + } > + > + /* The body walk above only covered uses of src_reduc_res inside > + src->header; catch any remaining uses (LC-SSA closures reached > + by the post-reduction store) before the PHI is removed. */ > + replace_uses_by (src_reduc_res, new_name); > + > + gphi_iterator psi_src_reduc = gsi_for_phi (src_reduc_phi); > + remove_phi_node (&psi_src_reduc, false); > +} > + > +/* Phase 4: redirect SRC's preheader edge straight past the now-empty > + SRC loop so cleanup_tree_cfg can remove it. SRC must have been > + drained by unroll_jam_reduction_fuse_sibling first. */ > + > +static void > +unroll_jam_reduction_bypass_sibling (class loop *src) > +{ > + edge src_preheader_e = loop_preheader_edge (src); > + edge src_exit_e = single_exit (src); > + if (!src_exit_e) > + return; > + basic_block src_exit_dest = src_exit_e->dest; > + > + /* src_exit_dest is a single-pred BB after strip-mining (the next > + sibling's preheader, or outer's latch for the last sibling), so > + its LC-SSA PHIs are single-arg. Flatten them before the redirect > + so no arg dangles once SRC's BBs are deleted: the driver rebuilds > + scalar LC-SSA afterward but not the virtual side, so a stale vop > + arg would otherwise survive. */ > + for (gphi_iterator psi = gsi_start_phis (src_exit_dest); > + !gsi_end_p (psi);) > + collapse_single_arg_phi (&psi); > + > + edge new_edge = redirect_edge_and_branch (src_preheader_e, src_exit_dest); > + gcc_assert (new_edge); > + > + /* Delete SRC's now-unreachable BBs and the empty loop entry, > + mirroring fuse_loops's delete_basic_block + delete_loop pattern > + on the IV-only path. Without this the loop tree carries stale > + entries until the driver's cleanup_tree_cfg picks them up. */ > + basic_block *bbs = get_loop_body (src); > + unsigned n = src->num_nodes; > + for (unsigned i = 0; i < n; i++) > + delete_basic_block (bbs[i]); > + free (bbs); > + delete_loop (src); > +} > + > /* Return true if any of the access functions for dataref A > isn't invariant with respect to loop LOOP_NEST. */ > static bool > @@ -497,6 +1177,221 @@ adjust_unroll_factor (class loop *inner, struct > data_dependence_relation *ddr, > return ret; > } > > +/* Walk OUTER's data dependences, running each through > + adjust_unroll_factor. *UNROLL_FACTOR receives the safety upper > + bound (UINT_MAX for no constraint, 0 for cannot unroll), > + *PROFIT_UNROLL the largest distance+1 yielding a removable ref > + under the (a,0) pattern, *REMOVED the count of removable refs and > + *NUM_DATAREFS the total. Returns false if the dependences cannot > + be computed. > + > + When *UNROLL_FACTOR comes back 0 the other out-params are only > + partially populated and must not be read for profitability. */ > + > +static bool > +compute_jam_dependence_factors (class loop *outer, class loop *inner, > + unsigned *unroll_factor, > + unsigned *profit_unroll, > + unsigned *removed, > + unsigned *num_datarefs) > +{ > + vec<data_reference_p> datarefs = vNULL; > + vec<ddr_p> dependences = vNULL; > + auto_vec<loop_p, 3> loop_nest; > + > + if (!compute_data_dependences_for_loop (outer, true, &loop_nest, > + &datarefs, &dependences)) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, "Cannot analyze data dependencies\n"); > + free_data_refs (datarefs); > + free_dependence_relations (dependences); > + return false; > + } > + > + *num_datarefs = datarefs.length (); > + *unroll_factor = (unsigned) -1; > + *profit_unroll = 1; > + *removed = 0; > + > + if (*num_datarefs) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + dump_data_dependence_relations (dump_file, dependences); > + > + unsigned i; > + ddr_p ddr; > + FOR_EACH_VEC_ELT (dependences, i, ddr) > + { > + if (DDR_ARE_DEPENDENT (ddr) == chrec_known) > + continue; > + data_reference *dra = DDR_A (ddr); > + data_reference *drb = DDR_B (ddr); > + /* Self-dependence: WAW with non-affine access could reorder > + writes to the same location. */ > + if (dra == drb) > + { > + if (DR_IS_WRITE (dra) > + && !DR_ACCESS_FNS (dra).is_empty () > + && DDR_ARE_DEPENDENT (ddr) == chrec_dont_know) > + { > + *unroll_factor = 0; > + break; > + } > + continue; > + } > + /* Couldn't determine the distance vector: harmless for two > + reads, but a write means we cannot prove safety. */ > + if (!adjust_unroll_factor (inner, ddr, unroll_factor, > + profit_unroll, removed)) > + if (DR_IS_WRITE (dra) || DR_IS_WRITE (drb)) > + { > + *unroll_factor = 0; > + break; > + } > + } > + } > + > + free_dependence_relations (dependences); > + free_data_refs (datarefs); > + return true; > +} > + > +/* Pick a strip factor for the reduction-jam path on OUTER / INNER. > + adjust_unroll_factor gives the data-dep safety bound, TARGET_FACTOR > + the target's preference and param_unroll_jam_reduction_streams > + the cap. We do not use adjust_unroll_factor's profit model here: > + it would cap N at 1 for GEMV, because the input-load DDR has > + dist_v=(0,0). The result is narrowed to a divisor of OUTER_TRIP, > + which keeps us off tree_unroll_loop's remainder path, and validated > + via can_unroll_loop_p. *OUT_FACTOR receives the chosen factor and > + *OUT_DESC the niter. */ > + > +static bool > +select_reduction_jam_factor (class loop *outer, class loop *inner, > + unsigned HOST_WIDE_INT outer_trip, > + unsigned target_factor, > + unsigned *out_factor, > + tree_niter_desc *out_desc) > +{ > + /* Only the safety bound matters here; the ref-removal outputs go > + unused. */ > + unsigned unroll_factor, profit_unroll, removed, num_datarefs; > + if (!compute_jam_dependence_factors (outer, inner, &unroll_factor, > + &profit_unroll, &removed, > + &num_datarefs)) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, > + "reduction-jam: data-dep analysis failed\n"); > + return false; > + } > + if (unroll_factor < 2) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, > + "reduction-jam: data-dep safety bound below 2\n"); > + return false; > + } > + > + unsigned N = MIN (target_factor, > + (unsigned) param_unroll_jam_reduction_streams); > + if (N > unroll_factor) > + N = unroll_factor; > + unsigned initial_N = N; > + while (N >= 2 && (outer_trip < N || outer_trip % N != 0)) > + --N; > + if (N < 2) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, > + "reduction-jam: outer trip " HOST_WIDE_INT_PRINT_UNSIGNED > + " has no divisor in [2, %u]\n", outer_trip, initial_N); > + return false; > + } > + > + if (!can_unroll_loop_p (outer, N, out_desc)) > + { > + if (dump_file && (dump_flags & TDF_DETAILS)) > + fprintf (dump_file, > + "reduction-jam: cannot unroll outer by %u\n", N); > + return false; > + } > + > + *out_factor = N; > + return true; > +} > + > +/* Strip-mine OUTER by N and fuse the resulting N inner siblings into > + one body that carries N parallel reduction PHIs. Caller has > + already validated OUTER and chosen N + DESC via > + unroll_jam_reduction_possible_p / select_reduction_jam_factor. */ > + > +static void > +do_unroll_jam_reductions (class loop *outer, unsigned N, > + tree_niter_desc *desc) > +{ > + initialize_original_copy_tables (); > + tree_unroll_loop (outer, N, desc); > + free_original_copy_tables (); > + > + /* dst->next chains siblings in CFG order (per tree_unroll_loop); > + the fuser relies on that when migrating SSA uses. */ > + class loop *dst = outer->inner; > + auto_vec<class loop *, 4> siblings; > + for (class loop *src = dst->next; src; src = src->next) > + siblings.safe_push (src); > + > + /* RPO of CFUN, shared across all siblings: the fuser only moves > + stmts and removes single-arg PHIs, neither of which alters BB > + structure, so one walk suffices for the whole nest. */ > + auto_vec<int> rpo; > + rpo.safe_grow (last_basic_block_for_fn (cfun)); > + rpo.truncate (pre_and_rev_post_order_compute_fn (cfun, NULL, rpo.address > (), > + false)); > + > + /* Phase 1a (nest-wide): collapse single-arg LC-SSA PHIs that the > + per-sibling migrate-walk in Phase 1b would otherwise have to > + chase through (it skips PHIs). Sweep the BBs that do not dominate > + DST's preheader, i.e. the nest and everything downstream where the > + sibling-exit closures live. One sweep suffices: later fuses only > + add two-arg reduction PHIs and make no CFG change that could > + reintroduce single-arg closures. */ > + basic_block dst_preheader = loop_preheader_edge (dst)->src; > + calculate_dominance_info (CDI_DOMINATORS); > + basic_block bb; > + FOR_EACH_BB_FN (bb, cfun) > + { > + if (bb == dst_preheader > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb)) > + continue; > + for (gphi_iterator psi = gsi_start_phis (bb); !gsi_end_p (psi);) > + { > + gphi *phi = psi.phi (); > + if (gimple_phi_num_args (phi) == 1 > + && !virtual_operand_p (gimple_phi_result (phi))) > + collapse_single_arg_phi (&psi); > + else > + gsi_next (&psi); > + } > + } > + > + for (class loop *src : siblings) > + unroll_jam_reduction_fuse_sibling (src, dst, rpo); > + > + /* Bypass the drained siblings only after every fuse: the CFG edits > + invalidate dominance / SCEV state the fuse loop above relies on. */ > + for (class loop *src : siblings) > + unroll_jam_reduction_bypass_sibling (src); > + > + if (dump_enabled_p ()) > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS | TDF_DETAILS, > + find_loop_location (outer), > + "applying reduction unroll-and-jam with %u streams\n", > + siblings.length () + 1); > +} > + > + > /* Main entry point for the unroll-and-jam transformation > described above. */ > > @@ -516,8 +1411,36 @@ tree_loop_unroll_and_jam (void) > || optimize_loop_nest_for_size_p (outer)) > continue; > > - if (!unroll_jam_possible_p (outer, loop)) > - continue; > + if (!flag_unroll_jam || !unroll_jam_possible_p (outer, loop)) > + { > + /* The IV-only path skips reduction-shaped nests (reduction PHIs, > + or outer-body stores consuming the reduction result). Try the > + reduction-jam extension on those. */ > + unsigned HOST_WIDE_INT outer_trip; > + unsigned target_factor, N; > + tree_niter_desc red_desc; > + if (flag_unroll_jam_reductions > + && unroll_jam_reduction_possible_p (outer, loop, > + &outer_trip, > + &target_factor) > + && select_reduction_jam_factor (outer, loop, outer_trip, > + target_factor, > + &N, &red_desc)) > + { > + do_unroll_jam_reductions (outer, N, &red_desc); > + free_dominance_info (CDI_DOMINATORS); > + cleanup_tree_cfg (); > + /* The fuser maintains virtual SSA from the matcher's > + invariants, but moving SRC's body into DST leaves scalar > + SSA names defined in DST yet used outside it (e.g. new_name's > + LC-SSA closures). Rebuild scalar LC-SSA so the next nest's > + tree_unroll_loop is well-formed, and verify the result. */ > + rewrite_into_loop_closed_ssa (NULL, 0); > + checking_verify_ssa (true, true); > + scev_reset (); > + } > + continue; > + } > > vec<data_reference_p> datarefs = vNULL; > vec<ddr_p> dependences = vNULL; > @@ -674,7 +1597,10 @@ public: > {} > > /* opt_pass methods: */ > - bool gate (function *) final override { return flag_unroll_jam != 0; } > + bool gate (function *) final override > + { > + return flag_unroll_jam != 0 || flag_unroll_jam_reductions; > + } > unsigned int execute (function *) final override; > > }; > diff --git a/gcc/hooks.cc b/gcc/hooks.cc > index 2b47fd3e50f8..976bae3c89a9 100644 > --- a/gcc/hooks.cc > +++ b/gcc/hooks.cc > @@ -424,6 +424,13 @@ hook_uint_mode_0 (machine_mode) > return 0; > } > > +/* Generic hook that takes two machine modes and returns an unsigned int 0. > */ > +unsigned int > +hook_uint_mode_mode_0 (machine_mode, machine_mode) > +{ > + return 0; > +} > + > /* Generic hook that takes no arguments and returns a NULL const string. */ > const char * > hook_constcharptr_void_null (void) > diff --git a/gcc/hooks.h b/gcc/hooks.h > index 1ad4abb5500c..0ec0b32fa1a5 100644 > --- a/gcc/hooks.h > +++ b/gcc/hooks.h > @@ -119,6 +119,7 @@ extern tree hook_tree_treeptr_tree_tree_int_boolptr_null > (tree *, tree, tree, in > > extern unsigned hook_uint_void_0 (void); > extern unsigned int hook_uint_mode_0 (machine_mode); > +extern unsigned int hook_uint_mode_mode_0 (machine_mode, machine_mode); > > extern bool default_can_output_mi_thunk_no_vcall (const_tree, HOST_WIDE_INT, > HOST_WIDE_INT, const_tree); > diff --git a/gcc/params.opt b/gcc/params.opt > index 0c6a3e514574..e99c4fc4173f 100644 > --- a/gcc/params.opt > +++ b/gcc/params.opt > @@ -1263,6 +1263,14 @@ Maximum unroll factor for the unroll-and-jam > transformation. > Common Joined UInteger Var(param_unroll_jam_min_percent) Init(1) > IntegerRange(0, 100) Param Optimization > Minimum percentage of memrefs that must go away for unroll-and-jam to be > considered profitable. > > +-param=unroll-jam-reduction-streams= > +Common Joined UInteger Var(param_unroll_jam_reduction_streams) Init(4) > IntegerRange(2, 16) Param Optimization > +Target number of parallel reduction streams produced by reduction-aware > unroll-and-jam. > + > +-param=unroll-jam-reduction-min-inner-iters= > +Common Joined UInteger Var(param_unroll_jam_reduction_min_inner_iters) > Init(32) IntegerRange(2, 65536) Param Optimization > +Minimum inner-loop trip count required for reduction-aware unroll-and-jam to > fire. > + > -param=use-after-scope-direct-emission-threshold= > Common Joined UInteger Var(param_use_after_scope_direct_emission_threshold) > Init(256) Param Optimization > Use direct poisoning/unpoisoning instructions for variables smaller or equal > to this number. > diff --git a/gcc/target.def b/gcc/target.def > index 66c20a9c9d2a..118aa9bb82ec 100644 > --- a/gcc/target.def > +++ b/gcc/target.def > @@ -2873,6 +2873,19 @@ number of memory accesses.", > unsigned, (unsigned nunroll, class loop *loop), > NULL) > > +DEFHOOK > +(loop_jam_reduction_factor, > + "Return the preferred number of parallel reduction streams\n\ > +@option{-floop-unroll-and-jam-reductions} should produce for an inner\n\ > +reduction that accumulates into mode @var{acc_mode} from operands of\n\ > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source\n\ > +of the widening multiply, and is always strictly narrower than\n\ > +@var{acc_mode}: plain reductions with no widening multiply are\n\ > +filtered out before the hook is consulted. Returning less than two\n\ > +disables the transform for that mode pair.", > + unsigned, (machine_mode acc_mode, machine_mode op_mode), > + hook_uint_mode_mode_0) > + > /* True if X is a legitimate MODE-mode immediate operand. */ > DEFHOOK > (legitimate_constant_p, > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c > new file mode 100644 > index 000000000000..d14711034bc0 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c > @@ -0,0 +1,78 @@ > +/* Positive tests: the pass must strip-mine and fuse each nest below. > + The outer trip count of 32 is divisible by the default factor of 4. > + Both arms of the aarch64 hook are covered: QImode / HImode -> SImode > + (DOT_PROD class) in affine, and SImode -> DImode (SMLAL_DI) in > + affine64. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions > -fdump-tree-unrolljam-details" } */ > + > +#include <stdint.h> > + > +#define IN_DIM 512 > +#define OUT_DIM 32 > + > +void > +affine (int32_t *__restrict out, > + const int8_t *weights, > + const int32_t *biases, > + const uint8_t *input) > +{ > + for (int i = 0; i < OUT_DIM; ++i) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < IN_DIM; ++j) > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; > + out[i] = s; > + } > +} > + > +void > +affine64 (int64_t *__restrict out, > + const int32_t *weights, > + const int64_t *biases, > + const int32_t *input) > +{ > + for (int i = 0; i < OUT_DIM; ++i) > + { > + int64_t s = biases[i]; > + for (int j = 0; j < IN_DIM; ++j) > + s += (int64_t) weights[i * IN_DIM + j] * (int64_t) input[j]; > + out[i] = s; > + } > +} > + > +/* Inner trip count at the min-inner-iters default of 32. */ > +void > +affine_min_inner (int32_t *__restrict out, > + const int8_t *weights, > + const int32_t *biases, > + const uint8_t *input) > +{ > + for (int i = 0; i < 32; ++i) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < 32; ++j) > + s += (int32_t) weights[i * 32 + j] * (int32_t) input[j]; > + out[i] = s; > + } > +} > + > +/* Outer trip count of 9 narrows N from 4 to 3. */ > +void > +affine_narrowed (int32_t *__restrict out, > + const int8_t *weights, > + const int32_t *biases, > + const uint8_t *input) > +{ > + for (int i = 0; i < 9; ++i) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < 512; ++j) > + s += (int32_t) weights[i * 512 + j] * (int32_t) input[j]; > + out[i] = s; > + } > +} > + > +/* { dg-final { scan-tree-dump-times "applying reduction unroll-and-jam" 4 > "unrolljam" } } */ > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 11 > "unrolljam" } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c > new file mode 100644 > index 000000000000..44301348a659 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c > @@ -0,0 +1,89 @@ > +/* Correctness test for the reduction-aware extension of unroll-and-jam > + on the GEMV pattern. Compares against an in-main reference > + computation. */ > + > +/* { dg-do run } */ > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */ > + > +#include <stdint.h> > + > +#define IN_DIM 512 > +#define OUT_DIM 32 > + > +extern void abort (void); > + > +void __attribute__((noipa)) > +affine (int32_t *__restrict out, > + const int8_t *weights, > + const int32_t *biases, > + const uint8_t *input) > +{ > + for (int i = 0; i < OUT_DIM; ++i) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < IN_DIM; ++j) > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; > + out[i] = s; > + } > +} > + > +void __attribute__((noipa)) > +affine_zero_init (int32_t *__restrict out, > + const int8_t *a, > + const int8_t *b) > +{ > + for (int i = 0; i < OUT_DIM; ++i) > + { > + int32_t s = 0; > + for (int j = 0; j < IN_DIM; ++j) > + s += a[i * IN_DIM + j] * b[j]; > + out[i] = s; > + } > +} > + > +int > +main (void) > +{ > + static int8_t weights[OUT_DIM * IN_DIM]; > + static uint8_t input[IN_DIM]; > + static int32_t biases[OUT_DIM]; > + static int32_t out[OUT_DIM]; > + static int32_t expected[OUT_DIM]; > + > + for (int k = 0; k < OUT_DIM * IN_DIM; k++) > + weights[k] = (k * 13) & 0x7f; > + for (int k = 0; k < IN_DIM; k++) > + input[k] = (k * 7) & 0xff; > + for (int k = 0; k < OUT_DIM; k++) > + biases[k] = k * 1000; > + > + for (int i = 0; i < OUT_DIM; i++) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < IN_DIM; j++) > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; > + expected[i] = s; > + } > + > + affine (out, weights, biases, input); > + > + for (int i = 0; i < OUT_DIM; i++) > + if (out[i] != expected[i]) > + abort (); > + > + for (int i = 0; i < OUT_DIM; i++) > + { > + int32_t s = 0; > + for (int j = 0; j < IN_DIM; j++) > + s += weights[i * IN_DIM + j] * weights[j]; > + expected[i] = s; > + } > + > + affine_zero_init (out, weights, weights); > + > + for (int i = 0; i < OUT_DIM; i++) > + if (out[i] != expected[i]) > + abort (); > + > + return 0; > +} > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c > new file mode 100644 > index 000000000000..6e4b590fefeb > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c > @@ -0,0 +1,189 @@ > +/* Negative tests: each function below hits one of > + unroll_jam_reduction_possible_p's rejection clauses and the pass > + must not fire on any of them. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions > -fdump-tree-unrolljam-details" } */ > + > +#include <stdint.h> > + > +/* Outer reduction: outer carries 'sum'. */ > +int > +double_reduc (const int *a) > +{ > + int sum = 0; > + for (int i = 0; i < 32; i++) > + for (int j = 0; j < 512; j++) > + sum += a[i * 512 + j]; > + return sum; > +} > + > +/* Inner body has a store: inner writes a[j]. */ > +void > +inner_store (int *__restrict a, int *__restrict b, const int *c, const int > *d) > +{ > + for (int i = 0; i < 32; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + { > + a[j] = c[j] + d[j]; > + s += a[j]; > + } > + b[i] = s; > + } > +} > + > +/* Inner trip count just below the min-inner-iters default of 32. */ > +void > +small_inner (int *out, const int *a) > +{ > + for (int i = 0; i < 32; i++) > + { > + int s = 0; > + for (int j = 0; j < 31; j++) > + s += a[i * 31 + j]; > + out[i] = s; > + } > +} > + > +/* Outer body stores to counter as well as out[i]. */ > +extern int counter; > +void > +outer_store (int *out, const int8_t *a, const int8_t *b) > +{ > + for (int i = 0; i < 32; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + s += (int) a[i * 512 + j] * (int) b[j]; > + out[i] = s; > + counter++; > + } > +} > + > +/* FP accumulator: the multiply has no narrower scalar-int operand, so > + it is not a widening MAC. */ > +void > +fp_mac (float *out, const float *a, const float *b) > +{ > + for (int i = 0; i < 32; i++) > + { > + float s = 0.0f; > + for (int j = 0; j < 512; j++) > + s += a[i * 512 + j] * b[j]; > + out[i] = s; > + } > +} > + > +/* Plain SImode add reduction: no multiply in the update chain. */ > +void > +plain_add (int *out, const int *a) > +{ > + for (int i = 0; i < 32; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + s += a[i * 512 + j]; > + out[i] = s; > + } > +} > + > +/* MAX reduction: no multiply in the update chain. */ > +void > +max_reduc (int *out, const int *a) > +{ > + for (int i = 0; i < 32; i++) > + { > + int m = a[i * 512]; > + for (int j = 1; j < 512; j++) > + { > + int v = a[i * 512 + j]; > + m = m > v ? m : v; > + } > + out[i] = m; > + } > +} > + > +/* Inner header carries two reduction PHIs, m and s. */ > +void > +multi_phi (int *out, const int *a) > +{ > + for (int i = 0; i < 32; i++) > + { > + int m = a[i * 512]; > + int s = 0; > + for (int j = 0; j < 512; j++) > + { > + int v = a[i * 512 + j]; > + m = m > v ? m : v; > + s += v; > + } > + out[i] = m + s; > + } > +} > + > +/* Parametric outer trip count: not a known UHWI. */ > +void > +parametric_outer (int *out, const int8_t *a, const int8_t *b, int n) > +{ > + for (int i = 0; i < n; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + s += (int) a[i * 512 + j] * (int) b[j]; > + out[i] = s; > + } > +} > + > +/* Outer trip count of 7 has no divisor in [2, 4]. */ > +void > +narrow_to_one (int *__restrict out, const int8_t *a, const int8_t *b) > +{ > + for (int i = 0; i < 7; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + s += (int) a[i * 512 + j] * (int) b[j]; > + out[i] = s; > + } > +} > + > +/* No __restrict: data-dep analysis cannot rule out out vs b > + aliasing. */ > +void > +aliasing (int *out, const int8_t *a, const int8_t *b) > +{ > + for (int i = 0; i < 32; i++) > + { > + int s = 0; > + for (int j = 0; j < 512; j++) > + s += (int) a[i * 512 + j] * (int) b[j]; > + out[i] = s; > + } > +} > + > +/* Conditional accumulator init: the "?:" lowers to a store-free diamond > + in the outer body whose merge PHI feeds the reduction, so the body is > + not straight-line. */ > +void > +cond_init (int32_t *__restrict out, const int8_t *__restrict a, > + const int32_t *__restrict b1, const int32_t *__restrict b2, > + const int32_t *__restrict c) > +{ > + for (int i = 0; i < 32; i++) > + { > + int32_t s = c[i] ? b1[i] : b2[i]; > + for (int j = 0; j < 512; j++) > + s += (int32_t) a[i * 512 + j] * (int32_t) a[j]; > + out[i] = s; > + } > +} > + > +/* { dg-final { scan-tree-dump-not "applying reduction unroll-and-jam" > "unrolljam" } } */ > +/* { dg-final { scan-tree-dump-not "reduction-jam: fusing src loop" > "unrolljam" } } */ > +/* { dg-final { scan-tree-dump "reduction-jam: outer trip 7 has no divisor > in \\\[2, 4\\\]" "unrolljam" } } */ > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer > body has extra stores" "unrolljam" } } */ > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer > body has control flow" "unrolljam" } } */ > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer > trip count unknown" "unrolljam" } } */ > +/* { dg-final { scan-tree-dump "reduction-jam: data-dep safety bound below > 2" "unrolljam" } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c > new file mode 100644 > index 000000000000..938f7328b049 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c > @@ -0,0 +1,30 @@ > +/* The reduction-jam pass should fire even when the IV-only path is > + disabled via -fno-loop-unroll-and-jam. Pins the gate change in > + pass_loop_jam::gate and the IV-only/reduction-jam fall-through in > + tree_loop_unroll_and_jam. */ > + > +/* { dg-do compile } */ > +/* { dg-options "-O3 -mtune=ampere1 -fno-loop-unroll-and-jam > -floop-unroll-and-jam-reductions -fdump-tree-unrolljam-details" } */ > + > +#include <stdint.h> > + > +#define IN_DIM 512 > +#define OUT_DIM 32 > + > +void > +affine (int32_t *__restrict out, > + const int8_t *weights, > + const int32_t *biases, > + const uint8_t *input) > +{ > + for (int i = 0; i < OUT_DIM; ++i) > + { > + int32_t s = biases[i]; > + for (int j = 0; j < IN_DIM; ++j) > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; > + out[i] = s; > + } > +} > + > +/* { dg-final { scan-tree-dump "applying reduction unroll-and-jam" > "unrolljam" } } */ > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 3 > "unrolljam" } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c > new file mode 100644 > index 000000000000..0a198bb27ecd > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c > @@ -0,0 +1,72 @@ > +/* Regression test for a multi-IV mispairing in the reduction-jam > + sibling matcher. The inner loop carries two IVs that share the > + preheader entry value 0 but differ in step (j has step 1 and > + indexes W, p has step 2 and indexes X); the matcher must pair each > + source IV with its own dst counterpart, not with whichever dst PHI > + happens to share an entry value. > + > + The reference loop in main uses a volatile accumulator so > + pass_loop_jam does not recognize a reduction PHI there. Without > + that guard the reference would be jammed with the same mispairing > + as gemv_2iv, hiding the miscompile behind a symmetric bug. */ > + > +/* { dg-do run } */ > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */ > + > +#include <stdint.h> > + > +#define OUT_DIM 32 > +#define IN_DIM 256 > +#define STRIDE 512 > + > +extern void abort (void); > + > +void __attribute__((noipa)) > +gemv_2iv (int32_t *__restrict out, const int8_t *w, const uint8_t *x) > +{ > + for (int i = 0; i < OUT_DIM; i++) > + { > + int32_t s = 0; > + int p = 0; > + for (int j = 0; j < IN_DIM; j++) > + { > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p]; > + p += 2; > + } > + out[i] = s; > + } > +} > + > +int > +main (void) > +{ > + static int8_t w[OUT_DIM * STRIDE]; > + static uint8_t x[IN_DIM * 2]; > + static int32_t out[OUT_DIM]; > + static int32_t expected[OUT_DIM]; > + > + for (int k = 0; k < OUT_DIM * STRIDE; k++) > + w[k] = (k * 13) & 0x7f; > + for (int k = 0; k < IN_DIM * 2; k++) > + x[k] = (k * 7) & 0xff; > + > + for (int i = 0; i < OUT_DIM; i++) > + { > + volatile int32_t s = 0; > + int p = 0; > + for (int j = 0; j < IN_DIM; j++) > + { > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p]; > + p += 2; > + } > + expected[i] = s; > + } > + > + gemv_2iv (out, w, x); > + > + for (int i = 0; i < OUT_DIM; i++) > + if (out[i] != expected[i]) > + abort (); > + > + return 0; > +} > -- > 2.55.0 >
