On Thu, Sep 10, 2026 at 1:30 PM Konstantinos Eleftheriou <[email protected]> wrote: > > Hi Richard, > > Supporting the weights access would remove the rejection, but not the > throughput problem behind it. DOT_PROD reduces within a register > across consecutive j: one usdot consumes 16 elements of j, and the > four lanes are summed once at loop exit. With i in the lanes the inner > loop still advances j one iteration at a time and a widening > multiply-accumulate performs 4 products per instruction where a > usdot performs 16. > > Thanks, > Konstantinos > > On Tue, Sep 8, 2026 at 5:17 PM Richard Biener <[email protected]> > wrote: >> >> On Mon, Sep 7, 2026 at 4:35 PM Konstantinos Eleftheriou >> <[email protected]> wrote: >> > >> > Extends pass_loop_jam with a parallel-reduction branch for the >> > GEMV / NNUE inference shape >> > >> > for (i) { >> > s = bias[i]; >> > for (j) s += w[i][j] * x[j]; >> > out[i] = s; >> > }
I guess this snippet lacks types. I infer that w/x are char and s is int? And when you do not split the reduction, why is this not picked up as DOT_PROD and a reduction chain? I do agree that unroll-and-jam for an inner loop with a reduction is somewhat pointless if the dependence cannot be split (assuming the loop stays scalar). So in principle this makes sense if we want to unroll-and-jam loops with reduction(s). Richard. >> > Stock unroll_jam_possible_p rejects this nest because `s' is a >> > reduction PHI, not a simple induction; even if it were accepted, >> > stock fuse_loops would collapse the post-unroll per-sibling reduction >> > PHIs into one serial chain, multiplying the inner loop's serial >> > dependency height by N and defeating the goal of unroll-and-jam. >> >> So given we do not unroll-and-jam this, wouldn't it be better to >> "simply" support outer loop vectorizing for it because outer loop >> vectorization is exactly unroll-and-jam under the hood? >> >> > unroll_jam_reduction_possible_p detects the candidate, the existing >> > tree_unroll_loop strip-mines the outer loop by N, and >> > unroll_jam_reduction_fuse_sibling fuses the N inner siblings into one >> > body that retains N parallel reduction PHIs in its header and hoists >> > each sibling's preamble (s_k = bias[i+k]) to the fused preheader. The >> > post-reduction stores stay on the linear sibling-exit chain; >> > cleanup_tree_cfg coalesces it and rewrite_into_loop_closed_ssa restores >> > LC-SSA. The vectorizer then sees one inner loop with N independent >> > chains and emits N parallel dot-product / FMA accumulators, cutting the >> > serial dependency height across the reduction by N. >> > >> > N is the min of TARGET_LOOP_JAM_REDUCTION_FACTOR (opt-in, default 0 >> > = "no benefit measured"), --param=unroll-jam-reduction-streams and >> > adjust_unroll_factor's data-dep safety bound, narrowed to a divisor of >> > the outer trip count. The hook receives the accumulator mode plus the >> > narrowest scalar-int mode found at a widening multiply in the update >> > chain; op_mode == acc_mode signals "no widening multiply", letting a >> > target opt in for MAC shapes only. Inner loops whose trip count is >> > below --param=unroll-jam-reduction-min-inner-iters are left alone: >> > their per-stream serial chain is too short to profit from parallelism. >> > >> > AArch64 opts in for QI/HI -> SI (DOT_PROD class) and SI -> DI >> > (SMLAL_DI); all other shapes (plain reductions, max/min, FP) return 0. >> > The per-CPU factor is sourced from a new tune_params field >> > loop_jam_reduction_factor rather than a hardcoded return value, so the >> > profitability decision lives in the tuning structure next to the other >> > latency-driven knobs. Cores with measured wins ship factor 4; every >> > other tuning leaves the field at 0 until measured. The new tests pin >> > -mtune to a factor-4 core so they exercise that path independently of >> > the configured default tuning. >> > >> > FP reductions are deferred to outer-loop vectorization's lane-broadcast >> > FMLA path. The chained-reduction case (single accumulator carried >> > across outer iterations) is left as a follow-up. >> > >> > ~8% speedup on SPEC CPU 2026 706.stockfish_r. >> > >> > gcc/ChangeLog: >> > >> > * common.opt (floop-unroll-and-jam-reductions): New option, >> > EnabledBy(floop-unroll-and-jam). >> > * config/aarch64/aarch64-protos.h (struct tune_params): >> > Add loop_jam_reduction_factor. >> > * config/aarch64/aarch64.cc (aarch64_loop_jam_reduction_factor): >> > New function; source factor from aarch64_tune_params and opt >> > in for QI/HI->SI and SI->DI MAC shapes only. >> > (TARGET_LOOP_JAM_REDUCTION_FACTOR): Define. >> > * config/aarch64/tuning_models/a64fx.h: Default >> > loop_jam_reduction_factor to 0 (opt-out). >> > * config/aarch64/tuning_models/ampere1.h: Set >> > loop_jam_reduction_factor to 4. >> > * config/aarch64/tuning_models/ampere1a.h: Likewise. >> > * config/aarch64/tuning_models/ampere1b.h: Likewise. >> > * config/aarch64/tuning_models/cortexa320.h: Default >> > loop_jam_reduction_factor to 0 (opt-out). >> > * config/aarch64/tuning_models/cortexa35.h: Likewise. >> > * config/aarch64/tuning_models/cortexa53.h: Likewise. >> > * config/aarch64/tuning_models/cortexa57.h: Likewise. >> > * config/aarch64/tuning_models/cortexa72.h: Likewise. >> > * config/aarch64/tuning_models/cortexa73.h: Likewise. >> > * config/aarch64/tuning_models/cortexx925.h: Likewise. >> > * config/aarch64/tuning_models/emag.h: Likewise. >> > * config/aarch64/tuning_models/exynosm1.h: Likewise. >> > * config/aarch64/tuning_models/fujitsu_monaka.h: Likewise. >> > * config/aarch64/tuning_models/generic.h: Likewise. >> > * config/aarch64/tuning_models/generic_armv8_a.h: Likewise. >> > * config/aarch64/tuning_models/generic_armv9_a.h: Likewise. >> > * config/aarch64/tuning_models/hip12.h: Likewise. >> > * config/aarch64/tuning_models/neoverse512tvb.h: Likewise. >> > * config/aarch64/tuning_models/neoversen1.h: Set >> > loop_jam_reduction_factor to 4. >> > * config/aarch64/tuning_models/neoversen2.h: Default >> > loop_jam_reduction_factor to 0 (opt-out). >> > * config/aarch64/tuning_models/neoversen3.h: Likewise. >> > * config/aarch64/tuning_models/neoversev1.h: Likewise. >> > * config/aarch64/tuning_models/neoversev2.h: Likewise. >> > * config/aarch64/tuning_models/neoversev3.h: Likewise. >> > * config/aarch64/tuning_models/neoversev3ae.h: Likewise. >> > * config/aarch64/tuning_models/olympus.h: Likewise. >> > * config/aarch64/tuning_models/qdf24xx.h: Likewise. >> > * config/aarch64/tuning_models/saphira.h: Likewise. >> > * config/aarch64/tuning_models/thunderx.h: Likewise. >> > * config/aarch64/tuning_models/thunderx2t99.h: Likewise. >> > * config/aarch64/tuning_models/thunderx3t110.h: Likewise. >> > * config/aarch64/tuning_models/thunderxt88.h: Likewise. >> > * config/aarch64/tuning_models/tsv110.h: Likewise. >> > * config/aarch64/tuning_models/xgene1.h: Likewise. >> > * doc/invoke.texi (-floop-unroll-and-jam-reductions): Document. >> > * doc/tm.texi: Regenerate. >> > * doc/tm.texi.in (TARGET_LOOP_JAM_REDUCTION_FACTOR): Add hook >> > reference. >> > * gimple-loop-jam.cc: Include cfganal.h and tree-into-ssa.h. >> > (inner_reduction_phi): New helper. >> > (narrowest_reduc_input_mode): New helper that walks the >> > reduction update chain for widening multiplies. >> > (gsi_move_to_bb_end_before_terminator): New helper. >> > (collapse_single_arg_phi): New helper. >> > (loop_is_dot_product_candidate_p): New static helper. >> > (unroll_jam_reduction_possible_p): New candidate detector; >> > calls loop_is_dot_product_candidate_p and layers outer-loop >> > guards on top. >> > (migrate_loop_invariant_defs_to_preheader): New helper that >> > hoists the SSA-use closure of a sibling's reduction init and >> > body operands to the destination preheader in RPO. >> > (match_sibling_header_phis): New helper pairing sibling non- >> > reduction PHIs positionally. >> > (unroll_jam_reduction_fuse_sibling): New per-sibling fuser. >> > (unroll_jam_reduction_bypass_sibling): New CFG bypass for the >> > drained source sibling. >> > (compute_jam_dependence_factors): New helper. Walks OUTER's >> > data dependences and accumulates the unroll-factor safety >> > bound, ref-removal profit signal, removed-ref count, and total >> > dataref count. >> > (select_reduction_jam_factor): New helper combining target >> > factor, param cap and data-dep safety bound, narrowed to a >> > divisor of the outer trip count. Emit a TDF_DETAILS note >> > when the outer trip has no divisor in [2, factor cap]. >> > (do_unroll_jam_reductions): New driver helper. >> > (tree_loop_unroll_and_jam): Try the reduction-jam path on >> > nests the IV-only path skips; cleanup CFG, rebuild LC-SSA and >> > reset SCEV per transformed nest. >> > (pass_loop_jam::gate): Also enable for >> > flag_unroll_jam_reductions. >> > * hooks.cc (hook_uint_mode_mode_0): New default hook. >> > * hooks.h (hook_uint_mode_mode_0): Declare. >> > * params.opt (unroll-jam-reduction-streams): New param. >> > (unroll-jam-reduction-min-inner-iters): New param. >> > * target.def (loop_jam_reduction_factor): New target hook. >> > >> > gcc/testsuite/ChangeLog: >> > >> > * gcc.target/aarch64/unroll-jam-reductions-1.c: New test. >> > * gcc.target/aarch64/unroll-jam-reductions-2.c: New test. >> > * gcc.target/aarch64/unroll-jam-reductions-3.c: New test. >> > * gcc.target/aarch64/unroll-jam-reductions-4.c: New test. >> > * gcc.target/aarch64/unroll-jam-reductions-5.c: New test. >> > >> > Co-Authored-By: Philipp Tomsich <[email protected]> >> > Signed-off-by: Konstantinos Eleftheriou <[email protected]> >> > --- >> > >> > gcc/common.opt | 5 + >> > gcc/config/aarch64/aarch64-protos.h | 8 + >> > gcc/config/aarch64/aarch64.cc | 29 + >> > gcc/config/aarch64/tuning_models/a64fx.h | 3 +- >> > gcc/config/aarch64/tuning_models/ampere1.h | 3 +- >> > gcc/config/aarch64/tuning_models/ampere1a.h | 3 +- >> > gcc/config/aarch64/tuning_models/ampere1b.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa320.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa35.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa53.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa57.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa72.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexa73.h | 3 +- >> > gcc/config/aarch64/tuning_models/cortexx925.h | 3 +- >> > gcc/config/aarch64/tuning_models/emag.h | 3 +- >> > gcc/config/aarch64/tuning_models/exynosm1.h | 3 +- >> > .../aarch64/tuning_models/fujitsu_monaka.h | 3 +- >> > gcc/config/aarch64/tuning_models/generic.h | 3 +- >> > .../aarch64/tuning_models/generic_armv8_a.h | 3 +- >> > .../aarch64/tuning_models/generic_armv9_a.h | 3 +- >> > gcc/config/aarch64/tuning_models/hip12.h | 3 +- >> > .../aarch64/tuning_models/neoverse512tvb.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversen1.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversen2.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversen3.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversev1.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversev2.h | 3 +- >> > gcc/config/aarch64/tuning_models/neoversev3.h | 3 +- >> > .../aarch64/tuning_models/neoversev3ae.h | 3 +- >> > gcc/config/aarch64/tuning_models/olympus.h | 3 +- >> > gcc/config/aarch64/tuning_models/qdf24xx.h | 3 +- >> > gcc/config/aarch64/tuning_models/saphira.h | 3 +- >> > gcc/config/aarch64/tuning_models/thunderx.h | 3 +- >> > .../aarch64/tuning_models/thunderx2t99.h | 3 +- >> > .../aarch64/tuning_models/thunderx3t110.h | 3 +- >> > .../aarch64/tuning_models/thunderxt88.h | 3 +- >> > gcc/config/aarch64/tuning_models/tsv110.h | 3 +- >> > gcc/config/aarch64/tuning_models/xgene1.h | 3 +- >> > gcc/doc/invoke.texi | 17 +- >> > gcc/doc/tm.texi | 11 + >> > gcc/doc/tm.texi.in | 2 + >> > gcc/gimple-loop-jam.cc | 932 +++++++++++++++++- >> > gcc/hooks.cc | 7 + >> > gcc/hooks.h | 1 + >> > gcc/params.opt | 8 + >> > gcc/target.def | 13 + >> > .../aarch64/unroll-jam-reductions-1.c | 78 ++ >> > .../aarch64/unroll-jam-reductions-2.c | 89 ++ >> > .../aarch64/unroll-jam-reductions-3.c | 189 ++++ >> > .../aarch64/unroll-jam-reductions-4.c | 30 + >> > .../aarch64/unroll-jam-reductions-5.c | 72 ++ >> > 51 files changed, 1557 insertions(+), 39 deletions(-) >> > create mode 100644 >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c >> > create mode 100644 >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c >> > create mode 100644 >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c >> > create mode 100644 >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c >> > create mode 100644 >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c >> > >> > diff --git a/gcc/common.opt b/gcc/common.opt >> > index 69a55b8d40d6..634b63569c67 100644 >> > --- a/gcc/common.opt >> > +++ b/gcc/common.opt >> > @@ -1928,6 +1928,11 @@ floop-unroll-and-jam >> > Common Var(flag_unroll_jam) Optimization >> > Perform unroll-and-jam on loops. >> > >> > +floop-unroll-and-jam-reductions >> > +Common Var(flag_unroll_jam_reductions) Optimization >> > EnabledBy(floop-unroll-and-jam) >> > +Perform reduction-aware unroll-and-jam on loops with inner reductions >> > +to enable multi-stream parallelism during vectorization. >> > + >> > fgnu-tm >> > Common Var(flag_tm) >> > Enable support for GNU transactional memory. >> > diff --git a/gcc/config/aarch64/aarch64-protos.h >> > b/gcc/config/aarch64/aarch64-protos.h >> > index e8ae3d427949..88237e243738 100644 >> > --- a/gcc/config/aarch64/aarch64-protos.h >> > +++ b/gcc/config/aarch64/aarch64-protos.h >> > @@ -590,6 +590,14 @@ struct tune_params >> > >> > /* Dispatch constraints for instruction scheduling. */ >> > const struct dispatch_constraint_info *dispatch_constraints; >> > + >> > + /* Preferred number of parallel reduction streams for >> > + -floop-unroll-and-jam-reductions on this core. Zero opts out; >> > + two or more enables the transform on reduction nests whose >> > + (acc_mode, op_mode) pair the target hook accepts. Cores with a >> > + two-wide int/FP pipe and several-cycle MAC latency typically >> > + want 4. */ >> > + unsigned int loop_jam_reduction_factor; >> > }; >> > >> > /* Classifies an address. >> > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc >> > index ccb4f5fbb10d..546912534499 100644 >> > --- a/gcc/config/aarch64/aarch64.cc >> > +++ b/gcc/config/aarch64/aarch64.cc >> > @@ -17402,6 +17402,32 @@ aarch64_sched_issue_rate (void) >> > return aarch64_tune_params.issue_rate; >> > } >> > >> > +/* Implement TARGET_LOOP_JAM_REDUCTION_FACTOR. Return the preferred >> > + number of parallel reduction streams for >> > + -floop-unroll-and-jam-reductions on a reduction accumulating into >> > + ACC_MODE from operands of OP_MODE. We opt in only for the widening >> > + MAC shapes the vectorizer lowers to a latency-bound >> > + DOT_PROD / SMLAL chain: >> > + * QImode / HImode -> SImode (DOT_PROD-class, USDOT / SDOT / UDOT) >> > + * SImode -> DImode (SMLAL_DI) >> > + The stream count comes from the per-core tuning structure; cores >> > + that have not been measured leave loop_jam_reduction_factor at 0, >> > + which suppresses the transform. */ >> > + >> > +static unsigned int >> > +aarch64_loop_jam_reduction_factor (machine_mode acc_mode, >> > + machine_mode op_mode) >> > +{ >> > + unsigned int factor = aarch64_tune_params.loop_jam_reduction_factor; >> > + if (factor < 2 || !SCALAR_INT_MODE_P (acc_mode)) >> > + return 0; >> > + >> > + if ((acc_mode == SImode && (op_mode == QImode || op_mode == HImode)) >> > + || (acc_mode == DImode && op_mode == SImode)) >> > + return factor; >> > + return 0; >> > +} >> > + >> > /* Implement TARGET_SCHED_VARIABLE_ISSUE. */ >> > static int >> > aarch64_sched_variable_issue (FILE *, int, rtx_insn *insn, int more) >> > @@ -34491,6 +34517,9 @@ aarch64_libgcc_floating_mode_supported_p >> > #undef TARGET_SCHED_ISSUE_RATE >> > #define TARGET_SCHED_ISSUE_RATE aarch64_sched_issue_rate >> > >> > +#undef TARGET_LOOP_JAM_REDUCTION_FACTOR >> > +#define TARGET_LOOP_JAM_REDUCTION_FACTOR aarch64_loop_jam_reduction_factor >> > + >> > #undef TARGET_SCHED_VARIABLE_ISSUE >> > #define TARGET_SCHED_VARIABLE_ISSUE aarch64_sched_variable_issue >> > >> > diff --git a/gcc/config/aarch64/tuning_models/a64fx.h >> > b/gcc/config/aarch64/tuning_models/a64fx.h >> > index c3222960f175..171e64ce23ad 100644 >> > --- a/gcc/config/aarch64/tuning_models/a64fx.h >> > +++ b/gcc/config/aarch64/tuning_models/a64fx.h >> > @@ -166,7 +166,8 @@ static const struct tune_params a64fx_tunings = >> > &a64fx_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_A64FX. */ >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1.h >> > b/gcc/config/aarch64/tuning_models/ampere1.h >> > index e2a42f585dd9..0057db3765b4 100644 >> > --- a/gcc/config/aarch64/tuning_models/ampere1.h >> > +++ b/gcc/config/aarch64/tuning_models/ampere1.h >> > @@ -136,7 +136,8 @@ static const struct tune_params ampere1_tunings = >> > &ere1_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 4 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_AMPERE1. */ >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1a.h >> > b/gcc/config/aarch64/tuning_models/ampere1a.h >> > index 0954ac551a0b..1350c0f3b5c5 100644 >> > --- a/gcc/config/aarch64/tuning_models/ampere1a.h >> > +++ b/gcc/config/aarch64/tuning_models/ampere1a.h >> > @@ -58,7 +58,8 @@ static const struct tune_params ampere1a_tunings = >> > &ere1_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 4 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_AMPERE1A. */ >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1b.h >> > b/gcc/config/aarch64/tuning_models/ampere1b.h >> > index 8ba434d27bbf..0fc6eb36bd2a 100644 >> > --- a/gcc/config/aarch64/tuning_models/ampere1b.h >> > +++ b/gcc/config/aarch64/tuning_models/ampere1b.h >> > @@ -137,7 +137,8 @@ static const struct tune_params ampere1b_tunings = >> > &ere1b_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 4 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_AMPERE1B */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa320.h >> > b/gcc/config/aarch64/tuning_models/cortexa320.h >> > index d5e1f2b29938..89ed24360008 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa320.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa320.h >> > @@ -189,7 +189,8 @@ static const struct tune_params cortexa320_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXA320. */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa35.h >> > b/gcc/config/aarch64/tuning_models/cortexa35.h >> > index fd2fc47f5e23..ff9eeb406799 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa35.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa35.h >> > @@ -57,7 +57,8 @@ static const struct tune_params cortexa35_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXA35. */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa53.h >> > b/gcc/config/aarch64/tuning_models/cortexa53.h >> > index e71cc1b26afb..13dbb152c6e9 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa53.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa53.h >> > @@ -66,7 +66,8 @@ static const struct tune_params cortexa53_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXA53. */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa57.h >> > b/gcc/config/aarch64/tuning_models/cortexa57.h >> > index 4f3ec60ba275..c78c15198158 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa57.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa57.h >> > @@ -103,7 +103,8 @@ static const struct tune_params cortexa57_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXA57. */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa72.h >> > b/gcc/config/aarch64/tuning_models/cortexa72.h >> > index df0fcf110867..8a4e5e9d8a36 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa72.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa72.h >> > @@ -55,7 +55,8 @@ static const struct tune_params cortexa72_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXA72. */ >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa73.h >> > b/gcc/config/aarch64/tuning_models/cortexa73.h >> > index c871b454d40f..40d46f23ae82 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexa73.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexa73.h >> > @@ -56,7 +56,8 @@ static const struct tune_params cortexa73_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > >> > diff --git a/gcc/config/aarch64/tuning_models/cortexx925.h >> > b/gcc/config/aarch64/tuning_models/cortexx925.h >> > index 2a73b3697065..b99d6b4300e5 100644 >> > --- a/gcc/config/aarch64/tuning_models/cortexx925.h >> > +++ b/gcc/config/aarch64/tuning_models/cortexx925.h >> > @@ -227,7 +227,8 @@ static const struct tune_params cortexx925_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_CORTEXX925. */ >> > diff --git a/gcc/config/aarch64/tuning_models/emag.h >> > b/gcc/config/aarch64/tuning_models/emag.h >> > index 82bbc554f41b..6e87d28173b2 100644 >> > --- a/gcc/config/aarch64/tuning_models/emag.h >> > +++ b/gcc/config/aarch64/tuning_models/emag.h >> > @@ -55,7 +55,8 @@ static const struct tune_params emag_tunings = >> > &xgene1_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_EMAG. */ >> > diff --git a/gcc/config/aarch64/tuning_models/exynosm1.h >> > b/gcc/config/aarch64/tuning_models/exynosm1.h >> > index a561ccf6fa71..928f3d4f3014 100644 >> > --- a/gcc/config/aarch64/tuning_models/exynosm1.h >> > +++ b/gcc/config/aarch64/tuning_models/exynosm1.h >> > @@ -139,7 +139,8 @@ static const struct tune_params exynosm1_tunings = >> > &exynosm1_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_EXYNOSM1. */ >> > diff --git a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h >> > b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h >> > index 71549bd0ea70..70d71bf253de 100644 >> > --- a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h >> > +++ b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h >> > @@ -59,7 +59,8 @@ static const struct tune_params fujitsu_monaka_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_FUJITSU_MONAKA. */ >> > diff --git a/gcc/config/aarch64/tuning_models/generic.h >> > b/gcc/config/aarch64/tuning_models/generic.h >> > index af013e478d45..71dbebeb47c0 100644 >> > --- a/gcc/config/aarch64/tuning_models/generic.h >> > +++ b/gcc/config/aarch64/tuning_models/generic.h >> > @@ -188,7 +188,8 @@ static const struct tune_params generic_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_GENERIC. */ >> > diff --git a/gcc/config/aarch64/tuning_models/generic_armv8_a.h >> > b/gcc/config/aarch64/tuning_models/generic_armv8_a.h >> > index 29cf3a5a7c7a..15900a7ba733 100644 >> > --- a/gcc/config/aarch64/tuning_models/generic_armv8_a.h >> > +++ b/gcc/config/aarch64/tuning_models/generic_armv8_a.h >> > @@ -180,7 +180,8 @@ static const struct tune_params >> > generic_armv8_a_tunings = >> > &generic_armv8_a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_GENERIC_ARMV8_A. */ >> > diff --git a/gcc/config/aarch64/tuning_models/generic_armv9_a.h >> > b/gcc/config/aarch64/tuning_models/generic_armv9_a.h >> > index 05e777d20efb..7c1b78f88a53 100644 >> > --- a/gcc/config/aarch64/tuning_models/generic_armv9_a.h >> > +++ b/gcc/config/aarch64/tuning_models/generic_armv9_a.h >> > @@ -255,7 +255,8 @@ static const struct tune_params >> > generic_armv9_a_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_GENERIC_ARMV9_A. */ >> > diff --git a/gcc/config/aarch64/tuning_models/hip12.h >> > b/gcc/config/aarch64/tuning_models/hip12.h >> > index cfb204b2e533..2d5c107d71d7 100644 >> > --- a/gcc/config/aarch64/tuning_models/hip12.h >> > +++ b/gcc/config/aarch64/tuning_models/hip12.h >> > @@ -221,7 +221,8 @@ static const struct tune_params hip12_tunings = >> > &generic_armv8_a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_HIP12. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoverse512tvb.h >> > b/gcc/config/aarch64/tuning_models/neoverse512tvb.h >> > index d6e2c2e8120d..a128d5ed7b3a 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoverse512tvb.h >> > +++ b/gcc/config/aarch64/tuning_models/neoverse512tvb.h >> > @@ -162,7 +162,8 @@ static const struct tune_params neoverse512tvb_tunings >> > = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSE512TVB. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen1.h >> > b/gcc/config/aarch64/tuning_models/neoversen1.h >> > index 87c7fbcb7f22..9b2ec9322bba 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversen1.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversen1.h >> > @@ -55,7 +55,8 @@ static const struct tune_params neoversen1_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 4 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEN1. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen2.h >> > b/gcc/config/aarch64/tuning_models/neoversen2.h >> > index 21597e6b50f8..e1648c88eb78 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversen2.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversen2.h >> > @@ -224,7 +224,8 @@ static const struct tune_params neoversen2_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEN2. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen3.h >> > b/gcc/config/aarch64/tuning_models/neoversen3.h >> > index 37f9819148f6..8796914772b4 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversen3.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversen3.h >> > @@ -223,7 +223,8 @@ static const struct tune_params neoversen3_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEN3. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev1.h >> > b/gcc/config/aarch64/tuning_models/neoversev1.h >> > index 253f11e87a68..7f93e4092550 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversev1.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversev1.h >> > @@ -233,7 +233,8 @@ static const struct tune_params neoversev1_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev2.h >> > b/gcc/config/aarch64/tuning_models/neoversev2.h >> > index 6df0cc444b80..8ceb7d5538fb 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversev2.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversev2.h >> > @@ -364,7 +364,8 @@ static const struct tune_params neoversev2_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - &neoversev2_dispatch_constraint_info /* dispatch_constraints. */ >> > + &neoversev2_dispatch_constraint_info, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEV2. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev3.h >> > b/gcc/config/aarch64/tuning_models/neoversev3.h >> > index 7e0144032dc6..ea840d2cbb7c 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversev3.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversev3.h >> > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEV3. */ >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev3ae.h >> > b/gcc/config/aarch64/tuning_models/neoversev3ae.h >> > index bbe4dc547015..fe39c7b67c9d 100644 >> > --- a/gcc/config/aarch64/tuning_models/neoversev3ae.h >> > +++ b/gcc/config/aarch64/tuning_models/neoversev3ae.h >> > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3ae_tunings = >> > &generic_armv9a_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_NEOVERSEV3AE. */ >> > diff --git a/gcc/config/aarch64/tuning_models/olympus.h >> > b/gcc/config/aarch64/tuning_models/olympus.h >> > index 1e701722e224..ab1ba2464d23 100644 >> > --- a/gcc/config/aarch64/tuning_models/olympus.h >> > +++ b/gcc/config/aarch64/tuning_models/olympus.h >> > @@ -565,7 +565,8 @@ static struct tune_params olympus_tunings = >> > &olympus_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - &olympus_dispatch_constraint_info /* dispatch_constraints. */ >> > + &olympus_dispatch_constraint_info, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_OLYMPUS. */ >> > diff --git a/gcc/config/aarch64/tuning_models/qdf24xx.h >> > b/gcc/config/aarch64/tuning_models/qdf24xx.h >> > index a15a4fc56897..73eb0b8c48cf 100644 >> > --- a/gcc/config/aarch64/tuning_models/qdf24xx.h >> > +++ b/gcc/config/aarch64/tuning_models/qdf24xx.h >> > @@ -131,7 +131,8 @@ static const struct tune_params qdf24xx_tunings = >> > &qdf24xx_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_QDF24XX. */ >> > diff --git a/gcc/config/aarch64/tuning_models/saphira.h >> > b/gcc/config/aarch64/tuning_models/saphira.h >> > index 369d3618f7f1..582c4517fcdc 100644 >> > --- a/gcc/config/aarch64/tuning_models/saphira.h >> > +++ b/gcc/config/aarch64/tuning_models/saphira.h >> > @@ -57,7 +57,8 @@ static const struct tune_params saphira_tunings = >> > &generic_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_SAPHIRA. */ >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx.h >> > b/gcc/config/aarch64/tuning_models/thunderx.h >> > index ed8de307276a..a275659c0b9e 100644 >> > --- a/gcc/config/aarch64/tuning_models/thunderx.h >> > +++ b/gcc/config/aarch64/tuning_models/thunderx.h >> > @@ -112,7 +112,8 @@ static const struct tune_params thunderx_tunings = >> > &thunderx_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_THUNDERX. */ >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx2t99.h >> > b/gcc/config/aarch64/tuning_models/thunderx2t99.h >> > index f6be657ac9c3..5afbc21ef40c 100644 >> > --- a/gcc/config/aarch64/tuning_models/thunderx2t99.h >> > +++ b/gcc/config/aarch64/tuning_models/thunderx2t99.h >> > @@ -132,7 +132,8 @@ static const struct tune_params thunderx2t99_tunings = >> > &thunderx2t99_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_THUNDERX2T99. */ >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx3t110.h >> > b/gcc/config/aarch64/tuning_models/thunderx3t110.h >> > index cdcbec680945..0d7f29bee209 100644 >> > --- a/gcc/config/aarch64/tuning_models/thunderx3t110.h >> > +++ b/gcc/config/aarch64/tuning_models/thunderx3t110.h >> > @@ -131,7 +131,8 @@ static const struct tune_params thunderx3t110_tunings = >> > &thunderx3t110_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_THUNDERX3T110. */ >> > diff --git a/gcc/config/aarch64/tuning_models/thunderxt88.h >> > b/gcc/config/aarch64/tuning_models/thunderxt88.h >> > index 44a1eac59a96..d0c39c07c88a 100644 >> > --- a/gcc/config/aarch64/tuning_models/thunderxt88.h >> > +++ b/gcc/config/aarch64/tuning_models/thunderxt88.h >> > @@ -67,7 +67,8 @@ static const struct tune_params thunderxt88_tunings = >> > &thunderxt88_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_THUNDERXT88. */ >> > diff --git a/gcc/config/aarch64/tuning_models/tsv110.h >> > b/gcc/config/aarch64/tuning_models/tsv110.h >> > index 366f2b3e2323..0800bdc59ced 100644 >> > --- a/gcc/config/aarch64/tuning_models/tsv110.h >> > +++ b/gcc/config/aarch64/tuning_models/tsv110.h >> > @@ -132,7 +132,8 @@ static const struct tune_params tsv110_tunings = >> > &tsv110_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_TSV110. */ >> > diff --git a/gcc/config/aarch64/tuning_models/xgene1.h >> > b/gcc/config/aarch64/tuning_models/xgene1.h >> > index 0d4aaf80b61a..5fd5e06932c3 100644 >> > --- a/gcc/config/aarch64/tuning_models/xgene1.h >> > +++ b/gcc/config/aarch64/tuning_models/xgene1.h >> > @@ -140,7 +140,8 @@ static const struct tune_params xgene1_tunings = >> > &xgene1_prefetch_tune, >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */ >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */ >> > - nullptr /* dispatch_constraints. */ >> > + nullptr, /* dispatch_constraints. */ >> > + 0 /* loop_jam_reduction_factor. */ >> > }; >> > >> > #endif /* GCC_AARCH64_H_XGENE1. */ >> > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi >> > index a74e99081d35..9f3215deb942 100644 >> > --- a/gcc/doc/invoke.texi >> > +++ b/gcc/doc/invoke.texi >> > @@ -641,7 +641,8 @@ Objective-C and Objective-C++ Dialects}. >> > -fivopts -fkeep-inline-functions -fkeep-static-functions >> > -fkeep-static-consts -flimit-function-alignment -flive-range-shrinkage >> > -floop-block -floop-interchange -floop-strip-mine >> > --floop-unroll-and-jam -floop-nest-optimize >> > +-floop-unroll-and-jam -floop-unroll-and-jam-reductions >> > +-floop-nest-optimize >> > -floop-parallelize-all -flra-remat -flto >> > -flto-compression-level=@var{n} >> > -flto-toplevel-asm-heuristics >> > -flto-partition=@var{alg} -flto-incremental=@var{path} >> > @@ -15307,6 +15308,20 @@ nest this unrolls the outer loop by some factor >> > and fuses the resulting >> > multiple inner loops. This flag is enabled by default at @option{-O3}. >> > It is also enabled by @option{-fprofile-use} and @option{-fauto-profile}. >> > >> > +@opindex floop-unroll-and-jam-reductions >> > +@opindex fno-loop-unroll-and-jam-reductions >> > +@item -floop-unroll-and-jam-reductions >> > +Apply a reduction-aware variant of unroll and jam to loop nests whose >> > +inner loop carries a reduction accumulator reset every outer iteration >> > +(the GEMV / dot-product shape), producing N parallel reduction streams. >> > +N is taken from @code{TARGET_LOOP_JAM_REDUCTION_FACTOR} (no-op on >> > +targets that do not opt in) and capped by >> > +@option{--param=unroll-jam-reduction-streams}. Inner loops whose >> > +trip count is below @option{--param=unroll-jam-reduction-min-inner-iters} >> > +are left alone, because the per-stream serial chain is too short to >> > +profit from parallelism. The outer loop trip count must be a >> > +compile-time constant. Enabled whenever @option{-floop-unroll-and-jam} >> > is. >> > + >> > @opindex ftree-loop-im >> > @opindex fno-tree-loop-im >> > @item -ftree-loop-im >> > diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi >> > index ca68bdc3e615..be0771e5348a 100644 >> > --- a/gcc/doc/tm.texi >> > +++ b/gcc/doc/tm.texi >> > @@ -12653,6 +12653,17 @@ is required only when the target has special >> > constraints like maximum >> > number of memory accesses. >> > @end deftypefn >> > >> > +@deftypefn {Target Hook} unsigned TARGET_LOOP_JAM_REDUCTION_FACTOR >> > (machine_mode @var{acc_mode}, machine_mode @var{op_mode}) >> > +Return the preferred number of parallel reduction streams >> > +@option{-floop-unroll-and-jam-reductions} should produce for an inner >> > +reduction that accumulates into mode @var{acc_mode} from operands of >> > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source >> > +of the widening multiply, and is always strictly narrower than >> > +@var{acc_mode}: plain reductions with no widening multiply are >> > +filtered out before the hook is consulted. Returning less than two >> > +disables the transform for that mode pair. >> > +@end deftypefn >> > + >> > @defmac POWI_MAX_MULTS >> > If defined, this macro is interpreted as a signed integer C expression >> > that specifies the maximum number of floating point multiplications >> > diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in >> > index 532e95ef216f..027a311f310e 100644 >> > --- a/gcc/doc/tm.texi.in >> > +++ b/gcc/doc/tm.texi.in >> > @@ -8011,6 +8011,8 @@ lists. >> > >> > @hook TARGET_LOOP_UNROLL_ADJUST >> > >> > +@hook TARGET_LOOP_JAM_REDUCTION_FACTOR >> > + >> > @defmac POWI_MAX_MULTS >> > If defined, this macro is interpreted as a signed integer C expression >> > that specifies the maximum number of floating point multiplications >> > diff --git a/gcc/gimple-loop-jam.cc b/gcc/gimple-loop-jam.cc >> > index 2c4162d17a2a..527f7c6cce35 100644 >> > --- a/gcc/gimple-loop-jam.cc >> > +++ b/gcc/gimple-loop-jam.cc >> > @@ -34,12 +34,14 @@ along with GCC; see the file COPYING3. If not see >> > #include "cfgloop.h" >> > #include "tree-scalar-evolution.h" >> > #include "gimple-iterator.h" >> > +#include "cfganal.h" >> > #include "cfghooks.h" >> > #include "tree-data-ref.h" >> > #include "tree-ssa-loop-ivopts.h" >> > #include "tree-vectorizer.h" >> > #include "tree-ssa-sccvn.h" >> > #include "tree-cfgcleanup.h" >> > +#include "tree-into-ssa.h" >> > >> > /* Unroll and Jam transformation >> > >> > @@ -376,6 +378,684 @@ fuse_loops (class loop *loop) >> > } >> > } >> > >> > +/* Reduction-jam: a parallel-reduction extension of unroll-and-jam. >> > + >> > + The transformation above accepts only inner PHIs that are simple >> > + inductions. A reduction PHI cannot be fused the same way: merging >> > + the streams would chain them into a single serial accumulation, >> > + multiplying the serial dependency height instead of cutting it. >> > + This handles the GEMV / NNUE shape >> > + >> > + for (i = 0; i < M; i++) { >> > + s = bias[i]; >> > + for (j = 0; j < K; j++) s += w[i][j] * x[j]; >> > + out[i] = s; >> > + } >> > + >> > + where s starts afresh every outer iteration. Unrolling the outer >> > + loop by N leaves N sibling inner loops, each with its own reduction >> > + PHI and its own init (bias[i], bias[i+1], ...). Instead of merging >> > + them we keep all N PHIs in the fused header and hoist each sibling's >> > + init ahead of the body; the post-reduction stores stay on the >> > + sibling-exit chain, coalesced afterward. The vectorizer then emits >> > + N independent DOT_PROD / FMA chains, so the serial dependency height >> > + across the reduction drops by a factor of N. >> > + >> > + The strip factor N is the smallest of the target's >> > + TARGET_LOOP_JAM_REDUCTION_FACTOR, param_unroll_jam_reduction_streams >> > + and the data-dep safety bound, narrowed to a divisor of the outer >> > + trip count. We ignore adjust_unroll_factor's profit signal on >> > + purpose: it scores the memory refs that fusion removes, which is >> > + not what this transform is after. Targets that have not measured a >> > + win return 0 from the hook and we skip the loop. See params.opt >> > + for the tunables. */ >> > + >> > +/* Return the single non-virtual, non-simple-IV PHI in LOOP's header >> > + (the reduction accumulator), or NULL if there is none or more than >> > + one. */ >> > + >> > +static gphi * >> > +inner_reduction_phi (class loop *loop) >> > +{ >> > + gphi *result = NULL; >> > + for (gphi_iterator psi = gsi_start_phis (loop->header); >> > + !gsi_end_p (psi); gsi_next (&psi)) >> > + { >> > + gphi *phi = psi.phi (); >> > + tree op = gimple_phi_result (phi); >> > + if (virtual_operand_p (op)) >> > + continue; >> > + affine_iv iv; >> > + if (simple_iv (loop, loop, op, &iv, true)) >> > + continue; >> > + if (result) >> > + return NULL; >> > + result = phi; >> > + } >> > + return result; >> > +} >> > + >> > +/* Walk the SSA def chain from RED_PHI's latch update and return the >> > + narrowest scalar-int mode feeding a multiply inside INNER, or >> > + ACC_MODE if nothing narrower turns up. The walk follows only >> > + PLUS / MINUS / CONVERT chains so that a multiply buried in an array >> > + index (the "w * j" in "s += a[w * j]") is not mistaken for the MAC >> > + source. */ >> > + >> > +static machine_mode >> > +narrowest_reduc_input_mode (gphi *red_phi, class loop *inner, >> > + machine_mode acc_mode) >> > +{ >> > + tree update = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner)); >> > + if (TREE_CODE (update) != SSA_NAME) >> > + return acc_mode; >> > + >> > + machine_mode narrow = VOIDmode; >> > + auto_vec<tree, 8> worklist; >> > + hash_set<tree> seen; >> > + worklist.safe_push (update); >> > + >> > + while (!worklist.is_empty ()) >> > + { >> > + tree op = worklist.pop (); >> > + if (seen.add (op)) >> > + continue; >> > + gimple *def = SSA_NAME_DEF_STMT (op); >> > + if (gimple_code (def) == GIMPLE_PHI >> > + || !gimple_bb (def) >> > + || !flow_bb_inside_loop_p (inner, gimple_bb (def)) >> > + || !is_gimple_assign (def)) >> > + continue; >> > + >> > + /* Stay in the reduction value chain. Loads and pointer >> > + arithmetic feed address computation, not the accumulator. */ >> > + if (gimple_assign_load_p (def) >> > + || POINTER_TYPE_P (TREE_TYPE (gimple_assign_lhs (def)))) >> > + continue; >> > + >> > + use_operand_p use_p; >> > + ssa_op_iter oi; >> > + tree_code code = gimple_assign_rhs_code (def); >> > + if (code == MULT_EXPR || code == WIDEN_MULT_EXPR) >> > + { >> > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE) >> > + { >> > + tree mop = USE_FROM_PTR (use_p); >> > + while (TREE_CODE (mop) == SSA_NAME) >> > + { >> > + gimple *mdef = SSA_NAME_DEF_STMT (mop); >> > + if (!is_gimple_assign (mdef) >> > + || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code >> > (mdef))) >> > + break; >> > + mop = gimple_assign_rhs1 (mdef); >> > + } >> > + machine_mode m = TYPE_MODE (TREE_TYPE (mop)); >> > + if (SCALAR_INT_MODE_P (m) >> > + && (narrow == VOIDmode >> > + || known_lt (GET_MODE_PRECISION (m), >> > + GET_MODE_PRECISION (narrow)))) >> > + narrow = m; >> > + } >> > + continue; >> > + } >> > + >> > + /* Only scalar-reduction operations continue the walk. */ >> > + if (code != PLUS_EXPR >> > + && code != MINUS_EXPR >> > + && !CONVERT_EXPR_CODE_P (code)) >> > + continue; >> > + >> > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE) >> > + worklist.safe_push (USE_FROM_PTR (use_p)); >> > + } >> > + >> > + if (narrow == VOIDmode >> > + || known_ge (GET_MODE_PRECISION (narrow), >> > + GET_MODE_PRECISION (acc_mode))) >> > + return acc_mode; >> > + return narrow; >> > +} >> > + >> > +/* Move *FROM to BB, immediately before BB's terminator branch (or to >> > + the end of BB if there is none). */ >> > + >> > +static void >> > +gsi_move_to_bb_end_before_terminator (gimple_stmt_iterator *from, >> > + basic_block bb) >> > +{ >> > + gimple_stmt_iterator to = gsi_last_bb (bb); >> > + if (!gsi_end_p (to) && is_ctrl_stmt (gsi_stmt (to))) >> > + gsi_move_before (from, &to); >> > + else >> > + gsi_move_after (from, &to); >> > +} >> > + >> > +/* Propagate the single argument of the PHI at PSI into its uses and >> > + remove the PHI, leaving PSI at the next one. */ >> > + >> > +static void >> > +collapse_single_arg_phi (gphi_iterator *psi) >> > +{ >> > + gphi *phi = psi->phi (); >> > + gcc_checking_assert (gimple_phi_num_args (phi) == 1); >> > + tree result = gimple_phi_result (phi); >> > + if (!has_zero_uses (result)) >> > + replace_uses_by (result, gimple_phi_arg_def (phi, 0)); >> > + remove_phi_node (psi, true); >> > +} >> > + >> > +/* True if LOOP is a widening MAC reduction the vectorizer can turn >> > + into a DOT_PROD_EXPR: a single-BB body with one exit, loads only, a >> > + constant trip count of at least MIN_INNER_ITERS, one non-IV >> > + reduction PHI alongside a unit-step counter IV, and a widening >> > + multiply in the update chain. If OUTER is given, every IV's base >> > + and step must be invariant in it. On success *OUT_RED_PHI, >> > + *OUT_ACC_MODE and *OUT_OP_MODE receive the reduction PHI, the >> > + accumulator mode and the narrowest operand mode. */ >> > + >> > +static bool >> > +loop_is_dot_product_candidate_p (class loop *loop, >> > + unsigned HOST_WIDE_INT min_inner_iters, >> > + class loop *outer, >> > + gphi **out_red_phi, >> > + machine_mode *out_acc_mode, >> > + machine_mode *out_op_mode) >> > +{ >> > + if (loop->num_nodes != 2 || !empty_block_p (loop->latch)) >> > + return false; >> > + edge exit = single_exit (loop); >> > + if (!exit) >> > + return false; >> > + >> > + /* Header is the body; loads only. */ >> > + bool has_vuse = false; >> > + for (gimple_stmt_iterator gsi = gsi_start_bb (loop->header); >> > + !gsi_end_p (gsi); gsi_next (&gsi)) >> > + { >> > + gimple *s = gsi_stmt (gsi); >> > + if (gimple_vdef (s)) >> > + return false; >> > + if (gimple_vuse (s)) >> > + has_vuse = true; >> > + } >> > + if (!has_vuse) >> > + return false; >> > + >> > + /* Single reduction PHI with init defined outside the loop. */ >> > + gphi *red_phi = inner_reduction_phi (loop); >> > + if (!red_phi) >> > + return false; >> > + tree init = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_preheader_edge (loop)); >> > + if (!expr_invariant_in_loop_p (loop, init)) >> > + return false; >> > + >> > + /* Trip count is a known integer >= MIN_INNER_ITERS. niter.niter is >> > + the latch count (trip - 1); reject HOST_WIDE_INT_M1U separately >> > + so the +1 conversion below does not wrap. */ >> > + tree_niter_desc niter; >> > + if (!number_of_iterations_exit (loop, exit, &niter, false, true) >> > + || niter.cmp == ERROR_MARK >> > + || !integer_zerop (niter.may_be_zero) >> > + || !tree_fits_uhwi_p (niter.niter)) >> > + return false; >> > + >> > + unsigned HOST_WIDE_INT latch_count = tree_to_uhwi (niter.niter); >> > + if (latch_count == HOST_WIDE_INT_M1U >> > + || latch_count + 1 < min_inner_iters) >> > + return false; >> > + >> > + /* Plus a unit-step counter IV in the same header. When OUTER is >> > + given, every IV's step and base must also be OUTER-invariant. */ >> > + bool found_unit_iv = false; >> > + for (gphi_iterator psi = gsi_start_phis (loop->header); >> > + !gsi_end_p (psi); gsi_next (&psi)) >> > + { >> > + gphi *phi = psi.phi (); >> > + tree op = gimple_phi_result (phi); >> > + if (virtual_operand_p (op)) >> > + continue; >> > + affine_iv iv; >> > + if (!simple_iv (loop, loop, op, &iv, true)) >> > + continue; >> > + if (outer >> > + && (!expr_invariant_in_loop_p (outer, iv.step) >> > + || !expr_invariant_in_loop_p (outer, iv.base))) >> > + return false; >> > + if (iv.step && integer_onep (iv.step)) >> > + found_unit_iv = true; >> > + } >> > + if (!found_unit_iv) >> > + return false; >> > + >> > + /* Widening multiply somewhere in the reduction update chain. */ >> > + machine_mode acc_mode = TYPE_MODE (TREE_TYPE (gimple_phi_result >> > (red_phi))); >> > + machine_mode op_mode = narrowest_reduc_input_mode (red_phi, loop, >> > acc_mode); >> > + if (op_mode == acc_mode) >> > + return false; >> > + >> > + if (out_red_phi) >> > + *out_red_phi = red_phi; >> > + if (out_acc_mode) >> > + *out_acc_mode = acc_mode; >> > + if (out_op_mode) >> > + *out_op_mode = op_mode; >> > + return true; >> > +} >> > + >> > +/* Detect whether the OUTER / INNER nest has the parallel-reduction >> > + shape that unroll_jam_reduction_fuse_sibling can transform. The >> > + rejection messages below name each requirement. The only outer-body >> > + VDEF must be the post-reduction store, so that after strip-mining >> > + every replicated store sits downstream of every sibling's preheader. >> > + And the outer trip count must be a multiple of N, which keeps us off >> > + tree_unroll_loop's remainder path. */ >> > + >> > +static bool >> > +unroll_jam_reduction_possible_p (class loop *outer, class loop *inner, >> > + unsigned HOST_WIDE_INT *out_outer_trip, >> > + unsigned *out_target_factor) >> > +{ >> > + auto reject = [&] (const char *reason) -> bool >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, "reduction-jam: rejecting outer loop %d: %s\n", >> > + outer->num, reason); >> > + return false; >> > + }; >> > + >> > + /* Loop-tree relationships. The driver iterates LI_ONLY_INNERMOST, >> > + so INNER has no children; check only that OUTER has a single >> > + inner sibling (INNER) and that INNER's parent is OUTER. */ >> > + if (!inner || inner != outer->inner || inner->next >> > + || loop_outer (inner) != outer) >> > + return reject ("not a perfect 2-level nest"); >> > + >> > + if (!dominated_by_p (CDI_DOMINATORS, outer->latch, inner->header)) >> > + return reject ("inner has unsuitable dominance"); >> > + >> > + /* Outer loop must carry only simple IVs. */ >> > + for (gphi_iterator psi = gsi_start_phis (outer->header); >> > + !gsi_end_p (psi); gsi_next (&psi)) >> > + { >> > + gphi *phi = psi.phi (); >> > + tree op = gimple_phi_result (phi); >> > + if (virtual_operand_p (op)) >> > + continue; >> > + affine_iv iv; >> > + if (!simple_iv (outer, outer, op, &iv, true)) >> > + return reject ("outer carries a non-simple-IV PHI"); >> > + } >> > + >> > + /* Loop-local dot-product checks; on success red_phi and the modes >> > + feed the target-hook gate below. */ >> > + gphi *red_phi = NULL; >> > + machine_mode acc_mode = VOIDmode; >> > + machine_mode op_mode = VOIDmode; >> > + if (!loop_is_dot_product_candidate_p >> > + (inner, >> > + (unsigned HOST_WIDE_INT) >> > param_unroll_jam_reduction_min_inner_iters, >> > + outer, &red_phi, &acc_mode, &op_mode)) >> > + return reject ("inner not a dot-product candidate"); >> > + >> > + /* Opt-in gate: ask the target whether the (accumulator, operand) >> > + mode pair benefits from N parallel reduction streams. Targets >> > + that have not measured a benefit return 0, in which case we skip >> > + the transform regardless of -floop-unroll-and-jam-reductions. >> > + A return of 1 is also opt-out (single-stream = no parallelism). */ >> > + unsigned target_factor >> > + = targetm.loop_jam_reduction_factor (acc_mode, op_mode); >> > + if (target_factor < 2) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, >> > + "reduction-jam: target opted out for acc=%s op=%s\n", >> > + GET_MODE_NAME (acc_mode), GET_MODE_NAME (op_mode)); >> > + return false; >> > + } >> > + *out_target_factor = target_factor; >> > + >> > + /* The fuser drains stmts from inner->header alone, so a non-empty >> > + latch would be silently mishandled; loop_is_dot_product_candidate_p >> > + already rejects one. */ >> > + gcc_checking_assert (empty_block_p (inner->latch)); >> > + >> > + /* Require the chain from red_phi back to itself via the latch arg to >> > + be a PLUS_EXPR reduction. This limits the transform to plain >> > + summation; the N streams are independent, so each is still summed >> > + in its original order and no reassociation takes place. */ >> > + tree latch_arg = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge >> > (inner)); >> > + if (!check_reduction_path (dump_user_location_t (), inner, red_phi, >> > + latch_arg, PLUS_EXPR)) >> > + return reject ("inner reduction chain not associative-PLUS"); >> > + >> > + /* loop_is_dot_product_candidate_p confirmed inner has a single exit. >> > */ >> > + edge inner_exit = single_exit (inner); >> > + >> > + /* The outer body may carry at most one VDEF, only in the inner >> > + exit's destination BB (the post-reduction store). Also reject any >> > + non-VDEF side effect (volatile load, asm volatile, side-effecting >> > + builtin), matching bb_prevents_fusion_p on the IV-only path. >> > + >> > + The store BB must be dominated by inner->header so that after >> > + tree_unroll_loop strip-mines outer by N every replicated store >> > + sits downstream of its corresponding inner copy, and therefore >> > + downstream of every sibling's preheader. The fuser relies on >> > + this when rewiring VUSEs. */ >> > + basic_block store_bb = inner_exit->dest; >> > + if (!dominated_by_p (CDI_DOMINATORS, store_bb, inner->header)) >> > + return reject ("outer-body store not downstream of inner header"); >> > + >> > + /* The outer body must be straight-line apart from the exit test. A >> > + control-flow merge there leaves a PHI that would not dominate the >> > + fused preheader after strip-mining, and the migrator leaves body >> > + PHIs in place, so the SSA fails verification. */ >> > + edge outer_exit = single_exit (outer); >> > + if (!outer_exit) >> > + return reject ("outer loop has no single exit"); >> > + >> > + basic_block *bbs = get_loop_body (outer); >> > + const char *fail = NULL; >> > + for (unsigned i = 0; i < outer->num_nodes && !fail; ++i) >> > + { >> > + basic_block bb = bbs[i]; >> > + if (flow_bb_inside_loop_p (inner, bb)) >> > + continue; >> > + if (EDGE_COUNT (bb->succs) > 1 && bb != outer_exit->src) >> > + { >> > + fail = "outer body has control flow"; >> > + break; >> > + } >> > + unsigned vdef_count = 0; >> > + for (gimple_stmt_iterator gsi = gsi_start_bb (bb); >> > + !gsi_end_p (gsi); gsi_next (&gsi)) >> > + { >> > + gimple *s = gsi_stmt (gsi); >> > + if (gimple_has_side_effects (s)) >> > + { >> > + fail = "outer body has side effects"; >> > + break; >> > + } >> > + if (gimple_vdef (s)) >> > + vdef_count++; >> > + } >> > + if (!fail && (bb == store_bb ? vdef_count > 1 : vdef_count > 0)) >> > + fail = "outer body has extra stores"; >> > + } >> > + free (bbs); >> > + if (fail) >> > + return reject (fail); >> > + >> > + /* Outer trip count must be a known constant (N is later narrowed to >> > + a divisor of it; the remainder codepath of tree_unroll_loop is >> > + correct in principle but currently untested for this pattern). >> > + niter from number_of_iterations_exit is the latch execution count, >> > + so the trip count is niter + 1. */ >> > + tree_niter_desc outer_niter; >> > + if (!number_of_iterations_exit (outer, outer_exit, &outer_niter, >> > + false, true) >> > + || outer_niter.cmp == ERROR_MARK >> > + || !integer_zerop (outer_niter.may_be_zero) >> > + || !tree_fits_uhwi_p (outer_niter.niter)) >> > + return reject ("outer trip count unknown"); >> > + >> > + unsigned HOST_WIDE_INT outer_latch_count = tree_to_uhwi >> > (outer_niter.niter); >> > + if (outer_latch_count == HOST_WIDE_INT_M1U) >> > + return reject ("outer trip count overflows HOST_WIDE_INT"); >> > + >> > + *out_outer_trip = outer_latch_count + 1; >> > + >> > + return true; >> > +} >> > + >> > +/* Move the transitive SSA-use closure of SRC_INIT, and of every >> > + operand SRC's header body references, into DST_PREHEADER so the >> > + defs stay live once SRC's body moves. Defs that already dominate >> > + DST_PREHEADER, and defs inside SRC, are left to the body move. >> > + >> > + RPO is a reverse-postorder over CFUN's BBs, computed once by the >> > + caller and shared across the nest's siblings; walking it keeps >> > + def-use order among the migrated stmts intact. DST_LIVE_VUSE is >> > + the vop live at DST_PREHEADER, and migrated stmts are rewired to it >> > + so none of them reference a vop defined downstream. */ >> > + >> > +static void >> > +migrate_loop_invariant_defs_to_preheader (class loop *src, tree src_init, >> > + basic_block dst_preheader, >> > + const auto_vec<int> &rpo, >> > + tree dst_live_vuse) >> > +{ >> > + hash_set<gimple *> needed; >> > + auto_vec<gimple *, 16> worklist; >> > + >> > + auto consider = [&] (tree op) >> > + { >> > + if (TREE_CODE (op) != SSA_NAME) >> > + return; >> > + gimple *def = SSA_NAME_DEF_STMT (op); >> > + if (!def || is_a <gphi *> (def)) >> > + return; >> > + basic_block def_bb = gimple_bb (def); >> > + if (!def_bb >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, def_bb) >> > + || flow_bb_inside_loop_p (src, def_bb)) >> > + return; >> > + if (needed.add (def)) >> > + return; >> > + worklist.safe_push (def); >> > + }; >> > + >> > + auto consider_uses = [&] (gimple *s) >> > + { >> > + use_operand_p use_p; >> > + ssa_op_iter oi; >> > + FOR_EACH_SSA_USE_OPERAND (use_p, s, oi, SSA_OP_USE) >> > + consider (USE_FROM_PTR (use_p)); >> > + }; >> > + >> > + consider (src_init); >> > + for (gimple_stmt_iterator si = gsi_start_bb (src->header); >> > + !gsi_end_p (si); gsi_next (&si)) >> > + { >> > + gimple *s = gsi_stmt (si); >> > + if (!is_a <gphi *> (s) && !is_ctrl_stmt (s)) >> > + consider_uses (s); >> > + } >> > + while (!worklist.is_empty ()) >> > + consider_uses (worklist.pop ()); >> > + >> > + if (needed.is_empty ()) >> > + return; >> > + >> > + /* Walk BBs in reverse-postorder so that, for any def-use edge among >> > + the migrated stmts, the def is moved into DST_PREHEADER ahead of >> > + the use. */ >> > + int n = rpo.length (); >> > + for (int i = 0; i < n; ++i) >> > + { >> > + basic_block bb = BASIC_BLOCK_FOR_FN (cfun, rpo[i]); >> > + if (bb == dst_preheader >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb)) >> > + continue; >> > + for (gimple_stmt_iterator si = gsi_start_bb (bb); >> > + !gsi_end_p (si);) >> > + { >> > + gimple *stmt = gsi_stmt (si); >> > + if (!needed.contains (stmt)) >> > + { >> > + gsi_next (&si); >> > + continue; >> > + } >> > + if (dst_live_vuse && gimple_vuse (stmt)) >> > + { >> > + gimple_set_vuse (stmt, dst_live_vuse); >> > + update_stmt (stmt); >> > + } >> > + gsi_move_to_bb_end_before_terminator (&si, dst_preheader); >> > + } >> > + } >> > +} >> > + >> > +/* Pair SRC's non-reduction header PHIs with DST's by position, >> > + recording each pair in PHI_RENAME. Both are inner siblings of the >> > + same strip-mined outer loop, so their header PHI lists start with >> > + the duplicated originals in the same order. DST may have picked up >> > + extra reduction PHIs at the tail from earlier sibling fuses, so >> > + walk SRC to completion against DST's prefix. SRC_REDUC_PHI is >> > + renamed separately and virtual PHIs are skipped. */ >> > + >> > +static void >> > +match_sibling_header_phis (class loop *src, class loop *dst, >> > + gphi *src_reduc_phi, >> > + hash_map<tree, tree> &phi_rename) >> > +{ >> > + gphi_iterator spsi = gsi_start_phis (src->header); >> > + gphi_iterator dpsi = gsi_start_phis (dst->header); >> > + for (; !gsi_end_p (spsi); gsi_next (&spsi), gsi_next (&dpsi)) >> > + { >> > + gcc_checking_assert (!gsi_end_p (dpsi)); >> > + gphi *src_phi = spsi.phi (); >> > + gphi *dst_phi = dpsi.phi (); >> > + tree src_res = gimple_phi_result (src_phi); >> > + tree dst_res = gimple_phi_result (dst_phi); >> > + gcc_checking_assert (virtual_operand_p (src_res) >> > + == virtual_operand_p (dst_res)); >> > + if (src_phi != src_reduc_phi && !virtual_operand_p (src_res)) >> > + phi_rename.put (src_res, dst_res); >> > + } >> > +} >> > + >> > +/* Fuse the SRC sibling inner loop into DST, keeping SRC's reduction >> > + PHI as a parallel chain in DST's header. Both are inner siblings >> > + of an outer loop that passed unroll_jam_reduction_possible_p and >> > + was then strip-mined by tree_unroll_loop. >> > + >> > + The caller has already flattened LC-SSA across the nest (Phase 1a). >> > + Here SRC's reduction init and body operands move up to DST's >> > + preheader (Phase 1b), a fresh reduction PHI for SRC's stream goes >> > + into DST's header (Phase 2), and SRC's body follows with its PHI >> > + uses renamed (Phase 3). The caller then bypasses the drained SRC >> > + (Phase 4); its cleanup_tree_cfg folds the leftover chain of exit >> > + BBs holding the post-reduction stores, and >> > + rewrite_into_loop_closed_ssa puts LC-SSA back. */ >> > + >> > +static void >> > +unroll_jam_reduction_fuse_sibling (class loop *src, class loop *dst, >> > + const auto_vec<int> &rpo) >> > +{ >> > + gphi *src_reduc_phi = inner_reduction_phi (src); >> > + tree src_reduc_res = gimple_phi_result (src_reduc_phi); >> > + tree src_init = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi, >> > + loop_preheader_edge (src)); >> > + tree src_update = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi, >> > + loop_latch_edge (src)); >> > + >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, "reduction-jam: fusing src loop %d into dst %d\n", >> > + src->num, dst->num); >> > + >> > + basic_block dst_preheader = loop_preheader_edge (dst)->src; >> > + >> > + /* The migrated stmts must reference the vop live at DST's preheader. >> > + The matcher's only outer-body VDEF is the post-reduction store, >> > + which sits below DST after strip-mining, so nothing redefines the >> > + vop before DST: it is outer->header's virtual PHI result. A >> > + load-only outer has no such PHI; its loads already share one >> > + loop-invariant vop, so leave their VUSE alone (NULL_TREE). */ >> > + gphi *outer_vphi = get_virtual_phi (loop_outer (dst)->header); >> > + tree dst_live_vuse = outer_vphi ? gimple_phi_result (outer_vphi) : >> > NULL_TREE; >> > + >> > + /* Phase 1b. */ >> > + migrate_loop_invariant_defs_to_preheader (src, src_init, dst_preheader, >> > + rpo, dst_live_vuse); >> > + >> > + /* Phase 2: a new reduction PHI in DST's header for SRC's stream. */ >> > + tree new_name = make_ssa_name (TREE_TYPE (src_reduc_res)); >> > + gphi *new_phi = create_phi_node (new_name, dst->header); >> > + add_phi_arg (new_phi, src_init, loop_preheader_edge (dst), >> > + UNKNOWN_LOCATION); >> > + add_phi_arg (new_phi, src_update, loop_latch_edge (dst), >> > + UNKNOWN_LOCATION); >> > + >> > + /* Phase 3: rewrite uses, then move SRC's body into DST's header. */ >> > + hash_map<tree, tree> phi_rename; >> > + phi_rename.put (src_reduc_res, new_name); >> > + match_sibling_header_phis (src, dst, src_reduc_phi, phi_rename); >> > + >> > + auto_vec<gimple *, 32> src_body; >> > + for (gimple_stmt_iterator si = gsi_start_bb (src->header); >> > + !gsi_end_p (si); gsi_next (&si)) >> > + { >> > + gimple *s = gsi_stmt (si); >> > + if (is_a <gphi *> (s) || is_ctrl_stmt (s)) >> > + continue; >> > + src_body.safe_push (s); >> > + } >> > + for (gimple *stmt : src_body) >> > + { >> > + use_operand_p use_p; >> > + ssa_op_iter oi; >> > + FOR_EACH_SSA_USE_OPERAND (use_p, stmt, oi, SSA_OP_USE) >> > + { >> > + tree *map = phi_rename.get (USE_FROM_PTR (use_p)); >> > + if (map) >> > + SET_USE (use_p, *map); >> > + } >> > + if (dst_live_vuse && gimple_vuse (stmt)) >> > + { >> > + gimple_set_vuse (stmt, dst_live_vuse); >> > + update_stmt (stmt); >> > + } >> > + gimple_stmt_iterator from = gsi_for_stmt (stmt); >> > + gsi_move_to_bb_end_before_terminator (&from, dst->header); >> > + } >> > + >> > + /* The body walk above only covered uses of src_reduc_res inside >> > + src->header; catch any remaining uses (LC-SSA closures reached >> > + by the post-reduction store) before the PHI is removed. */ >> > + replace_uses_by (src_reduc_res, new_name); >> > + >> > + gphi_iterator psi_src_reduc = gsi_for_phi (src_reduc_phi); >> > + remove_phi_node (&psi_src_reduc, false); >> > +} >> > + >> > +/* Phase 4: redirect SRC's preheader edge straight past the now-empty >> > + SRC loop so cleanup_tree_cfg can remove it. SRC must have been >> > + drained by unroll_jam_reduction_fuse_sibling first. */ >> > + >> > +static void >> > +unroll_jam_reduction_bypass_sibling (class loop *src) >> > +{ >> > + edge src_preheader_e = loop_preheader_edge (src); >> > + edge src_exit_e = single_exit (src); >> > + if (!src_exit_e) >> > + return; >> > + basic_block src_exit_dest = src_exit_e->dest; >> > + >> > + /* src_exit_dest is a single-pred BB after strip-mining (the next >> > + sibling's preheader, or outer's latch for the last sibling), so >> > + its LC-SSA PHIs are single-arg. Flatten them before the redirect >> > + so no arg dangles once SRC's BBs are deleted: the driver rebuilds >> > + scalar LC-SSA afterward but not the virtual side, so a stale vop >> > + arg would otherwise survive. */ >> > + for (gphi_iterator psi = gsi_start_phis (src_exit_dest); >> > + !gsi_end_p (psi);) >> > + collapse_single_arg_phi (&psi); >> > + >> > + edge new_edge = redirect_edge_and_branch (src_preheader_e, >> > src_exit_dest); >> > + gcc_assert (new_edge); >> > + >> > + /* Delete SRC's now-unreachable BBs and the empty loop entry, >> > + mirroring fuse_loops's delete_basic_block + delete_loop pattern >> > + on the IV-only path. Without this the loop tree carries stale >> > + entries until the driver's cleanup_tree_cfg picks them up. */ >> > + basic_block *bbs = get_loop_body (src); >> > + unsigned n = src->num_nodes; >> > + for (unsigned i = 0; i < n; i++) >> > + delete_basic_block (bbs[i]); >> > + free (bbs); >> > + delete_loop (src); >> > +} >> > + >> > /* Return true if any of the access functions for dataref A >> > isn't invariant with respect to loop LOOP_NEST. */ >> > static bool >> > @@ -497,6 +1177,221 @@ adjust_unroll_factor (class loop *inner, struct >> > data_dependence_relation *ddr, >> > return ret; >> > } >> > >> > +/* Walk OUTER's data dependences, running each through >> > + adjust_unroll_factor. *UNROLL_FACTOR receives the safety upper >> > + bound (UINT_MAX for no constraint, 0 for cannot unroll), >> > + *PROFIT_UNROLL the largest distance+1 yielding a removable ref >> > + under the (a,0) pattern, *REMOVED the count of removable refs and >> > + *NUM_DATAREFS the total. Returns false if the dependences cannot >> > + be computed. >> > + >> > + When *UNROLL_FACTOR comes back 0 the other out-params are only >> > + partially populated and must not be read for profitability. */ >> > + >> > +static bool >> > +compute_jam_dependence_factors (class loop *outer, class loop *inner, >> > + unsigned *unroll_factor, >> > + unsigned *profit_unroll, >> > + unsigned *removed, >> > + unsigned *num_datarefs) >> > +{ >> > + vec<data_reference_p> datarefs = vNULL; >> > + vec<ddr_p> dependences = vNULL; >> > + auto_vec<loop_p, 3> loop_nest; >> > + >> > + if (!compute_data_dependences_for_loop (outer, true, &loop_nest, >> > + &datarefs, &dependences)) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, "Cannot analyze data dependencies\n"); >> > + free_data_refs (datarefs); >> > + free_dependence_relations (dependences); >> > + return false; >> > + } >> > + >> > + *num_datarefs = datarefs.length (); >> > + *unroll_factor = (unsigned) -1; >> > + *profit_unroll = 1; >> > + *removed = 0; >> > + >> > + if (*num_datarefs) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + dump_data_dependence_relations (dump_file, dependences); >> > + >> > + unsigned i; >> > + ddr_p ddr; >> > + FOR_EACH_VEC_ELT (dependences, i, ddr) >> > + { >> > + if (DDR_ARE_DEPENDENT (ddr) == chrec_known) >> > + continue; >> > + data_reference *dra = DDR_A (ddr); >> > + data_reference *drb = DDR_B (ddr); >> > + /* Self-dependence: WAW with non-affine access could reorder >> > + writes to the same location. */ >> > + if (dra == drb) >> > + { >> > + if (DR_IS_WRITE (dra) >> > + && !DR_ACCESS_FNS (dra).is_empty () >> > + && DDR_ARE_DEPENDENT (ddr) == chrec_dont_know) >> > + { >> > + *unroll_factor = 0; >> > + break; >> > + } >> > + continue; >> > + } >> > + /* Couldn't determine the distance vector: harmless for two >> > + reads, but a write means we cannot prove safety. */ >> > + if (!adjust_unroll_factor (inner, ddr, unroll_factor, >> > + profit_unroll, removed)) >> > + if (DR_IS_WRITE (dra) || DR_IS_WRITE (drb)) >> > + { >> > + *unroll_factor = 0; >> > + break; >> > + } >> > + } >> > + } >> > + >> > + free_dependence_relations (dependences); >> > + free_data_refs (datarefs); >> > + return true; >> > +} >> > + >> > +/* Pick a strip factor for the reduction-jam path on OUTER / INNER. >> > + adjust_unroll_factor gives the data-dep safety bound, TARGET_FACTOR >> > + the target's preference and param_unroll_jam_reduction_streams >> > + the cap. We do not use adjust_unroll_factor's profit model here: >> > + it would cap N at 1 for GEMV, because the input-load DDR has >> > + dist_v=(0,0). The result is narrowed to a divisor of OUTER_TRIP, >> > + which keeps us off tree_unroll_loop's remainder path, and validated >> > + via can_unroll_loop_p. *OUT_FACTOR receives the chosen factor and >> > + *OUT_DESC the niter. */ >> > + >> > +static bool >> > +select_reduction_jam_factor (class loop *outer, class loop *inner, >> > + unsigned HOST_WIDE_INT outer_trip, >> > + unsigned target_factor, >> > + unsigned *out_factor, >> > + tree_niter_desc *out_desc) >> > +{ >> > + /* Only the safety bound matters here; the ref-removal outputs go >> > + unused. */ >> > + unsigned unroll_factor, profit_unroll, removed, num_datarefs; >> > + if (!compute_jam_dependence_factors (outer, inner, &unroll_factor, >> > + &profit_unroll, &removed, >> > + &num_datarefs)) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, >> > + "reduction-jam: data-dep analysis failed\n"); >> > + return false; >> > + } >> > + if (unroll_factor < 2) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, >> > + "reduction-jam: data-dep safety bound below 2\n"); >> > + return false; >> > + } >> > + >> > + unsigned N = MIN (target_factor, >> > + (unsigned) param_unroll_jam_reduction_streams); >> > + if (N > unroll_factor) >> > + N = unroll_factor; >> > + unsigned initial_N = N; >> > + while (N >= 2 && (outer_trip < N || outer_trip % N != 0)) >> > + --N; >> > + if (N < 2) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, >> > + "reduction-jam: outer trip " HOST_WIDE_INT_PRINT_UNSIGNED >> > + " has no divisor in [2, %u]\n", outer_trip, initial_N); >> > + return false; >> > + } >> > + >> > + if (!can_unroll_loop_p (outer, N, out_desc)) >> > + { >> > + if (dump_file && (dump_flags & TDF_DETAILS)) >> > + fprintf (dump_file, >> > + "reduction-jam: cannot unroll outer by %u\n", N); >> > + return false; >> > + } >> > + >> > + *out_factor = N; >> > + return true; >> > +} >> > + >> > +/* Strip-mine OUTER by N and fuse the resulting N inner siblings into >> > + one body that carries N parallel reduction PHIs. Caller has >> > + already validated OUTER and chosen N + DESC via >> > + unroll_jam_reduction_possible_p / select_reduction_jam_factor. */ >> > + >> > +static void >> > +do_unroll_jam_reductions (class loop *outer, unsigned N, >> > + tree_niter_desc *desc) >> > +{ >> > + initialize_original_copy_tables (); >> > + tree_unroll_loop (outer, N, desc); >> > + free_original_copy_tables (); >> > + >> > + /* dst->next chains siblings in CFG order (per tree_unroll_loop); >> > + the fuser relies on that when migrating SSA uses. */ >> > + class loop *dst = outer->inner; >> > + auto_vec<class loop *, 4> siblings; >> > + for (class loop *src = dst->next; src; src = src->next) >> > + siblings.safe_push (src); >> > + >> > + /* RPO of CFUN, shared across all siblings: the fuser only moves >> > + stmts and removes single-arg PHIs, neither of which alters BB >> > + structure, so one walk suffices for the whole nest. */ >> > + auto_vec<int> rpo; >> > + rpo.safe_grow (last_basic_block_for_fn (cfun)); >> > + rpo.truncate (pre_and_rev_post_order_compute_fn (cfun, NULL, >> > rpo.address (), >> > + false)); >> > + >> > + /* Phase 1a (nest-wide): collapse single-arg LC-SSA PHIs that the >> > + per-sibling migrate-walk in Phase 1b would otherwise have to >> > + chase through (it skips PHIs). Sweep the BBs that do not dominate >> > + DST's preheader, i.e. the nest and everything downstream where the >> > + sibling-exit closures live. One sweep suffices: later fuses only >> > + add two-arg reduction PHIs and make no CFG change that could >> > + reintroduce single-arg closures. */ >> > + basic_block dst_preheader = loop_preheader_edge (dst)->src; >> > + calculate_dominance_info (CDI_DOMINATORS); >> > + basic_block bb; >> > + FOR_EACH_BB_FN (bb, cfun) >> > + { >> > + if (bb == dst_preheader >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb)) >> > + continue; >> > + for (gphi_iterator psi = gsi_start_phis (bb); !gsi_end_p (psi);) >> > + { >> > + gphi *phi = psi.phi (); >> > + if (gimple_phi_num_args (phi) == 1 >> > + && !virtual_operand_p (gimple_phi_result (phi))) >> > + collapse_single_arg_phi (&psi); >> > + else >> > + gsi_next (&psi); >> > + } >> > + } >> > + >> > + for (class loop *src : siblings) >> > + unroll_jam_reduction_fuse_sibling (src, dst, rpo); >> > + >> > + /* Bypass the drained siblings only after every fuse: the CFG edits >> > + invalidate dominance / SCEV state the fuse loop above relies on. */ >> > + for (class loop *src : siblings) >> > + unroll_jam_reduction_bypass_sibling (src); >> > + >> > + if (dump_enabled_p ()) >> > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS | TDF_DETAILS, >> > + find_loop_location (outer), >> > + "applying reduction unroll-and-jam with %u streams\n", >> > + siblings.length () + 1); >> > +} >> > + >> > + >> > /* Main entry point for the unroll-and-jam transformation >> > described above. */ >> > >> > @@ -516,8 +1411,36 @@ tree_loop_unroll_and_jam (void) >> > || optimize_loop_nest_for_size_p (outer)) >> > continue; >> > >> > - if (!unroll_jam_possible_p (outer, loop)) >> > - continue; >> > + if (!flag_unroll_jam || !unroll_jam_possible_p (outer, loop)) >> > + { >> > + /* The IV-only path skips reduction-shaped nests (reduction PHIs, >> > + or outer-body stores consuming the reduction result). Try the >> > + reduction-jam extension on those. */ >> > + unsigned HOST_WIDE_INT outer_trip; >> > + unsigned target_factor, N; >> > + tree_niter_desc red_desc; >> > + if (flag_unroll_jam_reductions >> > + && unroll_jam_reduction_possible_p (outer, loop, >> > + &outer_trip, >> > + &target_factor) >> > + && select_reduction_jam_factor (outer, loop, outer_trip, >> > + target_factor, >> > + &N, &red_desc)) >> > + { >> > + do_unroll_jam_reductions (outer, N, &red_desc); >> > + free_dominance_info (CDI_DOMINATORS); >> > + cleanup_tree_cfg (); >> > + /* The fuser maintains virtual SSA from the matcher's >> > + invariants, but moving SRC's body into DST leaves scalar >> > + SSA names defined in DST yet used outside it (e.g. >> > new_name's >> > + LC-SSA closures). Rebuild scalar LC-SSA so the next >> > nest's >> > + tree_unroll_loop is well-formed, and verify the result. >> > */ >> > + rewrite_into_loop_closed_ssa (NULL, 0); >> > + checking_verify_ssa (true, true); >> > + scev_reset (); >> > + } >> > + continue; >> > + } >> > >> > vec<data_reference_p> datarefs = vNULL; >> > vec<ddr_p> dependences = vNULL; >> > @@ -674,7 +1597,10 @@ public: >> > {} >> > >> > /* opt_pass methods: */ >> > - bool gate (function *) final override { return flag_unroll_jam != 0; } >> > + bool gate (function *) final override >> > + { >> > + return flag_unroll_jam != 0 || flag_unroll_jam_reductions; >> > + } >> > unsigned int execute (function *) final override; >> > >> > }; >> > diff --git a/gcc/hooks.cc b/gcc/hooks.cc >> > index 2b47fd3e50f8..976bae3c89a9 100644 >> > --- a/gcc/hooks.cc >> > +++ b/gcc/hooks.cc >> > @@ -424,6 +424,13 @@ hook_uint_mode_0 (machine_mode) >> > return 0; >> > } >> > >> > +/* Generic hook that takes two machine modes and returns an unsigned int >> > 0. */ >> > +unsigned int >> > +hook_uint_mode_mode_0 (machine_mode, machine_mode) >> > +{ >> > + return 0; >> > +} >> > + >> > /* Generic hook that takes no arguments and returns a NULL const string. >> > */ >> > const char * >> > hook_constcharptr_void_null (void) >> > diff --git a/gcc/hooks.h b/gcc/hooks.h >> > index 1ad4abb5500c..0ec0b32fa1a5 100644 >> > --- a/gcc/hooks.h >> > +++ b/gcc/hooks.h >> > @@ -119,6 +119,7 @@ extern tree >> > hook_tree_treeptr_tree_tree_int_boolptr_null (tree *, tree, tree, in >> > >> > extern unsigned hook_uint_void_0 (void); >> > extern unsigned int hook_uint_mode_0 (machine_mode); >> > +extern unsigned int hook_uint_mode_mode_0 (machine_mode, machine_mode); >> > >> > extern bool default_can_output_mi_thunk_no_vcall (const_tree, >> > HOST_WIDE_INT, >> > HOST_WIDE_INT, >> > const_tree); >> > diff --git a/gcc/params.opt b/gcc/params.opt >> > index 0c6a3e514574..e99c4fc4173f 100644 >> > --- a/gcc/params.opt >> > +++ b/gcc/params.opt >> > @@ -1263,6 +1263,14 @@ Maximum unroll factor for the unroll-and-jam >> > transformation. >> > Common Joined UInteger Var(param_unroll_jam_min_percent) Init(1) >> > IntegerRange(0, 100) Param Optimization >> > Minimum percentage of memrefs that must go away for unroll-and-jam to be >> > considered profitable. >> > >> > +-param=unroll-jam-reduction-streams= >> > +Common Joined UInteger Var(param_unroll_jam_reduction_streams) Init(4) >> > IntegerRange(2, 16) Param Optimization >> > +Target number of parallel reduction streams produced by reduction-aware >> > unroll-and-jam. >> > + >> > +-param=unroll-jam-reduction-min-inner-iters= >> > +Common Joined UInteger Var(param_unroll_jam_reduction_min_inner_iters) >> > Init(32) IntegerRange(2, 65536) Param Optimization >> > +Minimum inner-loop trip count required for reduction-aware unroll-and-jam >> > to fire. >> > + >> > -param=use-after-scope-direct-emission-threshold= >> > Common Joined UInteger >> > Var(param_use_after_scope_direct_emission_threshold) Init(256) Param >> > Optimization >> > Use direct poisoning/unpoisoning instructions for variables smaller or >> > equal to this number. >> > diff --git a/gcc/target.def b/gcc/target.def >> > index 66c20a9c9d2a..118aa9bb82ec 100644 >> > --- a/gcc/target.def >> > +++ b/gcc/target.def >> > @@ -2873,6 +2873,19 @@ number of memory accesses.", >> > unsigned, (unsigned nunroll, class loop *loop), >> > NULL) >> > >> > +DEFHOOK >> > +(loop_jam_reduction_factor, >> > + "Return the preferred number of parallel reduction streams\n\ >> > +@option{-floop-unroll-and-jam-reductions} should produce for an inner\n\ >> > +reduction that accumulates into mode @var{acc_mode} from operands of\n\ >> > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source\n\ >> > +of the widening multiply, and is always strictly narrower than\n\ >> > +@var{acc_mode}: plain reductions with no widening multiply are\n\ >> > +filtered out before the hook is consulted. Returning less than two\n\ >> > +disables the transform for that mode pair.", >> > + unsigned, (machine_mode acc_mode, machine_mode op_mode), >> > + hook_uint_mode_mode_0) >> > + >> > /* True if X is a legitimate MODE-mode immediate operand. */ >> > DEFHOOK >> > (legitimate_constant_p, >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c >> > new file mode 100644 >> > index 000000000000..d14711034bc0 >> > --- /dev/null >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c >> > @@ -0,0 +1,78 @@ >> > +/* Positive tests: the pass must strip-mine and fuse each nest below. >> > + The outer trip count of 32 is divisible by the default factor of 4. >> > + Both arms of the aarch64 hook are covered: QImode / HImode -> SImode >> > + (DOT_PROD class) in affine, and SImode -> DImode (SMLAL_DI) in >> > + affine64. */ >> > + >> > +/* { dg-do compile } */ >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions >> > -fdump-tree-unrolljam-details" } */ >> > + >> > +#include <stdint.h> >> > + >> > +#define IN_DIM 512 >> > +#define OUT_DIM 32 >> > + >> > +void >> > +affine (int32_t *__restrict out, >> > + const int8_t *weights, >> > + const int32_t *biases, >> > + const uint8_t *input) >> > +{ >> > + for (int i = 0; i < OUT_DIM; ++i) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < IN_DIM; ++j) >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +void >> > +affine64 (int64_t *__restrict out, >> > + const int32_t *weights, >> > + const int64_t *biases, >> > + const int32_t *input) >> > +{ >> > + for (int i = 0; i < OUT_DIM; ++i) >> > + { >> > + int64_t s = biases[i]; >> > + for (int j = 0; j < IN_DIM; ++j) >> > + s += (int64_t) weights[i * IN_DIM + j] * (int64_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Inner trip count at the min-inner-iters default of 32. */ >> > +void >> > +affine_min_inner (int32_t *__restrict out, >> > + const int8_t *weights, >> > + const int32_t *biases, >> > + const uint8_t *input) >> > +{ >> > + for (int i = 0; i < 32; ++i) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < 32; ++j) >> > + s += (int32_t) weights[i * 32 + j] * (int32_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Outer trip count of 9 narrows N from 4 to 3. */ >> > +void >> > +affine_narrowed (int32_t *__restrict out, >> > + const int8_t *weights, >> > + const int32_t *biases, >> > + const uint8_t *input) >> > +{ >> > + for (int i = 0; i < 9; ++i) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < 512; ++j) >> > + s += (int32_t) weights[i * 512 + j] * (int32_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* { dg-final { scan-tree-dump-times "applying reduction unroll-and-jam" >> > 4 "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 11 >> > "unrolljam" } } */ >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c >> > new file mode 100644 >> > index 000000000000..44301348a659 >> > --- /dev/null >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c >> > @@ -0,0 +1,89 @@ >> > +/* Correctness test for the reduction-aware extension of unroll-and-jam >> > + on the GEMV pattern. Compares against an in-main reference >> > + computation. */ >> > + >> > +/* { dg-do run } */ >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */ >> > + >> > +#include <stdint.h> >> > + >> > +#define IN_DIM 512 >> > +#define OUT_DIM 32 >> > + >> > +extern void abort (void); >> > + >> > +void __attribute__((noipa)) >> > +affine (int32_t *__restrict out, >> > + const int8_t *weights, >> > + const int32_t *biases, >> > + const uint8_t *input) >> > +{ >> > + for (int i = 0; i < OUT_DIM; ++i) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < IN_DIM; ++j) >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +void __attribute__((noipa)) >> > +affine_zero_init (int32_t *__restrict out, >> > + const int8_t *a, >> > + const int8_t *b) >> > +{ >> > + for (int i = 0; i < OUT_DIM; ++i) >> > + { >> > + int32_t s = 0; >> > + for (int j = 0; j < IN_DIM; ++j) >> > + s += a[i * IN_DIM + j] * b[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +int >> > +main (void) >> > +{ >> > + static int8_t weights[OUT_DIM * IN_DIM]; >> > + static uint8_t input[IN_DIM]; >> > + static int32_t biases[OUT_DIM]; >> > + static int32_t out[OUT_DIM]; >> > + static int32_t expected[OUT_DIM]; >> > + >> > + for (int k = 0; k < OUT_DIM * IN_DIM; k++) >> > + weights[k] = (k * 13) & 0x7f; >> > + for (int k = 0; k < IN_DIM; k++) >> > + input[k] = (k * 7) & 0xff; >> > + for (int k = 0; k < OUT_DIM; k++) >> > + biases[k] = k * 1000; >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < IN_DIM; j++) >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; >> > + expected[i] = s; >> > + } >> > + >> > + affine (out, weights, biases, input); >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + if (out[i] != expected[i]) >> > + abort (); >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + { >> > + int32_t s = 0; >> > + for (int j = 0; j < IN_DIM; j++) >> > + s += weights[i * IN_DIM + j] * weights[j]; >> > + expected[i] = s; >> > + } >> > + >> > + affine_zero_init (out, weights, weights); >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + if (out[i] != expected[i]) >> > + abort (); >> > + >> > + return 0; >> > +} >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c >> > new file mode 100644 >> > index 000000000000..6e4b590fefeb >> > --- /dev/null >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c >> > @@ -0,0 +1,189 @@ >> > +/* Negative tests: each function below hits one of >> > + unroll_jam_reduction_possible_p's rejection clauses and the pass >> > + must not fire on any of them. */ >> > + >> > +/* { dg-do compile } */ >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions >> > -fdump-tree-unrolljam-details" } */ >> > + >> > +#include <stdint.h> >> > + >> > +/* Outer reduction: outer carries 'sum'. */ >> > +int >> > +double_reduc (const int *a) >> > +{ >> > + int sum = 0; >> > + for (int i = 0; i < 32; i++) >> > + for (int j = 0; j < 512; j++) >> > + sum += a[i * 512 + j]; >> > + return sum; >> > +} >> > + >> > +/* Inner body has a store: inner writes a[j]. */ >> > +void >> > +inner_store (int *__restrict a, int *__restrict b, const int *c, const >> > int *d) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + { >> > + a[j] = c[j] + d[j]; >> > + s += a[j]; >> > + } >> > + b[i] = s; >> > + } >> > +} >> > + >> > +/* Inner trip count just below the min-inner-iters default of 32. */ >> > +void >> > +small_inner (int *out, const int *a) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 31; j++) >> > + s += a[i * 31 + j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Outer body stores to counter as well as out[i]. */ >> > +extern int counter; >> > +void >> > +outer_store (int *out, const int8_t *a, const int8_t *b) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + s += (int) a[i * 512 + j] * (int) b[j]; >> > + out[i] = s; >> > + counter++; >> > + } >> > +} >> > + >> > +/* FP accumulator: the multiply has no narrower scalar-int operand, so >> > + it is not a widening MAC. */ >> > +void >> > +fp_mac (float *out, const float *a, const float *b) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + float s = 0.0f; >> > + for (int j = 0; j < 512; j++) >> > + s += a[i * 512 + j] * b[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Plain SImode add reduction: no multiply in the update chain. */ >> > +void >> > +plain_add (int *out, const int *a) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + s += a[i * 512 + j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* MAX reduction: no multiply in the update chain. */ >> > +void >> > +max_reduc (int *out, const int *a) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int m = a[i * 512]; >> > + for (int j = 1; j < 512; j++) >> > + { >> > + int v = a[i * 512 + j]; >> > + m = m > v ? m : v; >> > + } >> > + out[i] = m; >> > + } >> > +} >> > + >> > +/* Inner header carries two reduction PHIs, m and s. */ >> > +void >> > +multi_phi (int *out, const int *a) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int m = a[i * 512]; >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + { >> > + int v = a[i * 512 + j]; >> > + m = m > v ? m : v; >> > + s += v; >> > + } >> > + out[i] = m + s; >> > + } >> > +} >> > + >> > +/* Parametric outer trip count: not a known UHWI. */ >> > +void >> > +parametric_outer (int *out, const int8_t *a, const int8_t *b, int n) >> > +{ >> > + for (int i = 0; i < n; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + s += (int) a[i * 512 + j] * (int) b[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Outer trip count of 7 has no divisor in [2, 4]. */ >> > +void >> > +narrow_to_one (int *__restrict out, const int8_t *a, const int8_t *b) >> > +{ >> > + for (int i = 0; i < 7; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + s += (int) a[i * 512 + j] * (int) b[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* No __restrict: data-dep analysis cannot rule out out vs b >> > + aliasing. */ >> > +void >> > +aliasing (int *out, const int8_t *a, const int8_t *b) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int s = 0; >> > + for (int j = 0; j < 512; j++) >> > + s += (int) a[i * 512 + j] * (int) b[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* Conditional accumulator init: the "?:" lowers to a store-free diamond >> > + in the outer body whose merge PHI feeds the reduction, so the body is >> > + not straight-line. */ >> > +void >> > +cond_init (int32_t *__restrict out, const int8_t *__restrict a, >> > + const int32_t *__restrict b1, const int32_t *__restrict b2, >> > + const int32_t *__restrict c) >> > +{ >> > + for (int i = 0; i < 32; i++) >> > + { >> > + int32_t s = c[i] ? b1[i] : b2[i]; >> > + for (int j = 0; j < 512; j++) >> > + s += (int32_t) a[i * 512 + j] * (int32_t) a[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* { dg-final { scan-tree-dump-not "applying reduction unroll-and-jam" >> > "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump-not "reduction-jam: fusing src loop" >> > "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump "reduction-jam: outer trip 7 has no >> > divisor in \\\[2, 4\\\]" "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: >> > outer body has extra stores" "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: >> > outer body has control flow" "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: >> > outer trip count unknown" "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump "reduction-jam: data-dep safety bound >> > below 2" "unrolljam" } } */ >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c >> > new file mode 100644 >> > index 000000000000..938f7328b049 >> > --- /dev/null >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c >> > @@ -0,0 +1,30 @@ >> > +/* The reduction-jam pass should fire even when the IV-only path is >> > + disabled via -fno-loop-unroll-and-jam. Pins the gate change in >> > + pass_loop_jam::gate and the IV-only/reduction-jam fall-through in >> > + tree_loop_unroll_and_jam. */ >> > + >> > +/* { dg-do compile } */ >> > +/* { dg-options "-O3 -mtune=ampere1 -fno-loop-unroll-and-jam >> > -floop-unroll-and-jam-reductions -fdump-tree-unrolljam-details" } */ >> > + >> > +#include <stdint.h> >> > + >> > +#define IN_DIM 512 >> > +#define OUT_DIM 32 >> > + >> > +void >> > +affine (int32_t *__restrict out, >> > + const int8_t *weights, >> > + const int32_t *biases, >> > + const uint8_t *input) >> > +{ >> > + for (int i = 0; i < OUT_DIM; ++i) >> > + { >> > + int32_t s = biases[i]; >> > + for (int j = 0; j < IN_DIM; ++j) >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j]; >> > + out[i] = s; >> > + } >> > +} >> > + >> > +/* { dg-final { scan-tree-dump "applying reduction unroll-and-jam" >> > "unrolljam" } } */ >> > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 3 >> > "unrolljam" } } */ >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c >> > new file mode 100644 >> > index 000000000000..0a198bb27ecd >> > --- /dev/null >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c >> > @@ -0,0 +1,72 @@ >> > +/* Regression test for a multi-IV mispairing in the reduction-jam >> > + sibling matcher. The inner loop carries two IVs that share the >> > + preheader entry value 0 but differ in step (j has step 1 and >> > + indexes W, p has step 2 and indexes X); the matcher must pair each >> > + source IV with its own dst counterpart, not with whichever dst PHI >> > + happens to share an entry value. >> > + >> > + The reference loop in main uses a volatile accumulator so >> > + pass_loop_jam does not recognize a reduction PHI there. Without >> > + that guard the reference would be jammed with the same mispairing >> > + as gemv_2iv, hiding the miscompile behind a symmetric bug. */ >> > + >> > +/* { dg-do run } */ >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */ >> > + >> > +#include <stdint.h> >> > + >> > +#define OUT_DIM 32 >> > +#define IN_DIM 256 >> > +#define STRIDE 512 >> > + >> > +extern void abort (void); >> > + >> > +void __attribute__((noipa)) >> > +gemv_2iv (int32_t *__restrict out, const int8_t *w, const uint8_t *x) >> > +{ >> > + for (int i = 0; i < OUT_DIM; i++) >> > + { >> > + int32_t s = 0; >> > + int p = 0; >> > + for (int j = 0; j < IN_DIM; j++) >> > + { >> > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p]; >> > + p += 2; >> > + } >> > + out[i] = s; >> > + } >> > +} >> > + >> > +int >> > +main (void) >> > +{ >> > + static int8_t w[OUT_DIM * STRIDE]; >> > + static uint8_t x[IN_DIM * 2]; >> > + static int32_t out[OUT_DIM]; >> > + static int32_t expected[OUT_DIM]; >> > + >> > + for (int k = 0; k < OUT_DIM * STRIDE; k++) >> > + w[k] = (k * 13) & 0x7f; >> > + for (int k = 0; k < IN_DIM * 2; k++) >> > + x[k] = (k * 7) & 0xff; >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + { >> > + volatile int32_t s = 0; >> > + int p = 0; >> > + for (int j = 0; j < IN_DIM; j++) >> > + { >> > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p]; >> > + p += 2; >> > + } >> > + expected[i] = s; >> > + } >> > + >> > + gemv_2iv (out, w, x); >> > + >> > + for (int i = 0; i < OUT_DIM; i++) >> > + if (out[i] != expected[i]) >> > + abort (); >> > + >> > + return 0; >> > +} >> > -- >> > 2.55.0 >> >
