The testcase snippet originally was attached to PR126385:
https://gcc.gnu.org/bugzilla/attachment.cgi?id=65122
It's int8_t and int32_t, as you correctly inferred.
#include <stdint.h>
#define IN_DIM 512
#define OUT_DIM 32
void
affine (int32_t *__restrict out,
const int8_t *weights,
const int32_t *biases,
const uint8_t *input)
{
for (int i = 0; i < OUT_DIM; ++i)
{
int32_t s = biases[i];
for (int j = 0; j < IN_DIM; ++j)
s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
out[i] = s;
}
}
On Thu, 10 Sept 2026 at 13:56, Richard Biener
<[email protected]> wrote:
>
> On Thu, Sep 10, 2026 at 1:30 PM Konstantinos Eleftheriou
> <[email protected]> wrote:
> >
> > Hi Richard,
> >
> > Supporting the weights access would remove the rejection, but not the
> > throughput problem behind it. DOT_PROD reduces within a register
> > across consecutive j: one usdot consumes 16 elements of j, and the
> > four lanes are summed once at loop exit. With i in the lanes the inner
> > loop still advances j one iteration at a time and a widening
> > multiply-accumulate performs 4 products per instruction where a
> > usdot performs 16.
> >
> > Thanks,
> > Konstantinos
> >
> > On Tue, Sep 8, 2026 at 5:17 PM Richard Biener <[email protected]>
> > wrote:
> >>
> >> On Mon, Sep 7, 2026 at 4:35 PM Konstantinos Eleftheriou
> >> <[email protected]> wrote:
> >> >
> >> > Extends pass_loop_jam with a parallel-reduction branch for the
> >> > GEMV / NNUE inference shape
> >> >
> >> > for (i) {
> >> > s = bias[i];
> >> > for (j) s += w[i][j] * x[j];
> >> > out[i] = s;
> >> > }
>
> I guess this snippet lacks types. I infer that w/x are char and s is int?
>
> And when you do not split the reduction, why is this not picked up
> as DOT_PROD and a reduction chain?
>
> I do agree that unroll-and-jam for an inner loop with a reduction is
> somewhat pointless if the dependence cannot be split (assuming
> the loop stays scalar). So in principle this makes sense if we
> want to unroll-and-jam loops with reduction(s).
>
> Richard.
>
> >> > Stock unroll_jam_possible_p rejects this nest because `s' is a
> >> > reduction PHI, not a simple induction; even if it were accepted,
> >> > stock fuse_loops would collapse the post-unroll per-sibling reduction
> >> > PHIs into one serial chain, multiplying the inner loop's serial
> >> > dependency height by N and defeating the goal of unroll-and-jam.
> >>
> >> So given we do not unroll-and-jam this, wouldn't it be better to
> >> "simply" support outer loop vectorizing for it because outer loop
> >> vectorization is exactly unroll-and-jam under the hood?
> >>
> >> > unroll_jam_reduction_possible_p detects the candidate, the existing
> >> > tree_unroll_loop strip-mines the outer loop by N, and
> >> > unroll_jam_reduction_fuse_sibling fuses the N inner siblings into one
> >> > body that retains N parallel reduction PHIs in its header and hoists
> >> > each sibling's preamble (s_k = bias[i+k]) to the fused preheader. The
> >> > post-reduction stores stay on the linear sibling-exit chain;
> >> > cleanup_tree_cfg coalesces it and rewrite_into_loop_closed_ssa restores
> >> > LC-SSA. The vectorizer then sees one inner loop with N independent
> >> > chains and emits N parallel dot-product / FMA accumulators, cutting the
> >> > serial dependency height across the reduction by N.
> >> >
> >> > N is the min of TARGET_LOOP_JAM_REDUCTION_FACTOR (opt-in, default 0
> >> > = "no benefit measured"), --param=unroll-jam-reduction-streams and
> >> > adjust_unroll_factor's data-dep safety bound, narrowed to a divisor of
> >> > the outer trip count. The hook receives the accumulator mode plus the
> >> > narrowest scalar-int mode found at a widening multiply in the update
> >> > chain; op_mode == acc_mode signals "no widening multiply", letting a
> >> > target opt in for MAC shapes only. Inner loops whose trip count is
> >> > below --param=unroll-jam-reduction-min-inner-iters are left alone:
> >> > their per-stream serial chain is too short to profit from parallelism.
> >> >
> >> > AArch64 opts in for QI/HI -> SI (DOT_PROD class) and SI -> DI
> >> > (SMLAL_DI); all other shapes (plain reductions, max/min, FP) return 0.
> >> > The per-CPU factor is sourced from a new tune_params field
> >> > loop_jam_reduction_factor rather than a hardcoded return value, so the
> >> > profitability decision lives in the tuning structure next to the other
> >> > latency-driven knobs. Cores with measured wins ship factor 4; every
> >> > other tuning leaves the field at 0 until measured. The new tests pin
> >> > -mtune to a factor-4 core so they exercise that path independently of
> >> > the configured default tuning.
> >> >
> >> > FP reductions are deferred to outer-loop vectorization's lane-broadcast
> >> > FMLA path. The chained-reduction case (single accumulator carried
> >> > across outer iterations) is left as a follow-up.
> >> >
> >> > ~8% speedup on SPEC CPU 2026 706.stockfish_r.
> >> >
> >> > gcc/ChangeLog:
> >> >
> >> > * common.opt (floop-unroll-and-jam-reductions): New option,
> >> > EnabledBy(floop-unroll-and-jam).
> >> > * config/aarch64/aarch64-protos.h (struct tune_params):
> >> > Add loop_jam_reduction_factor.
> >> > * config/aarch64/aarch64.cc (aarch64_loop_jam_reduction_factor):
> >> > New function; source factor from aarch64_tune_params and opt
> >> > in for QI/HI->SI and SI->DI MAC shapes only.
> >> > (TARGET_LOOP_JAM_REDUCTION_FACTOR): Define.
> >> > * config/aarch64/tuning_models/a64fx.h: Default
> >> > loop_jam_reduction_factor to 0 (opt-out).
> >> > * config/aarch64/tuning_models/ampere1.h: Set
> >> > loop_jam_reduction_factor to 4.
> >> > * config/aarch64/tuning_models/ampere1a.h: Likewise.
> >> > * config/aarch64/tuning_models/ampere1b.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexa320.h: Default
> >> > loop_jam_reduction_factor to 0 (opt-out).
> >> > * config/aarch64/tuning_models/cortexa35.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexa53.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexa57.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexa72.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexa73.h: Likewise.
> >> > * config/aarch64/tuning_models/cortexx925.h: Likewise.
> >> > * config/aarch64/tuning_models/emag.h: Likewise.
> >> > * config/aarch64/tuning_models/exynosm1.h: Likewise.
> >> > * config/aarch64/tuning_models/fujitsu_monaka.h: Likewise.
> >> > * config/aarch64/tuning_models/generic.h: Likewise.
> >> > * config/aarch64/tuning_models/generic_armv8_a.h: Likewise.
> >> > * config/aarch64/tuning_models/generic_armv9_a.h: Likewise.
> >> > * config/aarch64/tuning_models/hip12.h: Likewise.
> >> > * config/aarch64/tuning_models/neoverse512tvb.h: Likewise.
> >> > * config/aarch64/tuning_models/neoversen1.h: Set
> >> > loop_jam_reduction_factor to 4.
> >> > * config/aarch64/tuning_models/neoversen2.h: Default
> >> > loop_jam_reduction_factor to 0 (opt-out).
> >> > * config/aarch64/tuning_models/neoversen3.h: Likewise.
> >> > * config/aarch64/tuning_models/neoversev1.h: Likewise.
> >> > * config/aarch64/tuning_models/neoversev2.h: Likewise.
> >> > * config/aarch64/tuning_models/neoversev3.h: Likewise.
> >> > * config/aarch64/tuning_models/neoversev3ae.h: Likewise.
> >> > * config/aarch64/tuning_models/olympus.h: Likewise.
> >> > * config/aarch64/tuning_models/qdf24xx.h: Likewise.
> >> > * config/aarch64/tuning_models/saphira.h: Likewise.
> >> > * config/aarch64/tuning_models/thunderx.h: Likewise.
> >> > * config/aarch64/tuning_models/thunderx2t99.h: Likewise.
> >> > * config/aarch64/tuning_models/thunderx3t110.h: Likewise.
> >> > * config/aarch64/tuning_models/thunderxt88.h: Likewise.
> >> > * config/aarch64/tuning_models/tsv110.h: Likewise.
> >> > * config/aarch64/tuning_models/xgene1.h: Likewise.
> >> > * doc/invoke.texi (-floop-unroll-and-jam-reductions): Document.
> >> > * doc/tm.texi: Regenerate.
> >> > * doc/tm.texi.in (TARGET_LOOP_JAM_REDUCTION_FACTOR): Add hook
> >> > reference.
> >> > * gimple-loop-jam.cc: Include cfganal.h and tree-into-ssa.h.
> >> > (inner_reduction_phi): New helper.
> >> > (narrowest_reduc_input_mode): New helper that walks the
> >> > reduction update chain for widening multiplies.
> >> > (gsi_move_to_bb_end_before_terminator): New helper.
> >> > (collapse_single_arg_phi): New helper.
> >> > (loop_is_dot_product_candidate_p): New static helper.
> >> > (unroll_jam_reduction_possible_p): New candidate detector;
> >> > calls loop_is_dot_product_candidate_p and layers outer-loop
> >> > guards on top.
> >> > (migrate_loop_invariant_defs_to_preheader): New helper that
> >> > hoists the SSA-use closure of a sibling's reduction init and
> >> > body operands to the destination preheader in RPO.
> >> > (match_sibling_header_phis): New helper pairing sibling non-
> >> > reduction PHIs positionally.
> >> > (unroll_jam_reduction_fuse_sibling): New per-sibling fuser.
> >> > (unroll_jam_reduction_bypass_sibling): New CFG bypass for the
> >> > drained source sibling.
> >> > (compute_jam_dependence_factors): New helper. Walks OUTER's
> >> > data dependences and accumulates the unroll-factor safety
> >> > bound, ref-removal profit signal, removed-ref count, and total
> >> > dataref count.
> >> > (select_reduction_jam_factor): New helper combining target
> >> > factor, param cap and data-dep safety bound, narrowed to a
> >> > divisor of the outer trip count. Emit a TDF_DETAILS note
> >> > when the outer trip has no divisor in [2, factor cap].
> >> > (do_unroll_jam_reductions): New driver helper.
> >> > (tree_loop_unroll_and_jam): Try the reduction-jam path on
> >> > nests the IV-only path skips; cleanup CFG, rebuild LC-SSA and
> >> > reset SCEV per transformed nest.
> >> > (pass_loop_jam::gate): Also enable for
> >> > flag_unroll_jam_reductions.
> >> > * hooks.cc (hook_uint_mode_mode_0): New default hook.
> >> > * hooks.h (hook_uint_mode_mode_0): Declare.
> >> > * params.opt (unroll-jam-reduction-streams): New param.
> >> > (unroll-jam-reduction-min-inner-iters): New param.
> >> > * target.def (loop_jam_reduction_factor): New target hook.
> >> >
> >> > gcc/testsuite/ChangeLog:
> >> >
> >> > * gcc.target/aarch64/unroll-jam-reductions-1.c: New test.
> >> > * gcc.target/aarch64/unroll-jam-reductions-2.c: New test.
> >> > * gcc.target/aarch64/unroll-jam-reductions-3.c: New test.
> >> > * gcc.target/aarch64/unroll-jam-reductions-4.c: New test.
> >> > * gcc.target/aarch64/unroll-jam-reductions-5.c: New test.
> >> >
> >> > Co-Authored-By: Philipp Tomsich <[email protected]>
> >> > Signed-off-by: Konstantinos Eleftheriou
> >> > <[email protected]>
> >> > ---
> >> >
> >> > gcc/common.opt | 5 +
> >> > gcc/config/aarch64/aarch64-protos.h | 8 +
> >> > gcc/config/aarch64/aarch64.cc | 29 +
> >> > gcc/config/aarch64/tuning_models/a64fx.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/ampere1.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/ampere1a.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/ampere1b.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa320.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa35.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa53.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa57.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa72.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexa73.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/cortexx925.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/emag.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/exynosm1.h | 3 +-
> >> > .../aarch64/tuning_models/fujitsu_monaka.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/generic.h | 3 +-
> >> > .../aarch64/tuning_models/generic_armv8_a.h | 3 +-
> >> > .../aarch64/tuning_models/generic_armv9_a.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/hip12.h | 3 +-
> >> > .../aarch64/tuning_models/neoverse512tvb.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversen1.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversen2.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversen3.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversev1.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversev2.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/neoversev3.h | 3 +-
> >> > .../aarch64/tuning_models/neoversev3ae.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/olympus.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/qdf24xx.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/saphira.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/thunderx.h | 3 +-
> >> > .../aarch64/tuning_models/thunderx2t99.h | 3 +-
> >> > .../aarch64/tuning_models/thunderx3t110.h | 3 +-
> >> > .../aarch64/tuning_models/thunderxt88.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/tsv110.h | 3 +-
> >> > gcc/config/aarch64/tuning_models/xgene1.h | 3 +-
> >> > gcc/doc/invoke.texi | 17 +-
> >> > gcc/doc/tm.texi | 11 +
> >> > gcc/doc/tm.texi.in | 2 +
> >> > gcc/gimple-loop-jam.cc | 932 +++++++++++++++++-
> >> > gcc/hooks.cc | 7 +
> >> > gcc/hooks.h | 1 +
> >> > gcc/params.opt | 8 +
> >> > gcc/target.def | 13 +
> >> > .../aarch64/unroll-jam-reductions-1.c | 78 ++
> >> > .../aarch64/unroll-jam-reductions-2.c | 89 ++
> >> > .../aarch64/unroll-jam-reductions-3.c | 189 ++++
> >> > .../aarch64/unroll-jam-reductions-4.c | 30 +
> >> > .../aarch64/unroll-jam-reductions-5.c | 72 ++
> >> > 51 files changed, 1557 insertions(+), 39 deletions(-)
> >> > create mode 100644
> >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> >> > create mode 100644
> >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> >> > create mode 100644
> >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> >> > create mode 100644
> >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> >> > create mode 100644
> >> > gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> >> >
> >> > diff --git a/gcc/common.opt b/gcc/common.opt
> >> > index 69a55b8d40d6..634b63569c67 100644
> >> > --- a/gcc/common.opt
> >> > +++ b/gcc/common.opt
> >> > @@ -1928,6 +1928,11 @@ floop-unroll-and-jam
> >> > Common Var(flag_unroll_jam) Optimization
> >> > Perform unroll-and-jam on loops.
> >> >
> >> > +floop-unroll-and-jam-reductions
> >> > +Common Var(flag_unroll_jam_reductions) Optimization
> >> > EnabledBy(floop-unroll-and-jam)
> >> > +Perform reduction-aware unroll-and-jam on loops with inner reductions
> >> > +to enable multi-stream parallelism during vectorization.
> >> > +
> >> > fgnu-tm
> >> > Common Var(flag_tm)
> >> > Enable support for GNU transactional memory.
> >> > diff --git a/gcc/config/aarch64/aarch64-protos.h
> >> > b/gcc/config/aarch64/aarch64-protos.h
> >> > index e8ae3d427949..88237e243738 100644
> >> > --- a/gcc/config/aarch64/aarch64-protos.h
> >> > +++ b/gcc/config/aarch64/aarch64-protos.h
> >> > @@ -590,6 +590,14 @@ struct tune_params
> >> >
> >> > /* Dispatch constraints for instruction scheduling. */
> >> > const struct dispatch_constraint_info *dispatch_constraints;
> >> > +
> >> > + /* Preferred number of parallel reduction streams for
> >> > + -floop-unroll-and-jam-reductions on this core. Zero opts out;
> >> > + two or more enables the transform on reduction nests whose
> >> > + (acc_mode, op_mode) pair the target hook accepts. Cores with a
> >> > + two-wide int/FP pipe and several-cycle MAC latency typically
> >> > + want 4. */
> >> > + unsigned int loop_jam_reduction_factor;
> >> > };
> >> >
> >> > /* Classifies an address.
> >> > diff --git a/gcc/config/aarch64/aarch64.cc
> >> > b/gcc/config/aarch64/aarch64.cc
> >> > index ccb4f5fbb10d..546912534499 100644
> >> > --- a/gcc/config/aarch64/aarch64.cc
> >> > +++ b/gcc/config/aarch64/aarch64.cc
> >> > @@ -17402,6 +17402,32 @@ aarch64_sched_issue_rate (void)
> >> > return aarch64_tune_params.issue_rate;
> >> > }
> >> >
> >> > +/* Implement TARGET_LOOP_JAM_REDUCTION_FACTOR. Return the preferred
> >> > + number of parallel reduction streams for
> >> > + -floop-unroll-and-jam-reductions on a reduction accumulating into
> >> > + ACC_MODE from operands of OP_MODE. We opt in only for the widening
> >> > + MAC shapes the vectorizer lowers to a latency-bound
> >> > + DOT_PROD / SMLAL chain:
> >> > + * QImode / HImode -> SImode (DOT_PROD-class, USDOT / SDOT / UDOT)
> >> > + * SImode -> DImode (SMLAL_DI)
> >> > + The stream count comes from the per-core tuning structure; cores
> >> > + that have not been measured leave loop_jam_reduction_factor at 0,
> >> > + which suppresses the transform. */
> >> > +
> >> > +static unsigned int
> >> > +aarch64_loop_jam_reduction_factor (machine_mode acc_mode,
> >> > + machine_mode op_mode)
> >> > +{
> >> > + unsigned int factor = aarch64_tune_params.loop_jam_reduction_factor;
> >> > + if (factor < 2 || !SCALAR_INT_MODE_P (acc_mode))
> >> > + return 0;
> >> > +
> >> > + if ((acc_mode == SImode && (op_mode == QImode || op_mode == HImode))
> >> > + || (acc_mode == DImode && op_mode == SImode))
> >> > + return factor;
> >> > + return 0;
> >> > +}
> >> > +
> >> > /* Implement TARGET_SCHED_VARIABLE_ISSUE. */
> >> > static int
> >> > aarch64_sched_variable_issue (FILE *, int, rtx_insn *insn, int more)
> >> > @@ -34491,6 +34517,9 @@ aarch64_libgcc_floating_mode_supported_p
> >> > #undef TARGET_SCHED_ISSUE_RATE
> >> > #define TARGET_SCHED_ISSUE_RATE aarch64_sched_issue_rate
> >> >
> >> > +#undef TARGET_LOOP_JAM_REDUCTION_FACTOR
> >> > +#define TARGET_LOOP_JAM_REDUCTION_FACTOR
> >> > aarch64_loop_jam_reduction_factor
> >> > +
> >> > #undef TARGET_SCHED_VARIABLE_ISSUE
> >> > #define TARGET_SCHED_VARIABLE_ISSUE aarch64_sched_variable_issue
> >> >
> >> > diff --git a/gcc/config/aarch64/tuning_models/a64fx.h
> >> > b/gcc/config/aarch64/tuning_models/a64fx.h
> >> > index c3222960f175..171e64ce23ad 100644
> >> > --- a/gcc/config/aarch64/tuning_models/a64fx.h
> >> > +++ b/gcc/config/aarch64/tuning_models/a64fx.h
> >> > @@ -166,7 +166,8 @@ static const struct tune_params a64fx_tunings =
> >> > &a64fx_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_A64FX. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1.h
> >> > b/gcc/config/aarch64/tuning_models/ampere1.h
> >> > index e2a42f585dd9..0057db3765b4 100644
> >> > --- a/gcc/config/aarch64/tuning_models/ampere1.h
> >> > +++ b/gcc/config/aarch64/tuning_models/ampere1.h
> >> > @@ -136,7 +136,8 @@ static const struct tune_params ampere1_tunings =
> >> > &ere1_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 4 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_AMPERE1. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1a.h
> >> > b/gcc/config/aarch64/tuning_models/ampere1a.h
> >> > index 0954ac551a0b..1350c0f3b5c5 100644
> >> > --- a/gcc/config/aarch64/tuning_models/ampere1a.h
> >> > +++ b/gcc/config/aarch64/tuning_models/ampere1a.h
> >> > @@ -58,7 +58,8 @@ static const struct tune_params ampere1a_tunings =
> >> > &ere1_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 4 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_AMPERE1A. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/ampere1b.h
> >> > b/gcc/config/aarch64/tuning_models/ampere1b.h
> >> > index 8ba434d27bbf..0fc6eb36bd2a 100644
> >> > --- a/gcc/config/aarch64/tuning_models/ampere1b.h
> >> > +++ b/gcc/config/aarch64/tuning_models/ampere1b.h
> >> > @@ -137,7 +137,8 @@ static const struct tune_params ampere1b_tunings =
> >> > &ere1b_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 4 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_AMPERE1B */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa320.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa320.h
> >> > index d5e1f2b29938..89ed24360008 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa320.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa320.h
> >> > @@ -189,7 +189,8 @@ static const struct tune_params cortexa320_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXA320. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa35.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa35.h
> >> > index fd2fc47f5e23..ff9eeb406799 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa35.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa35.h
> >> > @@ -57,7 +57,8 @@ static const struct tune_params cortexa35_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXA35. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa53.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa53.h
> >> > index e71cc1b26afb..13dbb152c6e9 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa53.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa53.h
> >> > @@ -66,7 +66,8 @@ static const struct tune_params cortexa53_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXA53. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa57.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa57.h
> >> > index 4f3ec60ba275..c78c15198158 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa57.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa57.h
> >> > @@ -103,7 +103,8 @@ static const struct tune_params cortexa57_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXA57. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa72.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa72.h
> >> > index df0fcf110867..8a4e5e9d8a36 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa72.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa72.h
> >> > @@ -55,7 +55,8 @@ static const struct tune_params cortexa72_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXA72. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexa73.h
> >> > b/gcc/config/aarch64/tuning_models/cortexa73.h
> >> > index c871b454d40f..40d46f23ae82 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexa73.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexa73.h
> >> > @@ -56,7 +56,8 @@ static const struct tune_params cortexa73_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> >
> >> > diff --git a/gcc/config/aarch64/tuning_models/cortexx925.h
> >> > b/gcc/config/aarch64/tuning_models/cortexx925.h
> >> > index 2a73b3697065..b99d6b4300e5 100644
> >> > --- a/gcc/config/aarch64/tuning_models/cortexx925.h
> >> > +++ b/gcc/config/aarch64/tuning_models/cortexx925.h
> >> > @@ -227,7 +227,8 @@ static const struct tune_params cortexx925_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_CORTEXX925. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/emag.h
> >> > b/gcc/config/aarch64/tuning_models/emag.h
> >> > index 82bbc554f41b..6e87d28173b2 100644
> >> > --- a/gcc/config/aarch64/tuning_models/emag.h
> >> > +++ b/gcc/config/aarch64/tuning_models/emag.h
> >> > @@ -55,7 +55,8 @@ static const struct tune_params emag_tunings =
> >> > &xgene1_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_EMAG. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/exynosm1.h
> >> > b/gcc/config/aarch64/tuning_models/exynosm1.h
> >> > index a561ccf6fa71..928f3d4f3014 100644
> >> > --- a/gcc/config/aarch64/tuning_models/exynosm1.h
> >> > +++ b/gcc/config/aarch64/tuning_models/exynosm1.h
> >> > @@ -139,7 +139,8 @@ static const struct tune_params exynosm1_tunings =
> >> > &exynosm1_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_EXYNOSM1. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> >> > b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> >> > index 71549bd0ea70..70d71bf253de 100644
> >> > --- a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> >> > +++ b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> >> > @@ -59,7 +59,8 @@ static const struct tune_params fujitsu_monaka_tunings
> >> > =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_FUJITSU_MONAKA. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/generic.h
> >> > b/gcc/config/aarch64/tuning_models/generic.h
> >> > index af013e478d45..71dbebeb47c0 100644
> >> > --- a/gcc/config/aarch64/tuning_models/generic.h
> >> > +++ b/gcc/config/aarch64/tuning_models/generic.h
> >> > @@ -188,7 +188,8 @@ static const struct tune_params generic_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_GENERIC. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> >> > b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> >> > index 29cf3a5a7c7a..15900a7ba733 100644
> >> > --- a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> >> > +++ b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> >> > @@ -180,7 +180,8 @@ static const struct tune_params
> >> > generic_armv8_a_tunings =
> >> > &generic_armv8_a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_GENERIC_ARMV8_A. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> >> > b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> >> > index 05e777d20efb..7c1b78f88a53 100644
> >> > --- a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> >> > +++ b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> >> > @@ -255,7 +255,8 @@ static const struct tune_params
> >> > generic_armv9_a_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_GENERIC_ARMV9_A. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/hip12.h
> >> > b/gcc/config/aarch64/tuning_models/hip12.h
> >> > index cfb204b2e533..2d5c107d71d7 100644
> >> > --- a/gcc/config/aarch64/tuning_models/hip12.h
> >> > +++ b/gcc/config/aarch64/tuning_models/hip12.h
> >> > @@ -221,7 +221,8 @@ static const struct tune_params hip12_tunings =
> >> > &generic_armv8_a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_HIP12. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> >> > b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> >> > index d6e2c2e8120d..a128d5ed7b3a 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> >> > @@ -162,7 +162,8 @@ static const struct tune_params
> >> > neoverse512tvb_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSE512TVB. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen1.h
> >> > b/gcc/config/aarch64/tuning_models/neoversen1.h
> >> > index 87c7fbcb7f22..9b2ec9322bba 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversen1.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversen1.h
> >> > @@ -55,7 +55,8 @@ static const struct tune_params neoversen1_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 4 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEN1. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen2.h
> >> > b/gcc/config/aarch64/tuning_models/neoversen2.h
> >> > index 21597e6b50f8..e1648c88eb78 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversen2.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversen2.h
> >> > @@ -224,7 +224,8 @@ static const struct tune_params neoversen2_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEN2. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversen3.h
> >> > b/gcc/config/aarch64/tuning_models/neoversen3.h
> >> > index 37f9819148f6..8796914772b4 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversen3.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversen3.h
> >> > @@ -223,7 +223,8 @@ static const struct tune_params neoversen3_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEN3. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev1.h
> >> > b/gcc/config/aarch64/tuning_models/neoversev1.h
> >> > index 253f11e87a68..7f93e4092550 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversev1.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversev1.h
> >> > @@ -233,7 +233,8 @@ static const struct tune_params neoversev1_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> >
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev2.h
> >> > b/gcc/config/aarch64/tuning_models/neoversev2.h
> >> > index 6df0cc444b80..8ceb7d5538fb 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversev2.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversev2.h
> >> > @@ -364,7 +364,8 @@ static const struct tune_params neoversev2_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - &neoversev2_dispatch_constraint_info /* dispatch_constraints. */
> >> > + &neoversev2_dispatch_constraint_info, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEV2. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev3.h
> >> > b/gcc/config/aarch64/tuning_models/neoversev3.h
> >> > index 7e0144032dc6..ea840d2cbb7c 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversev3.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversev3.h
> >> > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3_tunings =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEV3. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/neoversev3ae.h
> >> > b/gcc/config/aarch64/tuning_models/neoversev3ae.h
> >> > index bbe4dc547015..fe39c7b67c9d 100644
> >> > --- a/gcc/config/aarch64/tuning_models/neoversev3ae.h
> >> > +++ b/gcc/config/aarch64/tuning_models/neoversev3ae.h
> >> > @@ -225,7 +225,8 @@ static const struct tune_params neoversev3ae_tunings
> >> > =
> >> > &generic_armv9a_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_NEOVERSEV3AE. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/olympus.h
> >> > b/gcc/config/aarch64/tuning_models/olympus.h
> >> > index 1e701722e224..ab1ba2464d23 100644
> >> > --- a/gcc/config/aarch64/tuning_models/olympus.h
> >> > +++ b/gcc/config/aarch64/tuning_models/olympus.h
> >> > @@ -565,7 +565,8 @@ static struct tune_params olympus_tunings =
> >> > &olympus_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - &olympus_dispatch_constraint_info /* dispatch_constraints. */
> >> > + &olympus_dispatch_constraint_info, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_OLYMPUS. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/qdf24xx.h
> >> > b/gcc/config/aarch64/tuning_models/qdf24xx.h
> >> > index a15a4fc56897..73eb0b8c48cf 100644
> >> > --- a/gcc/config/aarch64/tuning_models/qdf24xx.h
> >> > +++ b/gcc/config/aarch64/tuning_models/qdf24xx.h
> >> > @@ -131,7 +131,8 @@ static const struct tune_params qdf24xx_tunings =
> >> > &qdf24xx_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_QDF24XX. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/saphira.h
> >> > b/gcc/config/aarch64/tuning_models/saphira.h
> >> > index 369d3618f7f1..582c4517fcdc 100644
> >> > --- a/gcc/config/aarch64/tuning_models/saphira.h
> >> > +++ b/gcc/config/aarch64/tuning_models/saphira.h
> >> > @@ -57,7 +57,8 @@ static const struct tune_params saphira_tunings =
> >> > &generic_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_SAPHIRA. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx.h
> >> > b/gcc/config/aarch64/tuning_models/thunderx.h
> >> > index ed8de307276a..a275659c0b9e 100644
> >> > --- a/gcc/config/aarch64/tuning_models/thunderx.h
> >> > +++ b/gcc/config/aarch64/tuning_models/thunderx.h
> >> > @@ -112,7 +112,8 @@ static const struct tune_params thunderx_tunings =
> >> > &thunderx_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_THUNDERX. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx2t99.h
> >> > b/gcc/config/aarch64/tuning_models/thunderx2t99.h
> >> > index f6be657ac9c3..5afbc21ef40c 100644
> >> > --- a/gcc/config/aarch64/tuning_models/thunderx2t99.h
> >> > +++ b/gcc/config/aarch64/tuning_models/thunderx2t99.h
> >> > @@ -132,7 +132,8 @@ static const struct tune_params thunderx2t99_tunings
> >> > =
> >> > &thunderx2t99_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_THUNDERX2T99. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/thunderx3t110.h
> >> > b/gcc/config/aarch64/tuning_models/thunderx3t110.h
> >> > index cdcbec680945..0d7f29bee209 100644
> >> > --- a/gcc/config/aarch64/tuning_models/thunderx3t110.h
> >> > +++ b/gcc/config/aarch64/tuning_models/thunderx3t110.h
> >> > @@ -131,7 +131,8 @@ static const struct tune_params
> >> > thunderx3t110_tunings =
> >> > &thunderx3t110_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_THUNDERX3T110. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/thunderxt88.h
> >> > b/gcc/config/aarch64/tuning_models/thunderxt88.h
> >> > index 44a1eac59a96..d0c39c07c88a 100644
> >> > --- a/gcc/config/aarch64/tuning_models/thunderxt88.h
> >> > +++ b/gcc/config/aarch64/tuning_models/thunderxt88.h
> >> > @@ -67,7 +67,8 @@ static const struct tune_params thunderxt88_tunings =
> >> > &thunderxt88_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_THUNDERXT88. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/tsv110.h
> >> > b/gcc/config/aarch64/tuning_models/tsv110.h
> >> > index 366f2b3e2323..0800bdc59ced 100644
> >> > --- a/gcc/config/aarch64/tuning_models/tsv110.h
> >> > +++ b/gcc/config/aarch64/tuning_models/tsv110.h
> >> > @@ -132,7 +132,8 @@ static const struct tune_params tsv110_tunings =
> >> > &tsv110_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_TSV110. */
> >> > diff --git a/gcc/config/aarch64/tuning_models/xgene1.h
> >> > b/gcc/config/aarch64/tuning_models/xgene1.h
> >> > index 0d4aaf80b61a..5fd5e06932c3 100644
> >> > --- a/gcc/config/aarch64/tuning_models/xgene1.h
> >> > +++ b/gcc/config/aarch64/tuning_models/xgene1.h
> >> > @@ -140,7 +140,8 @@ static const struct tune_params xgene1_tunings =
> >> > &xgene1_prefetch_tune,
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> >> > AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> >> > - nullptr /* dispatch_constraints. */
> >> > + nullptr, /* dispatch_constraints. */
> >> > + 0 /* loop_jam_reduction_factor. */
> >> > };
> >> >
> >> > #endif /* GCC_AARCH64_H_XGENE1. */
> >> > diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> >> > index a74e99081d35..9f3215deb942 100644
> >> > --- a/gcc/doc/invoke.texi
> >> > +++ b/gcc/doc/invoke.texi
> >> > @@ -641,7 +641,8 @@ Objective-C and Objective-C++ Dialects}.
> >> > -fivopts -fkeep-inline-functions -fkeep-static-functions
> >> > -fkeep-static-consts -flimit-function-alignment -flive-range-shrinkage
> >> > -floop-block -floop-interchange -floop-strip-mine
> >> > --floop-unroll-and-jam -floop-nest-optimize
> >> > +-floop-unroll-and-jam -floop-unroll-and-jam-reductions
> >> > +-floop-nest-optimize
> >> > -floop-parallelize-all -flra-remat -flto
> >> > -flto-compression-level=@var{n}
> >> > -flto-toplevel-asm-heuristics
> >> > -flto-partition=@var{alg} -flto-incremental=@var{path}
> >> > @@ -15307,6 +15308,20 @@ nest this unrolls the outer loop by some factor
> >> > and fuses the resulting
> >> > multiple inner loops. This flag is enabled by default at @option{-O3}.
> >> > It is also enabled by @option{-fprofile-use} and
> >> > @option{-fauto-profile}.
> >> >
> >> > +@opindex floop-unroll-and-jam-reductions
> >> > +@opindex fno-loop-unroll-and-jam-reductions
> >> > +@item -floop-unroll-and-jam-reductions
> >> > +Apply a reduction-aware variant of unroll and jam to loop nests whose
> >> > +inner loop carries a reduction accumulator reset every outer iteration
> >> > +(the GEMV / dot-product shape), producing N parallel reduction streams.
> >> > +N is taken from @code{TARGET_LOOP_JAM_REDUCTION_FACTOR} (no-op on
> >> > +targets that do not opt in) and capped by
> >> > +@option{--param=unroll-jam-reduction-streams}. Inner loops whose
> >> > +trip count is below
> >> > @option{--param=unroll-jam-reduction-min-inner-iters}
> >> > +are left alone, because the per-stream serial chain is too short to
> >> > +profit from parallelism. The outer loop trip count must be a
> >> > +compile-time constant. Enabled whenever @option{-floop-unroll-and-jam}
> >> > is.
> >> > +
> >> > @opindex ftree-loop-im
> >> > @opindex fno-tree-loop-im
> >> > @item -ftree-loop-im
> >> > diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
> >> > index ca68bdc3e615..be0771e5348a 100644
> >> > --- a/gcc/doc/tm.texi
> >> > +++ b/gcc/doc/tm.texi
> >> > @@ -12653,6 +12653,17 @@ is required only when the target has special
> >> > constraints like maximum
> >> > number of memory accesses.
> >> > @end deftypefn
> >> >
> >> > +@deftypefn {Target Hook} unsigned TARGET_LOOP_JAM_REDUCTION_FACTOR
> >> > (machine_mode @var{acc_mode}, machine_mode @var{op_mode})
> >> > +Return the preferred number of parallel reduction streams
> >> > +@option{-floop-unroll-and-jam-reductions} should produce for an inner
> >> > +reduction that accumulates into mode @var{acc_mode} from operands of
> >> > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source
> >> > +of the widening multiply, and is always strictly narrower than
> >> > +@var{acc_mode}: plain reductions with no widening multiply are
> >> > +filtered out before the hook is consulted. Returning less than two
> >> > +disables the transform for that mode pair.
> >> > +@end deftypefn
> >> > +
> >> > @defmac POWI_MAX_MULTS
> >> > If defined, this macro is interpreted as a signed integer C expression
> >> > that specifies the maximum number of floating point multiplications
> >> > diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
> >> > index 532e95ef216f..027a311f310e 100644
> >> > --- a/gcc/doc/tm.texi.in
> >> > +++ b/gcc/doc/tm.texi.in
> >> > @@ -8011,6 +8011,8 @@ lists.
> >> >
> >> > @hook TARGET_LOOP_UNROLL_ADJUST
> >> >
> >> > +@hook TARGET_LOOP_JAM_REDUCTION_FACTOR
> >> > +
> >> > @defmac POWI_MAX_MULTS
> >> > If defined, this macro is interpreted as a signed integer C expression
> >> > that specifies the maximum number of floating point multiplications
> >> > diff --git a/gcc/gimple-loop-jam.cc b/gcc/gimple-loop-jam.cc
> >> > index 2c4162d17a2a..527f7c6cce35 100644
> >> > --- a/gcc/gimple-loop-jam.cc
> >> > +++ b/gcc/gimple-loop-jam.cc
> >> > @@ -34,12 +34,14 @@ along with GCC; see the file COPYING3. If not see
> >> > #include "cfgloop.h"
> >> > #include "tree-scalar-evolution.h"
> >> > #include "gimple-iterator.h"
> >> > +#include "cfganal.h"
> >> > #include "cfghooks.h"
> >> > #include "tree-data-ref.h"
> >> > #include "tree-ssa-loop-ivopts.h"
> >> > #include "tree-vectorizer.h"
> >> > #include "tree-ssa-sccvn.h"
> >> > #include "tree-cfgcleanup.h"
> >> > +#include "tree-into-ssa.h"
> >> >
> >> > /* Unroll and Jam transformation
> >> >
> >> > @@ -376,6 +378,684 @@ fuse_loops (class loop *loop)
> >> > }
> >> > }
> >> >
> >> > +/* Reduction-jam: a parallel-reduction extension of unroll-and-jam.
> >> > +
> >> > + The transformation above accepts only inner PHIs that are simple
> >> > + inductions. A reduction PHI cannot be fused the same way: merging
> >> > + the streams would chain them into a single serial accumulation,
> >> > + multiplying the serial dependency height instead of cutting it.
> >> > + This handles the GEMV / NNUE shape
> >> > +
> >> > + for (i = 0; i < M; i++) {
> >> > + s = bias[i];
> >> > + for (j = 0; j < K; j++) s += w[i][j] * x[j];
> >> > + out[i] = s;
> >> > + }
> >> > +
> >> > + where s starts afresh every outer iteration. Unrolling the outer
> >> > + loop by N leaves N sibling inner loops, each with its own reduction
> >> > + PHI and its own init (bias[i], bias[i+1], ...). Instead of merging
> >> > + them we keep all N PHIs in the fused header and hoist each sibling's
> >> > + init ahead of the body; the post-reduction stores stay on the
> >> > + sibling-exit chain, coalesced afterward. The vectorizer then emits
> >> > + N independent DOT_PROD / FMA chains, so the serial dependency height
> >> > + across the reduction drops by a factor of N.
> >> > +
> >> > + The strip factor N is the smallest of the target's
> >> > + TARGET_LOOP_JAM_REDUCTION_FACTOR, param_unroll_jam_reduction_streams
> >> > + and the data-dep safety bound, narrowed to a divisor of the outer
> >> > + trip count. We ignore adjust_unroll_factor's profit signal on
> >> > + purpose: it scores the memory refs that fusion removes, which is
> >> > + not what this transform is after. Targets that have not measured a
> >> > + win return 0 from the hook and we skip the loop. See params.opt
> >> > + for the tunables. */
> >> > +
> >> > +/* Return the single non-virtual, non-simple-IV PHI in LOOP's header
> >> > + (the reduction accumulator), or NULL if there is none or more than
> >> > + one. */
> >> > +
> >> > +static gphi *
> >> > +inner_reduction_phi (class loop *loop)
> >> > +{
> >> > + gphi *result = NULL;
> >> > + for (gphi_iterator psi = gsi_start_phis (loop->header);
> >> > + !gsi_end_p (psi); gsi_next (&psi))
> >> > + {
> >> > + gphi *phi = psi.phi ();
> >> > + tree op = gimple_phi_result (phi);
> >> > + if (virtual_operand_p (op))
> >> > + continue;
> >> > + affine_iv iv;
> >> > + if (simple_iv (loop, loop, op, &iv, true))
> >> > + continue;
> >> > + if (result)
> >> > + return NULL;
> >> > + result = phi;
> >> > + }
> >> > + return result;
> >> > +}
> >> > +
> >> > +/* Walk the SSA def chain from RED_PHI's latch update and return the
> >> > + narrowest scalar-int mode feeding a multiply inside INNER, or
> >> > + ACC_MODE if nothing narrower turns up. The walk follows only
> >> > + PLUS / MINUS / CONVERT chains so that a multiply buried in an array
> >> > + index (the "w * j" in "s += a[w * j]") is not mistaken for the MAC
> >> > + source. */
> >> > +
> >> > +static machine_mode
> >> > +narrowest_reduc_input_mode (gphi *red_phi, class loop *inner,
> >> > + machine_mode acc_mode)
> >> > +{
> >> > + tree update = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge
> >> > (inner));
> >> > + if (TREE_CODE (update) != SSA_NAME)
> >> > + return acc_mode;
> >> > +
> >> > + machine_mode narrow = VOIDmode;
> >> > + auto_vec<tree, 8> worklist;
> >> > + hash_set<tree> seen;
> >> > + worklist.safe_push (update);
> >> > +
> >> > + while (!worklist.is_empty ())
> >> > + {
> >> > + tree op = worklist.pop ();
> >> > + if (seen.add (op))
> >> > + continue;
> >> > + gimple *def = SSA_NAME_DEF_STMT (op);
> >> > + if (gimple_code (def) == GIMPLE_PHI
> >> > + || !gimple_bb (def)
> >> > + || !flow_bb_inside_loop_p (inner, gimple_bb (def))
> >> > + || !is_gimple_assign (def))
> >> > + continue;
> >> > +
> >> > + /* Stay in the reduction value chain. Loads and pointer
> >> > + arithmetic feed address computation, not the accumulator. */
> >> > + if (gimple_assign_load_p (def)
> >> > + || POINTER_TYPE_P (TREE_TYPE (gimple_assign_lhs (def))))
> >> > + continue;
> >> > +
> >> > + use_operand_p use_p;
> >> > + ssa_op_iter oi;
> >> > + tree_code code = gimple_assign_rhs_code (def);
> >> > + if (code == MULT_EXPR || code == WIDEN_MULT_EXPR)
> >> > + {
> >> > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
> >> > + {
> >> > + tree mop = USE_FROM_PTR (use_p);
> >> > + while (TREE_CODE (mop) == SSA_NAME)
> >> > + {
> >> > + gimple *mdef = SSA_NAME_DEF_STMT (mop);
> >> > + if (!is_gimple_assign (mdef)
> >> > + || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code
> >> > (mdef)))
> >> > + break;
> >> > + mop = gimple_assign_rhs1 (mdef);
> >> > + }
> >> > + machine_mode m = TYPE_MODE (TREE_TYPE (mop));
> >> > + if (SCALAR_INT_MODE_P (m)
> >> > + && (narrow == VOIDmode
> >> > + || known_lt (GET_MODE_PRECISION (m),
> >> > + GET_MODE_PRECISION (narrow))))
> >> > + narrow = m;
> >> > + }
> >> > + continue;
> >> > + }
> >> > +
> >> > + /* Only scalar-reduction operations continue the walk. */
> >> > + if (code != PLUS_EXPR
> >> > + && code != MINUS_EXPR
> >> > + && !CONVERT_EXPR_CODE_P (code))
> >> > + continue;
> >> > +
> >> > + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
> >> > + worklist.safe_push (USE_FROM_PTR (use_p));
> >> > + }
> >> > +
> >> > + if (narrow == VOIDmode
> >> > + || known_ge (GET_MODE_PRECISION (narrow),
> >> > + GET_MODE_PRECISION (acc_mode)))
> >> > + return acc_mode;
> >> > + return narrow;
> >> > +}
> >> > +
> >> > +/* Move *FROM to BB, immediately before BB's terminator branch (or to
> >> > + the end of BB if there is none). */
> >> > +
> >> > +static void
> >> > +gsi_move_to_bb_end_before_terminator (gimple_stmt_iterator *from,
> >> > + basic_block bb)
> >> > +{
> >> > + gimple_stmt_iterator to = gsi_last_bb (bb);
> >> > + if (!gsi_end_p (to) && is_ctrl_stmt (gsi_stmt (to)))
> >> > + gsi_move_before (from, &to);
> >> > + else
> >> > + gsi_move_after (from, &to);
> >> > +}
> >> > +
> >> > +/* Propagate the single argument of the PHI at PSI into its uses and
> >> > + remove the PHI, leaving PSI at the next one. */
> >> > +
> >> > +static void
> >> > +collapse_single_arg_phi (gphi_iterator *psi)
> >> > +{
> >> > + gphi *phi = psi->phi ();
> >> > + gcc_checking_assert (gimple_phi_num_args (phi) == 1);
> >> > + tree result = gimple_phi_result (phi);
> >> > + if (!has_zero_uses (result))
> >> > + replace_uses_by (result, gimple_phi_arg_def (phi, 0));
> >> > + remove_phi_node (psi, true);
> >> > +}
> >> > +
> >> > +/* True if LOOP is a widening MAC reduction the vectorizer can turn
> >> > + into a DOT_PROD_EXPR: a single-BB body with one exit, loads only, a
> >> > + constant trip count of at least MIN_INNER_ITERS, one non-IV
> >> > + reduction PHI alongside a unit-step counter IV, and a widening
> >> > + multiply in the update chain. If OUTER is given, every IV's base
> >> > + and step must be invariant in it. On success *OUT_RED_PHI,
> >> > + *OUT_ACC_MODE and *OUT_OP_MODE receive the reduction PHI, the
> >> > + accumulator mode and the narrowest operand mode. */
> >> > +
> >> > +static bool
> >> > +loop_is_dot_product_candidate_p (class loop *loop,
> >> > + unsigned HOST_WIDE_INT min_inner_iters,
> >> > + class loop *outer,
> >> > + gphi **out_red_phi,
> >> > + machine_mode *out_acc_mode,
> >> > + machine_mode *out_op_mode)
> >> > +{
> >> > + if (loop->num_nodes != 2 || !empty_block_p (loop->latch))
> >> > + return false;
> >> > + edge exit = single_exit (loop);
> >> > + if (!exit)
> >> > + return false;
> >> > +
> >> > + /* Header is the body; loads only. */
> >> > + bool has_vuse = false;
> >> > + for (gimple_stmt_iterator gsi = gsi_start_bb (loop->header);
> >> > + !gsi_end_p (gsi); gsi_next (&gsi))
> >> > + {
> >> > + gimple *s = gsi_stmt (gsi);
> >> > + if (gimple_vdef (s))
> >> > + return false;
> >> > + if (gimple_vuse (s))
> >> > + has_vuse = true;
> >> > + }
> >> > + if (!has_vuse)
> >> > + return false;
> >> > +
> >> > + /* Single reduction PHI with init defined outside the loop. */
> >> > + gphi *red_phi = inner_reduction_phi (loop);
> >> > + if (!red_phi)
> >> > + return false;
> >> > + tree init = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_preheader_edge
> >> > (loop));
> >> > + if (!expr_invariant_in_loop_p (loop, init))
> >> > + return false;
> >> > +
> >> > + /* Trip count is a known integer >= MIN_INNER_ITERS. niter.niter is
> >> > + the latch count (trip - 1); reject HOST_WIDE_INT_M1U separately
> >> > + so the +1 conversion below does not wrap. */
> >> > + tree_niter_desc niter;
> >> > + if (!number_of_iterations_exit (loop, exit, &niter, false, true)
> >> > + || niter.cmp == ERROR_MARK
> >> > + || !integer_zerop (niter.may_be_zero)
> >> > + || !tree_fits_uhwi_p (niter.niter))
> >> > + return false;
> >> > +
> >> > + unsigned HOST_WIDE_INT latch_count = tree_to_uhwi (niter.niter);
> >> > + if (latch_count == HOST_WIDE_INT_M1U
> >> > + || latch_count + 1 < min_inner_iters)
> >> > + return false;
> >> > +
> >> > + /* Plus a unit-step counter IV in the same header. When OUTER is
> >> > + given, every IV's step and base must also be OUTER-invariant. */
> >> > + bool found_unit_iv = false;
> >> > + for (gphi_iterator psi = gsi_start_phis (loop->header);
> >> > + !gsi_end_p (psi); gsi_next (&psi))
> >> > + {
> >> > + gphi *phi = psi.phi ();
> >> > + tree op = gimple_phi_result (phi);
> >> > + if (virtual_operand_p (op))
> >> > + continue;
> >> > + affine_iv iv;
> >> > + if (!simple_iv (loop, loop, op, &iv, true))
> >> > + continue;
> >> > + if (outer
> >> > + && (!expr_invariant_in_loop_p (outer, iv.step)
> >> > + || !expr_invariant_in_loop_p (outer, iv.base)))
> >> > + return false;
> >> > + if (iv.step && integer_onep (iv.step))
> >> > + found_unit_iv = true;
> >> > + }
> >> > + if (!found_unit_iv)
> >> > + return false;
> >> > +
> >> > + /* Widening multiply somewhere in the reduction update chain. */
> >> > + machine_mode acc_mode = TYPE_MODE (TREE_TYPE (gimple_phi_result
> >> > (red_phi)));
> >> > + machine_mode op_mode = narrowest_reduc_input_mode (red_phi, loop,
> >> > acc_mode);
> >> > + if (op_mode == acc_mode)
> >> > + return false;
> >> > +
> >> > + if (out_red_phi)
> >> > + *out_red_phi = red_phi;
> >> > + if (out_acc_mode)
> >> > + *out_acc_mode = acc_mode;
> >> > + if (out_op_mode)
> >> > + *out_op_mode = op_mode;
> >> > + return true;
> >> > +}
> >> > +
> >> > +/* Detect whether the OUTER / INNER nest has the parallel-reduction
> >> > + shape that unroll_jam_reduction_fuse_sibling can transform. The
> >> > + rejection messages below name each requirement. The only outer-body
> >> > + VDEF must be the post-reduction store, so that after strip-mining
> >> > + every replicated store sits downstream of every sibling's preheader.
> >> > + And the outer trip count must be a multiple of N, which keeps us off
> >> > + tree_unroll_loop's remainder path. */
> >> > +
> >> > +static bool
> >> > +unroll_jam_reduction_possible_p (class loop *outer, class loop *inner,
> >> > + unsigned HOST_WIDE_INT *out_outer_trip,
> >> > + unsigned *out_target_factor)
> >> > +{
> >> > + auto reject = [&] (const char *reason) -> bool
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file, "reduction-jam: rejecting outer loop %d:
> >> > %s\n",
> >> > + outer->num, reason);
> >> > + return false;
> >> > + };
> >> > +
> >> > + /* Loop-tree relationships. The driver iterates LI_ONLY_INNERMOST,
> >> > + so INNER has no children; check only that OUTER has a single
> >> > + inner sibling (INNER) and that INNER's parent is OUTER. */
> >> > + if (!inner || inner != outer->inner || inner->next
> >> > + || loop_outer (inner) != outer)
> >> > + return reject ("not a perfect 2-level nest");
> >> > +
> >> > + if (!dominated_by_p (CDI_DOMINATORS, outer->latch, inner->header))
> >> > + return reject ("inner has unsuitable dominance");
> >> > +
> >> > + /* Outer loop must carry only simple IVs. */
> >> > + for (gphi_iterator psi = gsi_start_phis (outer->header);
> >> > + !gsi_end_p (psi); gsi_next (&psi))
> >> > + {
> >> > + gphi *phi = psi.phi ();
> >> > + tree op = gimple_phi_result (phi);
> >> > + if (virtual_operand_p (op))
> >> > + continue;
> >> > + affine_iv iv;
> >> > + if (!simple_iv (outer, outer, op, &iv, true))
> >> > + return reject ("outer carries a non-simple-IV PHI");
> >> > + }
> >> > +
> >> > + /* Loop-local dot-product checks; on success red_phi and the modes
> >> > + feed the target-hook gate below. */
> >> > + gphi *red_phi = NULL;
> >> > + machine_mode acc_mode = VOIDmode;
> >> > + machine_mode op_mode = VOIDmode;
> >> > + if (!loop_is_dot_product_candidate_p
> >> > + (inner,
> >> > + (unsigned HOST_WIDE_INT)
> >> > param_unroll_jam_reduction_min_inner_iters,
> >> > + outer, &red_phi, &acc_mode, &op_mode))
> >> > + return reject ("inner not a dot-product candidate");
> >> > +
> >> > + /* Opt-in gate: ask the target whether the (accumulator, operand)
> >> > + mode pair benefits from N parallel reduction streams. Targets
> >> > + that have not measured a benefit return 0, in which case we skip
> >> > + the transform regardless of -floop-unroll-and-jam-reductions.
> >> > + A return of 1 is also opt-out (single-stream = no parallelism). */
> >> > + unsigned target_factor
> >> > + = targetm.loop_jam_reduction_factor (acc_mode, op_mode);
> >> > + if (target_factor < 2)
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file,
> >> > + "reduction-jam: target opted out for acc=%s op=%s\n",
> >> > + GET_MODE_NAME (acc_mode), GET_MODE_NAME (op_mode));
> >> > + return false;
> >> > + }
> >> > + *out_target_factor = target_factor;
> >> > +
> >> > + /* The fuser drains stmts from inner->header alone, so a non-empty
> >> > + latch would be silently mishandled; loop_is_dot_product_candidate_p
> >> > + already rejects one. */
> >> > + gcc_checking_assert (empty_block_p (inner->latch));
> >> > +
> >> > + /* Require the chain from red_phi back to itself via the latch arg to
> >> > + be a PLUS_EXPR reduction. This limits the transform to plain
> >> > + summation; the N streams are independent, so each is still summed
> >> > + in its original order and no reassociation takes place. */
> >> > + tree latch_arg = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge
> >> > (inner));
> >> > + if (!check_reduction_path (dump_user_location_t (), inner, red_phi,
> >> > + latch_arg, PLUS_EXPR))
> >> > + return reject ("inner reduction chain not associative-PLUS");
> >> > +
> >> > + /* loop_is_dot_product_candidate_p confirmed inner has a single exit.
> >> > */
> >> > + edge inner_exit = single_exit (inner);
> >> > +
> >> > + /* The outer body may carry at most one VDEF, only in the inner
> >> > + exit's destination BB (the post-reduction store). Also reject any
> >> > + non-VDEF side effect (volatile load, asm volatile, side-effecting
> >> > + builtin), matching bb_prevents_fusion_p on the IV-only path.
> >> > +
> >> > + The store BB must be dominated by inner->header so that after
> >> > + tree_unroll_loop strip-mines outer by N every replicated store
> >> > + sits downstream of its corresponding inner copy, and therefore
> >> > + downstream of every sibling's preheader. The fuser relies on
> >> > + this when rewiring VUSEs. */
> >> > + basic_block store_bb = inner_exit->dest;
> >> > + if (!dominated_by_p (CDI_DOMINATORS, store_bb, inner->header))
> >> > + return reject ("outer-body store not downstream of inner header");
> >> > +
> >> > + /* The outer body must be straight-line apart from the exit test. A
> >> > + control-flow merge there leaves a PHI that would not dominate the
> >> > + fused preheader after strip-mining, and the migrator leaves body
> >> > + PHIs in place, so the SSA fails verification. */
> >> > + edge outer_exit = single_exit (outer);
> >> > + if (!outer_exit)
> >> > + return reject ("outer loop has no single exit");
> >> > +
> >> > + basic_block *bbs = get_loop_body (outer);
> >> > + const char *fail = NULL;
> >> > + for (unsigned i = 0; i < outer->num_nodes && !fail; ++i)
> >> > + {
> >> > + basic_block bb = bbs[i];
> >> > + if (flow_bb_inside_loop_p (inner, bb))
> >> > + continue;
> >> > + if (EDGE_COUNT (bb->succs) > 1 && bb != outer_exit->src)
> >> > + {
> >> > + fail = "outer body has control flow";
> >> > + break;
> >> > + }
> >> > + unsigned vdef_count = 0;
> >> > + for (gimple_stmt_iterator gsi = gsi_start_bb (bb);
> >> > + !gsi_end_p (gsi); gsi_next (&gsi))
> >> > + {
> >> > + gimple *s = gsi_stmt (gsi);
> >> > + if (gimple_has_side_effects (s))
> >> > + {
> >> > + fail = "outer body has side effects";
> >> > + break;
> >> > + }
> >> > + if (gimple_vdef (s))
> >> > + vdef_count++;
> >> > + }
> >> > + if (!fail && (bb == store_bb ? vdef_count > 1 : vdef_count > 0))
> >> > + fail = "outer body has extra stores";
> >> > + }
> >> > + free (bbs);
> >> > + if (fail)
> >> > + return reject (fail);
> >> > +
> >> > + /* Outer trip count must be a known constant (N is later narrowed to
> >> > + a divisor of it; the remainder codepath of tree_unroll_loop is
> >> > + correct in principle but currently untested for this pattern).
> >> > + niter from number_of_iterations_exit is the latch execution count,
> >> > + so the trip count is niter + 1. */
> >> > + tree_niter_desc outer_niter;
> >> > + if (!number_of_iterations_exit (outer, outer_exit, &outer_niter,
> >> > + false, true)
> >> > + || outer_niter.cmp == ERROR_MARK
> >> > + || !integer_zerop (outer_niter.may_be_zero)
> >> > + || !tree_fits_uhwi_p (outer_niter.niter))
> >> > + return reject ("outer trip count unknown");
> >> > +
> >> > + unsigned HOST_WIDE_INT outer_latch_count = tree_to_uhwi
> >> > (outer_niter.niter);
> >> > + if (outer_latch_count == HOST_WIDE_INT_M1U)
> >> > + return reject ("outer trip count overflows HOST_WIDE_INT");
> >> > +
> >> > + *out_outer_trip = outer_latch_count + 1;
> >> > +
> >> > + return true;
> >> > +}
> >> > +
> >> > +/* Move the transitive SSA-use closure of SRC_INIT, and of every
> >> > + operand SRC's header body references, into DST_PREHEADER so the
> >> > + defs stay live once SRC's body moves. Defs that already dominate
> >> > + DST_PREHEADER, and defs inside SRC, are left to the body move.
> >> > +
> >> > + RPO is a reverse-postorder over CFUN's BBs, computed once by the
> >> > + caller and shared across the nest's siblings; walking it keeps
> >> > + def-use order among the migrated stmts intact. DST_LIVE_VUSE is
> >> > + the vop live at DST_PREHEADER, and migrated stmts are rewired to it
> >> > + so none of them reference a vop defined downstream. */
> >> > +
> >> > +static void
> >> > +migrate_loop_invariant_defs_to_preheader (class loop *src, tree
> >> > src_init,
> >> > + basic_block dst_preheader,
> >> > + const auto_vec<int> &rpo,
> >> > + tree dst_live_vuse)
> >> > +{
> >> > + hash_set<gimple *> needed;
> >> > + auto_vec<gimple *, 16> worklist;
> >> > +
> >> > + auto consider = [&] (tree op)
> >> > + {
> >> > + if (TREE_CODE (op) != SSA_NAME)
> >> > + return;
> >> > + gimple *def = SSA_NAME_DEF_STMT (op);
> >> > + if (!def || is_a <gphi *> (def))
> >> > + return;
> >> > + basic_block def_bb = gimple_bb (def);
> >> > + if (!def_bb
> >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, def_bb)
> >> > + || flow_bb_inside_loop_p (src, def_bb))
> >> > + return;
> >> > + if (needed.add (def))
> >> > + return;
> >> > + worklist.safe_push (def);
> >> > + };
> >> > +
> >> > + auto consider_uses = [&] (gimple *s)
> >> > + {
> >> > + use_operand_p use_p;
> >> > + ssa_op_iter oi;
> >> > + FOR_EACH_SSA_USE_OPERAND (use_p, s, oi, SSA_OP_USE)
> >> > + consider (USE_FROM_PTR (use_p));
> >> > + };
> >> > +
> >> > + consider (src_init);
> >> > + for (gimple_stmt_iterator si = gsi_start_bb (src->header);
> >> > + !gsi_end_p (si); gsi_next (&si))
> >> > + {
> >> > + gimple *s = gsi_stmt (si);
> >> > + if (!is_a <gphi *> (s) && !is_ctrl_stmt (s))
> >> > + consider_uses (s);
> >> > + }
> >> > + while (!worklist.is_empty ())
> >> > + consider_uses (worklist.pop ());
> >> > +
> >> > + if (needed.is_empty ())
> >> > + return;
> >> > +
> >> > + /* Walk BBs in reverse-postorder so that, for any def-use edge among
> >> > + the migrated stmts, the def is moved into DST_PREHEADER ahead of
> >> > + the use. */
> >> > + int n = rpo.length ();
> >> > + for (int i = 0; i < n; ++i)
> >> > + {
> >> > + basic_block bb = BASIC_BLOCK_FOR_FN (cfun, rpo[i]);
> >> > + if (bb == dst_preheader
> >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
> >> > + continue;
> >> > + for (gimple_stmt_iterator si = gsi_start_bb (bb);
> >> > + !gsi_end_p (si);)
> >> > + {
> >> > + gimple *stmt = gsi_stmt (si);
> >> > + if (!needed.contains (stmt))
> >> > + {
> >> > + gsi_next (&si);
> >> > + continue;
> >> > + }
> >> > + if (dst_live_vuse && gimple_vuse (stmt))
> >> > + {
> >> > + gimple_set_vuse (stmt, dst_live_vuse);
> >> > + update_stmt (stmt);
> >> > + }
> >> > + gsi_move_to_bb_end_before_terminator (&si, dst_preheader);
> >> > + }
> >> > + }
> >> > +}
> >> > +
> >> > +/* Pair SRC's non-reduction header PHIs with DST's by position,
> >> > + recording each pair in PHI_RENAME. Both are inner siblings of the
> >> > + same strip-mined outer loop, so their header PHI lists start with
> >> > + the duplicated originals in the same order. DST may have picked up
> >> > + extra reduction PHIs at the tail from earlier sibling fuses, so
> >> > + walk SRC to completion against DST's prefix. SRC_REDUC_PHI is
> >> > + renamed separately and virtual PHIs are skipped. */
> >> > +
> >> > +static void
> >> > +match_sibling_header_phis (class loop *src, class loop *dst,
> >> > + gphi *src_reduc_phi,
> >> > + hash_map<tree, tree> &phi_rename)
> >> > +{
> >> > + gphi_iterator spsi = gsi_start_phis (src->header);
> >> > + gphi_iterator dpsi = gsi_start_phis (dst->header);
> >> > + for (; !gsi_end_p (spsi); gsi_next (&spsi), gsi_next (&dpsi))
> >> > + {
> >> > + gcc_checking_assert (!gsi_end_p (dpsi));
> >> > + gphi *src_phi = spsi.phi ();
> >> > + gphi *dst_phi = dpsi.phi ();
> >> > + tree src_res = gimple_phi_result (src_phi);
> >> > + tree dst_res = gimple_phi_result (dst_phi);
> >> > + gcc_checking_assert (virtual_operand_p (src_res)
> >> > + == virtual_operand_p (dst_res));
> >> > + if (src_phi != src_reduc_phi && !virtual_operand_p (src_res))
> >> > + phi_rename.put (src_res, dst_res);
> >> > + }
> >> > +}
> >> > +
> >> > +/* Fuse the SRC sibling inner loop into DST, keeping SRC's reduction
> >> > + PHI as a parallel chain in DST's header. Both are inner siblings
> >> > + of an outer loop that passed unroll_jam_reduction_possible_p and
> >> > + was then strip-mined by tree_unroll_loop.
> >> > +
> >> > + The caller has already flattened LC-SSA across the nest (Phase 1a).
> >> > + Here SRC's reduction init and body operands move up to DST's
> >> > + preheader (Phase 1b), a fresh reduction PHI for SRC's stream goes
> >> > + into DST's header (Phase 2), and SRC's body follows with its PHI
> >> > + uses renamed (Phase 3). The caller then bypasses the drained SRC
> >> > + (Phase 4); its cleanup_tree_cfg folds the leftover chain of exit
> >> > + BBs holding the post-reduction stores, and
> >> > + rewrite_into_loop_closed_ssa puts LC-SSA back. */
> >> > +
> >> > +static void
> >> > +unroll_jam_reduction_fuse_sibling (class loop *src, class loop *dst,
> >> > + const auto_vec<int> &rpo)
> >> > +{
> >> > + gphi *src_reduc_phi = inner_reduction_phi (src);
> >> > + tree src_reduc_res = gimple_phi_result (src_reduc_phi);
> >> > + tree src_init = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
> >> > + loop_preheader_edge (src));
> >> > + tree src_update = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
> >> > + loop_latch_edge (src));
> >> > +
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file, "reduction-jam: fusing src loop %d into dst
> >> > %d\n",
> >> > + src->num, dst->num);
> >> > +
> >> > + basic_block dst_preheader = loop_preheader_edge (dst)->src;
> >> > +
> >> > + /* The migrated stmts must reference the vop live at DST's preheader.
> >> > + The matcher's only outer-body VDEF is the post-reduction store,
> >> > + which sits below DST after strip-mining, so nothing redefines the
> >> > + vop before DST: it is outer->header's virtual PHI result. A
> >> > + load-only outer has no such PHI; its loads already share one
> >> > + loop-invariant vop, so leave their VUSE alone (NULL_TREE). */
> >> > + gphi *outer_vphi = get_virtual_phi (loop_outer (dst)->header);
> >> > + tree dst_live_vuse = outer_vphi ? gimple_phi_result (outer_vphi) :
> >> > NULL_TREE;
> >> > +
> >> > + /* Phase 1b. */
> >> > + migrate_loop_invariant_defs_to_preheader (src, src_init,
> >> > dst_preheader,
> >> > + rpo, dst_live_vuse);
> >> > +
> >> > + /* Phase 2: a new reduction PHI in DST's header for SRC's stream. */
> >> > + tree new_name = make_ssa_name (TREE_TYPE (src_reduc_res));
> >> > + gphi *new_phi = create_phi_node (new_name, dst->header);
> >> > + add_phi_arg (new_phi, src_init, loop_preheader_edge (dst),
> >> > + UNKNOWN_LOCATION);
> >> > + add_phi_arg (new_phi, src_update, loop_latch_edge (dst),
> >> > + UNKNOWN_LOCATION);
> >> > +
> >> > + /* Phase 3: rewrite uses, then move SRC's body into DST's header. */
> >> > + hash_map<tree, tree> phi_rename;
> >> > + phi_rename.put (src_reduc_res, new_name);
> >> > + match_sibling_header_phis (src, dst, src_reduc_phi, phi_rename);
> >> > +
> >> > + auto_vec<gimple *, 32> src_body;
> >> > + for (gimple_stmt_iterator si = gsi_start_bb (src->header);
> >> > + !gsi_end_p (si); gsi_next (&si))
> >> > + {
> >> > + gimple *s = gsi_stmt (si);
> >> > + if (is_a <gphi *> (s) || is_ctrl_stmt (s))
> >> > + continue;
> >> > + src_body.safe_push (s);
> >> > + }
> >> > + for (gimple *stmt : src_body)
> >> > + {
> >> > + use_operand_p use_p;
> >> > + ssa_op_iter oi;
> >> > + FOR_EACH_SSA_USE_OPERAND (use_p, stmt, oi, SSA_OP_USE)
> >> > + {
> >> > + tree *map = phi_rename.get (USE_FROM_PTR (use_p));
> >> > + if (map)
> >> > + SET_USE (use_p, *map);
> >> > + }
> >> > + if (dst_live_vuse && gimple_vuse (stmt))
> >> > + {
> >> > + gimple_set_vuse (stmt, dst_live_vuse);
> >> > + update_stmt (stmt);
> >> > + }
> >> > + gimple_stmt_iterator from = gsi_for_stmt (stmt);
> >> > + gsi_move_to_bb_end_before_terminator (&from, dst->header);
> >> > + }
> >> > +
> >> > + /* The body walk above only covered uses of src_reduc_res inside
> >> > + src->header; catch any remaining uses (LC-SSA closures reached
> >> > + by the post-reduction store) before the PHI is removed. */
> >> > + replace_uses_by (src_reduc_res, new_name);
> >> > +
> >> > + gphi_iterator psi_src_reduc = gsi_for_phi (src_reduc_phi);
> >> > + remove_phi_node (&psi_src_reduc, false);
> >> > +}
> >> > +
> >> > +/* Phase 4: redirect SRC's preheader edge straight past the now-empty
> >> > + SRC loop so cleanup_tree_cfg can remove it. SRC must have been
> >> > + drained by unroll_jam_reduction_fuse_sibling first. */
> >> > +
> >> > +static void
> >> > +unroll_jam_reduction_bypass_sibling (class loop *src)
> >> > +{
> >> > + edge src_preheader_e = loop_preheader_edge (src);
> >> > + edge src_exit_e = single_exit (src);
> >> > + if (!src_exit_e)
> >> > + return;
> >> > + basic_block src_exit_dest = src_exit_e->dest;
> >> > +
> >> > + /* src_exit_dest is a single-pred BB after strip-mining (the next
> >> > + sibling's preheader, or outer's latch for the last sibling), so
> >> > + its LC-SSA PHIs are single-arg. Flatten them before the redirect
> >> > + so no arg dangles once SRC's BBs are deleted: the driver rebuilds
> >> > + scalar LC-SSA afterward but not the virtual side, so a stale vop
> >> > + arg would otherwise survive. */
> >> > + for (gphi_iterator psi = gsi_start_phis (src_exit_dest);
> >> > + !gsi_end_p (psi);)
> >> > + collapse_single_arg_phi (&psi);
> >> > +
> >> > + edge new_edge = redirect_edge_and_branch (src_preheader_e,
> >> > src_exit_dest);
> >> > + gcc_assert (new_edge);
> >> > +
> >> > + /* Delete SRC's now-unreachable BBs and the empty loop entry,
> >> > + mirroring fuse_loops's delete_basic_block + delete_loop pattern
> >> > + on the IV-only path. Without this the loop tree carries stale
> >> > + entries until the driver's cleanup_tree_cfg picks them up. */
> >> > + basic_block *bbs = get_loop_body (src);
> >> > + unsigned n = src->num_nodes;
> >> > + for (unsigned i = 0; i < n; i++)
> >> > + delete_basic_block (bbs[i]);
> >> > + free (bbs);
> >> > + delete_loop (src);
> >> > +}
> >> > +
> >> > /* Return true if any of the access functions for dataref A
> >> > isn't invariant with respect to loop LOOP_NEST. */
> >> > static bool
> >> > @@ -497,6 +1177,221 @@ adjust_unroll_factor (class loop *inner, struct
> >> > data_dependence_relation *ddr,
> >> > return ret;
> >> > }
> >> >
> >> > +/* Walk OUTER's data dependences, running each through
> >> > + adjust_unroll_factor. *UNROLL_FACTOR receives the safety upper
> >> > + bound (UINT_MAX for no constraint, 0 for cannot unroll),
> >> > + *PROFIT_UNROLL the largest distance+1 yielding a removable ref
> >> > + under the (a,0) pattern, *REMOVED the count of removable refs and
> >> > + *NUM_DATAREFS the total. Returns false if the dependences cannot
> >> > + be computed.
> >> > +
> >> > + When *UNROLL_FACTOR comes back 0 the other out-params are only
> >> > + partially populated and must not be read for profitability. */
> >> > +
> >> > +static bool
> >> > +compute_jam_dependence_factors (class loop *outer, class loop *inner,
> >> > + unsigned *unroll_factor,
> >> > + unsigned *profit_unroll,
> >> > + unsigned *removed,
> >> > + unsigned *num_datarefs)
> >> > +{
> >> > + vec<data_reference_p> datarefs = vNULL;
> >> > + vec<ddr_p> dependences = vNULL;
> >> > + auto_vec<loop_p, 3> loop_nest;
> >> > +
> >> > + if (!compute_data_dependences_for_loop (outer, true, &loop_nest,
> >> > + &datarefs, &dependences))
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file, "Cannot analyze data dependencies\n");
> >> > + free_data_refs (datarefs);
> >> > + free_dependence_relations (dependences);
> >> > + return false;
> >> > + }
> >> > +
> >> > + *num_datarefs = datarefs.length ();
> >> > + *unroll_factor = (unsigned) -1;
> >> > + *profit_unroll = 1;
> >> > + *removed = 0;
> >> > +
> >> > + if (*num_datarefs)
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + dump_data_dependence_relations (dump_file, dependences);
> >> > +
> >> > + unsigned i;
> >> > + ddr_p ddr;
> >> > + FOR_EACH_VEC_ELT (dependences, i, ddr)
> >> > + {
> >> > + if (DDR_ARE_DEPENDENT (ddr) == chrec_known)
> >> > + continue;
> >> > + data_reference *dra = DDR_A (ddr);
> >> > + data_reference *drb = DDR_B (ddr);
> >> > + /* Self-dependence: WAW with non-affine access could reorder
> >> > + writes to the same location. */
> >> > + if (dra == drb)
> >> > + {
> >> > + if (DR_IS_WRITE (dra)
> >> > + && !DR_ACCESS_FNS (dra).is_empty ()
> >> > + && DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
> >> > + {
> >> > + *unroll_factor = 0;
> >> > + break;
> >> > + }
> >> > + continue;
> >> > + }
> >> > + /* Couldn't determine the distance vector: harmless for two
> >> > + reads, but a write means we cannot prove safety. */
> >> > + if (!adjust_unroll_factor (inner, ddr, unroll_factor,
> >> > + profit_unroll, removed))
> >> > + if (DR_IS_WRITE (dra) || DR_IS_WRITE (drb))
> >> > + {
> >> > + *unroll_factor = 0;
> >> > + break;
> >> > + }
> >> > + }
> >> > + }
> >> > +
> >> > + free_dependence_relations (dependences);
> >> > + free_data_refs (datarefs);
> >> > + return true;
> >> > +}
> >> > +
> >> > +/* Pick a strip factor for the reduction-jam path on OUTER / INNER.
> >> > + adjust_unroll_factor gives the data-dep safety bound, TARGET_FACTOR
> >> > + the target's preference and param_unroll_jam_reduction_streams
> >> > + the cap. We do not use adjust_unroll_factor's profit model here:
> >> > + it would cap N at 1 for GEMV, because the input-load DDR has
> >> > + dist_v=(0,0). The result is narrowed to a divisor of OUTER_TRIP,
> >> > + which keeps us off tree_unroll_loop's remainder path, and validated
> >> > + via can_unroll_loop_p. *OUT_FACTOR receives the chosen factor and
> >> > + *OUT_DESC the niter. */
> >> > +
> >> > +static bool
> >> > +select_reduction_jam_factor (class loop *outer, class loop *inner,
> >> > + unsigned HOST_WIDE_INT outer_trip,
> >> > + unsigned target_factor,
> >> > + unsigned *out_factor,
> >> > + tree_niter_desc *out_desc)
> >> > +{
> >> > + /* Only the safety bound matters here; the ref-removal outputs go
> >> > + unused. */
> >> > + unsigned unroll_factor, profit_unroll, removed, num_datarefs;
> >> > + if (!compute_jam_dependence_factors (outer, inner, &unroll_factor,
> >> > + &profit_unroll, &removed,
> >> > + &num_datarefs))
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file,
> >> > + "reduction-jam: data-dep analysis failed\n");
> >> > + return false;
> >> > + }
> >> > + if (unroll_factor < 2)
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file,
> >> > + "reduction-jam: data-dep safety bound below 2\n");
> >> > + return false;
> >> > + }
> >> > +
> >> > + unsigned N = MIN (target_factor,
> >> > + (unsigned) param_unroll_jam_reduction_streams);
> >> > + if (N > unroll_factor)
> >> > + N = unroll_factor;
> >> > + unsigned initial_N = N;
> >> > + while (N >= 2 && (outer_trip < N || outer_trip % N != 0))
> >> > + --N;
> >> > + if (N < 2)
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file,
> >> > + "reduction-jam: outer trip "
> >> > HOST_WIDE_INT_PRINT_UNSIGNED
> >> > + " has no divisor in [2, %u]\n", outer_trip, initial_N);
> >> > + return false;
> >> > + }
> >> > +
> >> > + if (!can_unroll_loop_p (outer, N, out_desc))
> >> > + {
> >> > + if (dump_file && (dump_flags & TDF_DETAILS))
> >> > + fprintf (dump_file,
> >> > + "reduction-jam: cannot unroll outer by %u\n", N);
> >> > + return false;
> >> > + }
> >> > +
> >> > + *out_factor = N;
> >> > + return true;
> >> > +}
> >> > +
> >> > +/* Strip-mine OUTER by N and fuse the resulting N inner siblings into
> >> > + one body that carries N parallel reduction PHIs. Caller has
> >> > + already validated OUTER and chosen N + DESC via
> >> > + unroll_jam_reduction_possible_p / select_reduction_jam_factor. */
> >> > +
> >> > +static void
> >> > +do_unroll_jam_reductions (class loop *outer, unsigned N,
> >> > + tree_niter_desc *desc)
> >> > +{
> >> > + initialize_original_copy_tables ();
> >> > + tree_unroll_loop (outer, N, desc);
> >> > + free_original_copy_tables ();
> >> > +
> >> > + /* dst->next chains siblings in CFG order (per tree_unroll_loop);
> >> > + the fuser relies on that when migrating SSA uses. */
> >> > + class loop *dst = outer->inner;
> >> > + auto_vec<class loop *, 4> siblings;
> >> > + for (class loop *src = dst->next; src; src = src->next)
> >> > + siblings.safe_push (src);
> >> > +
> >> > + /* RPO of CFUN, shared across all siblings: the fuser only moves
> >> > + stmts and removes single-arg PHIs, neither of which alters BB
> >> > + structure, so one walk suffices for the whole nest. */
> >> > + auto_vec<int> rpo;
> >> > + rpo.safe_grow (last_basic_block_for_fn (cfun));
> >> > + rpo.truncate (pre_and_rev_post_order_compute_fn (cfun, NULL,
> >> > rpo.address (),
> >> > + false));
> >> > +
> >> > + /* Phase 1a (nest-wide): collapse single-arg LC-SSA PHIs that the
> >> > + per-sibling migrate-walk in Phase 1b would otherwise have to
> >> > + chase through (it skips PHIs). Sweep the BBs that do not dominate
> >> > + DST's preheader, i.e. the nest and everything downstream where the
> >> > + sibling-exit closures live. One sweep suffices: later fuses only
> >> > + add two-arg reduction PHIs and make no CFG change that could
> >> > + reintroduce single-arg closures. */
> >> > + basic_block dst_preheader = loop_preheader_edge (dst)->src;
> >> > + calculate_dominance_info (CDI_DOMINATORS);
> >> > + basic_block bb;
> >> > + FOR_EACH_BB_FN (bb, cfun)
> >> > + {
> >> > + if (bb == dst_preheader
> >> > + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
> >> > + continue;
> >> > + for (gphi_iterator psi = gsi_start_phis (bb); !gsi_end_p (psi);)
> >> > + {
> >> > + gphi *phi = psi.phi ();
> >> > + if (gimple_phi_num_args (phi) == 1
> >> > + && !virtual_operand_p (gimple_phi_result (phi)))
> >> > + collapse_single_arg_phi (&psi);
> >> > + else
> >> > + gsi_next (&psi);
> >> > + }
> >> > + }
> >> > +
> >> > + for (class loop *src : siblings)
> >> > + unroll_jam_reduction_fuse_sibling (src, dst, rpo);
> >> > +
> >> > + /* Bypass the drained siblings only after every fuse: the CFG edits
> >> > + invalidate dominance / SCEV state the fuse loop above relies on.
> >> > */
> >> > + for (class loop *src : siblings)
> >> > + unroll_jam_reduction_bypass_sibling (src);
> >> > +
> >> > + if (dump_enabled_p ())
> >> > + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS | TDF_DETAILS,
> >> > + find_loop_location (outer),
> >> > + "applying reduction unroll-and-jam with %u
> >> > streams\n",
> >> > + siblings.length () + 1);
> >> > +}
> >> > +
> >> > +
> >> > /* Main entry point for the unroll-and-jam transformation
> >> > described above. */
> >> >
> >> > @@ -516,8 +1411,36 @@ tree_loop_unroll_and_jam (void)
> >> > || optimize_loop_nest_for_size_p (outer))
> >> > continue;
> >> >
> >> > - if (!unroll_jam_possible_p (outer, loop))
> >> > - continue;
> >> > + if (!flag_unroll_jam || !unroll_jam_possible_p (outer, loop))
> >> > + {
> >> > + /* The IV-only path skips reduction-shaped nests (reduction
> >> > PHIs,
> >> > + or outer-body stores consuming the reduction result). Try
> >> > the
> >> > + reduction-jam extension on those. */
> >> > + unsigned HOST_WIDE_INT outer_trip;
> >> > + unsigned target_factor, N;
> >> > + tree_niter_desc red_desc;
> >> > + if (flag_unroll_jam_reductions
> >> > + && unroll_jam_reduction_possible_p (outer, loop,
> >> > + &outer_trip,
> >> > + &target_factor)
> >> > + && select_reduction_jam_factor (outer, loop, outer_trip,
> >> > + target_factor,
> >> > + &N, &red_desc))
> >> > + {
> >> > + do_unroll_jam_reductions (outer, N, &red_desc);
> >> > + free_dominance_info (CDI_DOMINATORS);
> >> > + cleanup_tree_cfg ();
> >> > + /* The fuser maintains virtual SSA from the matcher's
> >> > + invariants, but moving SRC's body into DST leaves scalar
> >> > + SSA names defined in DST yet used outside it (e.g.
> >> > new_name's
> >> > + LC-SSA closures). Rebuild scalar LC-SSA so the next
> >> > nest's
> >> > + tree_unroll_loop is well-formed, and verify the result.
> >> > */
> >> > + rewrite_into_loop_closed_ssa (NULL, 0);
> >> > + checking_verify_ssa (true, true);
> >> > + scev_reset ();
> >> > + }
> >> > + continue;
> >> > + }
> >> >
> >> > vec<data_reference_p> datarefs = vNULL;
> >> > vec<ddr_p> dependences = vNULL;
> >> > @@ -674,7 +1597,10 @@ public:
> >> > {}
> >> >
> >> > /* opt_pass methods: */
> >> > - bool gate (function *) final override { return flag_unroll_jam != 0; }
> >> > + bool gate (function *) final override
> >> > + {
> >> > + return flag_unroll_jam != 0 || flag_unroll_jam_reductions;
> >> > + }
> >> > unsigned int execute (function *) final override;
> >> >
> >> > };
> >> > diff --git a/gcc/hooks.cc b/gcc/hooks.cc
> >> > index 2b47fd3e50f8..976bae3c89a9 100644
> >> > --- a/gcc/hooks.cc
> >> > +++ b/gcc/hooks.cc
> >> > @@ -424,6 +424,13 @@ hook_uint_mode_0 (machine_mode)
> >> > return 0;
> >> > }
> >> >
> >> > +/* Generic hook that takes two machine modes and returns an unsigned
> >> > int 0. */
> >> > +unsigned int
> >> > +hook_uint_mode_mode_0 (machine_mode, machine_mode)
> >> > +{
> >> > + return 0;
> >> > +}
> >> > +
> >> > /* Generic hook that takes no arguments and returns a NULL const
> >> > string. */
> >> > const char *
> >> > hook_constcharptr_void_null (void)
> >> > diff --git a/gcc/hooks.h b/gcc/hooks.h
> >> > index 1ad4abb5500c..0ec0b32fa1a5 100644
> >> > --- a/gcc/hooks.h
> >> > +++ b/gcc/hooks.h
> >> > @@ -119,6 +119,7 @@ extern tree
> >> > hook_tree_treeptr_tree_tree_int_boolptr_null (tree *, tree, tree, in
> >> >
> >> > extern unsigned hook_uint_void_0 (void);
> >> > extern unsigned int hook_uint_mode_0 (machine_mode);
> >> > +extern unsigned int hook_uint_mode_mode_0 (machine_mode, machine_mode);
> >> >
> >> > extern bool default_can_output_mi_thunk_no_vcall (const_tree,
> >> > HOST_WIDE_INT,
> >> > HOST_WIDE_INT,
> >> > const_tree);
> >> > diff --git a/gcc/params.opt b/gcc/params.opt
> >> > index 0c6a3e514574..e99c4fc4173f 100644
> >> > --- a/gcc/params.opt
> >> > +++ b/gcc/params.opt
> >> > @@ -1263,6 +1263,14 @@ Maximum unroll factor for the unroll-and-jam
> >> > transformation.
> >> > Common Joined UInteger Var(param_unroll_jam_min_percent) Init(1)
> >> > IntegerRange(0, 100) Param Optimization
> >> > Minimum percentage of memrefs that must go away for unroll-and-jam to
> >> > be considered profitable.
> >> >
> >> > +-param=unroll-jam-reduction-streams=
> >> > +Common Joined UInteger Var(param_unroll_jam_reduction_streams) Init(4)
> >> > IntegerRange(2, 16) Param Optimization
> >> > +Target number of parallel reduction streams produced by reduction-aware
> >> > unroll-and-jam.
> >> > +
> >> > +-param=unroll-jam-reduction-min-inner-iters=
> >> > +Common Joined UInteger Var(param_unroll_jam_reduction_min_inner_iters)
> >> > Init(32) IntegerRange(2, 65536) Param Optimization
> >> > +Minimum inner-loop trip count required for reduction-aware
> >> > unroll-and-jam to fire.
> >> > +
> >> > -param=use-after-scope-direct-emission-threshold=
> >> > Common Joined UInteger
> >> > Var(param_use_after_scope_direct_emission_threshold) Init(256) Param
> >> > Optimization
> >> > Use direct poisoning/unpoisoning instructions for variables smaller or
> >> > equal to this number.
> >> > diff --git a/gcc/target.def b/gcc/target.def
> >> > index 66c20a9c9d2a..118aa9bb82ec 100644
> >> > --- a/gcc/target.def
> >> > +++ b/gcc/target.def
> >> > @@ -2873,6 +2873,19 @@ number of memory accesses.",
> >> > unsigned, (unsigned nunroll, class loop *loop),
> >> > NULL)
> >> >
> >> > +DEFHOOK
> >> > +(loop_jam_reduction_factor,
> >> > + "Return the preferred number of parallel reduction streams\n\
> >> > +@option{-floop-unroll-and-jam-reductions} should produce for an inner\n\
> >> > +reduction that accumulates into mode @var{acc_mode} from operands of\n\
> >> > +mode @var{op_mode}. @var{op_mode} is the operand mode at the source\n\
> >> > +of the widening multiply, and is always strictly narrower than\n\
> >> > +@var{acc_mode}: plain reductions with no widening multiply are\n\
> >> > +filtered out before the hook is consulted. Returning less than two\n\
> >> > +disables the transform for that mode pair.",
> >> > + unsigned, (machine_mode acc_mode, machine_mode op_mode),
> >> > + hook_uint_mode_mode_0)
> >> > +
> >> > /* True if X is a legitimate MODE-mode immediate operand. */
> >> > DEFHOOK
> >> > (legitimate_constant_p,
> >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> >> > new file mode 100644
> >> > index 000000000000..d14711034bc0
> >> > --- /dev/null
> >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> >> > @@ -0,0 +1,78 @@
> >> > +/* Positive tests: the pass must strip-mine and fuse each nest below.
> >> > + The outer trip count of 32 is divisible by the default factor of 4.
> >> > + Both arms of the aarch64 hook are covered: QImode / HImode -> SImode
> >> > + (DOT_PROD class) in affine, and SImode -> DImode (SMLAL_DI) in
> >> > + affine64. */
> >> > +
> >> > +/* { dg-do compile } */
> >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
> >> > -fdump-tree-unrolljam-details" } */
> >> > +
> >> > +#include <stdint.h>
> >> > +
> >> > +#define IN_DIM 512
> >> > +#define OUT_DIM 32
> >> > +
> >> > +void
> >> > +affine (int32_t *__restrict out,
> >> > + const int8_t *weights,
> >> > + const int32_t *biases,
> >> > + const uint8_t *input)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; ++i)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < IN_DIM; ++j)
> >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +void
> >> > +affine64 (int64_t *__restrict out,
> >> > + const int32_t *weights,
> >> > + const int64_t *biases,
> >> > + const int32_t *input)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; ++i)
> >> > + {
> >> > + int64_t s = biases[i];
> >> > + for (int j = 0; j < IN_DIM; ++j)
> >> > + s += (int64_t) weights[i * IN_DIM + j] * (int64_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Inner trip count at the min-inner-iters default of 32. */
> >> > +void
> >> > +affine_min_inner (int32_t *__restrict out,
> >> > + const int8_t *weights,
> >> > + const int32_t *biases,
> >> > + const uint8_t *input)
> >> > +{
> >> > + for (int i = 0; i < 32; ++i)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < 32; ++j)
> >> > + s += (int32_t) weights[i * 32 + j] * (int32_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Outer trip count of 9 narrows N from 4 to 3. */
> >> > +void
> >> > +affine_narrowed (int32_t *__restrict out,
> >> > + const int8_t *weights,
> >> > + const int32_t *biases,
> >> > + const uint8_t *input)
> >> > +{
> >> > + for (int i = 0; i < 9; ++i)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < 512; ++j)
> >> > + s += (int32_t) weights[i * 512 + j] * (int32_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* { dg-final { scan-tree-dump-times "applying reduction
> >> > unroll-and-jam" 4 "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop"
> >> > 11 "unrolljam" } } */
> >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> >> > new file mode 100644
> >> > index 000000000000..44301348a659
> >> > --- /dev/null
> >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> >> > @@ -0,0 +1,89 @@
> >> > +/* Correctness test for the reduction-aware extension of unroll-and-jam
> >> > + on the GEMV pattern. Compares against an in-main reference
> >> > + computation. */
> >> > +
> >> > +/* { dg-do run } */
> >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" }
> >> > */
> >> > +
> >> > +#include <stdint.h>
> >> > +
> >> > +#define IN_DIM 512
> >> > +#define OUT_DIM 32
> >> > +
> >> > +extern void abort (void);
> >> > +
> >> > +void __attribute__((noipa))
> >> > +affine (int32_t *__restrict out,
> >> > + const int8_t *weights,
> >> > + const int32_t *biases,
> >> > + const uint8_t *input)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; ++i)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < IN_DIM; ++j)
> >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +void __attribute__((noipa))
> >> > +affine_zero_init (int32_t *__restrict out,
> >> > + const int8_t *a,
> >> > + const int8_t *b)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; ++i)
> >> > + {
> >> > + int32_t s = 0;
> >> > + for (int j = 0; j < IN_DIM; ++j)
> >> > + s += a[i * IN_DIM + j] * b[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +int
> >> > +main (void)
> >> > +{
> >> > + static int8_t weights[OUT_DIM * IN_DIM];
> >> > + static uint8_t input[IN_DIM];
> >> > + static int32_t biases[OUT_DIM];
> >> > + static int32_t out[OUT_DIM];
> >> > + static int32_t expected[OUT_DIM];
> >> > +
> >> > + for (int k = 0; k < OUT_DIM * IN_DIM; k++)
> >> > + weights[k] = (k * 13) & 0x7f;
> >> > + for (int k = 0; k < IN_DIM; k++)
> >> > + input[k] = (k * 7) & 0xff;
> >> > + for (int k = 0; k < OUT_DIM; k++)
> >> > + biases[k] = k * 1000;
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < IN_DIM; j++)
> >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> >> > + expected[i] = s;
> >> > + }
> >> > +
> >> > + affine (out, weights, biases, input);
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + if (out[i] != expected[i])
> >> > + abort ();
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + {
> >> > + int32_t s = 0;
> >> > + for (int j = 0; j < IN_DIM; j++)
> >> > + s += weights[i * IN_DIM + j] * weights[j];
> >> > + expected[i] = s;
> >> > + }
> >> > +
> >> > + affine_zero_init (out, weights, weights);
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + if (out[i] != expected[i])
> >> > + abort ();
> >> > +
> >> > + return 0;
> >> > +}
> >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> >> > new file mode 100644
> >> > index 000000000000..6e4b590fefeb
> >> > --- /dev/null
> >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> >> > @@ -0,0 +1,189 @@
> >> > +/* Negative tests: each function below hits one of
> >> > + unroll_jam_reduction_possible_p's rejection clauses and the pass
> >> > + must not fire on any of them. */
> >> > +
> >> > +/* { dg-do compile } */
> >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
> >> > -fdump-tree-unrolljam-details" } */
> >> > +
> >> > +#include <stdint.h>
> >> > +
> >> > +/* Outer reduction: outer carries 'sum'. */
> >> > +int
> >> > +double_reduc (const int *a)
> >> > +{
> >> > + int sum = 0;
> >> > + for (int i = 0; i < 32; i++)
> >> > + for (int j = 0; j < 512; j++)
> >> > + sum += a[i * 512 + j];
> >> > + return sum;
> >> > +}
> >> > +
> >> > +/* Inner body has a store: inner writes a[j]. */
> >> > +void
> >> > +inner_store (int *__restrict a, int *__restrict b, const int *c, const
> >> > int *d)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + {
> >> > + a[j] = c[j] + d[j];
> >> > + s += a[j];
> >> > + }
> >> > + b[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Inner trip count just below the min-inner-iters default of 32. */
> >> > +void
> >> > +small_inner (int *out, const int *a)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 31; j++)
> >> > + s += a[i * 31 + j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Outer body stores to counter as well as out[i]. */
> >> > +extern int counter;
> >> > +void
> >> > +outer_store (int *out, const int8_t *a, const int8_t *b)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += (int) a[i * 512 + j] * (int) b[j];
> >> > + out[i] = s;
> >> > + counter++;
> >> > + }
> >> > +}
> >> > +
> >> > +/* FP accumulator: the multiply has no narrower scalar-int operand, so
> >> > + it is not a widening MAC. */
> >> > +void
> >> > +fp_mac (float *out, const float *a, const float *b)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + float s = 0.0f;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += a[i * 512 + j] * b[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Plain SImode add reduction: no multiply in the update chain. */
> >> > +void
> >> > +plain_add (int *out, const int *a)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += a[i * 512 + j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* MAX reduction: no multiply in the update chain. */
> >> > +void
> >> > +max_reduc (int *out, const int *a)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int m = a[i * 512];
> >> > + for (int j = 1; j < 512; j++)
> >> > + {
> >> > + int v = a[i * 512 + j];
> >> > + m = m > v ? m : v;
> >> > + }
> >> > + out[i] = m;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Inner header carries two reduction PHIs, m and s. */
> >> > +void
> >> > +multi_phi (int *out, const int *a)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int m = a[i * 512];
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + {
> >> > + int v = a[i * 512 + j];
> >> > + m = m > v ? m : v;
> >> > + s += v;
> >> > + }
> >> > + out[i] = m + s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Parametric outer trip count: not a known UHWI. */
> >> > +void
> >> > +parametric_outer (int *out, const int8_t *a, const int8_t *b, int n)
> >> > +{
> >> > + for (int i = 0; i < n; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += (int) a[i * 512 + j] * (int) b[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Outer trip count of 7 has no divisor in [2, 4]. */
> >> > +void
> >> > +narrow_to_one (int *__restrict out, const int8_t *a, const int8_t *b)
> >> > +{
> >> > + for (int i = 0; i < 7; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += (int) a[i * 512 + j] * (int) b[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* No __restrict: data-dep analysis cannot rule out out vs b
> >> > + aliasing. */
> >> > +void
> >> > +aliasing (int *out, const int8_t *a, const int8_t *b)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int s = 0;
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += (int) a[i * 512 + j] * (int) b[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* Conditional accumulator init: the "?:" lowers to a store-free diamond
> >> > + in the outer body whose merge PHI feeds the reduction, so the body is
> >> > + not straight-line. */
> >> > +void
> >> > +cond_init (int32_t *__restrict out, const int8_t *__restrict a,
> >> > + const int32_t *__restrict b1, const int32_t *__restrict b2,
> >> > + const int32_t *__restrict c)
> >> > +{
> >> > + for (int i = 0; i < 32; i++)
> >> > + {
> >> > + int32_t s = c[i] ? b1[i] : b2[i];
> >> > + for (int j = 0; j < 512; j++)
> >> > + s += (int32_t) a[i * 512 + j] * (int32_t) a[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* { dg-final { scan-tree-dump-not "applying reduction unroll-and-jam"
> >> > "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump-not "reduction-jam: fusing src loop"
> >> > "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump "reduction-jam: outer trip 7 has no
> >> > divisor in \\\[2, 4\\\]" "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1:
> >> > outer body has extra stores" "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1:
> >> > outer body has control flow" "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1:
> >> > outer trip count unknown" "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump "reduction-jam: data-dep safety bound
> >> > below 2" "unrolljam" } } */
> >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> >> > new file mode 100644
> >> > index 000000000000..938f7328b049
> >> > --- /dev/null
> >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> >> > @@ -0,0 +1,30 @@
> >> > +/* The reduction-jam pass should fire even when the IV-only path is
> >> > + disabled via -fno-loop-unroll-and-jam. Pins the gate change in
> >> > + pass_loop_jam::gate and the IV-only/reduction-jam fall-through in
> >> > + tree_loop_unroll_and_jam. */
> >> > +
> >> > +/* { dg-do compile } */
> >> > +/* { dg-options "-O3 -mtune=ampere1 -fno-loop-unroll-and-jam
> >> > -floop-unroll-and-jam-reductions -fdump-tree-unrolljam-details" } */
> >> > +
> >> > +#include <stdint.h>
> >> > +
> >> > +#define IN_DIM 512
> >> > +#define OUT_DIM 32
> >> > +
> >> > +void
> >> > +affine (int32_t *__restrict out,
> >> > + const int8_t *weights,
> >> > + const int32_t *biases,
> >> > + const uint8_t *input)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; ++i)
> >> > + {
> >> > + int32_t s = biases[i];
> >> > + for (int j = 0; j < IN_DIM; ++j)
> >> > + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +/* { dg-final { scan-tree-dump "applying reduction unroll-and-jam"
> >> > "unrolljam" } } */
> >> > +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 3
> >> > "unrolljam" } } */
> >> > diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> >> > b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> >> > new file mode 100644
> >> > index 000000000000..0a198bb27ecd
> >> > --- /dev/null
> >> > +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> >> > @@ -0,0 +1,72 @@
> >> > +/* Regression test for a multi-IV mispairing in the reduction-jam
> >> > + sibling matcher. The inner loop carries two IVs that share the
> >> > + preheader entry value 0 but differ in step (j has step 1 and
> >> > + indexes W, p has step 2 and indexes X); the matcher must pair each
> >> > + source IV with its own dst counterpart, not with whichever dst PHI
> >> > + happens to share an entry value.
> >> > +
> >> > + The reference loop in main uses a volatile accumulator so
> >> > + pass_loop_jam does not recognize a reduction PHI there. Without
> >> > + that guard the reference would be jammed with the same mispairing
> >> > + as gemv_2iv, hiding the miscompile behind a symmetric bug. */
> >> > +
> >> > +/* { dg-do run } */
> >> > +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" }
> >> > */
> >> > +
> >> > +#include <stdint.h>
> >> > +
> >> > +#define OUT_DIM 32
> >> > +#define IN_DIM 256
> >> > +#define STRIDE 512
> >> > +
> >> > +extern void abort (void);
> >> > +
> >> > +void __attribute__((noipa))
> >> > +gemv_2iv (int32_t *__restrict out, const int8_t *w, const uint8_t *x)
> >> > +{
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + {
> >> > + int32_t s = 0;
> >> > + int p = 0;
> >> > + for (int j = 0; j < IN_DIM; j++)
> >> > + {
> >> > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
> >> > + p += 2;
> >> > + }
> >> > + out[i] = s;
> >> > + }
> >> > +}
> >> > +
> >> > +int
> >> > +main (void)
> >> > +{
> >> > + static int8_t w[OUT_DIM * STRIDE];
> >> > + static uint8_t x[IN_DIM * 2];
> >> > + static int32_t out[OUT_DIM];
> >> > + static int32_t expected[OUT_DIM];
> >> > +
> >> > + for (int k = 0; k < OUT_DIM * STRIDE; k++)
> >> > + w[k] = (k * 13) & 0x7f;
> >> > + for (int k = 0; k < IN_DIM * 2; k++)
> >> > + x[k] = (k * 7) & 0xff;
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + {
> >> > + volatile int32_t s = 0;
> >> > + int p = 0;
> >> > + for (int j = 0; j < IN_DIM; j++)
> >> > + {
> >> > + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
> >> > + p += 2;
> >> > + }
> >> > + expected[i] = s;
> >> > + }
> >> > +
> >> > + gemv_2iv (out, w, x);
> >> > +
> >> > + for (int i = 0; i < OUT_DIM; i++)
> >> > + if (out[i] != expected[i])
> >> > + abort ();
> >> > +
> >> > + return 0;
> >> > +}
> >> > --
> >> > 2.55.0
> >> >
