On Mon, Sep 7, 2026 at 7:36 AM Konstantinos Eleftheriou
<[email protected]> wrote:
>
> Extends pass_loop_jam with a parallel-reduction branch for the
> GEMV / NNUE inference shape
>
> for (i) {
> s = bias[i];
> for (j) s += w[i][j] * x[j];
> out[i] = s;
> }
>
> Stock unroll_jam_possible_p rejects this nest because `s' is a
> reduction PHI, not a simple induction; even if it were accepted,
> stock fuse_loops would collapse the post-unroll per-sibling reduction
> PHIs into one serial chain, multiplying the inner loop's serial
> dependency height by N and defeating the goal of unroll-and-jam.
>
> unroll_jam_reduction_possible_p detects the candidate, the existing
> tree_unroll_loop strip-mines the outer loop by N, and
> unroll_jam_reduction_fuse_sibling fuses the N inner siblings into one
> body that retains N parallel reduction PHIs in its header and hoists
> each sibling's preamble (s_k = bias[i+k]) to the fused preheader. The
> post-reduction stores stay on the linear sibling-exit chain;
> cleanup_tree_cfg coalesces it and rewrite_into_loop_closed_ssa restores
> LC-SSA. The vectorizer then sees one inner loop with N independent
> chains and emits N parallel dot-product / FMA accumulators, cutting the
> serial dependency height across the reduction by N.
>
> N is the min of TARGET_LOOP_JAM_REDUCTION_FACTOR (opt-in, default 0
> = "no benefit measured"), --param=unroll-jam-reduction-streams and
> adjust_unroll_factor's data-dep safety bound, narrowed to a divisor of
> the outer trip count. The hook receives the accumulator mode plus the
> narrowest scalar-int mode found at a widening multiply in the update
> chain; op_mode == acc_mode signals "no widening multiply", letting a
> target opt in for MAC shapes only. Inner loops whose trip count is
> below --param=unroll-jam-reduction-min-inner-iters are left alone:
> their per-stream serial chain is too short to profit from parallelism.
>
> AArch64 opts in for QI/HI -> SI (DOT_PROD class) and SI -> DI
> (SMLAL_DI); all other shapes (plain reductions, max/min, FP) return 0.
> The per-CPU factor is sourced from a new tune_params field
> loop_jam_reduction_factor rather than a hardcoded return value, so the
> profitability decision lives in the tuning structure next to the other
> latency-driven knobs. Cores with measured wins ship factor 4; every
> other tuning leaves the field at 0 until measured. The new tests pin
> -mtune to a factor-4 core so they exercise that path independently of
> the configured default tuning.
>
> FP reductions are deferred to outer-loop vectorization's lane-broadcast
> FMLA path. The chained-reduction case (single accumulator carried
> across outer iterations) is left as a follow-up.
>
> ~8% speedup on SPEC CPU 2026 706.stockfish_r.
>
> gcc/ChangeLog:
>
> * common.opt (floop-unroll-and-jam-reductions): New option,
> EnabledBy(floop-unroll-and-jam).
> * config/aarch64/aarch64-protos.h (struct tune_params):
> Add loop_jam_reduction_factor.
> * config/aarch64/aarch64.cc (aarch64_loop_jam_reduction_factor):
> New function; source factor from aarch64_tune_params and opt
> in for QI/HI->SI and SI->DI MAC shapes only.
> (TARGET_LOOP_JAM_REDUCTION_FACTOR): Define.
> * config/aarch64/tuning_models/a64fx.h: Default
> loop_jam_reduction_factor to 0 (opt-out).
> * config/aarch64/tuning_models/ampere1.h: Set
> loop_jam_reduction_factor to 4.
> * config/aarch64/tuning_models/ampere1a.h: Likewise.
> * config/aarch64/tuning_models/ampere1b.h: Likewise.
> * config/aarch64/tuning_models/cortexa320.h: Default
> loop_jam_reduction_factor to 0 (opt-out).
> * config/aarch64/tuning_models/cortexa35.h: Likewise.
> * config/aarch64/tuning_models/cortexa53.h: Likewise.
> * config/aarch64/tuning_models/cortexa57.h: Likewise.
> * config/aarch64/tuning_models/cortexa72.h: Likewise.
> * config/aarch64/tuning_models/cortexa73.h: Likewise.
> * config/aarch64/tuning_models/cortexx925.h: Likewise.
> * config/aarch64/tuning_models/emag.h: Likewise.
> * config/aarch64/tuning_models/exynosm1.h: Likewise.
> * config/aarch64/tuning_models/fujitsu_monaka.h: Likewise.
> * config/aarch64/tuning_models/generic.h: Likewise.
> * config/aarch64/tuning_models/generic_armv8_a.h: Likewise.
> * config/aarch64/tuning_models/generic_armv9_a.h: Likewise.
> * config/aarch64/tuning_models/hip12.h: Likewise.
> * config/aarch64/tuning_models/neoverse512tvb.h: Likewise.
> * config/aarch64/tuning_models/neoversen1.h: Set
> loop_jam_reduction_factor to 4.
> * config/aarch64/tuning_models/neoversen2.h: Default
> loop_jam_reduction_factor to 0 (opt-out).
> * config/aarch64/tuning_models/neoversen3.h: Likewise.
> * config/aarch64/tuning_models/neoversev1.h: Likewise.
> * config/aarch64/tuning_models/neoversev2.h: Likewise.
> * config/aarch64/tuning_models/neoversev3.h: Likewise.
> * config/aarch64/tuning_models/neoversev3ae.h: Likewise.
> * config/aarch64/tuning_models/olympus.h: Likewise.
> * config/aarch64/tuning_models/qdf24xx.h: Likewise.
> * config/aarch64/tuning_models/saphira.h: Likewise.
> * config/aarch64/tuning_models/thunderx.h: Likewise.
> * config/aarch64/tuning_models/thunderx2t99.h: Likewise.
> * config/aarch64/tuning_models/thunderx3t110.h: Likewise.
> * config/aarch64/tuning_models/thunderxt88.h: Likewise.
> * config/aarch64/tuning_models/tsv110.h: Likewise.
> * config/aarch64/tuning_models/xgene1.h: Likewise.
> * doc/invoke.texi (-floop-unroll-and-jam-reductions): Document.
> * doc/tm.texi: Regenerate.
> * doc/tm.texi.in (TARGET_LOOP_JAM_REDUCTION_FACTOR): Add hook
> reference.
> * gimple-loop-jam.cc: Include cfganal.h and tree-into-ssa.h.
> (inner_reduction_phi): New helper.
> (narrowest_reduc_input_mode): New helper that walks the
> reduction update chain for widening multiplies.
> (gsi_move_to_bb_end_before_terminator): New helper.
> (collapse_single_arg_phi): New helper.
> (loop_is_dot_product_candidate_p): New static helper.
> (unroll_jam_reduction_possible_p): New candidate detector;
> calls loop_is_dot_product_candidate_p and layers outer-loop
> guards on top.
> (migrate_loop_invariant_defs_to_preheader): New helper that
> hoists the SSA-use closure of a sibling's reduction init and
> body operands to the destination preheader in RPO.
> (match_sibling_header_phis): New helper pairing sibling non-
> reduction PHIs positionally.
> (unroll_jam_reduction_fuse_sibling): New per-sibling fuser.
> (unroll_jam_reduction_bypass_sibling): New CFG bypass for the
> drained source sibling.
> (compute_jam_dependence_factors): New helper. Walks OUTER's
> data dependences and accumulates the unroll-factor safety
> bound, ref-removal profit signal, removed-ref count, and total
> dataref count.
> (select_reduction_jam_factor): New helper combining target
> factor, param cap and data-dep safety bound, narrowed to a
> divisor of the outer trip count. Emit a TDF_DETAILS note
> when the outer trip has no divisor in [2, factor cap].
> (do_unroll_jam_reductions): New driver helper.
> (tree_loop_unroll_and_jam): Try the reduction-jam path on
> nests the IV-only path skips; cleanup CFG, rebuild LC-SSA and
> reset SCEV per transformed nest.
> (pass_loop_jam::gate): Also enable for
> flag_unroll_jam_reductions.
> * hooks.cc (hook_uint_mode_mode_0): New default hook.
> * hooks.h (hook_uint_mode_mode_0): Declare.
> * params.opt (unroll-jam-reduction-streams): New param.
> (unroll-jam-reduction-min-inner-iters): New param.
> * target.def (loop_jam_reduction_factor): New target hook.
>
> gcc/testsuite/ChangeLog:
>
> * gcc.target/aarch64/unroll-jam-reductions-1.c: New test.
> * gcc.target/aarch64/unroll-jam-reductions-2.c: New test.
> * gcc.target/aarch64/unroll-jam-reductions-3.c: New test.
> * gcc.target/aarch64/unroll-jam-reductions-4.c: New test.
> * gcc.target/aarch64/unroll-jam-reductions-5.c: New test.
>
> Co-Authored-By: Philipp Tomsich <[email protected]>
> Signed-off-by: Konstantinos Eleftheriou <[email protected]>
> ---
>
> gcc/common.opt | 5 +
> gcc/config/aarch64/aarch64-protos.h | 8 +
> gcc/config/aarch64/aarch64.cc | 29 +
> gcc/config/aarch64/tuning_models/a64fx.h | 3 +-
> gcc/config/aarch64/tuning_models/ampere1.h | 3 +-
> gcc/config/aarch64/tuning_models/ampere1a.h | 3 +-
> gcc/config/aarch64/tuning_models/ampere1b.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa320.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa35.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa53.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa57.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa72.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexa73.h | 3 +-
> gcc/config/aarch64/tuning_models/cortexx925.h | 3 +-
> gcc/config/aarch64/tuning_models/emag.h | 3 +-
> gcc/config/aarch64/tuning_models/exynosm1.h | 3 +-
> .../aarch64/tuning_models/fujitsu_monaka.h | 3 +-
> gcc/config/aarch64/tuning_models/generic.h | 3 +-
> .../aarch64/tuning_models/generic_armv8_a.h | 3 +-
> .../aarch64/tuning_models/generic_armv9_a.h | 3 +-
> gcc/config/aarch64/tuning_models/hip12.h | 3 +-
> .../aarch64/tuning_models/neoverse512tvb.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversen1.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversen2.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversen3.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversev1.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversev2.h | 3 +-
> gcc/config/aarch64/tuning_models/neoversev3.h | 3 +-
> .../aarch64/tuning_models/neoversev3ae.h | 3 +-
> gcc/config/aarch64/tuning_models/olympus.h | 3 +-
> gcc/config/aarch64/tuning_models/qdf24xx.h | 3 +-
> gcc/config/aarch64/tuning_models/saphira.h | 3 +-
> gcc/config/aarch64/tuning_models/thunderx.h | 3 +-
> .../aarch64/tuning_models/thunderx2t99.h | 3 +-
> .../aarch64/tuning_models/thunderx3t110.h | 3 +-
> .../aarch64/tuning_models/thunderxt88.h | 3 +-
> gcc/config/aarch64/tuning_models/tsv110.h | 3 +-
> gcc/config/aarch64/tuning_models/xgene1.h | 3 +-
> gcc/doc/invoke.texi | 17 +-
> gcc/doc/tm.texi | 11 +
> gcc/doc/tm.texi.in | 2 +
> gcc/gimple-loop-jam.cc | 932 +++++++++++++++++-
> gcc/hooks.cc | 7 +
> gcc/hooks.h | 1 +
> gcc/params.opt | 8 +
> gcc/target.def | 13 +
> .../aarch64/unroll-jam-reductions-1.c | 78 ++
> .../aarch64/unroll-jam-reductions-2.c | 89 ++
> .../aarch64/unroll-jam-reductions-3.c | 189 ++++
> .../aarch64/unroll-jam-reductions-4.c | 30 +
> .../aarch64/unroll-jam-reductions-5.c | 72 ++
> 51 files changed, 1557 insertions(+), 39 deletions(-)
> create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
>
> diff --git a/gcc/common.opt b/gcc/common.opt
> index 69a55b8d40d6..634b63569c67 100644
> --- a/gcc/common.opt
> +++ b/gcc/common.opt
> @@ -1928,6 +1928,11 @@ floop-unroll-and-jam
> Common Var(flag_unroll_jam) Optimization
> Perform unroll-and-jam on loops.
>
> +floop-unroll-and-jam-reductions
> +Common Var(flag_unroll_jam_reductions) Optimization
> EnabledBy(floop-unroll-and-jam)
> +Perform reduction-aware unroll-and-jam on loops with inner reductions
> +to enable multi-stream parallelism during vectorization.
> +
> fgnu-tm
> Common Var(flag_tm)
> Enable support for GNU transactional memory.
> diff --git a/gcc/config/aarch64/aarch64-protos.h
> b/gcc/config/aarch64/aarch64-protos.h
> index e8ae3d427949..88237e243738 100644
> --- a/gcc/config/aarch64/aarch64-protos.h
> +++ b/gcc/config/aarch64/aarch64-protos.h
> @@ -590,6 +590,14 @@ struct tune_params
>
> /* Dispatch constraints for instruction scheduling. */
> const struct dispatch_constraint_info *dispatch_constraints;
> +
> + /* Preferred number of parallel reduction streams for
> + -floop-unroll-and-jam-reductions on this core. Zero opts out;
> + two or more enables the transform on reduction nests whose
> + (acc_mode, op_mode) pair the target hook accepts. Cores with a
> + two-wide int/FP pipe and several-cycle MAC latency typically
> + want 4. */
> + unsigned int loop_jam_reduction_factor;
> };
>
> /* Classifies an address.
> diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
> index ccb4f5fbb10d..546912534499 100644
> --- a/gcc/config/aarch64/aarch64.cc
> +++ b/gcc/config/aarch64/aarch64.cc
> @@ -17402,6 +17402,32 @@ aarch64_sched_issue_rate (void)
> return aarch64_tune_params.issue_rate;
> }
>
> +/* Implement TARGET_LOOP_JAM_REDUCTION_FACTOR. Return the preferred
> + number of parallel reduction streams for
> + -floop-unroll-and-jam-reductions on a reduction accumulating into
> + ACC_MODE from operands of OP_MODE. We opt in only for the widening
> + MAC shapes the vectorizer lowers to a latency-bound
> + DOT_PROD / SMLAL chain:
> + * QImode / HImode -> SImode (DOT_PROD-class, USDOT / SDOT / UDOT)
> + * SImode -> DImode (SMLAL_DI)
> + The stream count comes from the per-core tuning structure; cores
> + that have not been measured leave loop_jam_reduction_factor at 0,
> + which suppresses the transform. */
> +
> +static unsigned int
> +aarch64_loop_jam_reduction_factor (machine_mode acc_mode,
> + machine_mode op_mode)
> +{
> + unsigned int factor = aarch64_tune_params.loop_jam_reduction_factor;
> + if (factor < 2 || !SCALAR_INT_MODE_P (acc_mode))
> + return 0;
> +
> + if ((acc_mode == SImode && (op_mode == QImode || op_mode == HImode))
> + || (acc_mode == DImode && op_mode == SImode))
> + return factor;
> + return 0;
> +}
> +
> /* Implement TARGET_SCHED_VARIABLE_ISSUE. */
> static int
> aarch64_sched_variable_issue (FILE *, int, rtx_insn *insn, int more)
> @@ -34491,6 +34517,9 @@ aarch64_libgcc_floating_mode_supported_p
> #undef TARGET_SCHED_ISSUE_RATE
> #define TARGET_SCHED_ISSUE_RATE aarch64_sched_issue_rate
>
> +#undef TARGET_LOOP_JAM_REDUCTION_FACTOR
> +#define TARGET_LOOP_JAM_REDUCTION_FACTOR aarch64_loop_jam_reduction_factor
> +
> #undef TARGET_SCHED_VARIABLE_ISSUE
> #define TARGET_SCHED_VARIABLE_ISSUE aarch64_sched_variable_issue
>
> diff --git a/gcc/config/aarch64/tuning_models/a64fx.h
> b/gcc/config/aarch64/tuning_models/a64fx.h
> index c3222960f175..171e64ce23ad 100644
> --- a/gcc/config/aarch64/tuning_models/a64fx.h
> +++ b/gcc/config/aarch64/tuning_models/a64fx.h
> @@ -166,7 +166,8 @@ static const struct tune_params a64fx_tunings =
> &a64fx_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_A64FX. */
> diff --git a/gcc/config/aarch64/tuning_models/ampere1.h
> b/gcc/config/aarch64/tuning_models/ampere1.h
> index e2a42f585dd9..0057db3765b4 100644
> --- a/gcc/config/aarch64/tuning_models/ampere1.h
> +++ b/gcc/config/aarch64/tuning_models/ampere1.h
> @@ -136,7 +136,8 @@ static const struct tune_params ampere1_tunings =
> &ere1_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 4 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_AMPERE1. */
> diff --git a/gcc/config/aarch64/tuning_models/ampere1a.h
> b/gcc/config/aarch64/tuning_models/ampere1a.h
> index 0954ac551a0b..1350c0f3b5c5 100644
> --- a/gcc/config/aarch64/tuning_models/ampere1a.h
> +++ b/gcc/config/aarch64/tuning_models/ampere1a.h
> @@ -58,7 +58,8 @@ static const struct tune_params ampere1a_tunings =
> &ere1_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 4 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_AMPERE1A. */
> diff --git a/gcc/config/aarch64/tuning_models/ampere1b.h
> b/gcc/config/aarch64/tuning_models/ampere1b.h
> index 8ba434d27bbf..0fc6eb36bd2a 100644
> --- a/gcc/config/aarch64/tuning_models/ampere1b.h
> +++ b/gcc/config/aarch64/tuning_models/ampere1b.h
> @@ -137,7 +137,8 @@ static const struct tune_params ampere1b_tunings =
> &ere1b_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 4 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_AMPERE1B */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa320.h
> b/gcc/config/aarch64/tuning_models/cortexa320.h
> index d5e1f2b29938..89ed24360008 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa320.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa320.h
> @@ -189,7 +189,8 @@ static const struct tune_params cortexa320_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXA320. */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa35.h
> b/gcc/config/aarch64/tuning_models/cortexa35.h
> index fd2fc47f5e23..ff9eeb406799 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa35.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa35.h
> @@ -57,7 +57,8 @@ static const struct tune_params cortexa35_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXA35. */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa53.h
> b/gcc/config/aarch64/tuning_models/cortexa53.h
> index e71cc1b26afb..13dbb152c6e9 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa53.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa53.h
> @@ -66,7 +66,8 @@ static const struct tune_params cortexa53_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXA53. */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa57.h
> b/gcc/config/aarch64/tuning_models/cortexa57.h
> index 4f3ec60ba275..c78c15198158 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa57.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa57.h
> @@ -103,7 +103,8 @@ static const struct tune_params cortexa57_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXA57. */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa72.h
> b/gcc/config/aarch64/tuning_models/cortexa72.h
> index df0fcf110867..8a4e5e9d8a36 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa72.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa72.h
> @@ -55,7 +55,8 @@ static const struct tune_params cortexa72_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXA72. */
> diff --git a/gcc/config/aarch64/tuning_models/cortexa73.h
> b/gcc/config/aarch64/tuning_models/cortexa73.h
> index c871b454d40f..40d46f23ae82 100644
> --- a/gcc/config/aarch64/tuning_models/cortexa73.h
> +++ b/gcc/config/aarch64/tuning_models/cortexa73.h
> @@ -56,7 +56,8 @@ static const struct tune_params cortexa73_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
>
> diff --git a/gcc/config/aarch64/tuning_models/cortexx925.h
> b/gcc/config/aarch64/tuning_models/cortexx925.h
> index 2a73b3697065..b99d6b4300e5 100644
> --- a/gcc/config/aarch64/tuning_models/cortexx925.h
> +++ b/gcc/config/aarch64/tuning_models/cortexx925.h
> @@ -227,7 +227,8 @@ static const struct tune_params cortexx925_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_CORTEXX925. */
> diff --git a/gcc/config/aarch64/tuning_models/emag.h
> b/gcc/config/aarch64/tuning_models/emag.h
> index 82bbc554f41b..6e87d28173b2 100644
> --- a/gcc/config/aarch64/tuning_models/emag.h
> +++ b/gcc/config/aarch64/tuning_models/emag.h
> @@ -55,7 +55,8 @@ static const struct tune_params emag_tunings =
> &xgene1_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_EMAG. */
> diff --git a/gcc/config/aarch64/tuning_models/exynosm1.h
> b/gcc/config/aarch64/tuning_models/exynosm1.h
> index a561ccf6fa71..928f3d4f3014 100644
> --- a/gcc/config/aarch64/tuning_models/exynosm1.h
> +++ b/gcc/config/aarch64/tuning_models/exynosm1.h
> @@ -139,7 +139,8 @@ static const struct tune_params exynosm1_tunings =
> &exynosm1_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_EXYNOSM1. */
> diff --git a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> index 71549bd0ea70..70d71bf253de 100644
> --- a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> +++ b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
> @@ -59,7 +59,8 @@ static const struct tune_params fujitsu_monaka_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_FUJITSU_MONAKA. */
> diff --git a/gcc/config/aarch64/tuning_models/generic.h
> b/gcc/config/aarch64/tuning_models/generic.h
> index af013e478d45..71dbebeb47c0 100644
> --- a/gcc/config/aarch64/tuning_models/generic.h
> +++ b/gcc/config/aarch64/tuning_models/generic.h
> @@ -188,7 +188,8 @@ static const struct tune_params generic_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_GENERIC. */
> diff --git a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> index 29cf3a5a7c7a..15900a7ba733 100644
> --- a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> +++ b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
> @@ -180,7 +180,8 @@ static const struct tune_params generic_armv8_a_tunings =
> &generic_armv8_a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_GENERIC_ARMV8_A. */
> diff --git a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> index 05e777d20efb..7c1b78f88a53 100644
> --- a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> +++ b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
> @@ -255,7 +255,8 @@ static const struct tune_params generic_armv9_a_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_GENERIC_ARMV9_A. */
> diff --git a/gcc/config/aarch64/tuning_models/hip12.h
> b/gcc/config/aarch64/tuning_models/hip12.h
> index cfb204b2e533..2d5c107d71d7 100644
> --- a/gcc/config/aarch64/tuning_models/hip12.h
> +++ b/gcc/config/aarch64/tuning_models/hip12.h
> @@ -221,7 +221,8 @@ static const struct tune_params hip12_tunings =
> &generic_armv8_a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_HIP12. */
> diff --git a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> index d6e2c2e8120d..a128d5ed7b3a 100644
> --- a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> +++ b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
> @@ -162,7 +162,8 @@ static const struct tune_params neoverse512tvb_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSE512TVB. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversen1.h
> b/gcc/config/aarch64/tuning_models/neoversen1.h
> index 87c7fbcb7f22..9b2ec9322bba 100644
> --- a/gcc/config/aarch64/tuning_models/neoversen1.h
> +++ b/gcc/config/aarch64/tuning_models/neoversen1.h
> @@ -55,7 +55,8 @@ static const struct tune_params neoversen1_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 4 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEN1. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversen2.h
> b/gcc/config/aarch64/tuning_models/neoversen2.h
> index 21597e6b50f8..e1648c88eb78 100644
> --- a/gcc/config/aarch64/tuning_models/neoversen2.h
> +++ b/gcc/config/aarch64/tuning_models/neoversen2.h
> @@ -224,7 +224,8 @@ static const struct tune_params neoversen2_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEN2. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversen3.h
> b/gcc/config/aarch64/tuning_models/neoversen3.h
> index 37f9819148f6..8796914772b4 100644
> --- a/gcc/config/aarch64/tuning_models/neoversen3.h
> +++ b/gcc/config/aarch64/tuning_models/neoversen3.h
> @@ -223,7 +223,8 @@ static const struct tune_params neoversen3_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEN3. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversev1.h
> b/gcc/config/aarch64/tuning_models/neoversev1.h
> index 253f11e87a68..7f93e4092550 100644
> --- a/gcc/config/aarch64/tuning_models/neoversev1.h
> +++ b/gcc/config/aarch64/tuning_models/neoversev1.h
> @@ -233,7 +233,8 @@ static const struct tune_params neoversev1_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
>
> diff --git a/gcc/config/aarch64/tuning_models/neoversev2.h
> b/gcc/config/aarch64/tuning_models/neoversev2.h
> index 6df0cc444b80..8ceb7d5538fb 100644
> --- a/gcc/config/aarch64/tuning_models/neoversev2.h
> +++ b/gcc/config/aarch64/tuning_models/neoversev2.h
> @@ -364,7 +364,8 @@ static const struct tune_params neoversev2_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - &neoversev2_dispatch_constraint_info /* dispatch_constraints. */
> + &neoversev2_dispatch_constraint_info, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEV2. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversev3.h
> b/gcc/config/aarch64/tuning_models/neoversev3.h
> index 7e0144032dc6..ea840d2cbb7c 100644
> --- a/gcc/config/aarch64/tuning_models/neoversev3.h
> +++ b/gcc/config/aarch64/tuning_models/neoversev3.h
> @@ -225,7 +225,8 @@ static const struct tune_params neoversev3_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEV3. */
> diff --git a/gcc/config/aarch64/tuning_models/neoversev3ae.h
> b/gcc/config/aarch64/tuning_models/neoversev3ae.h
> index bbe4dc547015..fe39c7b67c9d 100644
> --- a/gcc/config/aarch64/tuning_models/neoversev3ae.h
> +++ b/gcc/config/aarch64/tuning_models/neoversev3ae.h
> @@ -225,7 +225,8 @@ static const struct tune_params neoversev3ae_tunings =
> &generic_armv9a_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_NEOVERSEV3AE. */
> diff --git a/gcc/config/aarch64/tuning_models/olympus.h
> b/gcc/config/aarch64/tuning_models/olympus.h
> index 1e701722e224..ab1ba2464d23 100644
> --- a/gcc/config/aarch64/tuning_models/olympus.h
> +++ b/gcc/config/aarch64/tuning_models/olympus.h
> @@ -565,7 +565,8 @@ static struct tune_params olympus_tunings =
> &olympus_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - &olympus_dispatch_constraint_info /* dispatch_constraints. */
> + &olympus_dispatch_constraint_info, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_OLYMPUS. */
> diff --git a/gcc/config/aarch64/tuning_models/qdf24xx.h
> b/gcc/config/aarch64/tuning_models/qdf24xx.h
> index a15a4fc56897..73eb0b8c48cf 100644
> --- a/gcc/config/aarch64/tuning_models/qdf24xx.h
> +++ b/gcc/config/aarch64/tuning_models/qdf24xx.h
> @@ -131,7 +131,8 @@ static const struct tune_params qdf24xx_tunings =
> &qdf24xx_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_QDF24XX. */
> diff --git a/gcc/config/aarch64/tuning_models/saphira.h
> b/gcc/config/aarch64/tuning_models/saphira.h
> index 369d3618f7f1..582c4517fcdc 100644
> --- a/gcc/config/aarch64/tuning_models/saphira.h
> +++ b/gcc/config/aarch64/tuning_models/saphira.h
> @@ -57,7 +57,8 @@ static const struct tune_params saphira_tunings =
> &generic_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_SAPHIRA. */
> diff --git a/gcc/config/aarch64/tuning_models/thunderx.h
> b/gcc/config/aarch64/tuning_models/thunderx.h
> index ed8de307276a..a275659c0b9e 100644
> --- a/gcc/config/aarch64/tuning_models/thunderx.h
> +++ b/gcc/config/aarch64/tuning_models/thunderx.h
> @@ -112,7 +112,8 @@ static const struct tune_params thunderx_tunings =
> &thunderx_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_THUNDERX. */
> diff --git a/gcc/config/aarch64/tuning_models/thunderx2t99.h
> b/gcc/config/aarch64/tuning_models/thunderx2t99.h
> index f6be657ac9c3..5afbc21ef40c 100644
> --- a/gcc/config/aarch64/tuning_models/thunderx2t99.h
> +++ b/gcc/config/aarch64/tuning_models/thunderx2t99.h
> @@ -132,7 +132,8 @@ static const struct tune_params thunderx2t99_tunings =
> &thunderx2t99_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_THUNDERX2T99. */
> diff --git a/gcc/config/aarch64/tuning_models/thunderx3t110.h
> b/gcc/config/aarch64/tuning_models/thunderx3t110.h
> index cdcbec680945..0d7f29bee209 100644
> --- a/gcc/config/aarch64/tuning_models/thunderx3t110.h
> +++ b/gcc/config/aarch64/tuning_models/thunderx3t110.h
> @@ -131,7 +131,8 @@ static const struct tune_params thunderx3t110_tunings =
> &thunderx3t110_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_THUNDERX3T110. */
> diff --git a/gcc/config/aarch64/tuning_models/thunderxt88.h
> b/gcc/config/aarch64/tuning_models/thunderxt88.h
> index 44a1eac59a96..d0c39c07c88a 100644
> --- a/gcc/config/aarch64/tuning_models/thunderxt88.h
> +++ b/gcc/config/aarch64/tuning_models/thunderxt88.h
> @@ -67,7 +67,8 @@ static const struct tune_params thunderxt88_tunings =
> &thunderxt88_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_THUNDERXT88. */
> diff --git a/gcc/config/aarch64/tuning_models/tsv110.h
> b/gcc/config/aarch64/tuning_models/tsv110.h
> index 366f2b3e2323..0800bdc59ced 100644
> --- a/gcc/config/aarch64/tuning_models/tsv110.h
> +++ b/gcc/config/aarch64/tuning_models/tsv110.h
> @@ -132,7 +132,8 @@ static const struct tune_params tsv110_tunings =
> &tsv110_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_TSV110. */
> diff --git a/gcc/config/aarch64/tuning_models/xgene1.h
> b/gcc/config/aarch64/tuning_models/xgene1.h
> index 0d4aaf80b61a..5fd5e06932c3 100644
> --- a/gcc/config/aarch64/tuning_models/xgene1.h
> +++ b/gcc/config/aarch64/tuning_models/xgene1.h
> @@ -140,7 +140,8 @@ static const struct tune_params xgene1_tunings =
> &xgene1_prefetch_tune,
> AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
> AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
> - nullptr /* dispatch_constraints. */
> + nullptr, /* dispatch_constraints. */
> + 0 /* loop_jam_reduction_factor. */
> };
>
> #endif /* GCC_AARCH64_H_XGENE1. */
> diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
> index a74e99081d35..9f3215deb942 100644
> --- a/gcc/doc/invoke.texi
> +++ b/gcc/doc/invoke.texi
> @@ -641,7 +641,8 @@ Objective-C and Objective-C++ Dialects}.
> -fivopts -fkeep-inline-functions -fkeep-static-functions
> -fkeep-static-consts -flimit-function-alignment -flive-range-shrinkage
> -floop-block -floop-interchange -floop-strip-mine
> --floop-unroll-and-jam -floop-nest-optimize
> +-floop-unroll-and-jam -floop-unroll-and-jam-reductions
> +-floop-nest-optimize
> -floop-parallelize-all -flra-remat -flto -flto-compression-level=@var{n}
> -flto-toplevel-asm-heuristics
> -flto-partition=@var{alg} -flto-incremental=@var{path}
> @@ -15307,6 +15308,20 @@ nest this unrolls the outer loop by some factor and
> fuses the resulting
> multiple inner loops. This flag is enabled by default at @option{-O3}.
> It is also enabled by @option{-fprofile-use} and @option{-fauto-profile}.
>
> +@opindex floop-unroll-and-jam-reductions
> +@opindex fno-loop-unroll-and-jam-reductions
> +@item -floop-unroll-and-jam-reductions
> +Apply a reduction-aware variant of unroll and jam to loop nests whose
> +inner loop carries a reduction accumulator reset every outer iteration
> +(the GEMV / dot-product shape), producing N parallel reduction streams.
> +N is taken from @code{TARGET_LOOP_JAM_REDUCTION_FACTOR} (no-op on
> +targets that do not opt in) and capped by
> +@option{--param=unroll-jam-reduction-streams}. Inner loops whose
> +trip count is below @option{--param=unroll-jam-reduction-min-inner-iters}
> +are left alone, because the per-stream serial chain is too short to
> +profit from parallelism. The outer loop trip count must be a
> +compile-time constant. Enabled whenever @option{-floop-unroll-and-jam} is.
> +
> @opindex ftree-loop-im
> @opindex fno-tree-loop-im
> @item -ftree-loop-im
> diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
> index ca68bdc3e615..be0771e5348a 100644
> --- a/gcc/doc/tm.texi
> +++ b/gcc/doc/tm.texi
> @@ -12653,6 +12653,17 @@ is required only when the target has special
> constraints like maximum
> number of memory accesses.
> @end deftypefn
>
> +@deftypefn {Target Hook} unsigned TARGET_LOOP_JAM_REDUCTION_FACTOR
> (machine_mode @var{acc_mode}, machine_mode @var{op_mode})
> +Return the preferred number of parallel reduction streams
> +@option{-floop-unroll-and-jam-reductions} should produce for an inner
> +reduction that accumulates into mode @var{acc_mode} from operands of
> +mode @var{op_mode}. @var{op_mode} is the operand mode at the source
> +of the widening multiply, and is always strictly narrower than
> +@var{acc_mode}: plain reductions with no widening multiply are
> +filtered out before the hook is consulted. Returning less than two
> +disables the transform for that mode pair.
> +@end deftypefn
> +
> @defmac POWI_MAX_MULTS
> If defined, this macro is interpreted as a signed integer C expression
> that specifies the maximum number of floating point multiplications
> diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
> index 532e95ef216f..027a311f310e 100644
> --- a/gcc/doc/tm.texi.in
> +++ b/gcc/doc/tm.texi.in
> @@ -8011,6 +8011,8 @@ lists.
>
> @hook TARGET_LOOP_UNROLL_ADJUST
>
> +@hook TARGET_LOOP_JAM_REDUCTION_FACTOR
> +
> @defmac POWI_MAX_MULTS
> If defined, this macro is interpreted as a signed integer C expression
> that specifies the maximum number of floating point multiplications
> diff --git a/gcc/gimple-loop-jam.cc b/gcc/gimple-loop-jam.cc
> index 2c4162d17a2a..527f7c6cce35 100644
> --- a/gcc/gimple-loop-jam.cc
> +++ b/gcc/gimple-loop-jam.cc
> @@ -34,12 +34,14 @@ along with GCC; see the file COPYING3. If not see
> #include "cfgloop.h"
> #include "tree-scalar-evolution.h"
> #include "gimple-iterator.h"
> +#include "cfganal.h"
> #include "cfghooks.h"
> #include "tree-data-ref.h"
> #include "tree-ssa-loop-ivopts.h"
> #include "tree-vectorizer.h"
> #include "tree-ssa-sccvn.h"
> #include "tree-cfgcleanup.h"
> +#include "tree-into-ssa.h"
>
> /* Unroll and Jam transformation
>
> @@ -376,6 +378,684 @@ fuse_loops (class loop *loop)
> }
> }
>
> +/* Reduction-jam: a parallel-reduction extension of unroll-and-jam.
> +
> + The transformation above accepts only inner PHIs that are simple
> + inductions. A reduction PHI cannot be fused the same way: merging
> + the streams would chain them into a single serial accumulation,
> + multiplying the serial dependency height instead of cutting it.
> + This handles the GEMV / NNUE shape
> +
> + for (i = 0; i < M; i++) {
> + s = bias[i];
> + for (j = 0; j < K; j++) s += w[i][j] * x[j];
> + out[i] = s;
> + }
> +
> + where s starts afresh every outer iteration. Unrolling the outer
> + loop by N leaves N sibling inner loops, each with its own reduction
> + PHI and its own init (bias[i], bias[i+1], ...). Instead of merging
> + them we keep all N PHIs in the fused header and hoist each sibling's
> + init ahead of the body; the post-reduction stores stay on the
> + sibling-exit chain, coalesced afterward. The vectorizer then emits
> + N independent DOT_PROD / FMA chains, so the serial dependency height
> + across the reduction drops by a factor of N.
> +
> + The strip factor N is the smallest of the target's
> + TARGET_LOOP_JAM_REDUCTION_FACTOR, param_unroll_jam_reduction_streams
> + and the data-dep safety bound, narrowed to a divisor of the outer
> + trip count. We ignore adjust_unroll_factor's profit signal on
> + purpose: it scores the memory refs that fusion removes, which is
> + not what this transform is after. Targets that have not measured a
> + win return 0 from the hook and we skip the loop. See params.opt
> + for the tunables. */
> +
> +/* Return the single non-virtual, non-simple-IV PHI in LOOP's header
> + (the reduction accumulator), or NULL if there is none or more than
> + one. */
> +
> +static gphi *
> +inner_reduction_phi (class loop *loop)
> +{
> + gphi *result = NULL;
> + for (gphi_iterator psi = gsi_start_phis (loop->header);
> + !gsi_end_p (psi); gsi_next (&psi))
> + {
> + gphi *phi = psi.phi ();
> + tree op = gimple_phi_result (phi);
> + if (virtual_operand_p (op))
> + continue;
> + affine_iv iv;
> + if (simple_iv (loop, loop, op, &iv, true))
> + continue;
> + if (result)
> + return NULL;
> + result = phi;
> + }
> + return result;
> +}
> +
> +/* Walk the SSA def chain from RED_PHI's latch update and return the
> + narrowest scalar-int mode feeding a multiply inside INNER, or
> + ACC_MODE if nothing narrower turns up. The walk follows only
> + PLUS / MINUS / CONVERT chains so that a multiply buried in an array
> + index (the "w * j" in "s += a[w * j]") is not mistaken for the MAC
> + source. */
I am not understanding why you need this function?
> +
> +static machine_mode
> +narrowest_reduc_input_mode (gphi *red_phi, class loop *inner,
> + machine_mode acc_mode)
> +{
> + tree update = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner));
> + if (TREE_CODE (update) != SSA_NAME)
> + return acc_mode;
> +
> + machine_mode narrow = VOIDmode;
> + auto_vec<tree, 8> worklist;
> + hash_set<tree> seen;
Both of these should be bitmaps.
At least seen should be.
> + worklist.safe_push (update);
> +
> + while (!worklist.is_empty ())
> + {
> + tree op = worklist.pop ();
> + if (seen.add (op))
> + continue;
> + gimple *def = SSA_NAME_DEF_STMT (op);
> + if (gimple_code (def) == GIMPLE_PHI
is_a<gphi*> (def)
> + || !gimple_bb (def)
> + || !flow_bb_inside_loop_p (inner, gimple_bb (def))
is inner always an inner most loop?
If so then this check can just be gimple_bb (def)->loop_father != inner
> + || !is_gimple_assign (def))
!is_a<gassign*> (def)
And then move this first and remove the is_a<gphi*> check.
> + continue;
> +
> + /* Stay in the reduction value chain. Loads and pointer
> + arithmetic feed address computation, not the accumulator. */
> + if (gimple_assign_load_p (def)
You can just check gimple_vuse earlier because I assume you also want
to ignore stores.
> + || POINTER_TYPE_P (TREE_TYPE (gimple_assign_lhs (def))))
gimple_assign_lhs (def) better be op at this point. So you can just use op.
> + continue;
> +
> + use_operand_p use_p;
> + ssa_op_iter oi;
> + tree_code code = gimple_assign_rhs_code (def);
> + if (code == MULT_EXPR || code == WIDEN_MULT_EXPR)
> + {
> + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
> + {
> + tree mop = USE_FROM_PTR (use_p);
> + while (TREE_CODE (mop) == SSA_NAME)
> + {
> + gimple *mdef = SSA_NAME_DEF_STMT (mop);
> + if (!is_gimple_assign (mdef)
> + || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (mdef)))
> + break;
> + mop = gimple_assign_rhs1 (mdef);
> + }
This loop needs a comment, I am trying to understand why you are
skipping over converts.
> + machine_mode m = TYPE_MODE (TREE_TYPE (mop));
> + if (SCALAR_INT_MODE_P (m)
> + && (narrow == VOIDmode
> + || known_lt (GET_MODE_PRECISION (m),
> + GET_MODE_PRECISION (narrow))))
> + narrow = m;
> + }
> + continue;
> + }
> +
> + /* Only scalar-reduction operations continue the walk. */
> + if (code != PLUS_EXPR
> + && code != MINUS_EXPR
> + && !CONVERT_EXPR_CODE_P (code))
> + continue;
> +
> + FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
> + worklist.safe_push (USE_FROM_PTR (use_p));
> + }
> +
> + if (narrow == VOIDmode
> + || known_ge (GET_MODE_PRECISION (narrow),
> + GET_MODE_PRECISION (acc_mode)))
> + return acc_mode;
> + return narrow;
> +}
> +
> +/* Move *FROM to BB, immediately before BB's terminator branch (or to
> + the end of BB if there is none). */
> +
> +static void
> +gsi_move_to_bb_end_before_terminator (gimple_stmt_iterator *from,
> + basic_block bb)
> +{
> + gimple_stmt_iterator to = gsi_last_bb (bb);
> + if (!gsi_end_p (to) && is_ctrl_stmt (gsi_stmt (to)))
> + gsi_move_before (from, &to);
> + else
> + gsi_move_after (from, &to);
> +}
> +
> +/* Propagate the single argument of the PHI at PSI into its uses and
> + remove the PHI, leaving PSI at the next one. */
> +
> +static void
> +collapse_single_arg_phi (gphi_iterator *psi)
> +{
> + gphi *phi = psi->phi ();
> + gcc_checking_assert (gimple_phi_num_args (phi) == 1);
> + tree result = gimple_phi_result (phi);
> + if (!has_zero_uses (result))
> + replace_uses_by (result, gimple_phi_arg_def (phi, 0));
> + remove_phi_node (psi, true);
> +}
> +
> +/* True if LOOP is a widening MAC reduction the vectorizer can turn
> + into a DOT_PROD_EXPR: a single-BB body with one exit, loads only, a
> + constant trip count of at least MIN_INNER_ITERS, one non-IV
> + reduction PHI alongside a unit-step counter IV, and a widening
> + multiply in the update chain. If OUTER is given, every IV's base
> + and step must be invariant in it. On success *OUT_RED_PHI,
> + *OUT_ACC_MODE and *OUT_OP_MODE receive the reduction PHI, the
> + accumulator mode and the narrowest operand mode. */
> +
> +static bool
> +loop_is_dot_product_candidate_p (class loop *loop,
> + unsigned HOST_WIDE_INT min_inner_iters,
> + class loop *outer,
> + gphi **out_red_phi,
> + machine_mode *out_acc_mode,
> + machine_mode *out_op_mode)
> +{
> + if (loop->num_nodes != 2 || !empty_block_p (loop->latch))
> + return false;
> + edge exit = single_exit (loop);
> + if (!exit)
> + return false;
> +
> + /* Header is the body; loads only. */
> + bool has_vuse = false;
> + for (gimple_stmt_iterator gsi = gsi_start_bb (loop->header);
> + !gsi_end_p (gsi); gsi_next (&gsi))
> + {
> + gimple *s = gsi_stmt (gsi);
> + if (gimple_vdef (s))
> + return false;
> + if (gimple_vuse (s))
> + has_vuse = true;
> + }
> + if (!has_vuse)
> + return false;
> +
> + /* Single reduction PHI with init defined outside the loop. */
> + gphi *red_phi = inner_reduction_phi (loop);
> + if (!red_phi)
> + return false;
> + tree init = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_preheader_edge (loop));
> + if (!expr_invariant_in_loop_p (loop, init))
> + return false;
> +
> + /* Trip count is a known integer >= MIN_INNER_ITERS. niter.niter is
> + the latch count (trip - 1); reject HOST_WIDE_INT_M1U separately
> + so the +1 conversion below does not wrap. */
> + tree_niter_desc niter;
> + if (!number_of_iterations_exit (loop, exit, &niter, false, true)
> + || niter.cmp == ERROR_MARK
> + || !integer_zerop (niter.may_be_zero)
> + || !tree_fits_uhwi_p (niter.niter))
> + return false;
> +
> + unsigned HOST_WIDE_INT latch_count = tree_to_uhwi (niter.niter);
> + if (latch_count == HOST_WIDE_INT_M1U
> + || latch_count + 1 < min_inner_iters)
> + return false;
> +
> + /* Plus a unit-step counter IV in the same header. When OUTER is
> + given, every IV's step and base must also be OUTER-invariant. */
> + bool found_unit_iv = false;
> + for (gphi_iterator psi = gsi_start_phis (loop->header);
> + !gsi_end_p (psi); gsi_next (&psi))
> + {
> + gphi *phi = psi.phi ();
> + tree op = gimple_phi_result (phi);
> + if (virtual_operand_p (op))
> + continue;
> + affine_iv iv;
> + if (!simple_iv (loop, loop, op, &iv, true))
> + continue;
> + if (outer
> + && (!expr_invariant_in_loop_p (outer, iv.step)
> + || !expr_invariant_in_loop_p (outer, iv.base)))
> + return false;
> + if (iv.step && integer_onep (iv.step))
> + found_unit_iv = true;
> + }
> + if (!found_unit_iv)
> + return false;
> +
> + /* Widening multiply somewhere in the reduction update chain. */
> + machine_mode acc_mode = TYPE_MODE (TREE_TYPE (gimple_phi_result
> (red_phi)));
> + machine_mode op_mode = narrowest_reduc_input_mode (red_phi, loop,
> acc_mode);
> + if (op_mode == acc_mode)
> + return false;
> +
> + if (out_red_phi)
> + *out_red_phi = red_phi;
> + if (out_acc_mode)
> + *out_acc_mode = acc_mode;
> + if (out_op_mode)
> + *out_op_mode = op_mode;
> + return true;
> +}
> +
> +/* Detect whether the OUTER / INNER nest has the parallel-reduction
> + shape that unroll_jam_reduction_fuse_sibling can transform. The
> + rejection messages below name each requirement. The only outer-body
> + VDEF must be the post-reduction store, so that after strip-mining
> + every replicated store sits downstream of every sibling's preheader.
> + And the outer trip count must be a multiple of N, which keeps us off
> + tree_unroll_loop's remainder path. */
> +
> +static bool
> +unroll_jam_reduction_possible_p (class loop *outer, class loop *inner,
> + unsigned HOST_WIDE_INT *out_outer_trip,
> + unsigned *out_target_factor)
> +{
> + auto reject = [&] (const char *reason) -> bool
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "reduction-jam: rejecting outer loop %d: %s\n",
> + outer->num, reason);
> + return false;
> + };
> +
> + /* Loop-tree relationships. The driver iterates LI_ONLY_INNERMOST,
> + so INNER has no children; check only that OUTER has a single
> + inner sibling (INNER) and that INNER's parent is OUTER. */
> + if (!inner || inner != outer->inner || inner->next
> + || loop_outer (inner) != outer)
> + return reject ("not a perfect 2-level nest");
> +
> + if (!dominated_by_p (CDI_DOMINATORS, outer->latch, inner->header))
> + return reject ("inner has unsuitable dominance");
> +
> + /* Outer loop must carry only simple IVs. */
> + for (gphi_iterator psi = gsi_start_phis (outer->header);
> + !gsi_end_p (psi); gsi_next (&psi))
> + {
> + gphi *phi = psi.phi ();
> + tree op = gimple_phi_result (phi);
> + if (virtual_operand_p (op))
> + continue;
> + affine_iv iv;
> + if (!simple_iv (outer, outer, op, &iv, true))
> + return reject ("outer carries a non-simple-IV PHI");
> + }
> +
> + /* Loop-local dot-product checks; on success red_phi and the modes
> + feed the target-hook gate below. */
> + gphi *red_phi = NULL;
> + machine_mode acc_mode = VOIDmode;
> + machine_mode op_mode = VOIDmode;
> + if (!loop_is_dot_product_candidate_p
> + (inner,
> + (unsigned HOST_WIDE_INT) param_unroll_jam_reduction_min_inner_iters,
> + outer, &red_phi, &acc_mode, &op_mode))
> + return reject ("inner not a dot-product candidate");
> +
> + /* Opt-in gate: ask the target whether the (accumulator, operand)
> + mode pair benefits from N parallel reduction streams. Targets
> + that have not measured a benefit return 0, in which case we skip
> + the transform regardless of -floop-unroll-and-jam-reductions.
> + A return of 1 is also opt-out (single-stream = no parallelism). */
> + unsigned target_factor
> + = targetm.loop_jam_reduction_factor (acc_mode, op_mode);
> + if (target_factor < 2)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file,
> + "reduction-jam: target opted out for acc=%s op=%s\n",
> + GET_MODE_NAME (acc_mode), GET_MODE_NAME (op_mode));
> + return false;
> + }
> + *out_target_factor = target_factor;
> +
> + /* The fuser drains stmts from inner->header alone, so a non-empty
> + latch would be silently mishandled; loop_is_dot_product_candidate_p
> + already rejects one. */
> + gcc_checking_assert (empty_block_p (inner->latch));
> +
> + /* Require the chain from red_phi back to itself via the latch arg to
> + be a PLUS_EXPR reduction. This limits the transform to plain
> + summation; the N streams are independent, so each is still summed
> + in its original order and no reassociation takes place. */
> + tree latch_arg = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner));
> + if (!check_reduction_path (dump_user_location_t (), inner, red_phi,
> + latch_arg, PLUS_EXPR))
> + return reject ("inner reduction chain not associative-PLUS");
> +
> + /* loop_is_dot_product_candidate_p confirmed inner has a single exit. */
> + edge inner_exit = single_exit (inner);
> +
> + /* The outer body may carry at most one VDEF, only in the inner
> + exit's destination BB (the post-reduction store). Also reject any
> + non-VDEF side effect (volatile load, asm volatile, side-effecting
> + builtin), matching bb_prevents_fusion_p on the IV-only path.
> +
> + The store BB must be dominated by inner->header so that after
> + tree_unroll_loop strip-mines outer by N every replicated store
> + sits downstream of its corresponding inner copy, and therefore
> + downstream of every sibling's preheader. The fuser relies on
> + this when rewiring VUSEs. */
> + basic_block store_bb = inner_exit->dest;
> + if (!dominated_by_p (CDI_DOMINATORS, store_bb, inner->header))
> + return reject ("outer-body store not downstream of inner header");
> +
> + /* The outer body must be straight-line apart from the exit test. A
> + control-flow merge there leaves a PHI that would not dominate the
> + fused preheader after strip-mining, and the migrator leaves body
> + PHIs in place, so the SSA fails verification. */
> + edge outer_exit = single_exit (outer);
> + if (!outer_exit)
> + return reject ("outer loop has no single exit");
> +
> + basic_block *bbs = get_loop_body (outer);
> + const char *fail = NULL;
> + for (unsigned i = 0; i < outer->num_nodes && !fail; ++i)
> + {
> + basic_block bb = bbs[i];
> + if (flow_bb_inside_loop_p (inner, bb))
> + continue;
> + if (EDGE_COUNT (bb->succs) > 1 && bb != outer_exit->src)
> + {
> + fail = "outer body has control flow";
> + break;
> + }
> + unsigned vdef_count = 0;
> + for (gimple_stmt_iterator gsi = gsi_start_bb (bb);
> + !gsi_end_p (gsi); gsi_next (&gsi))
> + {
> + gimple *s = gsi_stmt (gsi);
> + if (gimple_has_side_effects (s))
> + {
> + fail = "outer body has side effects";
> + break;
> + }
> + if (gimple_vdef (s))
> + vdef_count++;
> + }
> + if (!fail && (bb == store_bb ? vdef_count > 1 : vdef_count > 0))
> + fail = "outer body has extra stores";
> + }
> + free (bbs);
> + if (fail)
> + return reject (fail);
> +
> + /* Outer trip count must be a known constant (N is later narrowed to
> + a divisor of it; the remainder codepath of tree_unroll_loop is
> + correct in principle but currently untested for this pattern).
> + niter from number_of_iterations_exit is the latch execution count,
> + so the trip count is niter + 1. */
> + tree_niter_desc outer_niter;
> + if (!number_of_iterations_exit (outer, outer_exit, &outer_niter,
> + false, true)
> + || outer_niter.cmp == ERROR_MARK
> + || !integer_zerop (outer_niter.may_be_zero)
> + || !tree_fits_uhwi_p (outer_niter.niter))
> + return reject ("outer trip count unknown");
> +
> + unsigned HOST_WIDE_INT outer_latch_count = tree_to_uhwi
> (outer_niter.niter);
> + if (outer_latch_count == HOST_WIDE_INT_M1U)
> + return reject ("outer trip count overflows HOST_WIDE_INT");
> +
> + *out_outer_trip = outer_latch_count + 1;
> +
> + return true;
> +}
> +
> +/* Move the transitive SSA-use closure of SRC_INIT, and of every
> + operand SRC's header body references, into DST_PREHEADER so the
> + defs stay live once SRC's body moves. Defs that already dominate
> + DST_PREHEADER, and defs inside SRC, are left to the body move.
> +
> + RPO is a reverse-postorder over CFUN's BBs, computed once by the
> + caller and shared across the nest's siblings; walking it keeps
> + def-use order among the migrated stmts intact. DST_LIVE_VUSE is
> + the vop live at DST_PREHEADER, and migrated stmts are rewired to it
> + so none of them reference a vop defined downstream. */
> +
> +static void
> +migrate_loop_invariant_defs_to_preheader (class loop *src, tree src_init,
> + basic_block dst_preheader,
> + const auto_vec<int> &rpo,
> + tree dst_live_vuse)
> +{
> + hash_set<gimple *> needed;
> + auto_vec<gimple *, 16> worklist;
> +
> + auto consider = [&] (tree op)
It would be better if you name each capture explictly rather than just
an implict by reference captures.
> + {
> + if (TREE_CODE (op) != SSA_NAME)
> + return;
> + gimple *def = SSA_NAME_DEF_STMT (op);
> + if (!def || is_a <gphi *> (def))
> + return;
> + basic_block def_bb = gimple_bb (def);
> + if (!def_bb
> + || dominated_by_p (CDI_DOMINATORS, dst_preheader, def_bb)
> + || flow_bb_inside_loop_p (src, def_bb))
> + return;
> + if (needed.add (def))
> + return;
> + worklist.safe_push (def);
Why not use a bitmap with the ssa name versions as the key for both
needed and worklist?
> + };
> +
> + auto consider_uses = [&] (gimple *s)
> + {
> + use_operand_p use_p;
> + ssa_op_iter oi;
> + FOR_EACH_SSA_USE_OPERAND (use_p, s, oi, SSA_OP_USE)
> + consider (USE_FROM_PTR (use_p));
> + };
Why not pass consider to the lamba rather than capturing it? C++14
has generic lambdas after all.
> +
> + consider (src_init);
> + for (gimple_stmt_iterator si = gsi_start_bb (src->header);
> + !gsi_end_p (si); gsi_next (&si))
> + {
> + gimple *s = gsi_stmt (si);
> + if (!is_a <gphi *> (s) && !is_ctrl_stmt (s))
> + consider_uses (s);
> + }
> + while (!worklist.is_empty ())
> + consider_uses (worklist.pop ());
> +
> + if (needed.is_empty ())
> + return;
> +
> + /* Walk BBs in reverse-postorder so that, for any def-use edge among
> + the migrated stmts, the def is moved into DST_PREHEADER ahead of
> + the use. */
> + int n = rpo.length ();
> + for (int i = 0; i < n; ++i)
> + {
> + basic_block bb = BASIC_BLOCK_FOR_FN (cfun, rpo[i]);
> + if (bb == dst_preheader
> + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
> + continue;
> + for (gimple_stmt_iterator si = gsi_start_bb (bb);
> + !gsi_end_p (si);)
> + {
> + gimple *stmt = gsi_stmt (si);
> + if (!needed.contains (stmt))
> + {
> + gsi_next (&si);
> + continue;
> + }
> + if (dst_live_vuse && gimple_vuse (stmt))
> + {
> + gimple_set_vuse (stmt, dst_live_vuse);
> + update_stmt (stmt);
> + }
> + gsi_move_to_bb_end_before_terminator (&si, dst_preheader);
> + }
> + }
> +}
> +
> +/* Pair SRC's non-reduction header PHIs with DST's by position,
> + recording each pair in PHI_RENAME. Both are inner siblings of the
> + same strip-mined outer loop, so their header PHI lists start with
> + the duplicated originals in the same order. DST may have picked up
> + extra reduction PHIs at the tail from earlier sibling fuses, so
> + walk SRC to completion against DST's prefix. SRC_REDUC_PHI is
> + renamed separately and virtual PHIs are skipped. */
> +
> +static void
> +match_sibling_header_phis (class loop *src, class loop *dst,
> + gphi *src_reduc_phi,
> + hash_map<tree, tree> &phi_rename)
> +{
> + gphi_iterator spsi = gsi_start_phis (src->header);
> + gphi_iterator dpsi = gsi_start_phis (dst->header);
> + for (; !gsi_end_p (spsi); gsi_next (&spsi), gsi_next (&dpsi))
> + {
> + gcc_checking_assert (!gsi_end_p (dpsi));
> + gphi *src_phi = spsi.phi ();
> + gphi *dst_phi = dpsi.phi ();
> + tree src_res = gimple_phi_result (src_phi);
> + tree dst_res = gimple_phi_result (dst_phi);
> + gcc_checking_assert (virtual_operand_p (src_res)
> + == virtual_operand_p (dst_res));
> + if (src_phi != src_reduc_phi && !virtual_operand_p (src_res))
> + phi_rename.put (src_res, dst_res);
> + }
> +}
> +
> +/* Fuse the SRC sibling inner loop into DST, keeping SRC's reduction
> + PHI as a parallel chain in DST's header. Both are inner siblings
> + of an outer loop that passed unroll_jam_reduction_possible_p and
> + was then strip-mined by tree_unroll_loop.
> +
> + The caller has already flattened LC-SSA across the nest (Phase 1a).
> + Here SRC's reduction init and body operands move up to DST's
> + preheader (Phase 1b), a fresh reduction PHI for SRC's stream goes
> + into DST's header (Phase 2), and SRC's body follows with its PHI
> + uses renamed (Phase 3). The caller then bypasses the drained SRC
> + (Phase 4); its cleanup_tree_cfg folds the leftover chain of exit
> + BBs holding the post-reduction stores, and
> + rewrite_into_loop_closed_ssa puts LC-SSA back. */
> +
> +static void
> +unroll_jam_reduction_fuse_sibling (class loop *src, class loop *dst,
> + const auto_vec<int> &rpo)
> +{
> + gphi *src_reduc_phi = inner_reduction_phi (src);
> + tree src_reduc_res = gimple_phi_result (src_reduc_phi);
> + tree src_init = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
> + loop_preheader_edge (src));
> + tree src_update = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
> + loop_latch_edge (src));
> +
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "reduction-jam: fusing src loop %d into dst %d\n",
> + src->num, dst->num);
> +
> + basic_block dst_preheader = loop_preheader_edge (dst)->src;
> +
> + /* The migrated stmts must reference the vop live at DST's preheader.
> + The matcher's only outer-body VDEF is the post-reduction store,
> + which sits below DST after strip-mining, so nothing redefines the
> + vop before DST: it is outer->header's virtual PHI result. A
> + load-only outer has no such PHI; its loads already share one
> + loop-invariant vop, so leave their VUSE alone (NULL_TREE). */
> + gphi *outer_vphi = get_virtual_phi (loop_outer (dst)->header);
> + tree dst_live_vuse = outer_vphi ? gimple_phi_result (outer_vphi) :
> NULL_TREE;
> +
> + /* Phase 1b. */
> + migrate_loop_invariant_defs_to_preheader (src, src_init, dst_preheader,
> + rpo, dst_live_vuse);
> +
> + /* Phase 2: a new reduction PHI in DST's header for SRC's stream. */
> + tree new_name = make_ssa_name (TREE_TYPE (src_reduc_res));
> + gphi *new_phi = create_phi_node (new_name, dst->header);
> + add_phi_arg (new_phi, src_init, loop_preheader_edge (dst),
> + UNKNOWN_LOCATION);
> + add_phi_arg (new_phi, src_update, loop_latch_edge (dst),
> + UNKNOWN_LOCATION);
> +
> + /* Phase 3: rewrite uses, then move SRC's body into DST's header. */
> + hash_map<tree, tree> phi_rename;
> + phi_rename.put (src_reduc_res, new_name);
> + match_sibling_header_phis (src, dst, src_reduc_phi, phi_rename);
> +
> + auto_vec<gimple *, 32> src_body;
> + for (gimple_stmt_iterator si = gsi_start_bb (src->header);
> + !gsi_end_p (si); gsi_next (&si))
> + {
> + gimple *s = gsi_stmt (si);
> + if (is_a <gphi *> (s) || is_ctrl_stmt (s))
> + continue;
phis won't appear in the body of a bb. So why is there a check here?
> + src_body.safe_push (s);
This seems broken. You are creating a vector of the body just to use
it below ...
Why not just walk the bb without the extra vector? You can do gsi_next
before the move to get the next stmt even.
> + }
> + for (gimple *stmt : src_body)
> + {
> + use_operand_p use_p;
> + ssa_op_iter oi;
> + FOR_EACH_SSA_USE_OPERAND (use_p, stmt, oi, SSA_OP_USE)
> + {
> + tree *map = phi_rename.get (USE_FROM_PTR (use_p));
> + if (map)
> + SET_USE (use_p, *map);
> + }
> + if (dst_live_vuse && gimple_vuse (stmt))
> + {
> + gimple_set_vuse (stmt, dst_live_vuse);
> + update_stmt (stmt);
> + }
> + gimple_stmt_iterator from = gsi_for_stmt (stmt);
> + gsi_move_to_bb_end_before_terminator (&from, dst->header);
> + }
> +
> + /* The body walk above only covered uses of src_reduc_res inside
> + src->header; catch any remaining uses (LC-SSA closures reached
> + by the post-reduction store) before the PHI is removed. */
> + replace_uses_by (src_reduc_res, new_name);
> +
> + gphi_iterator psi_src_reduc = gsi_for_phi (src_reduc_phi);
> + remove_phi_node (&psi_src_reduc, false);
> +}
> +
> +/* Phase 4: redirect SRC's preheader edge straight past the now-empty
> + SRC loop so cleanup_tree_cfg can remove it. SRC must have been
> + drained by unroll_jam_reduction_fuse_sibling first. */
> +
> +static void
> +unroll_jam_reduction_bypass_sibling (class loop *src)
> +{
> + edge src_preheader_e = loop_preheader_edge (src);
> + edge src_exit_e = single_exit (src);
> + if (!src_exit_e)
> + return;
> + basic_block src_exit_dest = src_exit_e->dest;
> +
> + /* src_exit_dest is a single-pred BB after strip-mining (the next
> + sibling's preheader, or outer's latch for the last sibling), so
> + its LC-SSA PHIs are single-arg. Flatten them before the redirect
> + so no arg dangles once SRC's BBs are deleted: the driver rebuilds
> + scalar LC-SSA afterward but not the virtual side, so a stale vop
> + arg would otherwise survive. */
> + for (gphi_iterator psi = gsi_start_phis (src_exit_dest);
> + !gsi_end_p (psi);)
> + collapse_single_arg_phi (&psi);
> +
> + edge new_edge = redirect_edge_and_branch (src_preheader_e, src_exit_dest);
> + gcc_assert (new_edge);
> +
> + /* Delete SRC's now-unreachable BBs and the empty loop entry,
> + mirroring fuse_loops's delete_basic_block + delete_loop pattern
> + on the IV-only path. Without this the loop tree carries stale
> + entries until the driver's cleanup_tree_cfg picks them up. */
> + basic_block *bbs = get_loop_body (src);
> + unsigned n = src->num_nodes;
> + for (unsigned i = 0; i < n; i++)
> + delete_basic_block (bbs[i]);
> + free (bbs);
> + delete_loop (src);
> +}
> +
> /* Return true if any of the access functions for dataref A
> isn't invariant with respect to loop LOOP_NEST. */
> static bool
> @@ -497,6 +1177,221 @@ adjust_unroll_factor (class loop *inner, struct
> data_dependence_relation *ddr,
> return ret;
> }
>
> +/* Walk OUTER's data dependences, running each through
> + adjust_unroll_factor. *UNROLL_FACTOR receives the safety upper
> + bound (UINT_MAX for no constraint, 0 for cannot unroll),
> + *PROFIT_UNROLL the largest distance+1 yielding a removable ref
> + under the (a,0) pattern, *REMOVED the count of removable refs and
> + *NUM_DATAREFS the total. Returns false if the dependences cannot
> + be computed.
> +
> + When *UNROLL_FACTOR comes back 0 the other out-params are only
> + partially populated and must not be read for profitability. */
> +
> +static bool
> +compute_jam_dependence_factors (class loop *outer, class loop *inner,
> + unsigned *unroll_factor,
> + unsigned *profit_unroll,
> + unsigned *removed,
> + unsigned *num_datarefs)
> +{
> + vec<data_reference_p> datarefs = vNULL;
> + vec<ddr_p> dependences = vNULL;
> + auto_vec<loop_p, 3> loop_nest;
> +
> + if (!compute_data_dependences_for_loop (outer, true, &loop_nest,
> + &datarefs, &dependences))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file, "Cannot analyze data dependencies\n");
> + free_data_refs (datarefs);
> + free_dependence_relations (dependences);
> + return false;
> + }
> +
> + *num_datarefs = datarefs.length ();
> + *unroll_factor = (unsigned) -1;
> + *profit_unroll = 1;
> + *removed = 0;
> +
> + if (*num_datarefs)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + dump_data_dependence_relations (dump_file, dependences);
> +
> + unsigned i;
> + ddr_p ddr;
> + FOR_EACH_VEC_ELT (dependences, i, ddr)
> + {
> + if (DDR_ARE_DEPENDENT (ddr) == chrec_known)
> + continue;
> + data_reference *dra = DDR_A (ddr);
> + data_reference *drb = DDR_B (ddr);
> + /* Self-dependence: WAW with non-affine access could reorder
> + writes to the same location. */
> + if (dra == drb)
> + {
> + if (DR_IS_WRITE (dra)
> + && !DR_ACCESS_FNS (dra).is_empty ()
> + && DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
> + {
> + *unroll_factor = 0;
> + break;
> + }
> + continue;
> + }
> + /* Couldn't determine the distance vector: harmless for two
> + reads, but a write means we cannot prove safety. */
> + if (!adjust_unroll_factor (inner, ddr, unroll_factor,
> + profit_unroll, removed))
> + if (DR_IS_WRITE (dra) || DR_IS_WRITE (drb))
> + {
> + *unroll_factor = 0;
> + break;
> + }
> + }
> + }
> +
> + free_dependence_relations (dependences);
> + free_data_refs (datarefs);
> + return true;
> +}
> +
> +/* Pick a strip factor for the reduction-jam path on OUTER / INNER.
> + adjust_unroll_factor gives the data-dep safety bound, TARGET_FACTOR
> + the target's preference and param_unroll_jam_reduction_streams
> + the cap. We do not use adjust_unroll_factor's profit model here:
> + it would cap N at 1 for GEMV, because the input-load DDR has
> + dist_v=(0,0). The result is narrowed to a divisor of OUTER_TRIP,
> + which keeps us off tree_unroll_loop's remainder path, and validated
> + via can_unroll_loop_p. *OUT_FACTOR receives the chosen factor and
> + *OUT_DESC the niter. */
> +
> +static bool
> +select_reduction_jam_factor (class loop *outer, class loop *inner,
> + unsigned HOST_WIDE_INT outer_trip,
> + unsigned target_factor,
> + unsigned *out_factor,
> + tree_niter_desc *out_desc)
> +{
> + /* Only the safety bound matters here; the ref-removal outputs go
> + unused. */
> + unsigned unroll_factor, profit_unroll, removed, num_datarefs;
> + if (!compute_jam_dependence_factors (outer, inner, &unroll_factor,
> + &profit_unroll, &removed,
> + &num_datarefs))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file,
> + "reduction-jam: data-dep analysis failed\n");
> + return false;
> + }
> + if (unroll_factor < 2)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file,
> + "reduction-jam: data-dep safety bound below 2\n");
> + return false;
> + }
> +
> + unsigned N = MIN (target_factor,
> + (unsigned) param_unroll_jam_reduction_streams);
> + if (N > unroll_factor)
> + N = unroll_factor;
> + unsigned initial_N = N;
> + while (N >= 2 && (outer_trip < N || outer_trip % N != 0))
> + --N;
> + if (N < 2)
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file,
> + "reduction-jam: outer trip " HOST_WIDE_INT_PRINT_UNSIGNED
> + " has no divisor in [2, %u]\n", outer_trip, initial_N);
> + return false;
> + }
> +
> + if (!can_unroll_loop_p (outer, N, out_desc))
> + {
> + if (dump_file && (dump_flags & TDF_DETAILS))
> + fprintf (dump_file,
> + "reduction-jam: cannot unroll outer by %u\n", N);
> + return false;
> + }
> +
> + *out_factor = N;
> + return true;
> +}
> +
> +/* Strip-mine OUTER by N and fuse the resulting N inner siblings into
> + one body that carries N parallel reduction PHIs. Caller has
> + already validated OUTER and chosen N + DESC via
> + unroll_jam_reduction_possible_p / select_reduction_jam_factor. */
> +
> +static void
> +do_unroll_jam_reductions (class loop *outer, unsigned N,
> + tree_niter_desc *desc)
> +{
> + initialize_original_copy_tables ();
> + tree_unroll_loop (outer, N, desc);
> + free_original_copy_tables ();
> +
> + /* dst->next chains siblings in CFG order (per tree_unroll_loop);
> + the fuser relies on that when migrating SSA uses. */
> + class loop *dst = outer->inner;
> + auto_vec<class loop *, 4> siblings;
> + for (class loop *src = dst->next; src; src = src->next)
> + siblings.safe_push (src);
> +
> + /* RPO of CFUN, shared across all siblings: the fuser only moves
> + stmts and removes single-arg PHIs, neither of which alters BB
> + structure, so one walk suffices for the whole nest. */
> + auto_vec<int> rpo;
> + rpo.safe_grow (last_basic_block_for_fn (cfun));
> + rpo.truncate (pre_and_rev_post_order_compute_fn (cfun, NULL, rpo.address
> (),
> + false));
> +
> + /* Phase 1a (nest-wide): collapse single-arg LC-SSA PHIs that the
> + per-sibling migrate-walk in Phase 1b would otherwise have to
> + chase through (it skips PHIs). Sweep the BBs that do not dominate
> + DST's preheader, i.e. the nest and everything downstream where the
> + sibling-exit closures live. One sweep suffices: later fuses only
> + add two-arg reduction PHIs and make no CFG change that could
> + reintroduce single-arg closures. */
> + basic_block dst_preheader = loop_preheader_edge (dst)->src;
> + calculate_dominance_info (CDI_DOMINATORS);
> + basic_block bb;
> + FOR_EACH_BB_FN (bb, cfun)
> + {
> + if (bb == dst_preheader
> + || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
> + continue;
> + for (gphi_iterator psi = gsi_start_phis (bb); !gsi_end_p (psi);)
> + {
> + gphi *phi = psi.phi ();
> + if (gimple_phi_num_args (phi) == 1
> + && !virtual_operand_p (gimple_phi_result (phi)))
> + collapse_single_arg_phi (&psi);
> + else
> + gsi_next (&psi);
> + }
> + }
> +
> + for (class loop *src : siblings)
> + unroll_jam_reduction_fuse_sibling (src, dst, rpo);
> +
> + /* Bypass the drained siblings only after every fuse: the CFG edits
> + invalidate dominance / SCEV state the fuse loop above relies on. */
> + for (class loop *src : siblings)
> + unroll_jam_reduction_bypass_sibling (src);
> +
> + if (dump_enabled_p ())
> + dump_printf_loc (MSG_OPTIMIZED_LOCATIONS | TDF_DETAILS,
> + find_loop_location (outer),
> + "applying reduction unroll-and-jam with %u streams\n",
> + siblings.length () + 1);
> +}
> +
> +
> /* Main entry point for the unroll-and-jam transformation
> described above. */
>
> @@ -516,8 +1411,36 @@ tree_loop_unroll_and_jam (void)
> || optimize_loop_nest_for_size_p (outer))
> continue;
>
> - if (!unroll_jam_possible_p (outer, loop))
> - continue;
> + if (!flag_unroll_jam || !unroll_jam_possible_p (outer, loop))
> + {
> + /* The IV-only path skips reduction-shaped nests (reduction PHIs,
> + or outer-body stores consuming the reduction result). Try the
> + reduction-jam extension on those. */
> + unsigned HOST_WIDE_INT outer_trip;
> + unsigned target_factor, N;
> + tree_niter_desc red_desc;
> + if (flag_unroll_jam_reductions
> + && unroll_jam_reduction_possible_p (outer, loop,
> + &outer_trip,
> + &target_factor)
> + && select_reduction_jam_factor (outer, loop, outer_trip,
> + target_factor,
> + &N, &red_desc))
> + {
> + do_unroll_jam_reductions (outer, N, &red_desc);
> + free_dominance_info (CDI_DOMINATORS);
> + cleanup_tree_cfg ();
> + /* The fuser maintains virtual SSA from the matcher's
> + invariants, but moving SRC's body into DST leaves scalar
> + SSA names defined in DST yet used outside it (e.g. new_name's
> + LC-SSA closures). Rebuild scalar LC-SSA so the next nest's
> + tree_unroll_loop is well-formed, and verify the result. */
> + rewrite_into_loop_closed_ssa (NULL, 0);
> + checking_verify_ssa (true, true);
> + scev_reset ();
> + }
> + continue;
> + }
>
> vec<data_reference_p> datarefs = vNULL;
> vec<ddr_p> dependences = vNULL;
> @@ -674,7 +1597,10 @@ public:
> {}
>
> /* opt_pass methods: */
> - bool gate (function *) final override { return flag_unroll_jam != 0; }
> + bool gate (function *) final override
> + {
> + return flag_unroll_jam != 0 || flag_unroll_jam_reductions;
> + }
> unsigned int execute (function *) final override;
>
> };
> diff --git a/gcc/hooks.cc b/gcc/hooks.cc
> index 2b47fd3e50f8..976bae3c89a9 100644
> --- a/gcc/hooks.cc
> +++ b/gcc/hooks.cc
> @@ -424,6 +424,13 @@ hook_uint_mode_0 (machine_mode)
> return 0;
> }
>
> +/* Generic hook that takes two machine modes and returns an unsigned int 0.
> */
> +unsigned int
> +hook_uint_mode_mode_0 (machine_mode, machine_mode)
> +{
> + return 0;
> +}
> +
> /* Generic hook that takes no arguments and returns a NULL const string. */
> const char *
> hook_constcharptr_void_null (void)
> diff --git a/gcc/hooks.h b/gcc/hooks.h
> index 1ad4abb5500c..0ec0b32fa1a5 100644
> --- a/gcc/hooks.h
> +++ b/gcc/hooks.h
> @@ -119,6 +119,7 @@ extern tree hook_tree_treeptr_tree_tree_int_boolptr_null
> (tree *, tree, tree, in
>
> extern unsigned hook_uint_void_0 (void);
> extern unsigned int hook_uint_mode_0 (machine_mode);
> +extern unsigned int hook_uint_mode_mode_0 (machine_mode, machine_mode);
>
> extern bool default_can_output_mi_thunk_no_vcall (const_tree, HOST_WIDE_INT,
> HOST_WIDE_INT, const_tree);
> diff --git a/gcc/params.opt b/gcc/params.opt
> index 0c6a3e514574..e99c4fc4173f 100644
> --- a/gcc/params.opt
> +++ b/gcc/params.opt
> @@ -1263,6 +1263,14 @@ Maximum unroll factor for the unroll-and-jam
> transformation.
> Common Joined UInteger Var(param_unroll_jam_min_percent) Init(1)
> IntegerRange(0, 100) Param Optimization
> Minimum percentage of memrefs that must go away for unroll-and-jam to be
> considered profitable.
>
> +-param=unroll-jam-reduction-streams=
> +Common Joined UInteger Var(param_unroll_jam_reduction_streams) Init(4)
> IntegerRange(2, 16) Param Optimization
> +Target number of parallel reduction streams produced by reduction-aware
> unroll-and-jam.
> +
> +-param=unroll-jam-reduction-min-inner-iters=
> +Common Joined UInteger Var(param_unroll_jam_reduction_min_inner_iters)
> Init(32) IntegerRange(2, 65536) Param Optimization
> +Minimum inner-loop trip count required for reduction-aware unroll-and-jam to
> fire.
> +
> -param=use-after-scope-direct-emission-threshold=
> Common Joined UInteger Var(param_use_after_scope_direct_emission_threshold)
> Init(256) Param Optimization
> Use direct poisoning/unpoisoning instructions for variables smaller or equal
> to this number.
> diff --git a/gcc/target.def b/gcc/target.def
> index 66c20a9c9d2a..118aa9bb82ec 100644
> --- a/gcc/target.def
> +++ b/gcc/target.def
> @@ -2873,6 +2873,19 @@ number of memory accesses.",
> unsigned, (unsigned nunroll, class loop *loop),
> NULL)
>
> +DEFHOOK
> +(loop_jam_reduction_factor,
> + "Return the preferred number of parallel reduction streams\n\
> +@option{-floop-unroll-and-jam-reductions} should produce for an inner\n\
> +reduction that accumulates into mode @var{acc_mode} from operands of\n\
> +mode @var{op_mode}. @var{op_mode} is the operand mode at the source\n\
> +of the widening multiply, and is always strictly narrower than\n\
> +@var{acc_mode}: plain reductions with no widening multiply are\n\
> +filtered out before the hook is consulted. Returning less than two\n\
> +disables the transform for that mode pair.",
> + unsigned, (machine_mode acc_mode, machine_mode op_mode),
> + hook_uint_mode_mode_0)
> +
> /* True if X is a legitimate MODE-mode immediate operand. */
> DEFHOOK
> (legitimate_constant_p,
> diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> new file mode 100644
> index 000000000000..d14711034bc0
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
> @@ -0,0 +1,78 @@
> +/* Positive tests: the pass must strip-mine and fuse each nest below.
> + The outer trip count of 32 is divisible by the default factor of 4.
> + Both arms of the aarch64 hook are covered: QImode / HImode -> SImode
> + (DOT_PROD class) in affine, and SImode -> DImode (SMLAL_DI) in
> + affine64. */
> +
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
> -fdump-tree-unrolljam-details" } */
> +
> +#include <stdint.h>
> +
> +#define IN_DIM 512
> +#define OUT_DIM 32
> +
> +void
> +affine (int32_t *__restrict out,
> + const int8_t *weights,
> + const int32_t *biases,
> + const uint8_t *input)
> +{
> + for (int i = 0; i < OUT_DIM; ++i)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < IN_DIM; ++j)
> + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +void
> +affine64 (int64_t *__restrict out,
> + const int32_t *weights,
> + const int64_t *biases,
> + const int32_t *input)
> +{
> + for (int i = 0; i < OUT_DIM; ++i)
> + {
> + int64_t s = biases[i];
> + for (int j = 0; j < IN_DIM; ++j)
> + s += (int64_t) weights[i * IN_DIM + j] * (int64_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +/* Inner trip count at the min-inner-iters default of 32. */
> +void
> +affine_min_inner (int32_t *__restrict out,
> + const int8_t *weights,
> + const int32_t *biases,
> + const uint8_t *input)
> +{
> + for (int i = 0; i < 32; ++i)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < 32; ++j)
> + s += (int32_t) weights[i * 32 + j] * (int32_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +/* Outer trip count of 9 narrows N from 4 to 3. */
> +void
> +affine_narrowed (int32_t *__restrict out,
> + const int8_t *weights,
> + const int32_t *biases,
> + const uint8_t *input)
> +{
> + for (int i = 0; i < 9; ++i)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < 512; ++j)
> + s += (int32_t) weights[i * 512 + j] * (int32_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-times "applying reduction unroll-and-jam" 4
> "unrolljam" } } */
> +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 11
> "unrolljam" } } */
> diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> new file mode 100644
> index 000000000000..44301348a659
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
> @@ -0,0 +1,89 @@
> +/* Correctness test for the reduction-aware extension of unroll-and-jam
> + on the GEMV pattern. Compares against an in-main reference
> + computation. */
> +
> +/* { dg-do run } */
> +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */
> +
> +#include <stdint.h>
> +
> +#define IN_DIM 512
> +#define OUT_DIM 32
> +
> +extern void abort (void);
> +
> +void __attribute__((noipa))
> +affine (int32_t *__restrict out,
> + const int8_t *weights,
> + const int32_t *biases,
> + const uint8_t *input)
> +{
> + for (int i = 0; i < OUT_DIM; ++i)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < IN_DIM; ++j)
> + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +void __attribute__((noipa))
> +affine_zero_init (int32_t *__restrict out,
> + const int8_t *a,
> + const int8_t *b)
> +{
> + for (int i = 0; i < OUT_DIM; ++i)
> + {
> + int32_t s = 0;
> + for (int j = 0; j < IN_DIM; ++j)
> + s += a[i * IN_DIM + j] * b[j];
> + out[i] = s;
> + }
> +}
> +
> +int
> +main (void)
> +{
> + static int8_t weights[OUT_DIM * IN_DIM];
> + static uint8_t input[IN_DIM];
> + static int32_t biases[OUT_DIM];
> + static int32_t out[OUT_DIM];
> + static int32_t expected[OUT_DIM];
> +
> + for (int k = 0; k < OUT_DIM * IN_DIM; k++)
> + weights[k] = (k * 13) & 0x7f;
> + for (int k = 0; k < IN_DIM; k++)
> + input[k] = (k * 7) & 0xff;
> + for (int k = 0; k < OUT_DIM; k++)
> + biases[k] = k * 1000;
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < IN_DIM; j++)
> + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> + expected[i] = s;
> + }
> +
> + affine (out, weights, biases, input);
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + if (out[i] != expected[i])
> + abort ();
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + {
> + int32_t s = 0;
> + for (int j = 0; j < IN_DIM; j++)
> + s += weights[i * IN_DIM + j] * weights[j];
> + expected[i] = s;
> + }
> +
> + affine_zero_init (out, weights, weights);
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + if (out[i] != expected[i])
> + abort ();
> +
> + return 0;
> +}
> diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> new file mode 100644
> index 000000000000..6e4b590fefeb
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
> @@ -0,0 +1,189 @@
> +/* Negative tests: each function below hits one of
> + unroll_jam_reduction_possible_p's rejection clauses and the pass
> + must not fire on any of them. */
> +
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
> -fdump-tree-unrolljam-details" } */
> +
> +#include <stdint.h>
> +
> +/* Outer reduction: outer carries 'sum'. */
> +int
> +double_reduc (const int *a)
> +{
> + int sum = 0;
> + for (int i = 0; i < 32; i++)
> + for (int j = 0; j < 512; j++)
> + sum += a[i * 512 + j];
> + return sum;
> +}
> +
> +/* Inner body has a store: inner writes a[j]. */
> +void
> +inner_store (int *__restrict a, int *__restrict b, const int *c, const int
> *d)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + {
> + a[j] = c[j] + d[j];
> + s += a[j];
> + }
> + b[i] = s;
> + }
> +}
> +
> +/* Inner trip count just below the min-inner-iters default of 32. */
> +void
> +small_inner (int *out, const int *a)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 31; j++)
> + s += a[i * 31 + j];
> + out[i] = s;
> + }
> +}
> +
> +/* Outer body stores to counter as well as out[i]. */
> +extern int counter;
> +void
> +outer_store (int *out, const int8_t *a, const int8_t *b)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + s += (int) a[i * 512 + j] * (int) b[j];
> + out[i] = s;
> + counter++;
> + }
> +}
> +
> +/* FP accumulator: the multiply has no narrower scalar-int operand, so
> + it is not a widening MAC. */
> +void
> +fp_mac (float *out, const float *a, const float *b)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + float s = 0.0f;
> + for (int j = 0; j < 512; j++)
> + s += a[i * 512 + j] * b[j];
> + out[i] = s;
> + }
> +}
> +
> +/* Plain SImode add reduction: no multiply in the update chain. */
> +void
> +plain_add (int *out, const int *a)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + s += a[i * 512 + j];
> + out[i] = s;
> + }
> +}
> +
> +/* MAX reduction: no multiply in the update chain. */
> +void
> +max_reduc (int *out, const int *a)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int m = a[i * 512];
> + for (int j = 1; j < 512; j++)
> + {
> + int v = a[i * 512 + j];
> + m = m > v ? m : v;
> + }
> + out[i] = m;
> + }
> +}
> +
> +/* Inner header carries two reduction PHIs, m and s. */
> +void
> +multi_phi (int *out, const int *a)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int m = a[i * 512];
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + {
> + int v = a[i * 512 + j];
> + m = m > v ? m : v;
> + s += v;
> + }
> + out[i] = m + s;
> + }
> +}
> +
> +/* Parametric outer trip count: not a known UHWI. */
> +void
> +parametric_outer (int *out, const int8_t *a, const int8_t *b, int n)
> +{
> + for (int i = 0; i < n; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + s += (int) a[i * 512 + j] * (int) b[j];
> + out[i] = s;
> + }
> +}
> +
> +/* Outer trip count of 7 has no divisor in [2, 4]. */
> +void
> +narrow_to_one (int *__restrict out, const int8_t *a, const int8_t *b)
> +{
> + for (int i = 0; i < 7; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + s += (int) a[i * 512 + j] * (int) b[j];
> + out[i] = s;
> + }
> +}
> +
> +/* No __restrict: data-dep analysis cannot rule out out vs b
> + aliasing. */
> +void
> +aliasing (int *out, const int8_t *a, const int8_t *b)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int s = 0;
> + for (int j = 0; j < 512; j++)
> + s += (int) a[i * 512 + j] * (int) b[j];
> + out[i] = s;
> + }
> +}
> +
> +/* Conditional accumulator init: the "?:" lowers to a store-free diamond
> + in the outer body whose merge PHI feeds the reduction, so the body is
> + not straight-line. */
> +void
> +cond_init (int32_t *__restrict out, const int8_t *__restrict a,
> + const int32_t *__restrict b1, const int32_t *__restrict b2,
> + const int32_t *__restrict c)
> +{
> + for (int i = 0; i < 32; i++)
> + {
> + int32_t s = c[i] ? b1[i] : b2[i];
> + for (int j = 0; j < 512; j++)
> + s += (int32_t) a[i * 512 + j] * (int32_t) a[j];
> + out[i] = s;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump-not "applying reduction unroll-and-jam"
> "unrolljam" } } */
> +/* { dg-final { scan-tree-dump-not "reduction-jam: fusing src loop"
> "unrolljam" } } */
> +/* { dg-final { scan-tree-dump "reduction-jam: outer trip 7 has no divisor
> in \\\[2, 4\\\]" "unrolljam" } } */
> +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
> body has extra stores" "unrolljam" } } */
> +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
> body has control flow" "unrolljam" } } */
> +/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
> trip count unknown" "unrolljam" } } */
> +/* { dg-final { scan-tree-dump "reduction-jam: data-dep safety bound below
> 2" "unrolljam" } } */
> diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> new file mode 100644
> index 000000000000..938f7328b049
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
> @@ -0,0 +1,30 @@
> +/* The reduction-jam pass should fire even when the IV-only path is
> + disabled via -fno-loop-unroll-and-jam. Pins the gate change in
> + pass_loop_jam::gate and the IV-only/reduction-jam fall-through in
> + tree_loop_unroll_and_jam. */
> +
> +/* { dg-do compile } */
> +/* { dg-options "-O3 -mtune=ampere1 -fno-loop-unroll-and-jam
> -floop-unroll-and-jam-reductions -fdump-tree-unrolljam-details" } */
> +
> +#include <stdint.h>
> +
> +#define IN_DIM 512
> +#define OUT_DIM 32
> +
> +void
> +affine (int32_t *__restrict out,
> + const int8_t *weights,
> + const int32_t *biases,
> + const uint8_t *input)
> +{
> + for (int i = 0; i < OUT_DIM; ++i)
> + {
> + int32_t s = biases[i];
> + for (int j = 0; j < IN_DIM; ++j)
> + s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
> + out[i] = s;
> + }
> +}
> +
> +/* { dg-final { scan-tree-dump "applying reduction unroll-and-jam"
> "unrolljam" } } */
> +/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 3
> "unrolljam" } } */
> diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> new file mode 100644
> index 000000000000..0a198bb27ecd
> --- /dev/null
> +++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
> @@ -0,0 +1,72 @@
> +/* Regression test for a multi-IV mispairing in the reduction-jam
> + sibling matcher. The inner loop carries two IVs that share the
> + preheader entry value 0 but differ in step (j has step 1 and
> + indexes W, p has step 2 and indexes X); the matcher must pair each
> + source IV with its own dst counterpart, not with whichever dst PHI
> + happens to share an entry value.
> +
> + The reference loop in main uses a volatile accumulator so
> + pass_loop_jam does not recognize a reduction PHI there. Without
> + that guard the reference would be jammed with the same mispairing
> + as gemv_2iv, hiding the miscompile behind a symmetric bug. */
> +
> +/* { dg-do run } */
> +/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */
> +
> +#include <stdint.h>
> +
> +#define OUT_DIM 32
> +#define IN_DIM 256
> +#define STRIDE 512
> +
> +extern void abort (void);
> +
> +void __attribute__((noipa))
> +gemv_2iv (int32_t *__restrict out, const int8_t *w, const uint8_t *x)
> +{
> + for (int i = 0; i < OUT_DIM; i++)
> + {
> + int32_t s = 0;
> + int p = 0;
> + for (int j = 0; j < IN_DIM; j++)
> + {
> + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
> + p += 2;
> + }
> + out[i] = s;
> + }
> +}
> +
> +int
> +main (void)
> +{
> + static int8_t w[OUT_DIM * STRIDE];
> + static uint8_t x[IN_DIM * 2];
> + static int32_t out[OUT_DIM];
> + static int32_t expected[OUT_DIM];
> +
> + for (int k = 0; k < OUT_DIM * STRIDE; k++)
> + w[k] = (k * 13) & 0x7f;
> + for (int k = 0; k < IN_DIM * 2; k++)
> + x[k] = (k * 7) & 0xff;
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + {
> + volatile int32_t s = 0;
> + int p = 0;
> + for (int j = 0; j < IN_DIM; j++)
> + {
> + s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
> + p += 2;
> + }
> + expected[i] = s;
> + }
> +
> + gemv_2iv (out, w, x);
> +
> + for (int i = 0; i < OUT_DIM; i++)
> + if (out[i] != expected[i])
> + abort ();
> +
> + return 0;
> +}
> --
> 2.55.0
>
