Extends pass_loop_jam with a parallel-reduction branch for the
GEMV / NNUE inference shape
for (i) {
s = bias[i];
for (j) s += w[i][j] * x[j];
out[i] = s;
}
Stock unroll_jam_possible_p rejects this nest because `s' is a
reduction PHI, not a simple induction; even if it were accepted,
stock fuse_loops would collapse the post-unroll per-sibling reduction
PHIs into one serial chain, multiplying the inner loop's serial
dependency height by N and defeating the goal of unroll-and-jam.
unroll_jam_reduction_possible_p detects the candidate, the existing
tree_unroll_loop strip-mines the outer loop by N, and
unroll_jam_reduction_fuse_sibling fuses the N inner siblings into one
body that retains N parallel reduction PHIs in its header and hoists
each sibling's preamble (s_k = bias[i+k]) to the fused preheader. The
post-reduction stores stay on the linear sibling-exit chain;
cleanup_tree_cfg coalesces it and rewrite_into_loop_closed_ssa restores
LC-SSA. The vectorizer then sees one inner loop with N independent
chains and emits N parallel dot-product / FMA accumulators, cutting the
serial dependency height across the reduction by N.
N is the min of TARGET_LOOP_JAM_REDUCTION_FACTOR (opt-in, default 0
= "no benefit measured"), --param=unroll-jam-reduction-streams and
adjust_unroll_factor's data-dep safety bound, narrowed to a divisor of
the outer trip count. The hook receives the accumulator mode plus the
narrowest scalar-int mode found at a widening multiply in the update
chain; op_mode == acc_mode signals "no widening multiply", letting a
target opt in for MAC shapes only. Inner loops whose trip count is
below --param=unroll-jam-reduction-min-inner-iters are left alone:
their per-stream serial chain is too short to profit from parallelism.
AArch64 opts in for QI/HI -> SI (DOT_PROD class) and SI -> DI
(SMLAL_DI); all other shapes (plain reductions, max/min, FP) return 0.
The per-CPU factor is sourced from a new tune_params field
loop_jam_reduction_factor rather than a hardcoded return value, so the
profitability decision lives in the tuning structure next to the other
latency-driven knobs. Cores with measured wins ship factor 4; every
other tuning leaves the field at 0 until measured. The new tests pin
-mtune to a factor-4 core so they exercise that path independently of
the configured default tuning.
FP reductions are deferred to outer-loop vectorization's lane-broadcast
FMLA path. The chained-reduction case (single accumulator carried
across outer iterations) is left as a follow-up.
~8% speedup on SPEC CPU 2026 706.stockfish_r.
gcc/ChangeLog:
* common.opt (floop-unroll-and-jam-reductions): New option,
EnabledBy(floop-unroll-and-jam).
* config/aarch64/aarch64-protos.h (struct tune_params):
Add loop_jam_reduction_factor.
* config/aarch64/aarch64.cc (aarch64_loop_jam_reduction_factor):
New function; source factor from aarch64_tune_params and opt
in for QI/HI->SI and SI->DI MAC shapes only.
(TARGET_LOOP_JAM_REDUCTION_FACTOR): Define.
* config/aarch64/tuning_models/a64fx.h: Default
loop_jam_reduction_factor to 0 (opt-out).
* config/aarch64/tuning_models/ampere1.h: Set
loop_jam_reduction_factor to 4.
* config/aarch64/tuning_models/ampere1a.h: Likewise.
* config/aarch64/tuning_models/ampere1b.h: Likewise.
* config/aarch64/tuning_models/cortexa320.h: Default
loop_jam_reduction_factor to 0 (opt-out).
* config/aarch64/tuning_models/cortexa35.h: Likewise.
* config/aarch64/tuning_models/cortexa53.h: Likewise.
* config/aarch64/tuning_models/cortexa57.h: Likewise.
* config/aarch64/tuning_models/cortexa72.h: Likewise.
* config/aarch64/tuning_models/cortexa73.h: Likewise.
* config/aarch64/tuning_models/cortexx925.h: Likewise.
* config/aarch64/tuning_models/emag.h: Likewise.
* config/aarch64/tuning_models/exynosm1.h: Likewise.
* config/aarch64/tuning_models/fujitsu_monaka.h: Likewise.
* config/aarch64/tuning_models/generic.h: Likewise.
* config/aarch64/tuning_models/generic_armv8_a.h: Likewise.
* config/aarch64/tuning_models/generic_armv9_a.h: Likewise.
* config/aarch64/tuning_models/hip12.h: Likewise.
* config/aarch64/tuning_models/neoverse512tvb.h: Likewise.
* config/aarch64/tuning_models/neoversen1.h: Set
loop_jam_reduction_factor to 4.
* config/aarch64/tuning_models/neoversen2.h: Default
loop_jam_reduction_factor to 0 (opt-out).
* config/aarch64/tuning_models/neoversen3.h: Likewise.
* config/aarch64/tuning_models/neoversev1.h: Likewise.
* config/aarch64/tuning_models/neoversev2.h: Likewise.
* config/aarch64/tuning_models/neoversev3.h: Likewise.
* config/aarch64/tuning_models/neoversev3ae.h: Likewise.
* config/aarch64/tuning_models/olympus.h: Likewise.
* config/aarch64/tuning_models/qdf24xx.h: Likewise.
* config/aarch64/tuning_models/saphira.h: Likewise.
* config/aarch64/tuning_models/thunderx.h: Likewise.
* config/aarch64/tuning_models/thunderx2t99.h: Likewise.
* config/aarch64/tuning_models/thunderx3t110.h: Likewise.
* config/aarch64/tuning_models/thunderxt88.h: Likewise.
* config/aarch64/tuning_models/tsv110.h: Likewise.
* config/aarch64/tuning_models/xgene1.h: Likewise.
* doc/invoke.texi (-floop-unroll-and-jam-reductions): Document.
* doc/tm.texi: Regenerate.
* doc/tm.texi.in (TARGET_LOOP_JAM_REDUCTION_FACTOR): Add hook
reference.
* gimple-loop-jam.cc: Include cfganal.h and tree-into-ssa.h.
(inner_reduction_phi): New helper.
(narrowest_reduc_input_mode): New helper that walks the
reduction update chain for widening multiplies.
(gsi_move_to_bb_end_before_terminator): New helper.
(collapse_single_arg_phi): New helper.
(loop_is_dot_product_candidate_p): New static helper.
(unroll_jam_reduction_possible_p): New candidate detector;
calls loop_is_dot_product_candidate_p and layers outer-loop
guards on top.
(migrate_loop_invariant_defs_to_preheader): New helper that
hoists the SSA-use closure of a sibling's reduction init and
body operands to the destination preheader in RPO.
(match_sibling_header_phis): New helper pairing sibling non-
reduction PHIs positionally.
(unroll_jam_reduction_fuse_sibling): New per-sibling fuser.
(unroll_jam_reduction_bypass_sibling): New CFG bypass for the
drained source sibling.
(compute_jam_dependence_factors): New helper. Walks OUTER's
data dependences and accumulates the unroll-factor safety
bound, ref-removal profit signal, removed-ref count, and total
dataref count.
(select_reduction_jam_factor): New helper combining target
factor, param cap and data-dep safety bound, narrowed to a
divisor of the outer trip count. Emit a TDF_DETAILS note
when the outer trip has no divisor in [2, factor cap].
(do_unroll_jam_reductions): New driver helper.
(tree_loop_unroll_and_jam): Try the reduction-jam path on
nests the IV-only path skips; cleanup CFG, rebuild LC-SSA and
reset SCEV per transformed nest.
(pass_loop_jam::gate): Also enable for
flag_unroll_jam_reductions.
* hooks.cc (hook_uint_mode_mode_0): New default hook.
* hooks.h (hook_uint_mode_mode_0): Declare.
* params.opt (unroll-jam-reduction-streams): New param.
(unroll-jam-reduction-min-inner-iters): New param.
* target.def (loop_jam_reduction_factor): New target hook.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/unroll-jam-reductions-1.c: New test.
* gcc.target/aarch64/unroll-jam-reductions-2.c: New test.
* gcc.target/aarch64/unroll-jam-reductions-3.c: New test.
* gcc.target/aarch64/unroll-jam-reductions-4.c: New test.
* gcc.target/aarch64/unroll-jam-reductions-5.c: New test.
Co-Authored-By: Philipp Tomsich <[email protected]>
Signed-off-by: Konstantinos Eleftheriou <[email protected]>
---
gcc/common.opt | 5 +
gcc/config/aarch64/aarch64-protos.h | 8 +
gcc/config/aarch64/aarch64.cc | 29 +
gcc/config/aarch64/tuning_models/a64fx.h | 3 +-
gcc/config/aarch64/tuning_models/ampere1.h | 3 +-
gcc/config/aarch64/tuning_models/ampere1a.h | 3 +-
gcc/config/aarch64/tuning_models/ampere1b.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa320.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa35.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa53.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa57.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa72.h | 3 +-
gcc/config/aarch64/tuning_models/cortexa73.h | 3 +-
gcc/config/aarch64/tuning_models/cortexx925.h | 3 +-
gcc/config/aarch64/tuning_models/emag.h | 3 +-
gcc/config/aarch64/tuning_models/exynosm1.h | 3 +-
.../aarch64/tuning_models/fujitsu_monaka.h | 3 +-
gcc/config/aarch64/tuning_models/generic.h | 3 +-
.../aarch64/tuning_models/generic_armv8_a.h | 3 +-
.../aarch64/tuning_models/generic_armv9_a.h | 3 +-
gcc/config/aarch64/tuning_models/hip12.h | 3 +-
.../aarch64/tuning_models/neoverse512tvb.h | 3 +-
gcc/config/aarch64/tuning_models/neoversen1.h | 3 +-
gcc/config/aarch64/tuning_models/neoversen2.h | 3 +-
gcc/config/aarch64/tuning_models/neoversen3.h | 3 +-
gcc/config/aarch64/tuning_models/neoversev1.h | 3 +-
gcc/config/aarch64/tuning_models/neoversev2.h | 3 +-
gcc/config/aarch64/tuning_models/neoversev3.h | 3 +-
.../aarch64/tuning_models/neoversev3ae.h | 3 +-
gcc/config/aarch64/tuning_models/olympus.h | 3 +-
gcc/config/aarch64/tuning_models/qdf24xx.h | 3 +-
gcc/config/aarch64/tuning_models/saphira.h | 3 +-
gcc/config/aarch64/tuning_models/thunderx.h | 3 +-
.../aarch64/tuning_models/thunderx2t99.h | 3 +-
.../aarch64/tuning_models/thunderx3t110.h | 3 +-
.../aarch64/tuning_models/thunderxt88.h | 3 +-
gcc/config/aarch64/tuning_models/tsv110.h | 3 +-
gcc/config/aarch64/tuning_models/xgene1.h | 3 +-
gcc/doc/invoke.texi | 17 +-
gcc/doc/tm.texi | 11 +
gcc/doc/tm.texi.in | 2 +
gcc/gimple-loop-jam.cc | 932 +++++++++++++++++-
gcc/hooks.cc | 7 +
gcc/hooks.h | 1 +
gcc/params.opt | 8 +
gcc/target.def | 13 +
.../aarch64/unroll-jam-reductions-1.c | 78 ++
.../aarch64/unroll-jam-reductions-2.c | 89 ++
.../aarch64/unroll-jam-reductions-3.c | 189 ++++
.../aarch64/unroll-jam-reductions-4.c | 30 +
.../aarch64/unroll-jam-reductions-5.c | 72 ++
51 files changed, 1557 insertions(+), 39 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
diff --git a/gcc/common.opt b/gcc/common.opt
index 69a55b8d40d6..634b63569c67 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -1928,6 +1928,11 @@ floop-unroll-and-jam
Common Var(flag_unroll_jam) Optimization
Perform unroll-and-jam on loops.
+floop-unroll-and-jam-reductions
+Common Var(flag_unroll_jam_reductions) Optimization
EnabledBy(floop-unroll-and-jam)
+Perform reduction-aware unroll-and-jam on loops with inner reductions
+to enable multi-stream parallelism during vectorization.
+
fgnu-tm
Common Var(flag_tm)
Enable support for GNU transactional memory.
diff --git a/gcc/config/aarch64/aarch64-protos.h
b/gcc/config/aarch64/aarch64-protos.h
index e8ae3d427949..88237e243738 100644
--- a/gcc/config/aarch64/aarch64-protos.h
+++ b/gcc/config/aarch64/aarch64-protos.h
@@ -590,6 +590,14 @@ struct tune_params
/* Dispatch constraints for instruction scheduling. */
const struct dispatch_constraint_info *dispatch_constraints;
+
+ /* Preferred number of parallel reduction streams for
+ -floop-unroll-and-jam-reductions on this core. Zero opts out;
+ two or more enables the transform on reduction nests whose
+ (acc_mode, op_mode) pair the target hook accepts. Cores with a
+ two-wide int/FP pipe and several-cycle MAC latency typically
+ want 4. */
+ unsigned int loop_jam_reduction_factor;
};
/* Classifies an address.
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index ccb4f5fbb10d..546912534499 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -17402,6 +17402,32 @@ aarch64_sched_issue_rate (void)
return aarch64_tune_params.issue_rate;
}
+/* Implement TARGET_LOOP_JAM_REDUCTION_FACTOR. Return the preferred
+ number of parallel reduction streams for
+ -floop-unroll-and-jam-reductions on a reduction accumulating into
+ ACC_MODE from operands of OP_MODE. We opt in only for the widening
+ MAC shapes the vectorizer lowers to a latency-bound
+ DOT_PROD / SMLAL chain:
+ * QImode / HImode -> SImode (DOT_PROD-class, USDOT / SDOT / UDOT)
+ * SImode -> DImode (SMLAL_DI)
+ The stream count comes from the per-core tuning structure; cores
+ that have not been measured leave loop_jam_reduction_factor at 0,
+ which suppresses the transform. */
+
+static unsigned int
+aarch64_loop_jam_reduction_factor (machine_mode acc_mode,
+ machine_mode op_mode)
+{
+ unsigned int factor = aarch64_tune_params.loop_jam_reduction_factor;
+ if (factor < 2 || !SCALAR_INT_MODE_P (acc_mode))
+ return 0;
+
+ if ((acc_mode == SImode && (op_mode == QImode || op_mode == HImode))
+ || (acc_mode == DImode && op_mode == SImode))
+ return factor;
+ return 0;
+}
+
/* Implement TARGET_SCHED_VARIABLE_ISSUE. */
static int
aarch64_sched_variable_issue (FILE *, int, rtx_insn *insn, int more)
@@ -34491,6 +34517,9 @@ aarch64_libgcc_floating_mode_supported_p
#undef TARGET_SCHED_ISSUE_RATE
#define TARGET_SCHED_ISSUE_RATE aarch64_sched_issue_rate
+#undef TARGET_LOOP_JAM_REDUCTION_FACTOR
+#define TARGET_LOOP_JAM_REDUCTION_FACTOR aarch64_loop_jam_reduction_factor
+
#undef TARGET_SCHED_VARIABLE_ISSUE
#define TARGET_SCHED_VARIABLE_ISSUE aarch64_sched_variable_issue
diff --git a/gcc/config/aarch64/tuning_models/a64fx.h
b/gcc/config/aarch64/tuning_models/a64fx.h
index c3222960f175..171e64ce23ad 100644
--- a/gcc/config/aarch64/tuning_models/a64fx.h
+++ b/gcc/config/aarch64/tuning_models/a64fx.h
@@ -166,7 +166,8 @@ static const struct tune_params a64fx_tunings =
&a64fx_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_A64FX. */
diff --git a/gcc/config/aarch64/tuning_models/ampere1.h
b/gcc/config/aarch64/tuning_models/ampere1.h
index e2a42f585dd9..0057db3765b4 100644
--- a/gcc/config/aarch64/tuning_models/ampere1.h
+++ b/gcc/config/aarch64/tuning_models/ampere1.h
@@ -136,7 +136,8 @@ static const struct tune_params ampere1_tunings =
&ere1_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 4 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_AMPERE1. */
diff --git a/gcc/config/aarch64/tuning_models/ampere1a.h
b/gcc/config/aarch64/tuning_models/ampere1a.h
index 0954ac551a0b..1350c0f3b5c5 100644
--- a/gcc/config/aarch64/tuning_models/ampere1a.h
+++ b/gcc/config/aarch64/tuning_models/ampere1a.h
@@ -58,7 +58,8 @@ static const struct tune_params ampere1a_tunings =
&ere1_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 4 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_AMPERE1A. */
diff --git a/gcc/config/aarch64/tuning_models/ampere1b.h
b/gcc/config/aarch64/tuning_models/ampere1b.h
index 8ba434d27bbf..0fc6eb36bd2a 100644
--- a/gcc/config/aarch64/tuning_models/ampere1b.h
+++ b/gcc/config/aarch64/tuning_models/ampere1b.h
@@ -137,7 +137,8 @@ static const struct tune_params ampere1b_tunings =
&ere1b_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 4 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_AMPERE1B */
diff --git a/gcc/config/aarch64/tuning_models/cortexa320.h
b/gcc/config/aarch64/tuning_models/cortexa320.h
index d5e1f2b29938..89ed24360008 100644
--- a/gcc/config/aarch64/tuning_models/cortexa320.h
+++ b/gcc/config/aarch64/tuning_models/cortexa320.h
@@ -189,7 +189,8 @@ static const struct tune_params cortexa320_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXA320. */
diff --git a/gcc/config/aarch64/tuning_models/cortexa35.h
b/gcc/config/aarch64/tuning_models/cortexa35.h
index fd2fc47f5e23..ff9eeb406799 100644
--- a/gcc/config/aarch64/tuning_models/cortexa35.h
+++ b/gcc/config/aarch64/tuning_models/cortexa35.h
@@ -57,7 +57,8 @@ static const struct tune_params cortexa35_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXA35. */
diff --git a/gcc/config/aarch64/tuning_models/cortexa53.h
b/gcc/config/aarch64/tuning_models/cortexa53.h
index e71cc1b26afb..13dbb152c6e9 100644
--- a/gcc/config/aarch64/tuning_models/cortexa53.h
+++ b/gcc/config/aarch64/tuning_models/cortexa53.h
@@ -66,7 +66,8 @@ static const struct tune_params cortexa53_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXA53. */
diff --git a/gcc/config/aarch64/tuning_models/cortexa57.h
b/gcc/config/aarch64/tuning_models/cortexa57.h
index 4f3ec60ba275..c78c15198158 100644
--- a/gcc/config/aarch64/tuning_models/cortexa57.h
+++ b/gcc/config/aarch64/tuning_models/cortexa57.h
@@ -103,7 +103,8 @@ static const struct tune_params cortexa57_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXA57. */
diff --git a/gcc/config/aarch64/tuning_models/cortexa72.h
b/gcc/config/aarch64/tuning_models/cortexa72.h
index df0fcf110867..8a4e5e9d8a36 100644
--- a/gcc/config/aarch64/tuning_models/cortexa72.h
+++ b/gcc/config/aarch64/tuning_models/cortexa72.h
@@ -55,7 +55,8 @@ static const struct tune_params cortexa72_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXA72. */
diff --git a/gcc/config/aarch64/tuning_models/cortexa73.h
b/gcc/config/aarch64/tuning_models/cortexa73.h
index c871b454d40f..40d46f23ae82 100644
--- a/gcc/config/aarch64/tuning_models/cortexa73.h
+++ b/gcc/config/aarch64/tuning_models/cortexa73.h
@@ -56,7 +56,8 @@ static const struct tune_params cortexa73_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
diff --git a/gcc/config/aarch64/tuning_models/cortexx925.h
b/gcc/config/aarch64/tuning_models/cortexx925.h
index 2a73b3697065..b99d6b4300e5 100644
--- a/gcc/config/aarch64/tuning_models/cortexx925.h
+++ b/gcc/config/aarch64/tuning_models/cortexx925.h
@@ -227,7 +227,8 @@ static const struct tune_params cortexx925_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_CORTEXX925. */
diff --git a/gcc/config/aarch64/tuning_models/emag.h
b/gcc/config/aarch64/tuning_models/emag.h
index 82bbc554f41b..6e87d28173b2 100644
--- a/gcc/config/aarch64/tuning_models/emag.h
+++ b/gcc/config/aarch64/tuning_models/emag.h
@@ -55,7 +55,8 @@ static const struct tune_params emag_tunings =
&xgene1_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_EMAG. */
diff --git a/gcc/config/aarch64/tuning_models/exynosm1.h
b/gcc/config/aarch64/tuning_models/exynosm1.h
index a561ccf6fa71..928f3d4f3014 100644
--- a/gcc/config/aarch64/tuning_models/exynosm1.h
+++ b/gcc/config/aarch64/tuning_models/exynosm1.h
@@ -139,7 +139,8 @@ static const struct tune_params exynosm1_tunings =
&exynosm1_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_EXYNOSM1. */
diff --git a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
index 71549bd0ea70..70d71bf253de 100644
--- a/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
+++ b/gcc/config/aarch64/tuning_models/fujitsu_monaka.h
@@ -59,7 +59,8 @@ static const struct tune_params fujitsu_monaka_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_FUJITSU_MONAKA. */
diff --git a/gcc/config/aarch64/tuning_models/generic.h
b/gcc/config/aarch64/tuning_models/generic.h
index af013e478d45..71dbebeb47c0 100644
--- a/gcc/config/aarch64/tuning_models/generic.h
+++ b/gcc/config/aarch64/tuning_models/generic.h
@@ -188,7 +188,8 @@ static const struct tune_params generic_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_GENERIC. */
diff --git a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
index 29cf3a5a7c7a..15900a7ba733 100644
--- a/gcc/config/aarch64/tuning_models/generic_armv8_a.h
+++ b/gcc/config/aarch64/tuning_models/generic_armv8_a.h
@@ -180,7 +180,8 @@ static const struct tune_params generic_armv8_a_tunings =
&generic_armv8_a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_GENERIC_ARMV8_A. */
diff --git a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
index 05e777d20efb..7c1b78f88a53 100644
--- a/gcc/config/aarch64/tuning_models/generic_armv9_a.h
+++ b/gcc/config/aarch64/tuning_models/generic_armv9_a.h
@@ -255,7 +255,8 @@ static const struct tune_params generic_armv9_a_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_GENERIC_ARMV9_A. */
diff --git a/gcc/config/aarch64/tuning_models/hip12.h
b/gcc/config/aarch64/tuning_models/hip12.h
index cfb204b2e533..2d5c107d71d7 100644
--- a/gcc/config/aarch64/tuning_models/hip12.h
+++ b/gcc/config/aarch64/tuning_models/hip12.h
@@ -221,7 +221,8 @@ static const struct tune_params hip12_tunings =
&generic_armv8_a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_HIP12. */
diff --git a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
index d6e2c2e8120d..a128d5ed7b3a 100644
--- a/gcc/config/aarch64/tuning_models/neoverse512tvb.h
+++ b/gcc/config/aarch64/tuning_models/neoverse512tvb.h
@@ -162,7 +162,8 @@ static const struct tune_params neoverse512tvb_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSE512TVB. */
diff --git a/gcc/config/aarch64/tuning_models/neoversen1.h
b/gcc/config/aarch64/tuning_models/neoversen1.h
index 87c7fbcb7f22..9b2ec9322bba 100644
--- a/gcc/config/aarch64/tuning_models/neoversen1.h
+++ b/gcc/config/aarch64/tuning_models/neoversen1.h
@@ -55,7 +55,8 @@ static const struct tune_params neoversen1_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 4 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEN1. */
diff --git a/gcc/config/aarch64/tuning_models/neoversen2.h
b/gcc/config/aarch64/tuning_models/neoversen2.h
index 21597e6b50f8..e1648c88eb78 100644
--- a/gcc/config/aarch64/tuning_models/neoversen2.h
+++ b/gcc/config/aarch64/tuning_models/neoversen2.h
@@ -224,7 +224,8 @@ static const struct tune_params neoversen2_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEN2. */
diff --git a/gcc/config/aarch64/tuning_models/neoversen3.h
b/gcc/config/aarch64/tuning_models/neoversen3.h
index 37f9819148f6..8796914772b4 100644
--- a/gcc/config/aarch64/tuning_models/neoversen3.h
+++ b/gcc/config/aarch64/tuning_models/neoversen3.h
@@ -223,7 +223,8 @@ static const struct tune_params neoversen3_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEN3. */
diff --git a/gcc/config/aarch64/tuning_models/neoversev1.h
b/gcc/config/aarch64/tuning_models/neoversev1.h
index 253f11e87a68..7f93e4092550 100644
--- a/gcc/config/aarch64/tuning_models/neoversev1.h
+++ b/gcc/config/aarch64/tuning_models/neoversev1.h
@@ -233,7 +233,8 @@ static const struct tune_params neoversev1_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
diff --git a/gcc/config/aarch64/tuning_models/neoversev2.h
b/gcc/config/aarch64/tuning_models/neoversev2.h
index 6df0cc444b80..8ceb7d5538fb 100644
--- a/gcc/config/aarch64/tuning_models/neoversev2.h
+++ b/gcc/config/aarch64/tuning_models/neoversev2.h
@@ -364,7 +364,8 @@ static const struct tune_params neoversev2_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- &neoversev2_dispatch_constraint_info /* dispatch_constraints. */
+ &neoversev2_dispatch_constraint_info, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEV2. */
diff --git a/gcc/config/aarch64/tuning_models/neoversev3.h
b/gcc/config/aarch64/tuning_models/neoversev3.h
index 7e0144032dc6..ea840d2cbb7c 100644
--- a/gcc/config/aarch64/tuning_models/neoversev3.h
+++ b/gcc/config/aarch64/tuning_models/neoversev3.h
@@ -225,7 +225,8 @@ static const struct tune_params neoversev3_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEV3. */
diff --git a/gcc/config/aarch64/tuning_models/neoversev3ae.h
b/gcc/config/aarch64/tuning_models/neoversev3ae.h
index bbe4dc547015..fe39c7b67c9d 100644
--- a/gcc/config/aarch64/tuning_models/neoversev3ae.h
+++ b/gcc/config/aarch64/tuning_models/neoversev3ae.h
@@ -225,7 +225,8 @@ static const struct tune_params neoversev3ae_tunings =
&generic_armv9a_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_NEOVERSEV3AE. */
diff --git a/gcc/config/aarch64/tuning_models/olympus.h
b/gcc/config/aarch64/tuning_models/olympus.h
index 1e701722e224..ab1ba2464d23 100644
--- a/gcc/config/aarch64/tuning_models/olympus.h
+++ b/gcc/config/aarch64/tuning_models/olympus.h
@@ -565,7 +565,8 @@ static struct tune_params olympus_tunings =
&olympus_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- &olympus_dispatch_constraint_info /* dispatch_constraints. */
+ &olympus_dispatch_constraint_info, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_OLYMPUS. */
diff --git a/gcc/config/aarch64/tuning_models/qdf24xx.h
b/gcc/config/aarch64/tuning_models/qdf24xx.h
index a15a4fc56897..73eb0b8c48cf 100644
--- a/gcc/config/aarch64/tuning_models/qdf24xx.h
+++ b/gcc/config/aarch64/tuning_models/qdf24xx.h
@@ -131,7 +131,8 @@ static const struct tune_params qdf24xx_tunings =
&qdf24xx_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_QDF24XX. */
diff --git a/gcc/config/aarch64/tuning_models/saphira.h
b/gcc/config/aarch64/tuning_models/saphira.h
index 369d3618f7f1..582c4517fcdc 100644
--- a/gcc/config/aarch64/tuning_models/saphira.h
+++ b/gcc/config/aarch64/tuning_models/saphira.h
@@ -57,7 +57,8 @@ static const struct tune_params saphira_tunings =
&generic_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_SAPHIRA. */
diff --git a/gcc/config/aarch64/tuning_models/thunderx.h
b/gcc/config/aarch64/tuning_models/thunderx.h
index ed8de307276a..a275659c0b9e 100644
--- a/gcc/config/aarch64/tuning_models/thunderx.h
+++ b/gcc/config/aarch64/tuning_models/thunderx.h
@@ -112,7 +112,8 @@ static const struct tune_params thunderx_tunings =
&thunderx_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_THUNDERX. */
diff --git a/gcc/config/aarch64/tuning_models/thunderx2t99.h
b/gcc/config/aarch64/tuning_models/thunderx2t99.h
index f6be657ac9c3..5afbc21ef40c 100644
--- a/gcc/config/aarch64/tuning_models/thunderx2t99.h
+++ b/gcc/config/aarch64/tuning_models/thunderx2t99.h
@@ -132,7 +132,8 @@ static const struct tune_params thunderx2t99_tunings =
&thunderx2t99_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_THUNDERX2T99. */
diff --git a/gcc/config/aarch64/tuning_models/thunderx3t110.h
b/gcc/config/aarch64/tuning_models/thunderx3t110.h
index cdcbec680945..0d7f29bee209 100644
--- a/gcc/config/aarch64/tuning_models/thunderx3t110.h
+++ b/gcc/config/aarch64/tuning_models/thunderx3t110.h
@@ -131,7 +131,8 @@ static const struct tune_params thunderx3t110_tunings =
&thunderx3t110_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_THUNDERX3T110. */
diff --git a/gcc/config/aarch64/tuning_models/thunderxt88.h
b/gcc/config/aarch64/tuning_models/thunderxt88.h
index 44a1eac59a96..d0c39c07c88a 100644
--- a/gcc/config/aarch64/tuning_models/thunderxt88.h
+++ b/gcc/config/aarch64/tuning_models/thunderxt88.h
@@ -67,7 +67,8 @@ static const struct tune_params thunderxt88_tunings =
&thunderxt88_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALIGNED, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALIGNED, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_THUNDERXT88. */
diff --git a/gcc/config/aarch64/tuning_models/tsv110.h
b/gcc/config/aarch64/tuning_models/tsv110.h
index 366f2b3e2323..0800bdc59ced 100644
--- a/gcc/config/aarch64/tuning_models/tsv110.h
+++ b/gcc/config/aarch64/tuning_models/tsv110.h
@@ -132,7 +132,8 @@ static const struct tune_params tsv110_tunings =
&tsv110_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_TSV110. */
diff --git a/gcc/config/aarch64/tuning_models/xgene1.h
b/gcc/config/aarch64/tuning_models/xgene1.h
index 0d4aaf80b61a..5fd5e06932c3 100644
--- a/gcc/config/aarch64/tuning_models/xgene1.h
+++ b/gcc/config/aarch64/tuning_models/xgene1.h
@@ -140,7 +140,8 @@ static const struct tune_params xgene1_tunings =
&xgene1_prefetch_tune,
AARCH64_LDP_STP_POLICY_ALWAYS, /* ldp_policy_model. */
AARCH64_LDP_STP_POLICY_ALWAYS, /* stp_policy_model. */
- nullptr /* dispatch_constraints. */
+ nullptr, /* dispatch_constraints. */
+ 0 /* loop_jam_reduction_factor. */
};
#endif /* GCC_AARCH64_H_XGENE1. */
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index a74e99081d35..9f3215deb942 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -641,7 +641,8 @@ Objective-C and Objective-C++ Dialects}.
-fivopts -fkeep-inline-functions -fkeep-static-functions
-fkeep-static-consts -flimit-function-alignment -flive-range-shrinkage
-floop-block -floop-interchange -floop-strip-mine
--floop-unroll-and-jam -floop-nest-optimize
+-floop-unroll-and-jam -floop-unroll-and-jam-reductions
+-floop-nest-optimize
-floop-parallelize-all -flra-remat -flto -flto-compression-level=@var{n}
-flto-toplevel-asm-heuristics
-flto-partition=@var{alg} -flto-incremental=@var{path}
@@ -15307,6 +15308,20 @@ nest this unrolls the outer loop by some factor and
fuses the resulting
multiple inner loops. This flag is enabled by default at @option{-O3}.
It is also enabled by @option{-fprofile-use} and @option{-fauto-profile}.
+@opindex floop-unroll-and-jam-reductions
+@opindex fno-loop-unroll-and-jam-reductions
+@item -floop-unroll-and-jam-reductions
+Apply a reduction-aware variant of unroll and jam to loop nests whose
+inner loop carries a reduction accumulator reset every outer iteration
+(the GEMV / dot-product shape), producing N parallel reduction streams.
+N is taken from @code{TARGET_LOOP_JAM_REDUCTION_FACTOR} (no-op on
+targets that do not opt in) and capped by
+@option{--param=unroll-jam-reduction-streams}. Inner loops whose
+trip count is below @option{--param=unroll-jam-reduction-min-inner-iters}
+are left alone, because the per-stream serial chain is too short to
+profit from parallelism. The outer loop trip count must be a
+compile-time constant. Enabled whenever @option{-floop-unroll-and-jam} is.
+
@opindex ftree-loop-im
@opindex fno-tree-loop-im
@item -ftree-loop-im
diff --git a/gcc/doc/tm.texi b/gcc/doc/tm.texi
index ca68bdc3e615..be0771e5348a 100644
--- a/gcc/doc/tm.texi
+++ b/gcc/doc/tm.texi
@@ -12653,6 +12653,17 @@ is required only when the target has special
constraints like maximum
number of memory accesses.
@end deftypefn
+@deftypefn {Target Hook} unsigned TARGET_LOOP_JAM_REDUCTION_FACTOR
(machine_mode @var{acc_mode}, machine_mode @var{op_mode})
+Return the preferred number of parallel reduction streams
+@option{-floop-unroll-and-jam-reductions} should produce for an inner
+reduction that accumulates into mode @var{acc_mode} from operands of
+mode @var{op_mode}. @var{op_mode} is the operand mode at the source
+of the widening multiply, and is always strictly narrower than
+@var{acc_mode}: plain reductions with no widening multiply are
+filtered out before the hook is consulted. Returning less than two
+disables the transform for that mode pair.
+@end deftypefn
+
@defmac POWI_MAX_MULTS
If defined, this macro is interpreted as a signed integer C expression
that specifies the maximum number of floating point multiplications
diff --git a/gcc/doc/tm.texi.in b/gcc/doc/tm.texi.in
index 532e95ef216f..027a311f310e 100644
--- a/gcc/doc/tm.texi.in
+++ b/gcc/doc/tm.texi.in
@@ -8011,6 +8011,8 @@ lists.
@hook TARGET_LOOP_UNROLL_ADJUST
+@hook TARGET_LOOP_JAM_REDUCTION_FACTOR
+
@defmac POWI_MAX_MULTS
If defined, this macro is interpreted as a signed integer C expression
that specifies the maximum number of floating point multiplications
diff --git a/gcc/gimple-loop-jam.cc b/gcc/gimple-loop-jam.cc
index 2c4162d17a2a..527f7c6cce35 100644
--- a/gcc/gimple-loop-jam.cc
+++ b/gcc/gimple-loop-jam.cc
@@ -34,12 +34,14 @@ along with GCC; see the file COPYING3. If not see
#include "cfgloop.h"
#include "tree-scalar-evolution.h"
#include "gimple-iterator.h"
+#include "cfganal.h"
#include "cfghooks.h"
#include "tree-data-ref.h"
#include "tree-ssa-loop-ivopts.h"
#include "tree-vectorizer.h"
#include "tree-ssa-sccvn.h"
#include "tree-cfgcleanup.h"
+#include "tree-into-ssa.h"
/* Unroll and Jam transformation
@@ -376,6 +378,684 @@ fuse_loops (class loop *loop)
}
}
+/* Reduction-jam: a parallel-reduction extension of unroll-and-jam.
+
+ The transformation above accepts only inner PHIs that are simple
+ inductions. A reduction PHI cannot be fused the same way: merging
+ the streams would chain them into a single serial accumulation,
+ multiplying the serial dependency height instead of cutting it.
+ This handles the GEMV / NNUE shape
+
+ for (i = 0; i < M; i++) {
+ s = bias[i];
+ for (j = 0; j < K; j++) s += w[i][j] * x[j];
+ out[i] = s;
+ }
+
+ where s starts afresh every outer iteration. Unrolling the outer
+ loop by N leaves N sibling inner loops, each with its own reduction
+ PHI and its own init (bias[i], bias[i+1], ...). Instead of merging
+ them we keep all N PHIs in the fused header and hoist each sibling's
+ init ahead of the body; the post-reduction stores stay on the
+ sibling-exit chain, coalesced afterward. The vectorizer then emits
+ N independent DOT_PROD / FMA chains, so the serial dependency height
+ across the reduction drops by a factor of N.
+
+ The strip factor N is the smallest of the target's
+ TARGET_LOOP_JAM_REDUCTION_FACTOR, param_unroll_jam_reduction_streams
+ and the data-dep safety bound, narrowed to a divisor of the outer
+ trip count. We ignore adjust_unroll_factor's profit signal on
+ purpose: it scores the memory refs that fusion removes, which is
+ not what this transform is after. Targets that have not measured a
+ win return 0 from the hook and we skip the loop. See params.opt
+ for the tunables. */
+
+/* Return the single non-virtual, non-simple-IV PHI in LOOP's header
+ (the reduction accumulator), or NULL if there is none or more than
+ one. */
+
+static gphi *
+inner_reduction_phi (class loop *loop)
+{
+ gphi *result = NULL;
+ for (gphi_iterator psi = gsi_start_phis (loop->header);
+ !gsi_end_p (psi); gsi_next (&psi))
+ {
+ gphi *phi = psi.phi ();
+ tree op = gimple_phi_result (phi);
+ if (virtual_operand_p (op))
+ continue;
+ affine_iv iv;
+ if (simple_iv (loop, loop, op, &iv, true))
+ continue;
+ if (result)
+ return NULL;
+ result = phi;
+ }
+ return result;
+}
+
+/* Walk the SSA def chain from RED_PHI's latch update and return the
+ narrowest scalar-int mode feeding a multiply inside INNER, or
+ ACC_MODE if nothing narrower turns up. The walk follows only
+ PLUS / MINUS / CONVERT chains so that a multiply buried in an array
+ index (the "w * j" in "s += a[w * j]") is not mistaken for the MAC
+ source. */
+
+static machine_mode
+narrowest_reduc_input_mode (gphi *red_phi, class loop *inner,
+ machine_mode acc_mode)
+{
+ tree update = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner));
+ if (TREE_CODE (update) != SSA_NAME)
+ return acc_mode;
+
+ machine_mode narrow = VOIDmode;
+ auto_vec<tree, 8> worklist;
+ hash_set<tree> seen;
+ worklist.safe_push (update);
+
+ while (!worklist.is_empty ())
+ {
+ tree op = worklist.pop ();
+ if (seen.add (op))
+ continue;
+ gimple *def = SSA_NAME_DEF_STMT (op);
+ if (gimple_code (def) == GIMPLE_PHI
+ || !gimple_bb (def)
+ || !flow_bb_inside_loop_p (inner, gimple_bb (def))
+ || !is_gimple_assign (def))
+ continue;
+
+ /* Stay in the reduction value chain. Loads and pointer
+ arithmetic feed address computation, not the accumulator. */
+ if (gimple_assign_load_p (def)
+ || POINTER_TYPE_P (TREE_TYPE (gimple_assign_lhs (def))))
+ continue;
+
+ use_operand_p use_p;
+ ssa_op_iter oi;
+ tree_code code = gimple_assign_rhs_code (def);
+ if (code == MULT_EXPR || code == WIDEN_MULT_EXPR)
+ {
+ FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
+ {
+ tree mop = USE_FROM_PTR (use_p);
+ while (TREE_CODE (mop) == SSA_NAME)
+ {
+ gimple *mdef = SSA_NAME_DEF_STMT (mop);
+ if (!is_gimple_assign (mdef)
+ || !CONVERT_EXPR_CODE_P (gimple_assign_rhs_code (mdef)))
+ break;
+ mop = gimple_assign_rhs1 (mdef);
+ }
+ machine_mode m = TYPE_MODE (TREE_TYPE (mop));
+ if (SCALAR_INT_MODE_P (m)
+ && (narrow == VOIDmode
+ || known_lt (GET_MODE_PRECISION (m),
+ GET_MODE_PRECISION (narrow))))
+ narrow = m;
+ }
+ continue;
+ }
+
+ /* Only scalar-reduction operations continue the walk. */
+ if (code != PLUS_EXPR
+ && code != MINUS_EXPR
+ && !CONVERT_EXPR_CODE_P (code))
+ continue;
+
+ FOR_EACH_SSA_USE_OPERAND (use_p, def, oi, SSA_OP_USE)
+ worklist.safe_push (USE_FROM_PTR (use_p));
+ }
+
+ if (narrow == VOIDmode
+ || known_ge (GET_MODE_PRECISION (narrow),
+ GET_MODE_PRECISION (acc_mode)))
+ return acc_mode;
+ return narrow;
+}
+
+/* Move *FROM to BB, immediately before BB's terminator branch (or to
+ the end of BB if there is none). */
+
+static void
+gsi_move_to_bb_end_before_terminator (gimple_stmt_iterator *from,
+ basic_block bb)
+{
+ gimple_stmt_iterator to = gsi_last_bb (bb);
+ if (!gsi_end_p (to) && is_ctrl_stmt (gsi_stmt (to)))
+ gsi_move_before (from, &to);
+ else
+ gsi_move_after (from, &to);
+}
+
+/* Propagate the single argument of the PHI at PSI into its uses and
+ remove the PHI, leaving PSI at the next one. */
+
+static void
+collapse_single_arg_phi (gphi_iterator *psi)
+{
+ gphi *phi = psi->phi ();
+ gcc_checking_assert (gimple_phi_num_args (phi) == 1);
+ tree result = gimple_phi_result (phi);
+ if (!has_zero_uses (result))
+ replace_uses_by (result, gimple_phi_arg_def (phi, 0));
+ remove_phi_node (psi, true);
+}
+
+/* True if LOOP is a widening MAC reduction the vectorizer can turn
+ into a DOT_PROD_EXPR: a single-BB body with one exit, loads only, a
+ constant trip count of at least MIN_INNER_ITERS, one non-IV
+ reduction PHI alongside a unit-step counter IV, and a widening
+ multiply in the update chain. If OUTER is given, every IV's base
+ and step must be invariant in it. On success *OUT_RED_PHI,
+ *OUT_ACC_MODE and *OUT_OP_MODE receive the reduction PHI, the
+ accumulator mode and the narrowest operand mode. */
+
+static bool
+loop_is_dot_product_candidate_p (class loop *loop,
+ unsigned HOST_WIDE_INT min_inner_iters,
+ class loop *outer,
+ gphi **out_red_phi,
+ machine_mode *out_acc_mode,
+ machine_mode *out_op_mode)
+{
+ if (loop->num_nodes != 2 || !empty_block_p (loop->latch))
+ return false;
+ edge exit = single_exit (loop);
+ if (!exit)
+ return false;
+
+ /* Header is the body; loads only. */
+ bool has_vuse = false;
+ for (gimple_stmt_iterator gsi = gsi_start_bb (loop->header);
+ !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ gimple *s = gsi_stmt (gsi);
+ if (gimple_vdef (s))
+ return false;
+ if (gimple_vuse (s))
+ has_vuse = true;
+ }
+ if (!has_vuse)
+ return false;
+
+ /* Single reduction PHI with init defined outside the loop. */
+ gphi *red_phi = inner_reduction_phi (loop);
+ if (!red_phi)
+ return false;
+ tree init = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_preheader_edge (loop));
+ if (!expr_invariant_in_loop_p (loop, init))
+ return false;
+
+ /* Trip count is a known integer >= MIN_INNER_ITERS. niter.niter is
+ the latch count (trip - 1); reject HOST_WIDE_INT_M1U separately
+ so the +1 conversion below does not wrap. */
+ tree_niter_desc niter;
+ if (!number_of_iterations_exit (loop, exit, &niter, false, true)
+ || niter.cmp == ERROR_MARK
+ || !integer_zerop (niter.may_be_zero)
+ || !tree_fits_uhwi_p (niter.niter))
+ return false;
+
+ unsigned HOST_WIDE_INT latch_count = tree_to_uhwi (niter.niter);
+ if (latch_count == HOST_WIDE_INT_M1U
+ || latch_count + 1 < min_inner_iters)
+ return false;
+
+ /* Plus a unit-step counter IV in the same header. When OUTER is
+ given, every IV's step and base must also be OUTER-invariant. */
+ bool found_unit_iv = false;
+ for (gphi_iterator psi = gsi_start_phis (loop->header);
+ !gsi_end_p (psi); gsi_next (&psi))
+ {
+ gphi *phi = psi.phi ();
+ tree op = gimple_phi_result (phi);
+ if (virtual_operand_p (op))
+ continue;
+ affine_iv iv;
+ if (!simple_iv (loop, loop, op, &iv, true))
+ continue;
+ if (outer
+ && (!expr_invariant_in_loop_p (outer, iv.step)
+ || !expr_invariant_in_loop_p (outer, iv.base)))
+ return false;
+ if (iv.step && integer_onep (iv.step))
+ found_unit_iv = true;
+ }
+ if (!found_unit_iv)
+ return false;
+
+ /* Widening multiply somewhere in the reduction update chain. */
+ machine_mode acc_mode = TYPE_MODE (TREE_TYPE (gimple_phi_result (red_phi)));
+ machine_mode op_mode = narrowest_reduc_input_mode (red_phi, loop, acc_mode);
+ if (op_mode == acc_mode)
+ return false;
+
+ if (out_red_phi)
+ *out_red_phi = red_phi;
+ if (out_acc_mode)
+ *out_acc_mode = acc_mode;
+ if (out_op_mode)
+ *out_op_mode = op_mode;
+ return true;
+}
+
+/* Detect whether the OUTER / INNER nest has the parallel-reduction
+ shape that unroll_jam_reduction_fuse_sibling can transform. The
+ rejection messages below name each requirement. The only outer-body
+ VDEF must be the post-reduction store, so that after strip-mining
+ every replicated store sits downstream of every sibling's preheader.
+ And the outer trip count must be a multiple of N, which keeps us off
+ tree_unroll_loop's remainder path. */
+
+static bool
+unroll_jam_reduction_possible_p (class loop *outer, class loop *inner,
+ unsigned HOST_WIDE_INT *out_outer_trip,
+ unsigned *out_target_factor)
+{
+ auto reject = [&] (const char *reason) -> bool
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "reduction-jam: rejecting outer loop %d: %s\n",
+ outer->num, reason);
+ return false;
+ };
+
+ /* Loop-tree relationships. The driver iterates LI_ONLY_INNERMOST,
+ so INNER has no children; check only that OUTER has a single
+ inner sibling (INNER) and that INNER's parent is OUTER. */
+ if (!inner || inner != outer->inner || inner->next
+ || loop_outer (inner) != outer)
+ return reject ("not a perfect 2-level nest");
+
+ if (!dominated_by_p (CDI_DOMINATORS, outer->latch, inner->header))
+ return reject ("inner has unsuitable dominance");
+
+ /* Outer loop must carry only simple IVs. */
+ for (gphi_iterator psi = gsi_start_phis (outer->header);
+ !gsi_end_p (psi); gsi_next (&psi))
+ {
+ gphi *phi = psi.phi ();
+ tree op = gimple_phi_result (phi);
+ if (virtual_operand_p (op))
+ continue;
+ affine_iv iv;
+ if (!simple_iv (outer, outer, op, &iv, true))
+ return reject ("outer carries a non-simple-IV PHI");
+ }
+
+ /* Loop-local dot-product checks; on success red_phi and the modes
+ feed the target-hook gate below. */
+ gphi *red_phi = NULL;
+ machine_mode acc_mode = VOIDmode;
+ machine_mode op_mode = VOIDmode;
+ if (!loop_is_dot_product_candidate_p
+ (inner,
+ (unsigned HOST_WIDE_INT) param_unroll_jam_reduction_min_inner_iters,
+ outer, &red_phi, &acc_mode, &op_mode))
+ return reject ("inner not a dot-product candidate");
+
+ /* Opt-in gate: ask the target whether the (accumulator, operand)
+ mode pair benefits from N parallel reduction streams. Targets
+ that have not measured a benefit return 0, in which case we skip
+ the transform regardless of -floop-unroll-and-jam-reductions.
+ A return of 1 is also opt-out (single-stream = no parallelism). */
+ unsigned target_factor
+ = targetm.loop_jam_reduction_factor (acc_mode, op_mode);
+ if (target_factor < 2)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "reduction-jam: target opted out for acc=%s op=%s\n",
+ GET_MODE_NAME (acc_mode), GET_MODE_NAME (op_mode));
+ return false;
+ }
+ *out_target_factor = target_factor;
+
+ /* The fuser drains stmts from inner->header alone, so a non-empty
+ latch would be silently mishandled; loop_is_dot_product_candidate_p
+ already rejects one. */
+ gcc_checking_assert (empty_block_p (inner->latch));
+
+ /* Require the chain from red_phi back to itself via the latch arg to
+ be a PLUS_EXPR reduction. This limits the transform to plain
+ summation; the N streams are independent, so each is still summed
+ in its original order and no reassociation takes place. */
+ tree latch_arg = PHI_ARG_DEF_FROM_EDGE (red_phi, loop_latch_edge (inner));
+ if (!check_reduction_path (dump_user_location_t (), inner, red_phi,
+ latch_arg, PLUS_EXPR))
+ return reject ("inner reduction chain not associative-PLUS");
+
+ /* loop_is_dot_product_candidate_p confirmed inner has a single exit. */
+ edge inner_exit = single_exit (inner);
+
+ /* The outer body may carry at most one VDEF, only in the inner
+ exit's destination BB (the post-reduction store). Also reject any
+ non-VDEF side effect (volatile load, asm volatile, side-effecting
+ builtin), matching bb_prevents_fusion_p on the IV-only path.
+
+ The store BB must be dominated by inner->header so that after
+ tree_unroll_loop strip-mines outer by N every replicated store
+ sits downstream of its corresponding inner copy, and therefore
+ downstream of every sibling's preheader. The fuser relies on
+ this when rewiring VUSEs. */
+ basic_block store_bb = inner_exit->dest;
+ if (!dominated_by_p (CDI_DOMINATORS, store_bb, inner->header))
+ return reject ("outer-body store not downstream of inner header");
+
+ /* The outer body must be straight-line apart from the exit test. A
+ control-flow merge there leaves a PHI that would not dominate the
+ fused preheader after strip-mining, and the migrator leaves body
+ PHIs in place, so the SSA fails verification. */
+ edge outer_exit = single_exit (outer);
+ if (!outer_exit)
+ return reject ("outer loop has no single exit");
+
+ basic_block *bbs = get_loop_body (outer);
+ const char *fail = NULL;
+ for (unsigned i = 0; i < outer->num_nodes && !fail; ++i)
+ {
+ basic_block bb = bbs[i];
+ if (flow_bb_inside_loop_p (inner, bb))
+ continue;
+ if (EDGE_COUNT (bb->succs) > 1 && bb != outer_exit->src)
+ {
+ fail = "outer body has control flow";
+ break;
+ }
+ unsigned vdef_count = 0;
+ for (gimple_stmt_iterator gsi = gsi_start_bb (bb);
+ !gsi_end_p (gsi); gsi_next (&gsi))
+ {
+ gimple *s = gsi_stmt (gsi);
+ if (gimple_has_side_effects (s))
+ {
+ fail = "outer body has side effects";
+ break;
+ }
+ if (gimple_vdef (s))
+ vdef_count++;
+ }
+ if (!fail && (bb == store_bb ? vdef_count > 1 : vdef_count > 0))
+ fail = "outer body has extra stores";
+ }
+ free (bbs);
+ if (fail)
+ return reject (fail);
+
+ /* Outer trip count must be a known constant (N is later narrowed to
+ a divisor of it; the remainder codepath of tree_unroll_loop is
+ correct in principle but currently untested for this pattern).
+ niter from number_of_iterations_exit is the latch execution count,
+ so the trip count is niter + 1. */
+ tree_niter_desc outer_niter;
+ if (!number_of_iterations_exit (outer, outer_exit, &outer_niter,
+ false, true)
+ || outer_niter.cmp == ERROR_MARK
+ || !integer_zerop (outer_niter.may_be_zero)
+ || !tree_fits_uhwi_p (outer_niter.niter))
+ return reject ("outer trip count unknown");
+
+ unsigned HOST_WIDE_INT outer_latch_count = tree_to_uhwi (outer_niter.niter);
+ if (outer_latch_count == HOST_WIDE_INT_M1U)
+ return reject ("outer trip count overflows HOST_WIDE_INT");
+
+ *out_outer_trip = outer_latch_count + 1;
+
+ return true;
+}
+
+/* Move the transitive SSA-use closure of SRC_INIT, and of every
+ operand SRC's header body references, into DST_PREHEADER so the
+ defs stay live once SRC's body moves. Defs that already dominate
+ DST_PREHEADER, and defs inside SRC, are left to the body move.
+
+ RPO is a reverse-postorder over CFUN's BBs, computed once by the
+ caller and shared across the nest's siblings; walking it keeps
+ def-use order among the migrated stmts intact. DST_LIVE_VUSE is
+ the vop live at DST_PREHEADER, and migrated stmts are rewired to it
+ so none of them reference a vop defined downstream. */
+
+static void
+migrate_loop_invariant_defs_to_preheader (class loop *src, tree src_init,
+ basic_block dst_preheader,
+ const auto_vec<int> &rpo,
+ tree dst_live_vuse)
+{
+ hash_set<gimple *> needed;
+ auto_vec<gimple *, 16> worklist;
+
+ auto consider = [&] (tree op)
+ {
+ if (TREE_CODE (op) != SSA_NAME)
+ return;
+ gimple *def = SSA_NAME_DEF_STMT (op);
+ if (!def || is_a <gphi *> (def))
+ return;
+ basic_block def_bb = gimple_bb (def);
+ if (!def_bb
+ || dominated_by_p (CDI_DOMINATORS, dst_preheader, def_bb)
+ || flow_bb_inside_loop_p (src, def_bb))
+ return;
+ if (needed.add (def))
+ return;
+ worklist.safe_push (def);
+ };
+
+ auto consider_uses = [&] (gimple *s)
+ {
+ use_operand_p use_p;
+ ssa_op_iter oi;
+ FOR_EACH_SSA_USE_OPERAND (use_p, s, oi, SSA_OP_USE)
+ consider (USE_FROM_PTR (use_p));
+ };
+
+ consider (src_init);
+ for (gimple_stmt_iterator si = gsi_start_bb (src->header);
+ !gsi_end_p (si); gsi_next (&si))
+ {
+ gimple *s = gsi_stmt (si);
+ if (!is_a <gphi *> (s) && !is_ctrl_stmt (s))
+ consider_uses (s);
+ }
+ while (!worklist.is_empty ())
+ consider_uses (worklist.pop ());
+
+ if (needed.is_empty ())
+ return;
+
+ /* Walk BBs in reverse-postorder so that, for any def-use edge among
+ the migrated stmts, the def is moved into DST_PREHEADER ahead of
+ the use. */
+ int n = rpo.length ();
+ for (int i = 0; i < n; ++i)
+ {
+ basic_block bb = BASIC_BLOCK_FOR_FN (cfun, rpo[i]);
+ if (bb == dst_preheader
+ || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
+ continue;
+ for (gimple_stmt_iterator si = gsi_start_bb (bb);
+ !gsi_end_p (si);)
+ {
+ gimple *stmt = gsi_stmt (si);
+ if (!needed.contains (stmt))
+ {
+ gsi_next (&si);
+ continue;
+ }
+ if (dst_live_vuse && gimple_vuse (stmt))
+ {
+ gimple_set_vuse (stmt, dst_live_vuse);
+ update_stmt (stmt);
+ }
+ gsi_move_to_bb_end_before_terminator (&si, dst_preheader);
+ }
+ }
+}
+
+/* Pair SRC's non-reduction header PHIs with DST's by position,
+ recording each pair in PHI_RENAME. Both are inner siblings of the
+ same strip-mined outer loop, so their header PHI lists start with
+ the duplicated originals in the same order. DST may have picked up
+ extra reduction PHIs at the tail from earlier sibling fuses, so
+ walk SRC to completion against DST's prefix. SRC_REDUC_PHI is
+ renamed separately and virtual PHIs are skipped. */
+
+static void
+match_sibling_header_phis (class loop *src, class loop *dst,
+ gphi *src_reduc_phi,
+ hash_map<tree, tree> &phi_rename)
+{
+ gphi_iterator spsi = gsi_start_phis (src->header);
+ gphi_iterator dpsi = gsi_start_phis (dst->header);
+ for (; !gsi_end_p (spsi); gsi_next (&spsi), gsi_next (&dpsi))
+ {
+ gcc_checking_assert (!gsi_end_p (dpsi));
+ gphi *src_phi = spsi.phi ();
+ gphi *dst_phi = dpsi.phi ();
+ tree src_res = gimple_phi_result (src_phi);
+ tree dst_res = gimple_phi_result (dst_phi);
+ gcc_checking_assert (virtual_operand_p (src_res)
+ == virtual_operand_p (dst_res));
+ if (src_phi != src_reduc_phi && !virtual_operand_p (src_res))
+ phi_rename.put (src_res, dst_res);
+ }
+}
+
+/* Fuse the SRC sibling inner loop into DST, keeping SRC's reduction
+ PHI as a parallel chain in DST's header. Both are inner siblings
+ of an outer loop that passed unroll_jam_reduction_possible_p and
+ was then strip-mined by tree_unroll_loop.
+
+ The caller has already flattened LC-SSA across the nest (Phase 1a).
+ Here SRC's reduction init and body operands move up to DST's
+ preheader (Phase 1b), a fresh reduction PHI for SRC's stream goes
+ into DST's header (Phase 2), and SRC's body follows with its PHI
+ uses renamed (Phase 3). The caller then bypasses the drained SRC
+ (Phase 4); its cleanup_tree_cfg folds the leftover chain of exit
+ BBs holding the post-reduction stores, and
+ rewrite_into_loop_closed_ssa puts LC-SSA back. */
+
+static void
+unroll_jam_reduction_fuse_sibling (class loop *src, class loop *dst,
+ const auto_vec<int> &rpo)
+{
+ gphi *src_reduc_phi = inner_reduction_phi (src);
+ tree src_reduc_res = gimple_phi_result (src_reduc_phi);
+ tree src_init = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
+ loop_preheader_edge (src));
+ tree src_update = PHI_ARG_DEF_FROM_EDGE (src_reduc_phi,
+ loop_latch_edge (src));
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "reduction-jam: fusing src loop %d into dst %d\n",
+ src->num, dst->num);
+
+ basic_block dst_preheader = loop_preheader_edge (dst)->src;
+
+ /* The migrated stmts must reference the vop live at DST's preheader.
+ The matcher's only outer-body VDEF is the post-reduction store,
+ which sits below DST after strip-mining, so nothing redefines the
+ vop before DST: it is outer->header's virtual PHI result. A
+ load-only outer has no such PHI; its loads already share one
+ loop-invariant vop, so leave their VUSE alone (NULL_TREE). */
+ gphi *outer_vphi = get_virtual_phi (loop_outer (dst)->header);
+ tree dst_live_vuse = outer_vphi ? gimple_phi_result (outer_vphi) : NULL_TREE;
+
+ /* Phase 1b. */
+ migrate_loop_invariant_defs_to_preheader (src, src_init, dst_preheader,
+ rpo, dst_live_vuse);
+
+ /* Phase 2: a new reduction PHI in DST's header for SRC's stream. */
+ tree new_name = make_ssa_name (TREE_TYPE (src_reduc_res));
+ gphi *new_phi = create_phi_node (new_name, dst->header);
+ add_phi_arg (new_phi, src_init, loop_preheader_edge (dst),
+ UNKNOWN_LOCATION);
+ add_phi_arg (new_phi, src_update, loop_latch_edge (dst),
+ UNKNOWN_LOCATION);
+
+ /* Phase 3: rewrite uses, then move SRC's body into DST's header. */
+ hash_map<tree, tree> phi_rename;
+ phi_rename.put (src_reduc_res, new_name);
+ match_sibling_header_phis (src, dst, src_reduc_phi, phi_rename);
+
+ auto_vec<gimple *, 32> src_body;
+ for (gimple_stmt_iterator si = gsi_start_bb (src->header);
+ !gsi_end_p (si); gsi_next (&si))
+ {
+ gimple *s = gsi_stmt (si);
+ if (is_a <gphi *> (s) || is_ctrl_stmt (s))
+ continue;
+ src_body.safe_push (s);
+ }
+ for (gimple *stmt : src_body)
+ {
+ use_operand_p use_p;
+ ssa_op_iter oi;
+ FOR_EACH_SSA_USE_OPERAND (use_p, stmt, oi, SSA_OP_USE)
+ {
+ tree *map = phi_rename.get (USE_FROM_PTR (use_p));
+ if (map)
+ SET_USE (use_p, *map);
+ }
+ if (dst_live_vuse && gimple_vuse (stmt))
+ {
+ gimple_set_vuse (stmt, dst_live_vuse);
+ update_stmt (stmt);
+ }
+ gimple_stmt_iterator from = gsi_for_stmt (stmt);
+ gsi_move_to_bb_end_before_terminator (&from, dst->header);
+ }
+
+ /* The body walk above only covered uses of src_reduc_res inside
+ src->header; catch any remaining uses (LC-SSA closures reached
+ by the post-reduction store) before the PHI is removed. */
+ replace_uses_by (src_reduc_res, new_name);
+
+ gphi_iterator psi_src_reduc = gsi_for_phi (src_reduc_phi);
+ remove_phi_node (&psi_src_reduc, false);
+}
+
+/* Phase 4: redirect SRC's preheader edge straight past the now-empty
+ SRC loop so cleanup_tree_cfg can remove it. SRC must have been
+ drained by unroll_jam_reduction_fuse_sibling first. */
+
+static void
+unroll_jam_reduction_bypass_sibling (class loop *src)
+{
+ edge src_preheader_e = loop_preheader_edge (src);
+ edge src_exit_e = single_exit (src);
+ if (!src_exit_e)
+ return;
+ basic_block src_exit_dest = src_exit_e->dest;
+
+ /* src_exit_dest is a single-pred BB after strip-mining (the next
+ sibling's preheader, or outer's latch for the last sibling), so
+ its LC-SSA PHIs are single-arg. Flatten them before the redirect
+ so no arg dangles once SRC's BBs are deleted: the driver rebuilds
+ scalar LC-SSA afterward but not the virtual side, so a stale vop
+ arg would otherwise survive. */
+ for (gphi_iterator psi = gsi_start_phis (src_exit_dest);
+ !gsi_end_p (psi);)
+ collapse_single_arg_phi (&psi);
+
+ edge new_edge = redirect_edge_and_branch (src_preheader_e, src_exit_dest);
+ gcc_assert (new_edge);
+
+ /* Delete SRC's now-unreachable BBs and the empty loop entry,
+ mirroring fuse_loops's delete_basic_block + delete_loop pattern
+ on the IV-only path. Without this the loop tree carries stale
+ entries until the driver's cleanup_tree_cfg picks them up. */
+ basic_block *bbs = get_loop_body (src);
+ unsigned n = src->num_nodes;
+ for (unsigned i = 0; i < n; i++)
+ delete_basic_block (bbs[i]);
+ free (bbs);
+ delete_loop (src);
+}
+
/* Return true if any of the access functions for dataref A
isn't invariant with respect to loop LOOP_NEST. */
static bool
@@ -497,6 +1177,221 @@ adjust_unroll_factor (class loop *inner, struct
data_dependence_relation *ddr,
return ret;
}
+/* Walk OUTER's data dependences, running each through
+ adjust_unroll_factor. *UNROLL_FACTOR receives the safety upper
+ bound (UINT_MAX for no constraint, 0 for cannot unroll),
+ *PROFIT_UNROLL the largest distance+1 yielding a removable ref
+ under the (a,0) pattern, *REMOVED the count of removable refs and
+ *NUM_DATAREFS the total. Returns false if the dependences cannot
+ be computed.
+
+ When *UNROLL_FACTOR comes back 0 the other out-params are only
+ partially populated and must not be read for profitability. */
+
+static bool
+compute_jam_dependence_factors (class loop *outer, class loop *inner,
+ unsigned *unroll_factor,
+ unsigned *profit_unroll,
+ unsigned *removed,
+ unsigned *num_datarefs)
+{
+ vec<data_reference_p> datarefs = vNULL;
+ vec<ddr_p> dependences = vNULL;
+ auto_vec<loop_p, 3> loop_nest;
+
+ if (!compute_data_dependences_for_loop (outer, true, &loop_nest,
+ &datarefs, &dependences))
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "Cannot analyze data dependencies\n");
+ free_data_refs (datarefs);
+ free_dependence_relations (dependences);
+ return false;
+ }
+
+ *num_datarefs = datarefs.length ();
+ *unroll_factor = (unsigned) -1;
+ *profit_unroll = 1;
+ *removed = 0;
+
+ if (*num_datarefs)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ dump_data_dependence_relations (dump_file, dependences);
+
+ unsigned i;
+ ddr_p ddr;
+ FOR_EACH_VEC_ELT (dependences, i, ddr)
+ {
+ if (DDR_ARE_DEPENDENT (ddr) == chrec_known)
+ continue;
+ data_reference *dra = DDR_A (ddr);
+ data_reference *drb = DDR_B (ddr);
+ /* Self-dependence: WAW with non-affine access could reorder
+ writes to the same location. */
+ if (dra == drb)
+ {
+ if (DR_IS_WRITE (dra)
+ && !DR_ACCESS_FNS (dra).is_empty ()
+ && DDR_ARE_DEPENDENT (ddr) == chrec_dont_know)
+ {
+ *unroll_factor = 0;
+ break;
+ }
+ continue;
+ }
+ /* Couldn't determine the distance vector: harmless for two
+ reads, but a write means we cannot prove safety. */
+ if (!adjust_unroll_factor (inner, ddr, unroll_factor,
+ profit_unroll, removed))
+ if (DR_IS_WRITE (dra) || DR_IS_WRITE (drb))
+ {
+ *unroll_factor = 0;
+ break;
+ }
+ }
+ }
+
+ free_dependence_relations (dependences);
+ free_data_refs (datarefs);
+ return true;
+}
+
+/* Pick a strip factor for the reduction-jam path on OUTER / INNER.
+ adjust_unroll_factor gives the data-dep safety bound, TARGET_FACTOR
+ the target's preference and param_unroll_jam_reduction_streams
+ the cap. We do not use adjust_unroll_factor's profit model here:
+ it would cap N at 1 for GEMV, because the input-load DDR has
+ dist_v=(0,0). The result is narrowed to a divisor of OUTER_TRIP,
+ which keeps us off tree_unroll_loop's remainder path, and validated
+ via can_unroll_loop_p. *OUT_FACTOR receives the chosen factor and
+ *OUT_DESC the niter. */
+
+static bool
+select_reduction_jam_factor (class loop *outer, class loop *inner,
+ unsigned HOST_WIDE_INT outer_trip,
+ unsigned target_factor,
+ unsigned *out_factor,
+ tree_niter_desc *out_desc)
+{
+ /* Only the safety bound matters here; the ref-removal outputs go
+ unused. */
+ unsigned unroll_factor, profit_unroll, removed, num_datarefs;
+ if (!compute_jam_dependence_factors (outer, inner, &unroll_factor,
+ &profit_unroll, &removed,
+ &num_datarefs))
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "reduction-jam: data-dep analysis failed\n");
+ return false;
+ }
+ if (unroll_factor < 2)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "reduction-jam: data-dep safety bound below 2\n");
+ return false;
+ }
+
+ unsigned N = MIN (target_factor,
+ (unsigned) param_unroll_jam_reduction_streams);
+ if (N > unroll_factor)
+ N = unroll_factor;
+ unsigned initial_N = N;
+ while (N >= 2 && (outer_trip < N || outer_trip % N != 0))
+ --N;
+ if (N < 2)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "reduction-jam: outer trip " HOST_WIDE_INT_PRINT_UNSIGNED
+ " has no divisor in [2, %u]\n", outer_trip, initial_N);
+ return false;
+ }
+
+ if (!can_unroll_loop_p (outer, N, out_desc))
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "reduction-jam: cannot unroll outer by %u\n", N);
+ return false;
+ }
+
+ *out_factor = N;
+ return true;
+}
+
+/* Strip-mine OUTER by N and fuse the resulting N inner siblings into
+ one body that carries N parallel reduction PHIs. Caller has
+ already validated OUTER and chosen N + DESC via
+ unroll_jam_reduction_possible_p / select_reduction_jam_factor. */
+
+static void
+do_unroll_jam_reductions (class loop *outer, unsigned N,
+ tree_niter_desc *desc)
+{
+ initialize_original_copy_tables ();
+ tree_unroll_loop (outer, N, desc);
+ free_original_copy_tables ();
+
+ /* dst->next chains siblings in CFG order (per tree_unroll_loop);
+ the fuser relies on that when migrating SSA uses. */
+ class loop *dst = outer->inner;
+ auto_vec<class loop *, 4> siblings;
+ for (class loop *src = dst->next; src; src = src->next)
+ siblings.safe_push (src);
+
+ /* RPO of CFUN, shared across all siblings: the fuser only moves
+ stmts and removes single-arg PHIs, neither of which alters BB
+ structure, so one walk suffices for the whole nest. */
+ auto_vec<int> rpo;
+ rpo.safe_grow (last_basic_block_for_fn (cfun));
+ rpo.truncate (pre_and_rev_post_order_compute_fn (cfun, NULL, rpo.address (),
+ false));
+
+ /* Phase 1a (nest-wide): collapse single-arg LC-SSA PHIs that the
+ per-sibling migrate-walk in Phase 1b would otherwise have to
+ chase through (it skips PHIs). Sweep the BBs that do not dominate
+ DST's preheader, i.e. the nest and everything downstream where the
+ sibling-exit closures live. One sweep suffices: later fuses only
+ add two-arg reduction PHIs and make no CFG change that could
+ reintroduce single-arg closures. */
+ basic_block dst_preheader = loop_preheader_edge (dst)->src;
+ calculate_dominance_info (CDI_DOMINATORS);
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ if (bb == dst_preheader
+ || dominated_by_p (CDI_DOMINATORS, dst_preheader, bb))
+ continue;
+ for (gphi_iterator psi = gsi_start_phis (bb); !gsi_end_p (psi);)
+ {
+ gphi *phi = psi.phi ();
+ if (gimple_phi_num_args (phi) == 1
+ && !virtual_operand_p (gimple_phi_result (phi)))
+ collapse_single_arg_phi (&psi);
+ else
+ gsi_next (&psi);
+ }
+ }
+
+ for (class loop *src : siblings)
+ unroll_jam_reduction_fuse_sibling (src, dst, rpo);
+
+ /* Bypass the drained siblings only after every fuse: the CFG edits
+ invalidate dominance / SCEV state the fuse loop above relies on. */
+ for (class loop *src : siblings)
+ unroll_jam_reduction_bypass_sibling (src);
+
+ if (dump_enabled_p ())
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS | TDF_DETAILS,
+ find_loop_location (outer),
+ "applying reduction unroll-and-jam with %u streams\n",
+ siblings.length () + 1);
+}
+
+
/* Main entry point for the unroll-and-jam transformation
described above. */
@@ -516,8 +1411,36 @@ tree_loop_unroll_and_jam (void)
|| optimize_loop_nest_for_size_p (outer))
continue;
- if (!unroll_jam_possible_p (outer, loop))
- continue;
+ if (!flag_unroll_jam || !unroll_jam_possible_p (outer, loop))
+ {
+ /* The IV-only path skips reduction-shaped nests (reduction PHIs,
+ or outer-body stores consuming the reduction result). Try the
+ reduction-jam extension on those. */
+ unsigned HOST_WIDE_INT outer_trip;
+ unsigned target_factor, N;
+ tree_niter_desc red_desc;
+ if (flag_unroll_jam_reductions
+ && unroll_jam_reduction_possible_p (outer, loop,
+ &outer_trip,
+ &target_factor)
+ && select_reduction_jam_factor (outer, loop, outer_trip,
+ target_factor,
+ &N, &red_desc))
+ {
+ do_unroll_jam_reductions (outer, N, &red_desc);
+ free_dominance_info (CDI_DOMINATORS);
+ cleanup_tree_cfg ();
+ /* The fuser maintains virtual SSA from the matcher's
+ invariants, but moving SRC's body into DST leaves scalar
+ SSA names defined in DST yet used outside it (e.g. new_name's
+ LC-SSA closures). Rebuild scalar LC-SSA so the next nest's
+ tree_unroll_loop is well-formed, and verify the result. */
+ rewrite_into_loop_closed_ssa (NULL, 0);
+ checking_verify_ssa (true, true);
+ scev_reset ();
+ }
+ continue;
+ }
vec<data_reference_p> datarefs = vNULL;
vec<ddr_p> dependences = vNULL;
@@ -674,7 +1597,10 @@ public:
{}
/* opt_pass methods: */
- bool gate (function *) final override { return flag_unroll_jam != 0; }
+ bool gate (function *) final override
+ {
+ return flag_unroll_jam != 0 || flag_unroll_jam_reductions;
+ }
unsigned int execute (function *) final override;
};
diff --git a/gcc/hooks.cc b/gcc/hooks.cc
index 2b47fd3e50f8..976bae3c89a9 100644
--- a/gcc/hooks.cc
+++ b/gcc/hooks.cc
@@ -424,6 +424,13 @@ hook_uint_mode_0 (machine_mode)
return 0;
}
+/* Generic hook that takes two machine modes and returns an unsigned int 0. */
+unsigned int
+hook_uint_mode_mode_0 (machine_mode, machine_mode)
+{
+ return 0;
+}
+
/* Generic hook that takes no arguments and returns a NULL const string. */
const char *
hook_constcharptr_void_null (void)
diff --git a/gcc/hooks.h b/gcc/hooks.h
index 1ad4abb5500c..0ec0b32fa1a5 100644
--- a/gcc/hooks.h
+++ b/gcc/hooks.h
@@ -119,6 +119,7 @@ extern tree hook_tree_treeptr_tree_tree_int_boolptr_null
(tree *, tree, tree, in
extern unsigned hook_uint_void_0 (void);
extern unsigned int hook_uint_mode_0 (machine_mode);
+extern unsigned int hook_uint_mode_mode_0 (machine_mode, machine_mode);
extern bool default_can_output_mi_thunk_no_vcall (const_tree, HOST_WIDE_INT,
HOST_WIDE_INT, const_tree);
diff --git a/gcc/params.opt b/gcc/params.opt
index 0c6a3e514574..e99c4fc4173f 100644
--- a/gcc/params.opt
+++ b/gcc/params.opt
@@ -1263,6 +1263,14 @@ Maximum unroll factor for the unroll-and-jam
transformation.
Common Joined UInteger Var(param_unroll_jam_min_percent) Init(1)
IntegerRange(0, 100) Param Optimization
Minimum percentage of memrefs that must go away for unroll-and-jam to be
considered profitable.
+-param=unroll-jam-reduction-streams=
+Common Joined UInteger Var(param_unroll_jam_reduction_streams) Init(4)
IntegerRange(2, 16) Param Optimization
+Target number of parallel reduction streams produced by reduction-aware
unroll-and-jam.
+
+-param=unroll-jam-reduction-min-inner-iters=
+Common Joined UInteger Var(param_unroll_jam_reduction_min_inner_iters)
Init(32) IntegerRange(2, 65536) Param Optimization
+Minimum inner-loop trip count required for reduction-aware unroll-and-jam to
fire.
+
-param=use-after-scope-direct-emission-threshold=
Common Joined UInteger Var(param_use_after_scope_direct_emission_threshold)
Init(256) Param Optimization
Use direct poisoning/unpoisoning instructions for variables smaller or equal
to this number.
diff --git a/gcc/target.def b/gcc/target.def
index 66c20a9c9d2a..118aa9bb82ec 100644
--- a/gcc/target.def
+++ b/gcc/target.def
@@ -2873,6 +2873,19 @@ number of memory accesses.",
unsigned, (unsigned nunroll, class loop *loop),
NULL)
+DEFHOOK
+(loop_jam_reduction_factor,
+ "Return the preferred number of parallel reduction streams\n\
+@option{-floop-unroll-and-jam-reductions} should produce for an inner\n\
+reduction that accumulates into mode @var{acc_mode} from operands of\n\
+mode @var{op_mode}. @var{op_mode} is the operand mode at the source\n\
+of the widening multiply, and is always strictly narrower than\n\
+@var{acc_mode}: plain reductions with no widening multiply are\n\
+filtered out before the hook is consulted. Returning less than two\n\
+disables the transform for that mode pair.",
+ unsigned, (machine_mode acc_mode, machine_mode op_mode),
+ hook_uint_mode_mode_0)
+
/* True if X is a legitimate MODE-mode immediate operand. */
DEFHOOK
(legitimate_constant_p,
diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
new file mode 100644
index 000000000000..d14711034bc0
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-1.c
@@ -0,0 +1,78 @@
+/* Positive tests: the pass must strip-mine and fuse each nest below.
+ The outer trip count of 32 is divisible by the default factor of 4.
+ Both arms of the aarch64 hook are covered: QImode / HImode -> SImode
+ (DOT_PROD class) in affine, and SImode -> DImode (SMLAL_DI) in
+ affine64. */
+
+/* { dg-do compile } */
+/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
-fdump-tree-unrolljam-details" } */
+
+#include <stdint.h>
+
+#define IN_DIM 512
+#define OUT_DIM 32
+
+void
+affine (int32_t *__restrict out,
+ const int8_t *weights,
+ const int32_t *biases,
+ const uint8_t *input)
+{
+ for (int i = 0; i < OUT_DIM; ++i)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < IN_DIM; ++j)
+ s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
+ out[i] = s;
+ }
+}
+
+void
+affine64 (int64_t *__restrict out,
+ const int32_t *weights,
+ const int64_t *biases,
+ const int32_t *input)
+{
+ for (int i = 0; i < OUT_DIM; ++i)
+ {
+ int64_t s = biases[i];
+ for (int j = 0; j < IN_DIM; ++j)
+ s += (int64_t) weights[i * IN_DIM + j] * (int64_t) input[j];
+ out[i] = s;
+ }
+}
+
+/* Inner trip count at the min-inner-iters default of 32. */
+void
+affine_min_inner (int32_t *__restrict out,
+ const int8_t *weights,
+ const int32_t *biases,
+ const uint8_t *input)
+{
+ for (int i = 0; i < 32; ++i)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < 32; ++j)
+ s += (int32_t) weights[i * 32 + j] * (int32_t) input[j];
+ out[i] = s;
+ }
+}
+
+/* Outer trip count of 9 narrows N from 4 to 3. */
+void
+affine_narrowed (int32_t *__restrict out,
+ const int8_t *weights,
+ const int32_t *biases,
+ const uint8_t *input)
+{
+ for (int i = 0; i < 9; ++i)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < 512; ++j)
+ s += (int32_t) weights[i * 512 + j] * (int32_t) input[j];
+ out[i] = s;
+ }
+}
+
+/* { dg-final { scan-tree-dump-times "applying reduction unroll-and-jam" 4
"unrolljam" } } */
+/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 11
"unrolljam" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
new file mode 100644
index 000000000000..44301348a659
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-2.c
@@ -0,0 +1,89 @@
+/* Correctness test for the reduction-aware extension of unroll-and-jam
+ on the GEMV pattern. Compares against an in-main reference
+ computation. */
+
+/* { dg-do run } */
+/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */
+
+#include <stdint.h>
+
+#define IN_DIM 512
+#define OUT_DIM 32
+
+extern void abort (void);
+
+void __attribute__((noipa))
+affine (int32_t *__restrict out,
+ const int8_t *weights,
+ const int32_t *biases,
+ const uint8_t *input)
+{
+ for (int i = 0; i < OUT_DIM; ++i)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < IN_DIM; ++j)
+ s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
+ out[i] = s;
+ }
+}
+
+void __attribute__((noipa))
+affine_zero_init (int32_t *__restrict out,
+ const int8_t *a,
+ const int8_t *b)
+{
+ for (int i = 0; i < OUT_DIM; ++i)
+ {
+ int32_t s = 0;
+ for (int j = 0; j < IN_DIM; ++j)
+ s += a[i * IN_DIM + j] * b[j];
+ out[i] = s;
+ }
+}
+
+int
+main (void)
+{
+ static int8_t weights[OUT_DIM * IN_DIM];
+ static uint8_t input[IN_DIM];
+ static int32_t biases[OUT_DIM];
+ static int32_t out[OUT_DIM];
+ static int32_t expected[OUT_DIM];
+
+ for (int k = 0; k < OUT_DIM * IN_DIM; k++)
+ weights[k] = (k * 13) & 0x7f;
+ for (int k = 0; k < IN_DIM; k++)
+ input[k] = (k * 7) & 0xff;
+ for (int k = 0; k < OUT_DIM; k++)
+ biases[k] = k * 1000;
+
+ for (int i = 0; i < OUT_DIM; i++)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < IN_DIM; j++)
+ s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
+ expected[i] = s;
+ }
+
+ affine (out, weights, biases, input);
+
+ for (int i = 0; i < OUT_DIM; i++)
+ if (out[i] != expected[i])
+ abort ();
+
+ for (int i = 0; i < OUT_DIM; i++)
+ {
+ int32_t s = 0;
+ for (int j = 0; j < IN_DIM; j++)
+ s += weights[i * IN_DIM + j] * weights[j];
+ expected[i] = s;
+ }
+
+ affine_zero_init (out, weights, weights);
+
+ for (int i = 0; i < OUT_DIM; i++)
+ if (out[i] != expected[i])
+ abort ();
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
new file mode 100644
index 000000000000..6e4b590fefeb
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-3.c
@@ -0,0 +1,189 @@
+/* Negative tests: each function below hits one of
+ unroll_jam_reduction_possible_p's rejection clauses and the pass
+ must not fire on any of them. */
+
+/* { dg-do compile } */
+/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions
-fdump-tree-unrolljam-details" } */
+
+#include <stdint.h>
+
+/* Outer reduction: outer carries 'sum'. */
+int
+double_reduc (const int *a)
+{
+ int sum = 0;
+ for (int i = 0; i < 32; i++)
+ for (int j = 0; j < 512; j++)
+ sum += a[i * 512 + j];
+ return sum;
+}
+
+/* Inner body has a store: inner writes a[j]. */
+void
+inner_store (int *__restrict a, int *__restrict b, const int *c, const int *d)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ {
+ a[j] = c[j] + d[j];
+ s += a[j];
+ }
+ b[i] = s;
+ }
+}
+
+/* Inner trip count just below the min-inner-iters default of 32. */
+void
+small_inner (int *out, const int *a)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 31; j++)
+ s += a[i * 31 + j];
+ out[i] = s;
+ }
+}
+
+/* Outer body stores to counter as well as out[i]. */
+extern int counter;
+void
+outer_store (int *out, const int8_t *a, const int8_t *b)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ s += (int) a[i * 512 + j] * (int) b[j];
+ out[i] = s;
+ counter++;
+ }
+}
+
+/* FP accumulator: the multiply has no narrower scalar-int operand, so
+ it is not a widening MAC. */
+void
+fp_mac (float *out, const float *a, const float *b)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ float s = 0.0f;
+ for (int j = 0; j < 512; j++)
+ s += a[i * 512 + j] * b[j];
+ out[i] = s;
+ }
+}
+
+/* Plain SImode add reduction: no multiply in the update chain. */
+void
+plain_add (int *out, const int *a)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ s += a[i * 512 + j];
+ out[i] = s;
+ }
+}
+
+/* MAX reduction: no multiply in the update chain. */
+void
+max_reduc (int *out, const int *a)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int m = a[i * 512];
+ for (int j = 1; j < 512; j++)
+ {
+ int v = a[i * 512 + j];
+ m = m > v ? m : v;
+ }
+ out[i] = m;
+ }
+}
+
+/* Inner header carries two reduction PHIs, m and s. */
+void
+multi_phi (int *out, const int *a)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int m = a[i * 512];
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ {
+ int v = a[i * 512 + j];
+ m = m > v ? m : v;
+ s += v;
+ }
+ out[i] = m + s;
+ }
+}
+
+/* Parametric outer trip count: not a known UHWI. */
+void
+parametric_outer (int *out, const int8_t *a, const int8_t *b, int n)
+{
+ for (int i = 0; i < n; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ s += (int) a[i * 512 + j] * (int) b[j];
+ out[i] = s;
+ }
+}
+
+/* Outer trip count of 7 has no divisor in [2, 4]. */
+void
+narrow_to_one (int *__restrict out, const int8_t *a, const int8_t *b)
+{
+ for (int i = 0; i < 7; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ s += (int) a[i * 512 + j] * (int) b[j];
+ out[i] = s;
+ }
+}
+
+/* No __restrict: data-dep analysis cannot rule out out vs b
+ aliasing. */
+void
+aliasing (int *out, const int8_t *a, const int8_t *b)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int s = 0;
+ for (int j = 0; j < 512; j++)
+ s += (int) a[i * 512 + j] * (int) b[j];
+ out[i] = s;
+ }
+}
+
+/* Conditional accumulator init: the "?:" lowers to a store-free diamond
+ in the outer body whose merge PHI feeds the reduction, so the body is
+ not straight-line. */
+void
+cond_init (int32_t *__restrict out, const int8_t *__restrict a,
+ const int32_t *__restrict b1, const int32_t *__restrict b2,
+ const int32_t *__restrict c)
+{
+ for (int i = 0; i < 32; i++)
+ {
+ int32_t s = c[i] ? b1[i] : b2[i];
+ for (int j = 0; j < 512; j++)
+ s += (int32_t) a[i * 512 + j] * (int32_t) a[j];
+ out[i] = s;
+ }
+}
+
+/* { dg-final { scan-tree-dump-not "applying reduction unroll-and-jam"
"unrolljam" } } */
+/* { dg-final { scan-tree-dump-not "reduction-jam: fusing src loop"
"unrolljam" } } */
+/* { dg-final { scan-tree-dump "reduction-jam: outer trip 7 has no divisor in
\\\[2, 4\\\]" "unrolljam" } } */
+/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
body has extra stores" "unrolljam" } } */
+/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
body has control flow" "unrolljam" } } */
+/* { dg-final { scan-tree-dump "reduction-jam: rejecting outer loop 1: outer
trip count unknown" "unrolljam" } } */
+/* { dg-final { scan-tree-dump "reduction-jam: data-dep safety bound below 2"
"unrolljam" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
new file mode 100644
index 000000000000..938f7328b049
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-4.c
@@ -0,0 +1,30 @@
+/* The reduction-jam pass should fire even when the IV-only path is
+ disabled via -fno-loop-unroll-and-jam. Pins the gate change in
+ pass_loop_jam::gate and the IV-only/reduction-jam fall-through in
+ tree_loop_unroll_and_jam. */
+
+/* { dg-do compile } */
+/* { dg-options "-O3 -mtune=ampere1 -fno-loop-unroll-and-jam
-floop-unroll-and-jam-reductions -fdump-tree-unrolljam-details" } */
+
+#include <stdint.h>
+
+#define IN_DIM 512
+#define OUT_DIM 32
+
+void
+affine (int32_t *__restrict out,
+ const int8_t *weights,
+ const int32_t *biases,
+ const uint8_t *input)
+{
+ for (int i = 0; i < OUT_DIM; ++i)
+ {
+ int32_t s = biases[i];
+ for (int j = 0; j < IN_DIM; ++j)
+ s += (int32_t) weights[i * IN_DIM + j] * (int32_t) input[j];
+ out[i] = s;
+ }
+}
+
+/* { dg-final { scan-tree-dump "applying reduction unroll-and-jam" "unrolljam"
} } */
+/* { dg-final { scan-tree-dump-times "reduction-jam: fusing src loop" 3
"unrolljam" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
new file mode 100644
index 000000000000..0a198bb27ecd
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/unroll-jam-reductions-5.c
@@ -0,0 +1,72 @@
+/* Regression test for a multi-IV mispairing in the reduction-jam
+ sibling matcher. The inner loop carries two IVs that share the
+ preheader entry value 0 but differ in step (j has step 1 and
+ indexes W, p has step 2 and indexes X); the matcher must pair each
+ source IV with its own dst counterpart, not with whichever dst PHI
+ happens to share an entry value.
+
+ The reference loop in main uses a volatile accumulator so
+ pass_loop_jam does not recognize a reduction PHI there. Without
+ that guard the reference would be jammed with the same mispairing
+ as gemv_2iv, hiding the miscompile behind a symmetric bug. */
+
+/* { dg-do run } */
+/* { dg-options "-O3 -mtune=ampere1 -floop-unroll-and-jam-reductions" } */
+
+#include <stdint.h>
+
+#define OUT_DIM 32
+#define IN_DIM 256
+#define STRIDE 512
+
+extern void abort (void);
+
+void __attribute__((noipa))
+gemv_2iv (int32_t *__restrict out, const int8_t *w, const uint8_t *x)
+{
+ for (int i = 0; i < OUT_DIM; i++)
+ {
+ int32_t s = 0;
+ int p = 0;
+ for (int j = 0; j < IN_DIM; j++)
+ {
+ s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
+ p += 2;
+ }
+ out[i] = s;
+ }
+}
+
+int
+main (void)
+{
+ static int8_t w[OUT_DIM * STRIDE];
+ static uint8_t x[IN_DIM * 2];
+ static int32_t out[OUT_DIM];
+ static int32_t expected[OUT_DIM];
+
+ for (int k = 0; k < OUT_DIM * STRIDE; k++)
+ w[k] = (k * 13) & 0x7f;
+ for (int k = 0; k < IN_DIM * 2; k++)
+ x[k] = (k * 7) & 0xff;
+
+ for (int i = 0; i < OUT_DIM; i++)
+ {
+ volatile int32_t s = 0;
+ int p = 0;
+ for (int j = 0; j < IN_DIM; j++)
+ {
+ s += (int32_t) w[i * STRIDE + j] * (int32_t) x[p];
+ p += 2;
+ }
+ expected[i] = s;
+ }
+
+ gemv_2iv (out, w, x);
+
+ for (int i = 0; i < OUT_DIM; i++)
+ if (out[i] != expected[i])
+ abort ();
+
+ return 0;
+}
--
2.55.0
