On Wed, 24 Jun 2026, Tamar Christina wrote:
> > -----Original Message-----
> > From: Richard Biener <[email protected]>
> > Sent: 24 June 2026 13:02
> > To: Alfie Richards <[email protected]>
> > Cc: [email protected]; Tamar Christina <[email protected]>;
> > [email protected]; [email protected]; [email protected]
> > Subject: Re: [PATCH v1 0/8] RFC: vect: Add first faulting read support
> >
> > On Tue, 23 Jun 2026, Alfie Richards wrote:
> >
> > > Hi All,
> > >
> > > This is my first attempt at an implementation of FFR.
> >
> > I'm replying (late) to this mail before reading all the followup
> > discussion. Sorry if parts are answered already.
> >
> > > I think this needs some background and explanation
> > > (probably should be a comment at some point)
> > >
> > > == AArch64 first faulting read ==
> > >
> > > AArch64 first faulting reads are safe to execute speculatively because the
> > > hardware is able to return less than a full vector read.
> > >
> > > The loads (both "first faulting" (FF) and "non-faulting" (NF)) set a
> > > predicate
> > > register (first fault register) with a mask of loaded elements.
> > >
> > > The FF variant will fault only if the first active element to be loaded
> > > causes a fault. This guarantees forwards progress.
> > >
> > > The NF variant will never cause a fault, at the expense of possibly
> > > loading no elements.
> > >
> > > Architecturally, these loads are guaranteed to never cause a fault unless
> > > it is a FF load and there is a fault for the first element.
> > >
> > > However, the hardware can also do partial loads whenever it wants.
> > > For instance this could happen at page boundaries or cache faults.
> > >
> > > This means an FFR vectorized loop needs to be able to recover from partial
> > > loads back to vectoried code.
> > > (this seems to not be the case for riscv's equivalent feature).
> >
> > Alternatively such "bad" hardware could be forced to fall to a scalar
> > epilog.
> >
> > > Also of note, the first fault register starts as all 1's, and each
> > > subsequent
> > > load updates it if they fail to load any elements by diabling the bits of
> > > elements not loaded and all bits afterwards.
> > >
> > > For instance, it may start as
> > > [1,1,1,1,1,1,1,1]
> > > then a load is executed and could change that to
> > > [1,1,1,1,1,1,0,0]
> > >
> > > But then the next load, even if it loads all elements, will not re-enable
> > > the last two bits.
> > >
> > > Additionally, you never get a state like
> > > [1,1,1,1,1,0,0,1]
> > > where there are inactive elements followed by an active element.
> > >
> > > == Vectorized code structure ==
> > >
> > > The use case currently is for loops with mutually misaligned pointers,
> > > both
> > > of which require safe speculative reads, where we cannot peel for
> > > alignment,
> > or
> > > would not want to incur the code size cost of versioning to check if we
> > > can
> > > peel.
> > >
> > > For aarch64 this is only for early break as there aren't any other
> > > situations
> > > where we need safe speculative reads.
> > >
> > > Then for a loop such as
> > >
> > > int
> > > foo_no_vect (uint16_t *const restrict src1,
> > > uint16_t *const restrict src2,
> > > unsigned int n)
> > > {
> > > for (int i = 0; i < n; i++)
> > > {
> > > uint16_t v1 = src1[i];
> > > uint16_t v2 = src2[i];
> > > if (v1 + v2 == 0)
> > > return 1;
> > > }
> > > return 0;
> > > }
> > >
> > > We generate the following:
> > >
> > > GIMPLE (after dce):
> > >
> > > _78 = max_mask_75 & _77;
> > > ffr_preservation_82 = .READ_FAULT_STATE ();
> > > .SET_FAULT_STATE ({ -1, ... });
> > >
> > > <bb 3> [local count: 1014686025]:
> > > # vectp_src1.7_45 = PHI <vectp_src1.7_46(7), vectp_src1.8_41(13)>
> > > # vectp_src2.11_60 = PHI <vectp_src2.11_61(7), vectp_src2.12_56(13)>
> > > # ivtmp_72 = PHI <ivtmp_73(7), 0(13)>
> > > # loop_mask_48 = PHI <next_mask_ffr_81(7), _78(13)>
> > > vect_v1_14.9_49 = .MASK_FIRSTFAULT_LOAD (vectp_src1.7_45, 64B,
> > loop_mask_48, { 0, ... });
> > > vect__5.10_55 = (vector([4,4]) int) vect_v1_14.9_49;
> > > vect_v2_16.13_63 = .MASK_FIRSTFAULT_LOAD (vectp_src2.11_60, 64B,
> > loop_mask_48, { 0, ... });
> > > vect__6.14_65 = (vector([4,4]) int) vect_v2_16.13_63;
> > > ffr_mask_83 = .READ_FAULT_STATE ();
> >
> > There seems to be no data dependence between .MASK_FIRSTFAULT_LOAD
> > and .READ_FAULT_STATE which IMO is bad design (see all the issues
> > we have with FPU state accesses). You supposedly do not want to
> > have a dependence (due to the FFR) between the two
> > .MASK_FIRSTFAULT_LOAD
> > so an extra inout argument to that isn't intended. So the
> > alternative is to have separate virtual outputs and have
> > .READ_FAULT_STATE merge them via having N inputs?
>
> I think the above isn't a verbose dump, there should be a dependency on
> The VOPS chain to prevent both the re-ordering between fault groups and
> stores.
>
> This mirrors how the intrinsics are modelled so we kept the same dependency
> so that we can tell when interacting with intrinsics code.
Hmm. How does that translate to RTL then? Is this UNSPEC_VOLATILE
and we treat that as a scheduling barrier (huh? do we?)?
> At least that was the idea. I haven't looked at the implementation yet but I
> am
> expecting a virtual use chain.
That of course means that .MASK_FIRSTFAULT_LOAD are actually stores
to GIMPLE. Maybe that isn't bad and we hopefully do not need any
followup optimizations to such code anywhere ...
Richard.
>
> Thanks,
> Tamar
> >
> > > if (ffr_mask_83 == { -1, ... })
> > > goto <bb 14>; [99.95%]
> > > else
> > > goto <bb 15>; [0.05%]
> > >
> > > <bb 15> [local count: 10146860]:
> > > .SET_FAULT_STATE ({ -1, ... });
> > > _84 = ~ffr_mask_83;
> > > ffr_loop_mask_85 = loop_mask_48 & ffr_mask_83;
> > >
> > > <bb 14> [local count: 1014686025]:
> > > # ffr_loop_mask_68 = PHI <loop_mask_48(3), ffr_loop_mask_85(15)>
> > > # ffr_num_iters_36 = PHI <POLY_INT_CST [4, 4](3), 0(15)>
> > > # next_mask_ffr_80 = PHI <{ -1, ... }(3), _84(15)>
> > > vect__7.15_66 = vect__5.10_55 + vect__6.14_65;
> > > mask_patt_28.16_67 = vect__7.15_66 == { 0, ... };
> > > vec_mask_and_69 = mask_patt_28.16_67 & ffr_loop_mask_68;
> > > if (vec_mask_and_69 != { 0, ... })
> > > goto <bb 9>; [5.50%]
> > > else
> > > goto <bb 4>; [94.50%]
> > >
> > > <bb 9> [local count: 55807731]:
> > > .SET_FAULT_STATE (ffr_preservation_82);
> > > goto <bb 5>; [100.00%]
> > >
> > > <bb 4> [local count: 958878295]:
> > > _47 = ffr_num_iters_36 * 2;
> > > vectp_src1.7_46 = vectp_src1.7_45 + _47;
> > > vectp_src2.11_61 = vectp_src2.11_60 + _47;
> > > ivtmp_73 = ivtmp_72 + ffr_num_iters_36;
> > > next_mask_79 = .WHILE_ULT (ivtmp_73, _74, { 0, ... });
> > > next_mask_ffr_81 = next_mask_79 & next_mask_ffr_80;
> > > if (next_mask_ffr_81 != { 0, ... })
> > > goto <bb 7>; [94.50%]
> > > else
> > > goto <bb 12>; [5.50%]
> > >
> > > <bb 12> [local count: 52738306]:
> > > .SET_FAULT_STATE (ffr_preservation_82);
> > > goto <bb 5>; [100.00%]
> > >
> > > <bb 7> [local count: 906139989]:
> > > goto <bb 3>; [100.00%]
> > >
> > > <bb 5> [local count: 114863531]:
> > > # _10 = PHI <1(9), 0(12), 0(2)>
> > > return _10;
> > >
> > > Or final assembly:
> > >
> > > .L5:
> > > add x4, x4, x3
> > > whilelo p7.s, x4, x2
> > > add x0, x0, x3, lsl 1
> > > and p7.b, p7/z, p14.b, p14.b
> > > add x1, x1, x3, lsl 1
> > > ptest p15, p7.b
> > > b.none .L7
> > > .L6:
> > > ldff1h z31.s, p7/z, [x0]
> > > ldff1h z30.s, p7/z, [x1]
> > > cntw x3
> > > rdffr p14.b
> > > nots p13.b, p15/z, p14.b
> > > b.any .L11
> > > .L4:
> > > add z31.s, z31.s, z30.s
> > > cmpeq p7.s, p7/z, z31.s, #0
> > > b.none .L5
> > > mov w0, 1
> > > ret
> > >
> > > == Notes ==
> > >
> > > - When there is a "partial read", instead of treating that as a partial
> > > iteration
> > > and advancing by the number of loaded elements, we intead advance by 0
> > > iterations ard repeat the same iteration with the previously processed
> > > elements masked out. This preserves alignment with the starting position
> > > and avoids having to do anything awkward such as possibly rotating
> > > invariant vectors.
> >
> > So the advantage is that we never need a scalar fallback? But ISTR
> > there's other reasons why we might need that for early break still?
> >
> > > - This prioritises the "good" case, by trying to keep
> > > the "full read" path as tight as possible, and adding
> > > a fixup branch to handle the case where there is a partial read.
> > >
> > > - We preserve the state of the FFR register over the vectoried loop.
> > > This is to prevent code written with intrinsics breaking by clobbering
> > > their value in the FFR register. However, as the FFR is not preserved by
> > > the AAPCS, this is nearly always optimized out.
> > >
> > > - One downside of this approach is I dont think it will translate cleanly
> > > for
> > > len based loop vectorization (riscv). However, as I understand it, the
> > > riscv equivalent feature never does partial loads unless there is a
> > > genuine
> > > fault, so we will not need to worry about recovering back to
> > > vectorized code after a partial read (as it will either take the early
> > > break
> > > or the fault). So no fixup should be needed and can use a subset of
> > > this.
> > >
> > > == Remaining work to do ==
> > >
> > > - Versioning
> > > As FFR introduces overhead, it will always be slower than a mutually
> > > aligned loop. So my ideal code generation for a vector with two pointers
> > > requiring safe speculative reads is:
> > >
> > > At O2, where we do not want to incur the code size cost of versioning,
> > > instead just use FFR to vectorize the loop (if profitable).
> > >
> > > At O3, version to create a mutually aligned non-FFR case and a (current
> > code gen)
> > > if the pointers are mutually misaligned.
> > >
> > > This "FFR versioning" is not yet implemented.
> > >
> > > - Costing
> > > I haven't done anything to cost FFR yet, we will want accurate costing
> > > of
> > FFR
> > > vs scalar and vs non-FFR vectorized to make the decisions on versioning
> > > and
> > > what to use.
> > >
> > > - Optimization of the generated code
> > > The generated code has room for improvement, primarily using rdffrs
> > would
> > > be a performance gain, and reducing some of the moves within the hot
> > > section of the loop.
> > >
> > > == Feedback wanted ==
> > >
> > > - Overall design of this
> > > - What to call this within GCC (it's not really "first fault reads", maybe
> > > "hardware safe speculative reads" (HSSR)?)
> > > - How we can sensibly cost this to allow the choice to do scalar over
> > > FFR.
> >
> > When a load is partial you need to consider the FFR for all computations
> > that require masking by loop masking. Your example is simplified
> > to not have any, but consider a division - you show an else value
> > of zero, so before any use of loop_mask on an operation that is
> > dependent on the FFR affecting load value you need to read the FFR
> > and combine it with the loop mask, right?
> >
> > > Sorry for the essay!
> >
> > No need to sorry, it's very helpful.
> >
> > ISTR AVX10.x introduces some non-faulting loads but I have to check.
> >
> > Richard.
> >
> > >
> > > Bootstrapped and reg tested for aarch64, x68-64, arm32hf, riscv.
> > >
> > > I also ran vect.exp with --param=vect-ffr-usage=2 and -msve-vector-
> > bits=128
> > > with no errors.
> > >
> > > Thoughts?
> > >
> > > King regards,
> > > Alfie
> > >
> > > Alfie Richards (8):
> > > vect: Add internal functions and optabs for first fault loads
> > > vect: Add SLP_NODE argument to vect_get_loop_mask.
> > > vect: Make vect_maybe_permute_loop_masks optional and retargetable
> > > vect: Add EXCLUDE_VIRTUALS argument to _slp_tree::push_vec_def.
> > > vect: Add vect-ffr-usage param.
> > > vect: Add FFR analysis
> > > vect: Add FFR transformation
> > > vect: Enable FFR
> > >
> > > .../aarch64/aarch64-sve-builtins-base.cc | 78 +++--
> > > gcc/config/aarch64/aarch64-sve-builtins.cc | 24 ++
> > > gcc/config/aarch64/aarch64-sve-builtins.h | 1 +
> > > gcc/config/aarch64/aarch64-sve.md | 201 +++++++++--
> > > gcc/config/aarch64/aarch64.cc | 10 +
> > > gcc/config/aarch64/iterators.md | 2 +
> > > gcc/internal-fn.cc | 24 ++
> > > gcc/internal-fn.def | 18 +
> > > gcc/optabs-tree.cc | 57 +++
> > > gcc/optabs-tree.h | 2 +
> > > gcc/optabs.def | 5 +
> > > gcc/params.opt | 4 +
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c | 16 +
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c | 35 ++
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c | 42 +++
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c | 25 ++
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c | 25 ++
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c | 43 +++
> > > .../gcc.target/aarch64/sve/ffr_6_run.c | 81 +++++
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c | 16 +
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c | 16 +
> > > gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c | 18 +
> > > .../gcc.target/aarch64/sve/noeffect11.c | 2 +-
> > > .../gcc.target/aarch64/sve/peel_ind_12.c | 2 +-
> > > .../gcc.target/aarch64/sve/peel_ind_12_run.c | 2 +-
> > > .../gcc.target/aarch64/sve/pfalse-load.c | 6 +-
> > > gcc/tree-data-ref.cc | 4 +
> > > gcc/tree-ssa-alias.cc | 2 +
> > > gcc/tree-ssa-loop-ivopts.cc | 2 +
> > > gcc/tree-vect-data-refs.cc | 3 +-
> > > gcc/tree-vect-loop-manip.cc | 328 +++++++++++++++++-
> > > gcc/tree-vect-loop.cc | 264 +++++++++++++-
> > > gcc/tree-vect-slp.cc | 32 +-
> > > gcc/tree-vect-stmts.cc | 165 +++++++--
> > > gcc/tree-vectorizer.h | 63 +++-
> > > 35 files changed, 1496 insertions(+), 122 deletions(-)
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6_run.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c
> > > create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c
> > >
> > >
> >
> > --
> > Richard Biener <[email protected]>
> > SUSE Software Solutions Germany GmbH,
> > Frankenstrasse 146, 90461 Nuernberg, Germany;
> > GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG
> > Nuernberg)
>
--
Richard Biener <[email protected]>
SUSE Software Solutions Germany GmbH,
Frankenstrasse 146, 90461 Nuernberg, Germany;
GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG Nuernberg)
