> -----Original Message-----
> From: Alfie Richards <[email protected]>
> Sent: 24 June 2026 14:13
> To: Richard Biener <[email protected]>; Tamar Christina
> <[email protected]>
> Cc: [email protected]; [email protected]; [email protected];
> [email protected]
> Subject: Re: [PATCH v1 0/8] RFC: vect: Add first faulting read support
>
> On 24/06/2026 13:22, Richard Biener wrote:
> > On Wed, 24 Jun 2026, Tamar Christina wrote:
> >
> >>> -----Original Message-----
> >>> From: Richard Biener <[email protected]>
> >>> Sent: 24 June 2026 13:02
> >>> To: Alfie Richards <[email protected]>
> >>> Cc: [email protected]; Tamar Christina
> <[email protected]>;
> >>> [email protected]; [email protected]; [email protected]
> >>> Subject: Re: [PATCH v1 0/8] RFC: vect: Add first faulting read support
> >>>
> >>> On Tue, 23 Jun 2026, Alfie Richards wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> This is my first attempt at an implementation of FFR.
> >>>
> >>> I'm replying (late) to this mail before reading all the followup
> >>> discussion. Sorry if parts are answered already.
> >>>
> >>>> I think this needs some background and explanation
> >>>> (probably should be a comment at some point)
> >>>>
> >>>> == AArch64 first faulting read ==
> >>>>
> >>>> AArch64 first faulting reads are safe to execute speculatively because
> >>>> the
> >>>> hardware is able to return less than a full vector read.
> >>>>
> >>>> The loads (both "first faulting" (FF) and "non-faulting" (NF)) set a
> predicate
> >>>> register (first fault register) with a mask of loaded elements.
> >>>>
> >>>> The FF variant will fault only if the first active element to be loaded
> >>>> causes a fault. This guarantees forwards progress.
> >>>>
> >>>> The NF variant will never cause a fault, at the expense of possibly
> >>>> loading no elements.
> >>>>
> >>>> Architecturally, these loads are guaranteed to never cause a fault unless
> >>>> it is a FF load and there is a fault for the first element.
> >>>>
> >>>> However, the hardware can also do partial loads whenever it wants.
> >>>> For instance this could happen at page boundaries or cache faults.
> >>>>
> >>>> This means an FFR vectorized loop needs to be able to recover from
> partial
> >>>> loads back to vectoried code.
> >>>> (this seems to not be the case for riscv's equivalent feature).
> >>>
> >>> Alternatively such "bad" hardware could be forced to fall to a scalar
> >>> epilog.
> >>>
> >>>> Also of note, the first fault register starts as all 1's, and each
> >>>> subsequent
> >>>> load updates it if they fail to load any elements by diabling the bits of
> >>>> elements not loaded and all bits afterwards.
> >>>>
> >>>> For instance, it may start as
> >>>> [1,1,1,1,1,1,1,1]
> >>>> then a load is executed and could change that to
> >>>> [1,1,1,1,1,1,0,0]
> >>>>
> >>>> But then the next load, even if it loads all elements, will not re-enable
> >>>> the last two bits.
> >>>>
> >>>> Additionally, you never get a state like
> >>>> [1,1,1,1,1,0,0,1]
> >>>> where there are inactive elements followed by an active element.
> >>>>
> >>>> == Vectorized code structure ==
> >>>>
> >>>> The use case currently is for loops with mutually misaligned pointers,
> both
> >>>> of which require safe speculative reads, where we cannot peel for
> alignment,
> >>> or
> >>>> would not want to incur the code size cost of versioning to check if we
> can
> >>>> peel.
> >>>>
> >>>> For aarch64 this is only for early break as there aren't any other
> >>>> situations
> >>>> where we need safe speculative reads.
> >>>>
> >>>> Then for a loop such as
> >>>>
> >>>> int
> >>>> foo_no_vect (uint16_t *const restrict src1,
> >>>> uint16_t *const restrict src2,
> >>>> unsigned int n)
> >>>> {
> >>>> for (int i = 0; i < n; i++)
> >>>> {
> >>>> uint16_t v1 = src1[i];
> >>>> uint16_t v2 = src2[i];
> >>>> if (v1 + v2 == 0)
> >>>> return 1;
> >>>> }
> >>>> return 0;
> >>>> }
> >>>>
> >>>> We generate the following:
> >>>>
> >>>> GIMPLE (after dce):
> >>>>
> >>>> _78 = max_mask_75 & _77;
> >>>> ffr_preservation_82 = .READ_FAULT_STATE ();
> >>>> .SET_FAULT_STATE ({ -1, ... });
> >>>>
> >>>> <bb 3> [local count: 1014686025]:
> >>>> # vectp_src1.7_45 = PHI <vectp_src1.7_46(7), vectp_src1.8_41(13)>
> >>>> # vectp_src2.11_60 = PHI <vectp_src2.11_61(7),
> vectp_src2.12_56(13)>
> >>>> # ivtmp_72 = PHI <ivtmp_73(7), 0(13)>
> >>>> # loop_mask_48 = PHI <next_mask_ffr_81(7), _78(13)>
> >>>> vect_v1_14.9_49 = .MASK_FIRSTFAULT_LOAD (vectp_src1.7_45, 64B,
> >>> loop_mask_48, { 0, ... });
> >>>> vect__5.10_55 = (vector([4,4]) int) vect_v1_14.9_49;
> >>>> vect_v2_16.13_63 = .MASK_FIRSTFAULT_LOAD (vectp_src2.11_60,
> 64B,
> >>> loop_mask_48, { 0, ... });
> >>>> vect__6.14_65 = (vector([4,4]) int) vect_v2_16.13_63;
> >>>> ffr_mask_83 = .READ_FAULT_STATE ();
> >>>
> >>> There seems to be no data dependence between
> .MASK_FIRSTFAULT_LOAD
> >>> and .READ_FAULT_STATE which IMO is bad design (see all the issues
> >>> we have with FPU state accesses). You supposedly do not want to
> >>> have a dependence (due to the FFR) between the two
> >>> .MASK_FIRSTFAULT_LOAD
> >>> so an extra inout argument to that isn't intended. So the
> >>> alternative is to have separate virtual outputs and have
> >>> .READ_FAULT_STATE merge them via having N inputs?
> >>
> >> I think the above isn't a verbose dump, there should be a dependency on
> >> The VOPS chain to prevent both the re-ordering between fault groups and
> >> stores.
> >>
> >> This mirrors how the intrinsics are modelled so we kept the same
> dependency
> >> so that we can tell when interacting with intrinsics code.
> >
> > Hmm. How does that translate to RTL then? Is this UNSPEC_VOLATILE
> > and we treat that as a scheduling barrier (huh? do we?)?
>
> No in RTL this all uses the aarch64 FFR register and the FFRT (first
> faulting read token) to represent the scheduling dependencies. Though
> that is Richard S's invention so I will defer to him as the expert.
>
Yeah, In RTL there's a pseudo register FFRT which isn't an actual register but
there to deal with the global dependencies. By doing this and not having them
as unspecs we get optimizations like removing spurious READ_FFRS, while still
allowing scheduling within an FFR region.
Since FFR is also call globbered reload *should* handle the case where the
region
is split due to a call, like a libmvec call.
This also frees the representation of the RTL to model that it also sets
CC_FLAGS
and prevent ordering issues with order operations setting or clobbering CC
flags.
So generally this works pretty good. It allows CSE and the rest to optimize
the FFR
usages.
Thanks,
Tamar
> >
> >> At least that was the idea. I haven't looked at the implementation yet but
> >> I
> am
> >> expecting a virtual use chain.
> >
> > That of course means that .MASK_FIRSTFAULT_LOAD are actually stores
> > to GIMPLE. Maybe that isn't bad and we hopefully do not need any
> > followup optimizations to such code anywhere ...
>
> Yes this is correct and we represent the interaction with the "global
> FFR state" (read, the FFR register in aarch64) via virtual use chains.
>
> Sorry for not including that!
>
> Thanks,
> Alfie
>
> >
> > Richard.
> >
> >>
> >> Thanks,
> >> Tamar
> >>>
> >>>> if (ffr_mask_83 == { -1, ... })
> >>>> goto <bb 14>; [99.95%]
> >>>> else
> >>>> goto <bb 15>; [0.05%]
> >>>>
> >>>> <bb 15> [local count: 10146860]:
> >>>> .SET_FAULT_STATE ({ -1, ... });
> >>>> _84 = ~ffr_mask_83;
> >>>> ffr_loop_mask_85 = loop_mask_48 & ffr_mask_83;
> >>>>
> >>>> <bb 14> [local count: 1014686025]:
> >>>> # ffr_loop_mask_68 = PHI <loop_mask_48(3), ffr_loop_mask_85(15)>
> >>>> # ffr_num_iters_36 = PHI <POLY_INT_CST [4, 4](3), 0(15)>
> >>>> # next_mask_ffr_80 = PHI <{ -1, ... }(3), _84(15)>
> >>>> vect__7.15_66 = vect__5.10_55 + vect__6.14_65;
> >>>> mask_patt_28.16_67 = vect__7.15_66 == { 0, ... };
> >>>> vec_mask_and_69 = mask_patt_28.16_67 & ffr_loop_mask_68;
> >>>> if (vec_mask_and_69 != { 0, ... })
> >>>> goto <bb 9>; [5.50%]
> >>>> else
> >>>> goto <bb 4>; [94.50%]
> >>>>
> >>>> <bb 9> [local count: 55807731]:
> >>>> .SET_FAULT_STATE (ffr_preservation_82);
> >>>> goto <bb 5>; [100.00%]
> >>>>
> >>>> <bb 4> [local count: 958878295]:
> >>>> _47 = ffr_num_iters_36 * 2;
> >>>> vectp_src1.7_46 = vectp_src1.7_45 + _47;
> >>>> vectp_src2.11_61 = vectp_src2.11_60 + _47;
> >>>> ivtmp_73 = ivtmp_72 + ffr_num_iters_36;
> >>>> next_mask_79 = .WHILE_ULT (ivtmp_73, _74, { 0, ... });
> >>>> next_mask_ffr_81 = next_mask_79 & next_mask_ffr_80;
> >>>> if (next_mask_ffr_81 != { 0, ... })
> >>>> goto <bb 7>; [94.50%]
> >>>> else
> >>>> goto <bb 12>; [5.50%]
> >>>>
> >>>> <bb 12> [local count: 52738306]:
> >>>> .SET_FAULT_STATE (ffr_preservation_82);
> >>>> goto <bb 5>; [100.00%]
> >>>>
> >>>> <bb 7> [local count: 906139989]:
> >>>> goto <bb 3>; [100.00%]
> >>>>
> >>>> <bb 5> [local count: 114863531]:
> >>>> # _10 = PHI <1(9), 0(12), 0(2)>
> >>>> return _10;
> >>>>
> >>>> Or final assembly:
> >>>>
> >>>> .L5:
> >>>> add x4, x4, x3
> >>>> whilelo p7.s, x4, x2
> >>>> add x0, x0, x3, lsl 1
> >>>> and p7.b, p7/z, p14.b, p14.b
> >>>> add x1, x1, x3, lsl 1
> >>>> ptest p15, p7.b
> >>>> b.none .L7
> >>>> .L6:
> >>>> ldff1h z31.s, p7/z, [x0]
> >>>> ldff1h z30.s, p7/z, [x1]
> >>>> cntw x3
> >>>> rdffr p14.b
> >>>> nots p13.b, p15/z, p14.b
> >>>> b.any .L11
> >>>> .L4:
> >>>> add z31.s, z31.s, z30.s
> >>>> cmpeq p7.s, p7/z, z31.s, #0
> >>>> b.none .L5
> >>>> mov w0, 1
> >>>> ret
> >>>>
> >>>> == Notes ==
> >>>>
> >>>> - When there is a "partial read", instead of treating that as a partial
> iteration
> >>>> and advancing by the number of loaded elements, we intead advance
> by 0
> >>>> iterations ard repeat the same iteration with the previously processed
> >>>> elements masked out. This preserves alignment with the starting
> position
> >>>> and avoids having to do anything awkward such as possibly rotating
> >>>> invariant vectors.
> >>>
> >>> So the advantage is that we never need a scalar fallback? But ISTR
> >>> there's other reasons why we might need that for early break still?
> >>>
> >>>> - This prioritises the "good" case, by trying to keep
> >>>> the "full read" path as tight as possible, and adding
> >>>> a fixup branch to handle the case where there is a partial read.
> >>>>
> >>>> - We preserve the state of the FFR register over the vectoried loop.
> >>>> This is to prevent code written with intrinsics breaking by clobbering
> >>>> their value in the FFR register. However, as the FFR is not preserved
> >>>> by
> >>>> the AAPCS, this is nearly always optimized out.
> >>>>
> >>>> - One downside of this approach is I dont think it will translate
> >>>> cleanly for
> >>>> len based loop vectorization (riscv). However, as I understand it, the
> >>>> riscv equivalent feature never does partial loads unless there is a
> genuine
> >>>> fault, so we will not need to worry about recovering back to
> >>>> vectorized code after a partial read (as it will either take the
> >>>> early break
> >>>> or the fault). So no fixup should be needed and can use a subset of
> >>>> this.
> >>>>
> >>>> == Remaining work to do ==
> >>>>
> >>>> - Versioning
> >>>> As FFR introduces overhead, it will always be slower than a mutually
> >>>> aligned loop. So my ideal code generation for a vector with two
> >>>> pointers
> >>>> requiring safe speculative reads is:
> >>>>
> >>>> At O2, where we do not want to incur the code size cost of versioning,
> >>>> instead just use FFR to vectorize the loop (if profitable).
> >>>>
> >>>> At O3, version to create a mutually aligned non-FFR case and a
> >>>> (current
> >>> code gen)
> >>>> if the pointers are mutually misaligned.
> >>>>
> >>>> This "FFR versioning" is not yet implemented.
> >>>>
> >>>> - Costing
> >>>> I haven't done anything to cost FFR yet, we will want accurate costing
> of
> >>> FFR
> >>>> vs scalar and vs non-FFR vectorized to make the decisions on
> >>>> versioning
> and
> >>>> what to use.
> >>>>
> >>>> - Optimization of the generated code
> >>>> The generated code has room for improvement, primarily using rdffrs
> >>> would
> >>>> be a performance gain, and reducing some of the moves within the hot
> >>>> section of the loop.
> >>>>
> >>>> == Feedback wanted ==
> >>>>
> >>>> - Overall design of this
> >>>> - What to call this within GCC (it's not really "first fault reads",
> >>>> maybe
> >>>> "hardware safe speculative reads" (HSSR)?)
> >>>> - How we can sensibly cost this to allow the choice to do scalar over
> >>>> FFR.
> >>>
> >>> When a load is partial you need to consider the FFR for all computations
> >>> that require masking by loop masking. Your example is simplified
> >>> to not have any, but consider a division - you show an else value
> >>> of zero, so before any use of loop_mask on an operation that is
> >>> dependent on the FFR affecting load value you need to read the FFR
> >>> and combine it with the loop mask, right?
> >>>
> >>>> Sorry for the essay!
> >>>
> >>> No need to sorry, it's very helpful.
> >>>
> >>> ISTR AVX10.x introduces some non-faulting loads but I have to check.
> >>>
> >>> Richard.
> >>>
> >>>>
> >>>> Bootstrapped and reg tested for aarch64, x68-64, arm32hf, riscv.
> >>>>
> >>>> I also ran vect.exp with --param=vect-ffr-usage=2 and -msve-vector-
> >>> bits=128
> >>>> with no errors.
> >>>>
> >>>> Thoughts?
> >>>>
> >>>> King regards,
> >>>> Alfie
> >>>>
> >>>> Alfie Richards (8):
> >>>> vect: Add internal functions and optabs for first fault loads
> >>>> vect: Add SLP_NODE argument to vect_get_loop_mask.
> >>>> vect: Make vect_maybe_permute_loop_masks optional and
> retargetable
> >>>> vect: Add EXCLUDE_VIRTUALS argument to _slp_tree::push_vec_def.
> >>>> vect: Add vect-ffr-usage param.
> >>>> vect: Add FFR analysis
> >>>> vect: Add FFR transformation
> >>>> vect: Enable FFR
> >>>>
> >>>> .../aarch64/aarch64-sve-builtins-base.cc | 78 +++--
> >>>> gcc/config/aarch64/aarch64-sve-builtins.cc | 24 ++
> >>>> gcc/config/aarch64/aarch64-sve-builtins.h | 1 +
> >>>> gcc/config/aarch64/aarch64-sve.md | 201 +++++++++--
> >>>> gcc/config/aarch64/aarch64.cc | 10 +
> >>>> gcc/config/aarch64/iterators.md | 2 +
> >>>> gcc/internal-fn.cc | 24 ++
> >>>> gcc/internal-fn.def | 18 +
> >>>> gcc/optabs-tree.cc | 57 +++
> >>>> gcc/optabs-tree.h | 2 +
> >>>> gcc/optabs.def | 5 +
> >>>> gcc/params.opt | 4 +
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c | 16 +
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c | 35 ++
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c | 42 +++
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c | 25 ++
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c | 25 ++
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c | 43 +++
> >>>> .../gcc.target/aarch64/sve/ffr_6_run.c | 81 +++++
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c | 16 +
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c | 16 +
> >>>> gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c | 18 +
> >>>> .../gcc.target/aarch64/sve/noeffect11.c | 2 +-
> >>>> .../gcc.target/aarch64/sve/peel_ind_12.c | 2 +-
> >>>> .../gcc.target/aarch64/sve/peel_ind_12_run.c | 2 +-
> >>>> .../gcc.target/aarch64/sve/pfalse-load.c | 6 +-
> >>>> gcc/tree-data-ref.cc | 4 +
> >>>> gcc/tree-ssa-alias.cc | 2 +
> >>>> gcc/tree-ssa-loop-ivopts.cc | 2 +
> >>>> gcc/tree-vect-data-refs.cc | 3 +-
> >>>> gcc/tree-vect-loop-manip.cc | 328 +++++++++++++++++-
> >>>> gcc/tree-vect-loop.cc | 264 +++++++++++++-
> >>>> gcc/tree-vect-slp.cc | 32 +-
> >>>> gcc/tree-vect-stmts.cc | 165 +++++++--
> >>>> gcc/tree-vectorizer.h | 63 +++-
> >>>> 35 files changed, 1496 insertions(+), 122 deletions(-)
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6_run.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c
> >>>> create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c
> >>>>
> >>>>
> >>>
> >>> --
> >>> Richard Biener <[email protected]>
> >>> SUSE Software Solutions Germany GmbH,
> >>> Frankenstrasse 146, 90461 Nuernberg, Germany;
> >>> GF: Jochen Jaser, Andrew McDonald, Werner Knoblich; (HRB 36809, AG
> >>> Nuernberg)
> >>
> >
