On 24/06/2026 13:01, Richard Biener wrote:
On Tue, 23 Jun 2026, Alfie Richards wrote:
Hi All,
This is my first attempt at an implementation of FFR.
I'm replying (late) to this mail before reading all the followup
discussion. Sorry if parts are answered already.
Hi Richi,
Np, thanks for the feedback!
I think this needs some background and explanation
(probably should be a comment at some point)
== AArch64 first faulting read ==
AArch64 first faulting reads are safe to execute speculatively because the
hardware is able to return less than a full vector read.
The loads (both "first faulting" (FF) and "non-faulting" (NF)) set a predicate
register (first fault register) with a mask of loaded elements.
The FF variant will fault only if the first active element to be loaded
causes a fault. This guarantees forwards progress.
The NF variant will never cause a fault, at the expense of possibly
loading no elements.
Architecturally, these loads are guaranteed to never cause a fault unless
it is a FF load and there is a fault for the first element.
However, the hardware can also do partial loads whenever it wants.
For instance this could happen at page boundaries or cache faults.
This means an FFR vectorized loop needs to be able to recover from partial
loads back to vectoried code.
(this seems to not be the case for riscv's equivalent feature).
Alternatively such "bad" hardware could be forced to fall to a scalar
epilog.
I wouldn't be as strong as calling it as "bad" hardware personally. It
seems to be within the design intention of both aarch64 and riscv to
specifically allow for this as a feature.
For example in the extreme case, hardware doing partial loads at page
faults to avoid waiting for the rest of the load of data from hard disk.
Especially when that data may not end up being needed if the early break
is taken before any of that data.
In practice, from my experiments, these partial loads are pretty common.
So falling back to scalar with no way to recover to vectorised would be
make this feature pretty pointless.
Also of note, the first fault register starts as all 1's, and each subsequent
load updates it if they fail to load any elements by diabling the bits of
elements not loaded and all bits afterwards.
For instance, it may start as
[1,1,1,1,1,1,1,1]
then a load is executed and could change that to
[1,1,1,1,1,1,0,0]
But then the next load, even if it loads all elements, will not re-enable
the last two bits.
Additionally, you never get a state like
[1,1,1,1,1,0,0,1]
where there are inactive elements followed by an active element.
== Vectorized code structure ==
The use case currently is for loops with mutually misaligned pointers, both
of which require safe speculative reads, where we cannot peel for alignment, or
would not want to incur the code size cost of versioning to check if we can
peel.
For aarch64 this is only for early break as there aren't any other situations
where we need safe speculative reads.
Then for a loop such as
int
foo_no_vect (uint16_t *const restrict src1,
uint16_t *const restrict src2,
unsigned int n)
{
for (int i = 0; i < n; i++)
{
uint16_t v1 = src1[i];
uint16_t v2 = src2[i];
if (v1 + v2 == 0)
return 1;
}
return 0;
}
We generate the following:
GIMPLE (after dce):
_78 = max_mask_75 & _77;
ffr_preservation_82 = .READ_FAULT_STATE ();
.SET_FAULT_STATE ({ -1, ... });
<bb 3> [local count: 1014686025]:
# vectp_src1.7_45 = PHI <vectp_src1.7_46(7), vectp_src1.8_41(13)>
# vectp_src2.11_60 = PHI <vectp_src2.11_61(7), vectp_src2.12_56(13)>
# ivtmp_72 = PHI <ivtmp_73(7), 0(13)>
# loop_mask_48 = PHI <next_mask_ffr_81(7), _78(13)>
vect_v1_14.9_49 = .MASK_FIRSTFAULT_LOAD (vectp_src1.7_45, 64B, loop_mask_48,
{ 0, ... });
vect__5.10_55 = (vector([4,4]) int) vect_v1_14.9_49;
vect_v2_16.13_63 = .MASK_FIRSTFAULT_LOAD (vectp_src2.11_60, 64B,
loop_mask_48, { 0, ... });
vect__6.14_65 = (vector([4,4]) int) vect_v2_16.13_63;
ffr_mask_83 = .READ_FAULT_STATE ();
There seems to be no data dependence between .MASK_FIRSTFAULT_LOAD
and .READ_FAULT_STATE which IMO is bad design (see all the issues
we have with FPU state accesses). You supposedly do not want to
have a dependence (due to the FFR) between the two .MASK_FIRSTFAULT_LOAD
so an extra inout argument to that isn't intended. So the
alternative is to have separate virtual outputs and have
.READ_FAULT_STATE merge them via having N inputs?
Will follow up in the thread with Tamar.
if (ffr_mask_83 == { -1, ... })
goto <bb 14>; [99.95%]
else
goto <bb 15>; [0.05%]
<bb 15> [local count: 10146860]:
.SET_FAULT_STATE ({ -1, ... });
_84 = ~ffr_mask_83;
ffr_loop_mask_85 = loop_mask_48 & ffr_mask_83;
<bb 14> [local count: 1014686025]:
# ffr_loop_mask_68 = PHI <loop_mask_48(3), ffr_loop_mask_85(15)>
# ffr_num_iters_36 = PHI <POLY_INT_CST [4, 4](3), 0(15)>
# next_mask_ffr_80 = PHI <{ -1, ... }(3), _84(15)>
vect__7.15_66 = vect__5.10_55 + vect__6.14_65;
mask_patt_28.16_67 = vect__7.15_66 == { 0, ... };
vec_mask_and_69 = mask_patt_28.16_67 & ffr_loop_mask_68;
if (vec_mask_and_69 != { 0, ... })
goto <bb 9>; [5.50%]
else
goto <bb 4>; [94.50%]
<bb 9> [local count: 55807731]:
.SET_FAULT_STATE (ffr_preservation_82);
goto <bb 5>; [100.00%]
<bb 4> [local count: 958878295]:
_47 = ffr_num_iters_36 * 2;
vectp_src1.7_46 = vectp_src1.7_45 + _47;
vectp_src2.11_61 = vectp_src2.11_60 + _47;
ivtmp_73 = ivtmp_72 + ffr_num_iters_36;
next_mask_79 = .WHILE_ULT (ivtmp_73, _74, { 0, ... });
next_mask_ffr_81 = next_mask_79 & next_mask_ffr_80;
if (next_mask_ffr_81 != { 0, ... })
goto <bb 7>; [94.50%]
else
goto <bb 12>; [5.50%]
<bb 12> [local count: 52738306]:
.SET_FAULT_STATE (ffr_preservation_82);
goto <bb 5>; [100.00%]
<bb 7> [local count: 906139989]:
goto <bb 3>; [100.00%]
<bb 5> [local count: 114863531]:
# _10 = PHI <1(9), 0(12), 0(2)>
return _10;
Or final assembly:
.L5:
add x4, x4, x3
whilelo p7.s, x4, x2
add x0, x0, x3, lsl 1
and p7.b, p7/z, p14.b, p14.b
add x1, x1, x3, lsl 1
ptest p15, p7.b
b.none .L7
.L6:
ldff1h z31.s, p7/z, [x0]
ldff1h z30.s, p7/z, [x1]
cntw x3
rdffr p14.b
nots p13.b, p15/z, p14.b
b.any .L11
.L4:
add z31.s, z31.s, z30.s
cmpeq p7.s, p7/z, z31.s, #0
b.none .L5
mov w0, 1
ret
== Notes ==
- When there is a "partial read", instead of treating that as a partial
iteration
and advancing by the number of loaded elements, we intead advance by 0
iterations ard repeat the same iteration with the previously processed
elements masked out. This preserves alignment with the starting position
and avoids having to do anything awkward such as possibly rotating
invariant vectors.
So the advantage is that we never need a scalar fallback? But ISTR
there's other reasons why we might need that for early break still?
Yes this FFR implementation itself never needs scalar fallback. Tamar is
obviously the expert on early break but I believe early break still
needs epilogue at least for dealing with the side effects of the last
elements in the breaking iteration.
- This prioritises the "good" case, by trying to keep
the "full read" path as tight as possible, and adding
a fixup branch to handle the case where there is a partial read.
- We preserve the state of the FFR register over the vectoried loop.
This is to prevent code written with intrinsics breaking by clobbering
their value in the FFR register. However, as the FFR is not preserved by
the AAPCS, this is nearly always optimized out.
- One downside of this approach is I dont think it will translate cleanly for
len based loop vectorization (riscv). However, as I understand it, the
riscv equivalent feature never does partial loads unless there is a genuine
fault, so we will not need to worry about recovering back to
vectorized code after a partial read (as it will either take the early break
or the fault). So no fixup should be needed and can use a subset of this.
== Remaining work to do ==
- Versioning
As FFR introduces overhead, it will always be slower than a mutually
aligned loop. So my ideal code generation for a vector with two pointers
requiring safe speculative reads is:
At O2, where we do not want to incur the code size cost of versioning,
instead just use FFR to vectorize the loop (if profitable).
At O3, version to create a mutually aligned non-FFR case and a (current code
gen)
if the pointers are mutually misaligned.
This "FFR versioning" is not yet implemented.
- Costing
I haven't done anything to cost FFR yet, we will want accurate costing of FFR
vs scalar and vs non-FFR vectorized to make the decisions on versioning and
what to use.
- Optimization of the generated code
The generated code has room for improvement, primarily using rdffrs would
be a performance gain, and reducing some of the moves within the hot
section of the loop.
== Feedback wanted ==
- Overall design of this
- What to call this within GCC (it's not really "first fault reads", maybe
"hardware safe speculative reads" (HSSR)?)
- How we can sensibly cost this to allow the choice to do scalar over
FFR.
When a load is partial you need to consider the FFR for all computations
that require masking by loop masking. Your example is simplified
to not have any, but consider a division - you show an else value
of zero, so before any use of loop_mask on an operation that is
dependent on the FFR affecting load value you need to read the FFR
and combine it with the loop mask, right?
Yes exactly, this patch series handles this and masks all the operations
using the data from the FFR and the loop mask correctly. Apologies for
the light weight example.
There is a better example in the thread with Richard S.
Additionally, we have a behaviour to mark certain operations as not
needing a fully accurate mask. Which allows them to be scheduled with
the loads rather than after the READ_FAULT_STATE.
This is used for widening conversions to allow them to be combined in
RTL to make widening first fault reads.
Sorry for the essay!
No need to sorry, it's very helpful.
ISTR AVX10.x introduces some non-faulting loads but I have to check.
Ah thats really good to know. Seems hardware safe speculative reads (my
name) are popular these days.
Thanks,
Alfie
Richard.
Bootstrapped and reg tested for aarch64, x68-64, arm32hf, riscv.
I also ran vect.exp with --param=vect-ffr-usage=2 and -msve-vector-bits=128
with no errors.
Thoughts?
King regards,
Alfie
Alfie Richards (8):
vect: Add internal functions and optabs for first fault loads
vect: Add SLP_NODE argument to vect_get_loop_mask.
vect: Make vect_maybe_permute_loop_masks optional and retargetable
vect: Add EXCLUDE_VIRTUALS argument to _slp_tree::push_vec_def.
vect: Add vect-ffr-usage param.
vect: Add FFR analysis
vect: Add FFR transformation
vect: Enable FFR
.../aarch64/aarch64-sve-builtins-base.cc | 78 +++--
gcc/config/aarch64/aarch64-sve-builtins.cc | 24 ++
gcc/config/aarch64/aarch64-sve-builtins.h | 1 +
gcc/config/aarch64/aarch64-sve.md | 201 +++++++++--
gcc/config/aarch64/aarch64.cc | 10 +
gcc/config/aarch64/iterators.md | 2 +
gcc/internal-fn.cc | 24 ++
gcc/internal-fn.def | 18 +
gcc/optabs-tree.cc | 57 +++
gcc/optabs-tree.h | 2 +
gcc/optabs.def | 5 +
gcc/params.opt | 4 +
gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c | 16 +
gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c | 35 ++
gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c | 42 +++
gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c | 25 ++
gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c | 25 ++
gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c | 43 +++
.../gcc.target/aarch64/sve/ffr_6_run.c | 81 +++++
gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c | 16 +
gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c | 16 +
gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c | 18 +
.../gcc.target/aarch64/sve/noeffect11.c | 2 +-
.../gcc.target/aarch64/sve/peel_ind_12.c | 2 +-
.../gcc.target/aarch64/sve/peel_ind_12_run.c | 2 +-
.../gcc.target/aarch64/sve/pfalse-load.c | 6 +-
gcc/tree-data-ref.cc | 4 +
gcc/tree-ssa-alias.cc | 2 +
gcc/tree-ssa-loop-ivopts.cc | 2 +
gcc/tree-vect-data-refs.cc | 3 +-
gcc/tree-vect-loop-manip.cc | 328 +++++++++++++++++-
gcc/tree-vect-loop.cc | 264 +++++++++++++-
gcc/tree-vect-slp.cc | 32 +-
gcc/tree-vect-stmts.cc | 165 +++++++--
gcc/tree-vectorizer.h | 63 +++-
35 files changed, 1496 insertions(+), 122 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_1.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_2.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_3.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_4.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_5.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_6_run.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_7.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_8.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/ffr_9.c
