> -----Original Message-----
> From: Alfie Richards <[email protected]>
> Sent: 17 September 2026 15:40
> To: [email protected]
> Cc: Alex Coplan <[email protected]>; Alice Carlotti
> <[email protected]>; Andrea Pinski
> <[email protected]>; Richard Biener <[email protected]>;
> Tamar Christina <[email protected]>; Zdenek Dvorak <[email protected]>;
> Robin Dapp <[email protected]>; Richard Sandiford
> <[email protected]>; Kyrylo Tkachov <[email protected]>;
> Alfie Richards <[email protected]>
> Subject: [PATCH v3 03/16] ifn: optab: Add HSSR ifns and optabs
>
> Adds 4 new optabs and respective internal functions for hardware
> safe speculative reads:
>
> - mask_ff_hssr_load: a contiguous masked HSSR load
> - read_hssr_state: reads the global HSSR state
> - set_hssr_state: sets the global HSSR state
> - detect_hssr_fault: takes a loaded HSSR state value and returns a
> boolean value for if the mask implies a partial read happens.
>
> As this is initially only for targets that support partial vectors, the
> HSSR state is initially assumed to be masks.
>
> The last optab is required to hide from the gimple representation the
> value of the hssr mask. This is because, otherwise later optimizations
> try use it when it is a full mask, which introduces a very unfortunate
> data dependency.
>
> gcc/ChangeLog:
>
> * doc/ifn.texi: Document new internal functions.
> * doc/md.texi: Document new optabs.
> * gimple-fold.cc (gimple_fold_partial_load_store_mem_ref):
> Add handling of IFN_MASK_FF_HSSR_LOAD.
> (gimple_fold_partial_load_store): Likewise.
> (gimple_fold_call): Likewise.
> * internal-fn.cc (read_hssr_state_direct): New macro.
> (set_hssr_state_direct): New macro.
> (detect_hssr_fault_direct): New macro.
> (expand_read_hssr_state_optab_fn): New macro.
> (expand_detect_hssr_fault_optab_fn): New macro.
> (expand_set_hssr_state_optab_fn): New function.
> (direct_read_hssr_state_optab_supported_p): New macro.
> (direct_set_hssr_state_optab_supported_p): New macro.
> (direct_detect_hssr_fault_optab_supported_p): New macro.
> (internal_load_fn_p): Add support for IFN_MASK_FF_HSSR_LOAD.
> (internal_hssr_load_fn_p): New function.
> (internal_fn_else_index): Add support for IFN_MASK_FF_HSSR_LOAD.
> (internal_fn_mask_index): Add support for
> IFN_MASK_FF_HSSR_LOAD.
> (internal_fn_alias_ptr_index): Add support for
> IFN_MASK_FF_HSSR_LOAD.
> * internal-fn.def (MASK_FF_HSSR_LOAD): New IFN.
> (SET_HSSR_STATE): New IFN.
> (READ_HSSR_STATE): New IFN.
> (DETECT_HSSR_FAULT): New IFN.
> * internal-fn.h (internal_hssr_load_fn_p): New function.
> * optabs.def (mask_ff_hssr_load_optab): New optab.
> (read_hssr_state_optab): New optab.
> (set_hssr_state_optab): New optab.
> (detect_hssr_fault_optab): New optab.
> * tree-data-ref.cc (get_references_in_stmt):
> Add support for IFN_MASK_FF_HSSR_LOAD.
> * tree-ssa-alias.cc (ref_maybe_used_by_call_p_1):
> Add support for IFN_MASK_FF_HSSR_LOAD.
> * tree-ssa-loop-ivopts.cc (get_mem_type_for_internal_fn):
> Add support for IFN_MASK_FF_HSSR_LOAD.
> (get_alias_ptr_type_for_ptr_address):
> Add support for IFN_MASK_FF_HSSR_LOAD.
> * tree-vect-slp.cc (vect_get_operand_map):
> Add support for IFN_MASK_FF_HSSR_LOAD.
>
> -- >8 --
>
> Just to note, I fixed the issue with the test by instead making
> gimple_fold_call corectly handle IFN_MASK_FF_HSSR_LOAD and fold it
> in this situation.
Oro? What issue? Cover letter doesn't mention any :)
> ---
> gcc/doc/ifn.texi | 29 +++++++++++++++++++++++++
> gcc/doc/md.texi | 42 +++++++++++++++++++++++++++++++++++++
> gcc/gimple-fold.cc | 9 ++++++++
> gcc/internal-fn.cc | 39 ++++++++++++++++++++++++++++++++++
> gcc/internal-fn.def | 18 ++++++++++++++++
> gcc/internal-fn.h | 1 +
> gcc/optabs.def | 8 +++++++
> gcc/tree-data-ref.cc | 2 ++
> gcc/tree-ssa-alias.cc | 1 +
> gcc/tree-ssa-loop-ivopts.cc | 2 ++
> gcc/tree-vect-slp.cc | 3 +++
> 11 files changed, 154 insertions(+)
>
> diff --git a/gcc/doc/ifn.texi b/gcc/doc/ifn.texi
> index 7cfc80eecf2..0742803ccd4 100644
> --- a/gcc/doc/ifn.texi
> +++ b/gcc/doc/ifn.texi
> @@ -122,6 +122,35 @@ instruction pattern.
> The @code{IFN_MASK_LEN_LOAD} internal function directly maps to the
> @code{mask_len_load} instruction pattern.
>
> +@cindex @code{IFN_MASK_FF_HSSR_LOAD} internal function
> +@item @samp{IFN_MASK_FF_HSSR_LOAD}
> +The @code{IFN_MASK_FF_HSSR_LOAD} internal function directly maps to
> the
> +@code{mask_ff_hssr_load} instruction pattern.
We've now started documenting IFNs more usefully these days. See e.g.
https://patchwork.sourceware.org/project/gcc/patch/[email protected]/
also what does the _ff_ stand for? If it's first_faulting, is that not already
implicit in the
hssr?
> +
> +This internal function has a VDEF to represent it changing the global
> +HSSR state.
> +
> +@cindex @code{IFN_READ_HSSR_STATE} internal function
> +@item @samp{IFN_READ_HSSR_STATE}
> +The @code{IFN_READ_HSSR_STATE} internal function directly maps to the
> +@code{read_hssr_state} instruction pattern.
> +
> +This internal function has a VUSE to represent it reading the global
> +HSSR state.
> +
> +@cindex @code{IFN_SET_HSSR_STATE} internal function
> +@item @samp{IFN_SET_HSSR_STATE}
> +The @code{IFN_SET_HSSR_STATE} internal function directly maps to the
> +@code{set_hssr_state} instruction pattern.
> +
> +This internal function has a VDEF to represent it reading the global
> +HSSR state.
> +
> +@cindex @code{IFN_DETECT_HSSR_FAULT} internal function
> +@item @samp{IFN_DETECT_HSSR_FAULT}
> +The @code{IFN_DETECT_HSSR_FAULT} internal function directly maps to the
> +@code{detect_hssr_fault} instruction pattern.
> +
> @cindex @code{IFN_SCATTER_STORE} internal function
> @item @samp{IFN_SCATTER_STORE}
> The @code{IFN_SCATTER_STORE} internal function directly maps to the
> diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
> index 80761b0a8a6..1dd9050b6e3 100644
> --- a/gcc/doc/md.texi
> +++ b/gcc/doc/md.texi
> @@ -7823,6 +7823,48 @@ of @code{QI} elements.
>
> This pattern is not allowed to @code{FAIL}.
>
> +@mdindex mask_ff_hssr_load@var{m}@var{n}
> +@anchor{mask_ff_hssr_load@var{m}@var{n}}
> +@item @samp{mask_ff_hssr_load@var{m}@var{n}}
> +Perform a masked hardware-safe-speculative load of vector from memory
> +operand 1 of mode @var{m} into register operand 0. The mask is provided in
> +register operand 2 of mode @var{n}. Operand 3 (the ``else value'') is of
> mode
> +@var{m} and specifies which value is loaded when the mask is unset.
> +The predicate of operand 3 must only accept the else values that the target
> +actually supports. Currently three values are attempted, zero, -1, and
> +undefined. GCC handles an else value of zero more efficiently than -1 or
> +undefined.
> +
> +The instruction must also set the global HSSR state so it can be read by
> +@samp{read_hssr_state@var{m}}.
> +
> +This pattern is not allowed to @code{FAIL}.
> +
> +@mdindex read_hssr_state@var{m}
> +@anchor{read_hssr_state@var{m}}
> +@item @samp{read_hssr_state@var{m}}
> +Reads the global hardware safe speculative read state as a mask of mode
> @var{m}
> +into register operand 0.
> +
> +This pattern is not allowed to @code{FAIL}.
> +
> +@mdindex set_hssr_state@var{m}
> +@anchor{set_hssr_state@var{m}}
> +@item @samp{set_hssr_state@var{m}}
> +Sets the global HSSR state to be the mask in operand 0 of mode @var{m}.
> +
> +This pattern is not allowed to @code{FAIL}.
> +
> +@mdindex detect_hssr_fault@var{m}
> +@anchor{detect_hssr_fault@var{m}}
> +@item @samp{detect_hssr_fault@var{m}}
> +Checks if the given hardware safe speculative read mask of mode @var{m} in
> +operand 1 contains a partial read requiring a fixup path to be taken.
> +
> +Sets operand 0 of mode @samp{QI} to 1 if there is, and 0 otherwise.
> +
> +This pattern is not allowed to @code{FAIL}.
So this is purely a boolean? Why not make it a vect param? That allows for
better
testing and experimentation and would allow a backend to set it based on a CPU
cost model.
I assume the branch is on the actual result of the read_hssr_state? So codegen
is
state <- read_hssr_state
if (state && param_hssr_avoid_dependency)
...
use state.
else
state = param_hssr_avoid_dependency ? loop_mask : state;
...
so that when 0 it folds the branch away?
I haven't looked at it's usages yet but I am assuming that's sorta what it
looks like?
Thanks,
Tamar
> +
> @mdindex vec_perm@var{m}
> @item @samp{vec_perm@var{m}}
> Output a (variable) vector permutation. Operand 0 is the destination
> diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
> index e3df0d1163c..fbd06f9dddd 100644
> --- a/gcc/gimple-fold.cc
> +++ b/gcc/gimple-fold.cc
> @@ -6049,6 +6049,10 @@ partial_load_store_mask_state (gcall *call, tree
> vectype)
> static tree
> gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype)
> {
> + internal_fn ifn = gimple_call_internal_fn (call);
> + /* HSSR loads cannot be folded to MEM_REFs. */
> + if (internal_hssr_load_fn_p (ifn))
> + return NULL_TREE;
> /* Only fold if all lanes are active (unconditional). */
> if (partial_load_store_mask_state (call, vectype) != MASK_ALL_ACTIVE)
> return NULL_TREE;
> @@ -6106,6 +6110,10 @@ gimple_fold_partial_load_store
> (gimple_stmt_iterator *gsi, gcall *call)
> a store, move the vops from the old stmt to the new one. */
> if (!is_gimple_reg (lhs))
> gimple_move_vops (new_stmt, call);
> + /* HSSR loads have a VDEF in addition to represent changes to the
> + global HSSR state. */
> + if (internal_hssr_load_fn_p (ifn))
> + unlink_stmt_vdef (call);
> gsi_replace (gsi, new_stmt, false);
> return true;
> }
> @@ -6387,6 +6395,7 @@ gimple_fold_call (gimple_stmt_iterator *gsi, bool
> inplace)
> case IFN_MASK_LEN_SCATTER_STORE:
> case IFN_MASK_STORE_LANES:
> case IFN_MASK_LEN_STORE_LANES:
> + case IFN_MASK_FF_HSSR_LOAD:
> changed |= gimple_fold_partial_load_store (gsi, stmt);
> break;
> default:
> diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
> index 3fdf89e0e6e..6f451f9e3ff 100644
> --- a/gcc/internal-fn.cc
> +++ b/gcc/internal-fn.cc
> @@ -197,6 +197,9 @@ init_internal_fns ()
> #define crc_direct { 1, -1, true }
> #define reduc_sbool_direct { 0, 0, true }
> #define select_vl_direct { 2, 0, false }
> +#define read_hssr_state_direct { -1, -1, false }
> +#define set_hssr_state_direct { 0, 0, false }
> +#define detect_hssr_fault_direct { 0, 0, false }
>
> const direct_internal_fn_info direct_internal_fn_array[IFN_LAST + 1] = {
> #define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) not_direct,
> @@ -4317,6 +4320,21 @@ expand_reduc_sbool_optab_fn (internal_fn fn,
> gcall *stmt, direct_optab optab)
> #define expand_select_vl_optab_fn(FN, STMT, OPTAB) \
> expand_convert_optab_fn (FN, STMT, OPTAB, 3)
>
> +#define expand_read_hssr_state_optab_fn(FN, STMT, OPTAB) \
> + expand_direct_optab_fn (FN, STMT, OPTAB, 0)
> +
> +#define expand_detect_hssr_fault_optab_fn(FN, STMT, OPTAB) \
> + expand_direct_optab_fn (FN, STMT, OPTAB, 1)
> +
> +static void
> +expand_set_hssr_state_optab_fn (internal_fn fn, gcall *stmt,
> + direct_optab optab)
> +{
> + tree_pair types = direct_internal_fn_types (fn, stmt);
> + insn_code icode = direct_optab_handler (optab, TYPE_MODE (types.first));
> + expand_fn_using_insn (stmt, icode, 0, 1);
> +}
> +
> /* Expanders for optabs that can use expand_convert_optab_fn. */
>
> #define expand_unary_convert_optab_fn(FN, STMT, OPTAB) \
> @@ -4434,6 +4452,9 @@ multi_vector_optab_supported_p (convert_optab
> optab, tree_pair types,
> #define direct_vec_extract_optab_supported_p convert_optab_supported_p
> #define direct_reduc_sbool_optab_supported_p direct_optab_supported_p
> #define direct_select_vl_optab_supported_p convert_optab_supported_p
> +#define direct_read_hssr_state_optab_supported_p
> direct_optab_supported_p
> +#define direct_set_hssr_state_optab_supported_p
> direct_optab_supported_p
> +#define direct_detect_hssr_fault_optab_supported_p
> direct_optab_supported_p
>
> /* Return the optab used by internal function FN. */
>
> @@ -5083,6 +5104,7 @@ internal_load_fn_p (internal_fn fn)
> case IFN_MASK_LEN_GATHER_LOAD:
> case IFN_LEN_LOAD:
> case IFN_MASK_LEN_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> return true;
>
> default:
> @@ -5090,6 +5112,20 @@ internal_load_fn_p (internal_fn fn)
> }
> }
>
> +/* Return true if IFN is a HSSR load. */
> +
> +bool
> +internal_hssr_load_fn_p (internal_fn fn)
> +{
> + switch (fn)
> + {
> + case IFN_MASK_FF_HSSR_LOAD:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> /* Return true if IFN is some form of store to memory. */
>
> bool
> @@ -5255,6 +5291,7 @@ internal_fn_else_index (internal_fn fn)
> case IFN_MASK_LEN_LOAD:
> case IFN_MASK_LOAD_LANES:
> case IFN_MASK_LEN_LOAD_LANES:
> + case IFN_MASK_FF_HSSR_LOAD:
> return 3;
>
> case IFN_COND_FMA:
> @@ -5295,6 +5332,7 @@ internal_fn_mask_index (internal_fn fn)
> case IFN_MASK_LEN_STORE_LANES:
> case IFN_MASK_LEN_LOAD:
> case IFN_MASK_LEN_STORE:
> + case IFN_MASK_FF_HSSR_LOAD:
> return 2;
>
> case IFN_MASK_LEN_STRIDED_LOAD:
> @@ -5363,6 +5401,7 @@ internal_fn_alias_ptr_index (internal_fn fn)
> case IFN_SCATTER_STORE:
> case IFN_MASK_SCATTER_STORE:
> case IFN_MASK_LEN_SCATTER_STORE:
> + case IFN_MASK_FF_HSSR_LOAD:
> return 1;
>
> default:
> diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
> index 28ad5b7809d..68397c7bced 100644
> --- a/gcc/internal-fn.def
> +++ b/gcc/internal-fn.def
> @@ -651,6 +651,24 @@ DEF_INTERNAL_FN (BITINTTOFLOAT, ECF_PURE |
> ECF_LEAF, ". R . ")
> DEF_INTERNAL_OPTAB_FN (BIT_ANDN, ECF_CONST, andn, binary)
> DEF_INTERNAL_OPTAB_FN (BIT_IORN, ECF_CONST, iorn, binary)
>
> +/* Hardware Safe Speculative Reads internal functions. */
> +DEF_INTERNAL_OPTAB_FN (MASK_FF_HSSR_LOAD,
> + 0,
> + mask_ff_hssr_load,
> + mask_load)
> +DEF_INTERNAL_OPTAB_FN (SET_HSSR_STATE,
> + ECF_NOTHROW,
> + set_hssr_state,
> + set_hssr_state)
> +DEF_INTERNAL_OPTAB_FN (READ_HSSR_STATE,
> + ECF_NOTHROW,
> + read_hssr_state,
> + read_hssr_state)
> +DEF_INTERNAL_OPTAB_FN (DETECT_HSSR_FAULT,
> + ECF_CONST,
> + detect_hssr_fault,
> + detect_hssr_fault)
> +
> #undef DEF_INTERNAL_WIDENING_OPTAB_FN
> #undef DEF_INTERNAL_SIGNED_COND_FN
> #undef DEF_INTERNAL_COND_FN
> diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
> index 30930f388b0..d6a28a47d55 100644
> --- a/gcc/internal-fn.h
> +++ b/gcc/internal-fn.h
> @@ -234,6 +234,7 @@ extern bool can_interpret_as_conditional_op_p
> (gimple *, tree *,
> tree *, tree *, tree *);
>
> extern bool internal_load_fn_p (internal_fn);
> +extern bool internal_hssr_load_fn_p (internal_fn);
> extern bool internal_store_fn_p (internal_fn);
> extern bool internal_gather_scatter_fn_p (internal_fn);
> extern int internal_fn_mask_index (internal_fn);
> diff --git a/gcc/optabs.def b/gcc/optabs.def
> index d1f70a9b44a..4ae3ef8b78d 100644
> --- a/gcc/optabs.def
> +++ b/gcc/optabs.def
> @@ -118,6 +118,9 @@ OPTAB_CD (usdot_prod_optab,
> "usdot_prod$I$a$b")
> OPTAB_CD (while_ult_optab, "while_ult$a$b")
> OPTAB_CD (select_vl_optab, "select_vl$a$b")
>
> +/* Hardware Safe Speculative Reads optabs. */
> +OPTAB_CD (mask_ff_hssr_load_optab, "mask_ff_hssr_load$a$b")
> +
> OPTAB_NL(add_optab, "add$P$a3", PLUS, "add", '3',
> gen_int_fp_fixed_libfunc)
> OPTAB_NX(add_optab, "add$F$a3")
> OPTAB_NX(add_optab, "add$Q$a3")
> @@ -241,6 +244,11 @@ OPTAB_D (push_optab, "push$a1")
> OPTAB_D (reload_in_optab, "reload_in$a")
> OPTAB_D (reload_out_optab, "reload_out$a")
>
> +/* Hardware Safe Speculative Reads optabs. */
> +OPTAB_D (read_hssr_state_optab, "read_hssr_state$a")
> +OPTAB_D (set_hssr_state_optab, "set_hssr_state$a")
> +OPTAB_D (detect_hssr_fault_optab, "detect_hssr_fault$a")
> +
> OPTAB_DC(cbranch_optab, "cbranch$a4", COMPARE)
> OPTAB_D (tbranch_eq_optab, "tbranch_eq$a3")
> OPTAB_D (tbranch_ne_optab, "tbranch_ne$a3")
> diff --git a/gcc/tree-data-ref.cc b/gcc/tree-data-ref.cc
> index d6da0fe58c3..4100c66521b 100644
> --- a/gcc/tree-data-ref.cc
> +++ b/gcc/tree-data-ref.cc
> @@ -5905,6 +5905,7 @@ get_references_in_stmt (gimple *stmt,
> vec<data_ref_loc, va_heap> *references)
> break;
> }
> case IFN_MASK_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> case IFN_MASK_STORE:
> break;
> case IFN_MASK_CALL:
> @@ -5962,6 +5963,7 @@ get_references_in_stmt (gimple *stmt,
> vec<data_ref_loc, va_heap> *references)
> switch (gimple_call_internal_fn (stmt))
> {
> case IFN_MASK_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> if (gimple_call_lhs (stmt) == NULL_TREE)
> break;
> ref.is_read = true;
> diff --git a/gcc/tree-ssa-alias.cc b/gcc/tree-ssa-alias.cc
> index 7fc87b3b944..6122a6505c0 100644
> --- a/gcc/tree-ssa-alias.cc
> +++ b/gcc/tree-ssa-alias.cc
> @@ -2888,6 +2888,7 @@ ref_maybe_used_by_call_p_1 (gcall *call, ao_ref
> *ref, bool tbaa_p)
> case IFN_MASK_LEN_STORE_LANES:
> goto process_args;
> case IFN_MASK_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> case IFN_LEN_LOAD:
> case IFN_MASK_LEN_LOAD:
> case IFN_MASK_LOAD_LANES:
> diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc
> index fa3c2c136fc..3765b7070e9 100644
> --- a/gcc/tree-ssa-loop-ivopts.cc
> +++ b/gcc/tree-ssa-loop-ivopts.cc
> @@ -2371,6 +2371,7 @@ get_mem_type_for_internal_fn (gcall *call, tree
> *op_p)
> switch (gimple_call_internal_fn (call))
> {
> case IFN_MASK_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> case IFN_MASK_LOAD_LANES:
> case IFN_MASK_LEN_LOAD_LANES:
> case IFN_LEN_LOAD:
> @@ -7639,6 +7640,7 @@ get_alias_ptr_type_for_ptr_address (iv_use *use)
> switch (gimple_call_internal_fn (call))
> {
> case IFN_MASK_LOAD:
> + case IFN_MASK_FF_HSSR_LOAD:
> case IFN_MASK_STORE:
> case IFN_MASK_LOAD_LANES:
> case IFN_MASK_STORE_LANES:
> diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
> index eee02124d5b..e0a32ade79a 100644
> --- a/gcc/tree-vect-slp.cc
> +++ b/gcc/tree-vect-slp.cc
> @@ -587,6 +587,9 @@ vect_get_operand_map (const gimple *stmt, bool
> gather_scatter_p,
> case IFN_MASK_LOAD:
> return gather_scatter_p ? off_arg2_arg3_map : arg2_arg3_map;
>
> + case IFN_MASK_FF_HSSR_LOAD:
> + return arg2_arg3_map;
> +
> case IFN_GATHER_LOAD:
> return arg2_map;
>
> --
> 2.43.0
