Adds 4 new optabs and respective internal functions for hardware
safe speculative reads:
- mask_ff_hssr_load: a contiguous masked HSSR load
- read_hssr_state: reads the global HSSR state
- set_hssr_state: sets the global HSSR state
- detect_hssr_fault: takes a loaded HSSR state value and returns a
boolean value for if the mask implies a partial read happens.
As this is initially only for targets that support partial vectors, the
HSSR state is initially assumed to be masks.
The last optab is required to hide from the gimple representation the
value of the hssr mask. This is because, otherwise later optimizations
try use it when it is a full mask, which introduces a very unfortunate
data dependency.
gcc/ChangeLog:
* doc/ifn.texi: Document new internal functions.
* doc/md.texi: Document new optabs.
* gimple-fold.cc (gimple_fold_partial_load_store_mem_ref):
Add handling of IFN_MASK_FF_HSSR_LOAD.
(gimple_fold_partial_load_store): Likewise.
(gimple_fold_call): Likewise.
* internal-fn.cc (read_hssr_state_direct): New macro.
(set_hssr_state_direct): New macro.
(detect_hssr_fault_direct): New macro.
(expand_read_hssr_state_optab_fn): New macro.
(expand_detect_hssr_fault_optab_fn): New macro.
(expand_set_hssr_state_optab_fn): New function.
(direct_read_hssr_state_optab_supported_p): New macro.
(direct_set_hssr_state_optab_supported_p): New macro.
(direct_detect_hssr_fault_optab_supported_p): New macro.
(internal_load_fn_p): Add support for IFN_MASK_FF_HSSR_LOAD.
(internal_hssr_load_fn_p): New function.
(internal_fn_else_index): Add support for IFN_MASK_FF_HSSR_LOAD.
(internal_fn_mask_index): Add support for IFN_MASK_FF_HSSR_LOAD.
(internal_fn_alias_ptr_index): Add support for IFN_MASK_FF_HSSR_LOAD.
* internal-fn.def (MASK_FF_HSSR_LOAD): New IFN.
(SET_HSSR_STATE): New IFN.
(READ_HSSR_STATE): New IFN.
(DETECT_HSSR_FAULT): New IFN.
* internal-fn.h (internal_hssr_load_fn_p): New function.
* optabs.def (mask_ff_hssr_load_optab): New optab.
(read_hssr_state_optab): New optab.
(set_hssr_state_optab): New optab.
(detect_hssr_fault_optab): New optab.
* tree-data-ref.cc (get_references_in_stmt):
Add support for IFN_MASK_FF_HSSR_LOAD.
* tree-ssa-alias.cc (ref_maybe_used_by_call_p_1):
Add support for IFN_MASK_FF_HSSR_LOAD.
* tree-ssa-loop-ivopts.cc (get_mem_type_for_internal_fn):
Add support for IFN_MASK_FF_HSSR_LOAD.
(get_alias_ptr_type_for_ptr_address):
Add support for IFN_MASK_FF_HSSR_LOAD.
* tree-vect-slp.cc (vect_get_operand_map):
Add support for IFN_MASK_FF_HSSR_LOAD.
-- >8 --
Just to note, I fixed the issue with the test by instead making
gimple_fold_call corectly handle IFN_MASK_FF_HSSR_LOAD and fold it
in this situation.
---
gcc/doc/ifn.texi | 29 +++++++++++++++++++++++++
gcc/doc/md.texi | 42 +++++++++++++++++++++++++++++++++++++
gcc/gimple-fold.cc | 9 ++++++++
gcc/internal-fn.cc | 39 ++++++++++++++++++++++++++++++++++
gcc/internal-fn.def | 18 ++++++++++++++++
gcc/internal-fn.h | 1 +
gcc/optabs.def | 8 +++++++
gcc/tree-data-ref.cc | 2 ++
gcc/tree-ssa-alias.cc | 1 +
gcc/tree-ssa-loop-ivopts.cc | 2 ++
gcc/tree-vect-slp.cc | 3 +++
11 files changed, 154 insertions(+)
diff --git a/gcc/doc/ifn.texi b/gcc/doc/ifn.texi
index 7cfc80eecf2..0742803ccd4 100644
--- a/gcc/doc/ifn.texi
+++ b/gcc/doc/ifn.texi
@@ -122,6 +122,35 @@ instruction pattern.
The @code{IFN_MASK_LEN_LOAD} internal function directly maps to the
@code{mask_len_load} instruction pattern.
+@cindex @code{IFN_MASK_FF_HSSR_LOAD} internal function
+@item @samp{IFN_MASK_FF_HSSR_LOAD}
+The @code{IFN_MASK_FF_HSSR_LOAD} internal function directly maps to the
+@code{mask_ff_hssr_load} instruction pattern.
+
+This internal function has a VDEF to represent it changing the global
+HSSR state.
+
+@cindex @code{IFN_READ_HSSR_STATE} internal function
+@item @samp{IFN_READ_HSSR_STATE}
+The @code{IFN_READ_HSSR_STATE} internal function directly maps to the
+@code{read_hssr_state} instruction pattern.
+
+This internal function has a VUSE to represent it reading the global
+HSSR state.
+
+@cindex @code{IFN_SET_HSSR_STATE} internal function
+@item @samp{IFN_SET_HSSR_STATE}
+The @code{IFN_SET_HSSR_STATE} internal function directly maps to the
+@code{set_hssr_state} instruction pattern.
+
+This internal function has a VDEF to represent it reading the global
+HSSR state.
+
+@cindex @code{IFN_DETECT_HSSR_FAULT} internal function
+@item @samp{IFN_DETECT_HSSR_FAULT}
+The @code{IFN_DETECT_HSSR_FAULT} internal function directly maps to the
+@code{detect_hssr_fault} instruction pattern.
+
@cindex @code{IFN_SCATTER_STORE} internal function
@item @samp{IFN_SCATTER_STORE}
The @code{IFN_SCATTER_STORE} internal function directly maps to the
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index 80761b0a8a6..1dd9050b6e3 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7823,6 +7823,48 @@ of @code{QI} elements.
This pattern is not allowed to @code{FAIL}.
+@mdindex mask_ff_hssr_load@var{m}@var{n}
+@anchor{mask_ff_hssr_load@var{m}@var{n}}
+@item @samp{mask_ff_hssr_load@var{m}@var{n}}
+Perform a masked hardware-safe-speculative load of vector from memory
+operand 1 of mode @var{m} into register operand 0. The mask is provided in
+register operand 2 of mode @var{n}. Operand 3 (the ``else value'') is of mode
+@var{m} and specifies which value is loaded when the mask is unset.
+The predicate of operand 3 must only accept the else values that the target
+actually supports. Currently three values are attempted, zero, -1, and
+undefined. GCC handles an else value of zero more efficiently than -1 or
+undefined.
+
+The instruction must also set the global HSSR state so it can be read by
+@samp{read_hssr_state@var{m}}.
+
+This pattern is not allowed to @code{FAIL}.
+
+@mdindex read_hssr_state@var{m}
+@anchor{read_hssr_state@var{m}}
+@item @samp{read_hssr_state@var{m}}
+Reads the global hardware safe speculative read state as a mask of mode @var{m}
+into register operand 0.
+
+This pattern is not allowed to @code{FAIL}.
+
+@mdindex set_hssr_state@var{m}
+@anchor{set_hssr_state@var{m}}
+@item @samp{set_hssr_state@var{m}}
+Sets the global HSSR state to be the mask in operand 0 of mode @var{m}.
+
+This pattern is not allowed to @code{FAIL}.
+
+@mdindex detect_hssr_fault@var{m}
+@anchor{detect_hssr_fault@var{m}}
+@item @samp{detect_hssr_fault@var{m}}
+Checks if the given hardware safe speculative read mask of mode @var{m} in
+operand 1 contains a partial read requiring a fixup path to be taken.
+
+Sets operand 0 of mode @samp{QI} to 1 if there is, and 0 otherwise.
+
+This pattern is not allowed to @code{FAIL}.
+
@mdindex vec_perm@var{m}
@item @samp{vec_perm@var{m}}
Output a (variable) vector permutation. Operand 0 is the destination
diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
index e3df0d1163c..fbd06f9dddd 100644
--- a/gcc/gimple-fold.cc
+++ b/gcc/gimple-fold.cc
@@ -6049,6 +6049,10 @@ partial_load_store_mask_state (gcall *call, tree vectype)
static tree
gimple_fold_partial_load_store_mem_ref (gcall *call, tree vectype)
{
+ internal_fn ifn = gimple_call_internal_fn (call);
+ /* HSSR loads cannot be folded to MEM_REFs. */
+ if (internal_hssr_load_fn_p (ifn))
+ return NULL_TREE;
/* Only fold if all lanes are active (unconditional). */
if (partial_load_store_mask_state (call, vectype) != MASK_ALL_ACTIVE)
return NULL_TREE;
@@ -6106,6 +6110,10 @@ gimple_fold_partial_load_store (gimple_stmt_iterator
*gsi, gcall *call)
a store, move the vops from the old stmt to the new one. */
if (!is_gimple_reg (lhs))
gimple_move_vops (new_stmt, call);
+ /* HSSR loads have a VDEF in addition to represent changes to the
+ global HSSR state. */
+ if (internal_hssr_load_fn_p (ifn))
+ unlink_stmt_vdef (call);
gsi_replace (gsi, new_stmt, false);
return true;
}
@@ -6387,6 +6395,7 @@ gimple_fold_call (gimple_stmt_iterator *gsi, bool inplace)
case IFN_MASK_LEN_SCATTER_STORE:
case IFN_MASK_STORE_LANES:
case IFN_MASK_LEN_STORE_LANES:
+ case IFN_MASK_FF_HSSR_LOAD:
changed |= gimple_fold_partial_load_store (gsi, stmt);
break;
default:
diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc
index 3fdf89e0e6e..6f451f9e3ff 100644
--- a/gcc/internal-fn.cc
+++ b/gcc/internal-fn.cc
@@ -197,6 +197,9 @@ init_internal_fns ()
#define crc_direct { 1, -1, true }
#define reduc_sbool_direct { 0, 0, true }
#define select_vl_direct { 2, 0, false }
+#define read_hssr_state_direct { -1, -1, false }
+#define set_hssr_state_direct { 0, 0, false }
+#define detect_hssr_fault_direct { 0, 0, false }
const direct_internal_fn_info direct_internal_fn_array[IFN_LAST + 1] = {
#define DEF_INTERNAL_FN(CODE, FLAGS, FNSPEC) not_direct,
@@ -4317,6 +4320,21 @@ expand_reduc_sbool_optab_fn (internal_fn fn, gcall
*stmt, direct_optab optab)
#define expand_select_vl_optab_fn(FN, STMT, OPTAB) \
expand_convert_optab_fn (FN, STMT, OPTAB, 3)
+#define expand_read_hssr_state_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 0)
+
+#define expand_detect_hssr_fault_optab_fn(FN, STMT, OPTAB) \
+ expand_direct_optab_fn (FN, STMT, OPTAB, 1)
+
+static void
+expand_set_hssr_state_optab_fn (internal_fn fn, gcall *stmt,
+ direct_optab optab)
+{
+ tree_pair types = direct_internal_fn_types (fn, stmt);
+ insn_code icode = direct_optab_handler (optab, TYPE_MODE (types.first));
+ expand_fn_using_insn (stmt, icode, 0, 1);
+}
+
/* Expanders for optabs that can use expand_convert_optab_fn. */
#define expand_unary_convert_optab_fn(FN, STMT, OPTAB) \
@@ -4434,6 +4452,9 @@ multi_vector_optab_supported_p (convert_optab optab,
tree_pair types,
#define direct_vec_extract_optab_supported_p convert_optab_supported_p
#define direct_reduc_sbool_optab_supported_p direct_optab_supported_p
#define direct_select_vl_optab_supported_p convert_optab_supported_p
+#define direct_read_hssr_state_optab_supported_p direct_optab_supported_p
+#define direct_set_hssr_state_optab_supported_p direct_optab_supported_p
+#define direct_detect_hssr_fault_optab_supported_p direct_optab_supported_p
/* Return the optab used by internal function FN. */
@@ -5083,6 +5104,7 @@ internal_load_fn_p (internal_fn fn)
case IFN_MASK_LEN_GATHER_LOAD:
case IFN_LEN_LOAD:
case IFN_MASK_LEN_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
return true;
default:
@@ -5090,6 +5112,20 @@ internal_load_fn_p (internal_fn fn)
}
}
+/* Return true if IFN is a HSSR load. */
+
+bool
+internal_hssr_load_fn_p (internal_fn fn)
+{
+ switch (fn)
+ {
+ case IFN_MASK_FF_HSSR_LOAD:
+ return true;
+ default:
+ return false;
+ }
+}
+
/* Return true if IFN is some form of store to memory. */
bool
@@ -5255,6 +5291,7 @@ internal_fn_else_index (internal_fn fn)
case IFN_MASK_LEN_LOAD:
case IFN_MASK_LOAD_LANES:
case IFN_MASK_LEN_LOAD_LANES:
+ case IFN_MASK_FF_HSSR_LOAD:
return 3;
case IFN_COND_FMA:
@@ -5295,6 +5332,7 @@ internal_fn_mask_index (internal_fn fn)
case IFN_MASK_LEN_STORE_LANES:
case IFN_MASK_LEN_LOAD:
case IFN_MASK_LEN_STORE:
+ case IFN_MASK_FF_HSSR_LOAD:
return 2;
case IFN_MASK_LEN_STRIDED_LOAD:
@@ -5363,6 +5401,7 @@ internal_fn_alias_ptr_index (internal_fn fn)
case IFN_SCATTER_STORE:
case IFN_MASK_SCATTER_STORE:
case IFN_MASK_LEN_SCATTER_STORE:
+ case IFN_MASK_FF_HSSR_LOAD:
return 1;
default:
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index 28ad5b7809d..68397c7bced 100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -651,6 +651,24 @@ DEF_INTERNAL_FN (BITINTTOFLOAT, ECF_PURE | ECF_LEAF, ". R
. ")
DEF_INTERNAL_OPTAB_FN (BIT_ANDN, ECF_CONST, andn, binary)
DEF_INTERNAL_OPTAB_FN (BIT_IORN, ECF_CONST, iorn, binary)
+/* Hardware Safe Speculative Reads internal functions. */
+DEF_INTERNAL_OPTAB_FN (MASK_FF_HSSR_LOAD,
+ 0,
+ mask_ff_hssr_load,
+ mask_load)
+DEF_INTERNAL_OPTAB_FN (SET_HSSR_STATE,
+ ECF_NOTHROW,
+ set_hssr_state,
+ set_hssr_state)
+DEF_INTERNAL_OPTAB_FN (READ_HSSR_STATE,
+ ECF_NOTHROW,
+ read_hssr_state,
+ read_hssr_state)
+DEF_INTERNAL_OPTAB_FN (DETECT_HSSR_FAULT,
+ ECF_CONST,
+ detect_hssr_fault,
+ detect_hssr_fault)
+
#undef DEF_INTERNAL_WIDENING_OPTAB_FN
#undef DEF_INTERNAL_SIGNED_COND_FN
#undef DEF_INTERNAL_COND_FN
diff --git a/gcc/internal-fn.h b/gcc/internal-fn.h
index 30930f388b0..d6a28a47d55 100644
--- a/gcc/internal-fn.h
+++ b/gcc/internal-fn.h
@@ -234,6 +234,7 @@ extern bool can_interpret_as_conditional_op_p (gimple *,
tree *,
tree *, tree *, tree *);
extern bool internal_load_fn_p (internal_fn);
+extern bool internal_hssr_load_fn_p (internal_fn);
extern bool internal_store_fn_p (internal_fn);
extern bool internal_gather_scatter_fn_p (internal_fn);
extern int internal_fn_mask_index (internal_fn);
diff --git a/gcc/optabs.def b/gcc/optabs.def
index d1f70a9b44a..4ae3ef8b78d 100644
--- a/gcc/optabs.def
+++ b/gcc/optabs.def
@@ -118,6 +118,9 @@ OPTAB_CD (usdot_prod_optab, "usdot_prod$I$a$b")
OPTAB_CD (while_ult_optab, "while_ult$a$b")
OPTAB_CD (select_vl_optab, "select_vl$a$b")
+/* Hardware Safe Speculative Reads optabs. */
+OPTAB_CD (mask_ff_hssr_load_optab, "mask_ff_hssr_load$a$b")
+
OPTAB_NL(add_optab, "add$P$a3", PLUS, "add", '3', gen_int_fp_fixed_libfunc)
OPTAB_NX(add_optab, "add$F$a3")
OPTAB_NX(add_optab, "add$Q$a3")
@@ -241,6 +244,11 @@ OPTAB_D (push_optab, "push$a1")
OPTAB_D (reload_in_optab, "reload_in$a")
OPTAB_D (reload_out_optab, "reload_out$a")
+/* Hardware Safe Speculative Reads optabs. */
+OPTAB_D (read_hssr_state_optab, "read_hssr_state$a")
+OPTAB_D (set_hssr_state_optab, "set_hssr_state$a")
+OPTAB_D (detect_hssr_fault_optab, "detect_hssr_fault$a")
+
OPTAB_DC(cbranch_optab, "cbranch$a4", COMPARE)
OPTAB_D (tbranch_eq_optab, "tbranch_eq$a3")
OPTAB_D (tbranch_ne_optab, "tbranch_ne$a3")
diff --git a/gcc/tree-data-ref.cc b/gcc/tree-data-ref.cc
index d6da0fe58c3..4100c66521b 100644
--- a/gcc/tree-data-ref.cc
+++ b/gcc/tree-data-ref.cc
@@ -5905,6 +5905,7 @@ get_references_in_stmt (gimple *stmt, vec<data_ref_loc,
va_heap> *references)
break;
}
case IFN_MASK_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
case IFN_MASK_STORE:
break;
case IFN_MASK_CALL:
@@ -5962,6 +5963,7 @@ get_references_in_stmt (gimple *stmt, vec<data_ref_loc,
va_heap> *references)
switch (gimple_call_internal_fn (stmt))
{
case IFN_MASK_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
if (gimple_call_lhs (stmt) == NULL_TREE)
break;
ref.is_read = true;
diff --git a/gcc/tree-ssa-alias.cc b/gcc/tree-ssa-alias.cc
index 7fc87b3b944..6122a6505c0 100644
--- a/gcc/tree-ssa-alias.cc
+++ b/gcc/tree-ssa-alias.cc
@@ -2888,6 +2888,7 @@ ref_maybe_used_by_call_p_1 (gcall *call, ao_ref *ref,
bool tbaa_p)
case IFN_MASK_LEN_STORE_LANES:
goto process_args;
case IFN_MASK_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
case IFN_LEN_LOAD:
case IFN_MASK_LEN_LOAD:
case IFN_MASK_LOAD_LANES:
diff --git a/gcc/tree-ssa-loop-ivopts.cc b/gcc/tree-ssa-loop-ivopts.cc
index fa3c2c136fc..3765b7070e9 100644
--- a/gcc/tree-ssa-loop-ivopts.cc
+++ b/gcc/tree-ssa-loop-ivopts.cc
@@ -2371,6 +2371,7 @@ get_mem_type_for_internal_fn (gcall *call, tree *op_p)
switch (gimple_call_internal_fn (call))
{
case IFN_MASK_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
case IFN_MASK_LOAD_LANES:
case IFN_MASK_LEN_LOAD_LANES:
case IFN_LEN_LOAD:
@@ -7639,6 +7640,7 @@ get_alias_ptr_type_for_ptr_address (iv_use *use)
switch (gimple_call_internal_fn (call))
{
case IFN_MASK_LOAD:
+ case IFN_MASK_FF_HSSR_LOAD:
case IFN_MASK_STORE:
case IFN_MASK_LOAD_LANES:
case IFN_MASK_STORE_LANES:
diff --git a/gcc/tree-vect-slp.cc b/gcc/tree-vect-slp.cc
index eee02124d5b..e0a32ade79a 100644
--- a/gcc/tree-vect-slp.cc
+++ b/gcc/tree-vect-slp.cc
@@ -587,6 +587,9 @@ vect_get_operand_map (const gimple *stmt, bool
gather_scatter_p,
case IFN_MASK_LOAD:
return gather_scatter_p ? off_arg2_arg3_map : arg2_arg3_map;
+ case IFN_MASK_FF_HSSR_LOAD:
+ return arg2_arg3_map;
+
case IFN_GATHER_LOAD:
return arg2_map;
--
2.43.0
