Update to match the change in initialisation for patch 5/X. MM
---
There are four main features to this change:
1) Check pointer tags match address tags.
When sanitizing for hwasan we now put HWASAN_CHECK internal functions before
memory accesses in the `asan` pass. This checks that a tag in the pointer
being used match the tag stored in shadow memory for the memory region being
used.
These internal functions are expanded into actual checks in the sanopt
pass that happens just before expansion into RTL.
We use the same mechanism that currently inserts ASAN_CHECK internal
functions to insert the new HWASAN_CHECK functions.
2) Instrument known builtin function calls.
Handle all builtin functions that we know use memory accesses.
This commit uses the machinery added for ASAN to identify builtin
functions that access memory.
The main differences between the approaches for HWASAN and ASAN are:
- libhwasan intercepts much less builtin functions.
- Alloca needs to be transformed differently (instead of adding
redzones it needs to tag shadow memory and return a tagged pointer).
- stack_restore needs to untag the shadow stack between the current
position and where it's going.
- `noreturn` functions can not be handled by simply unpoisoning the
entire shadow stack -- there is no "always valid" tag.
(exceptions and things such as longjmp need to be handled in a
different way, usually in the runtime).
For hardware implemented checking (such as AArch64's memory tagging
extension) alloca and stack_restore will need to be handled by hooks in
the backend rather than transformation at the gimple level. This will
allow architecture specific handling of such stack modifications.
3) Introduce HWASAN block-scope poisoning
Here we use exactly the same mechanism as ASAN_MARK to poison/unpoison
variables on entry/exit of a block.
In order to simply use the exact same machinery we're using the same
internal functions until the SANOPT pass. This means that all handling
of ASAN_MARK is the same.
This has the negative that the naming may be a little confusing, but a
positive that handling of the internal function doesn't have to be
duplicated for a function that behaves exactly the same but has a
different name.
gcc/ChangeLog:
* asan.c (asan_instrument_reads): New.
(asan_instrument_writes): New.
(asan_memintrin): New.
(handle_builtin_stack_restore): Account for HWASAN.
(hwasan_emit_round_up): New.
(handle_builtin_alloca): Account for HWASAN.
(get_mem_refs_of_builtin_call): Special case strlen for HWASAN.
(hwasan_instrument_reads): New.
(hwasan_instrument_writes): New.
(hwasan_memintrin): New.
(report_error_func): Assert not HWASAN.
(build_check_stmt): Make HWASAN_CHECK instead of ASAN_CHECK.
(instrument_derefs): HWASAN does not tag globals.
(instrument_builtin_call): Use new helper functions.
(maybe_instrument_call): Don't instrument `noreturn` functions.
(initialize_sanitizer_builtins): Add new type.
(asan_expand_mark_ifn): Account for HWASAN.
(asan_expand_check_ifn): Assert never called by HWASAN.
(asan_expand_poison_ifn): Account for HWASAN.
(asan_instrument): Branch based on whether using HWASAN or ASAN.
(pass_asan::gate): Return true if sanitizing HWASAN.
(pass_asan_O0::gate): Return true if sanitizing HWASAN.
(hwasan_check_func): New.
(hwasan_expand_check_ifn): New.
(hwasan_expand_mark_ifn): New.
(gate_hwasan): New.
* asan.h (hwasan_expand_check_ifn): New decl.
(hwasan_expand_mark_ifn): New decl.
(gate_hwasan): New decl.
(asan_intercepted_p): Always false for hwasan.
(asan_sanitize_use_after_scope): Account for HWASAN.
* builtin-types.def (BT_FN_PTR_CONST_PTR_UINT8): New.
* gimple-pretty-print.c (dump_gimple_call_args): Account for
HWASAN.
* gimplify.c (asan_poison_variable): Account for HWASAN.
(gimplify_function_tree): Remove requirement of
SANITIZE_ADDRESS, requiring asan or hwasan is accounted for in
`asan_sanitize_use_after_scope`.
* internal-fn.c (expand_HWASAN_CHECK): New.
(expand_HWASAN_ALLOCA_UNPOISON): New.
(expand_HWASAN_CHOOSE_TAG): New.
(expand_HWASAN_MARK): New.
(expand_HWASAN_SET_TAG): New.
* internal-fn.def (HWASAN_ALLOCA_UNPOISON): New.
(HWASAN_CHOOSE_TAG): New.
(HWASAN_CHECK): New.
(HWASAN_MARK): New.
(HWASAN_SET_TAG): New.
* sanitizer.def (BUILT_IN_HWASAN_LOAD1): New.
(BUILT_IN_HWASAN_LOAD2): New.
(BUILT_IN_HWASAN_LOAD4): New.
(BUILT_IN_HWASAN_LOAD8): New.
(BUILT_IN_HWASAN_LOAD16): New.
(BUILT_IN_HWASAN_LOADN): New.
(BUILT_IN_HWASAN_STORE1): New.
(BUILT_IN_HWASAN_STORE2): New.
(BUILT_IN_HWASAN_STORE4): New.
(BUILT_IN_HWASAN_STORE8): New.
(BUILT_IN_HWASAN_STORE16): New.
(BUILT_IN_HWASAN_STOREN): New.
(BUILT_IN_HWASAN_LOAD1_NOABORT): New.
(BUILT_IN_HWASAN_LOAD2_NOABORT): New.
(BUILT_IN_HWASAN_LOAD4_NOABORT): New.
(BUILT_IN_HWASAN_LOAD8_NOABORT): New.
(BUILT_IN_HWASAN_LOAD16_NOABORT): New.
(BUILT_IN_HWASAN_LOADN_NOABORT): New.
(BUILT_IN_HWASAN_STORE1_NOABORT): New.
(BUILT_IN_HWASAN_STORE2_NOABORT): New.
(BUILT_IN_HWASAN_STORE4_NOABORT): New.
(BUILT_IN_HWASAN_STORE8_NOABORT): New.
(BUILT_IN_HWASAN_STORE16_NOABORT): New.
(BUILT_IN_HWASAN_STOREN_NOABORT): New.
(BUILT_IN_HWASAN_TAG_MISMATCH4): New.
(BUILT_IN_HWASAN_HANDLE_LONGJMP): New.
(BUILT_IN_HWASAN_TAG_PTR): New.
* sanopt.c (sanopt_optimize_walker): Act during hwasan.
(pass_sanopt::execute): Act during hwasan.
* toplev.c (compile_file): Use `gate_hwasan` function.
############### Attachment also inlined for ease of reply ###############
diff --git a/gcc/asan.h b/gcc/asan.h
index
8d5271e6b575d74da277420798557f3274e966ce..4b873d6696026839025926581ccc9aceca225196
100644
--- a/gcc/asan.h
+++ b/gcc/asan.h
@@ -49,6 +49,9 @@ extern void hwasan_finish_file (void);
extern bool hwasan_sanitize_p (void);
extern bool hwasan_sanitize_stack_p (void);
extern bool hwasan_sanitize_allocas_p (void);
+extern bool hwasan_expand_check_ifn (gimple_stmt_iterator *, bool);
+extern bool hwasan_expand_mark_ifn (gimple_stmt_iterator *);
+extern bool gate_hwasan (void);
extern gimple_stmt_iterator create_cond_insert_point
(gimple_stmt_iterator *, bool, bool, bool, basic_block *, basic_block *);
@@ -181,6 +184,9 @@ extern hash_set<tree> *asan_handled_variables;
static inline bool
asan_intercepted_p (enum built_in_function fcode)
{
+ if (hwasan_sanitize_p ())
+ return false;
+
return fcode == BUILT_IN_INDEX
|| fcode == BUILT_IN_MEMCHR
|| fcode == BUILT_IN_MEMCMP
@@ -209,7 +215,8 @@ asan_intercepted_p (enum built_in_function fcode)
static inline bool
asan_sanitize_use_after_scope (void)
{
- return (flag_sanitize_address_use_after_scope && asan_sanitize_stack_p ());
+ return (flag_sanitize_address_use_after_scope
+ && (asan_sanitize_stack_p () || hwasan_sanitize_stack_p ()));
}
/* Return true if DECL should be guarded on the stack. */
diff --git a/gcc/asan.c b/gcc/asan.c
index
b8e3e9f35535549967f72b0a60eb8b9b17ef5baf..39db40fd0fd558abaf72612eec2d36120ebb7362
100644
--- a/gcc/asan.c
+++ b/gcc/asan.c
@@ -370,6 +370,25 @@ asan_sanitize_allocas_p (void)
return (asan_sanitize_stack_p () && param_asan_protect_allocas);
}
+bool
+asan_instrument_reads (void)
+{
+ return (sanitize_flags_p (SANITIZE_ADDRESS) && param_asan_instrument_reads);
+}
+
+bool
+asan_instrument_writes (void)
+{
+ return (sanitize_flags_p (SANITIZE_ADDRESS) && param_asan_instrument_writes);
+}
+
+bool
+asan_memintrin (void)
+{
+ return (sanitize_flags_p (SANITIZE_ADDRESS) && param_asan_memintrin);
+}
+
+
/* Checks whether section SEC should be sanitized. */
static bool
@@ -630,23 +649,73 @@ get_last_alloca_addr ()
To overcome the issue we use following trick: pass new_sp as a second
parameter to __asan_allocas_unpoison and rewrite it during expansion with
new_sp + (virtual_dynamic_stack_rtx - sp) later in
- expand_asan_emit_allocas_unpoison function. */
+ expand_asan_emit_allocas_unpoison function.
+
+ HWASAN needs to do very similar, the eventual pseudocode should be:
+ __hwasan_tag_memory (virtual_stack_dynamic_rtx,
+ 0,
+ new_sp - sp);
+ __builtin_stack_restore (new_sp)
+
+ Need to use the same trick to handle STACK_DYNAMIC_OFFSET as described
+ above. */
static void
handle_builtin_stack_restore (gcall *call, gimple_stmt_iterator *iter)
{
- if (!iter || !asan_sanitize_allocas_p ())
+ if (!iter
+ || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()))
return;
- tree last_alloca = get_last_alloca_addr ();
tree restored_stack = gimple_call_arg (call, 0);
- tree fn = builtin_decl_implicit (BUILT_IN_ASAN_ALLOCAS_UNPOISON);
- gimple *g = gimple_build_call (fn, 2, last_alloca, restored_stack);
- gsi_insert_before (iter, g, GSI_SAME_STMT);
- g = gimple_build_assign (last_alloca, restored_stack);
+
+ gimple *g;
+
+ if (hwasan_sanitize_allocas_p ())
+ {
+ enum internal_fn fn = IFN_HWASAN_ALLOCA_UNPOISON;
+ /* There is only one piece of information `expand_HWASAN_ALLOCA_UNPOISON`
+ needs to work. This is the length of the area that we're
+ deallocating. Since the stack pointer is known at expand time, the
+ position of the new stack pointer after deallocation is enough
+ information to calculate this length. */
+ g = gimple_build_call_internal (fn, 1, restored_stack);
+ }
+ else
+ {
+ tree last_alloca = get_last_alloca_addr ();
+ tree fn = builtin_decl_implicit (BUILT_IN_ASAN_ALLOCAS_UNPOISON);
+ g = gimple_build_call (fn, 2, last_alloca, restored_stack);
+ gsi_insert_before (iter, g, GSI_SAME_STMT);
+ g = gimple_build_assign (last_alloca, restored_stack);
+ }
+
gsi_insert_before (iter, g, GSI_SAME_STMT);
}
+/* Emit gimple statements into &stmts that take the size given in `len` and
+ generate a size that is guaranteed to be rounded upwards to `align`.
+
+ This is a helper function for both handle_builtin_alloca and
+ asan_expand_mark_ifn when using HWASAN.
+
+ Return the tree node representing this size, it is of TREE_TYPE
+ size_type_node. */
+
+static tree
+hwasan_emit_round_up (gimple_seq *seq, location_t loc, tree old_size,
+ uint8_t align)
+{
+ uint8_t tg_mask = align - 1;
+ /* tree new_size = (old_size + tg_mask) & ~tg_mask; */
+ tree tree_mask = build_int_cst (size_type_node, tg_mask);
+ tree oversize = gimple_build (seq, loc, PLUS_EXPR, size_type_node, old_size,
+ tree_mask);
+
+ tree mask = build_int_cst (size_type_node, -align);
+ return gimple_build (seq, loc, BIT_AND_EXPR, size_type_node, oversize, mask);
+}
+
/* Deploy and poison redzones around __builtin_alloca call. To do this, we
should replace this call with another one with changed parameters and
replace all its uses with new address, so
@@ -673,14 +742,12 @@ handle_builtin_stack_restore (gcall *call,
gimple_stmt_iterator *iter)
static void
handle_builtin_alloca (gcall *call, gimple_stmt_iterator *iter)
{
- if (!iter || !asan_sanitize_allocas_p ())
+ if (!iter
+ || !(asan_sanitize_allocas_p () || hwasan_sanitize_allocas_p ()))
return;
gassign *g;
gcall *gg;
- const HOST_WIDE_INT redzone_mask = ASAN_RED_ZONE_SIZE - 1;
-
- tree last_alloca = get_last_alloca_addr ();
tree callee = gimple_call_fndecl (call);
tree old_size = gimple_call_arg (call, 0);
tree ptr_type = gimple_call_lhs (call) ? TREE_TYPE (gimple_call_lhs (call))
@@ -690,6 +757,71 @@ handle_builtin_alloca (gcall *call, gimple_stmt_iterator
*iter)
= DECL_FUNCTION_CODE (callee) == BUILT_IN_ALLOCA
? 0 : tree_to_uhwi (gimple_call_arg (call, 1));
+ if (hwasan_sanitize_allocas_p ())
+ {
+ gimple_seq stmts = NULL;
+ location_t loc = gimple_location (gsi_stmt (*iter));
+ /*
+ HWASAN needs a different expansion.
+
+ addr = __builtin_alloca (size, align);
+
+ should be replaced by
+
+ new_size = size rounded up to HWASAN_TAG_GRANULE_SIZE byte alignment;
+ untagged_addr = __builtin_alloca (new_size, align);
+ tag = __hwasan_choose_alloca_tag ();
+ addr = ifn_HWASAN_SET_TAG (untagged_addr, tag);
+ __hwasan_tag_memory (untagged_addr, tag, new_size);
+ */
+ /* Ensure alignment at least HWASAN_TAG_GRANULE_SIZE bytes so we start on
+ a tag granule. */
+ align = align > HWASAN_TAG_GRANULE_SIZE ? align :
HWASAN_TAG_GRANULE_SIZE;
+
+ tree old_size = gimple_call_arg (call, 0);
+ tree new_size = hwasan_emit_round_up (&stmts, loc, old_size,
+ HWASAN_TAG_GRANULE_SIZE);
+
+ /* Make the alloca call */
+ tree untagged_addr
+ = gimple_build (&stmts, loc,
+ as_combined_fn (BUILT_IN_ALLOCA_WITH_ALIGN), ptr_type,
+ new_size, build_int_cst (size_type_node, align));
+
+ /* Choose the tag.
+ Here we use an internal function so we can choose the tag at expand
+ time. We need the decision to be made after stack variables have been
+ assigned their tag (i.e. once the hwasan_frame_tag_offset variable has
+ been set to one after the last stack variables tag). */
+ gcall *stmt = gimple_build_call_internal (IFN_HWASAN_CHOOSE_TAG, 0);
+ tree tag = make_ssa_name (unsigned_char_type_node);
+ gimple_call_set_lhs (stmt, tag);
+ gimple_set_location (stmt, loc);
+ gimple_seq_add_stmt_without_update (&stmts, stmt);
+
+ /* Add tag to pointer. */
+ tree addr
+ = gimple_build (&stmts, loc, as_combined_fn (IFN_HWASAN_SET_TAG),
+ ptr_type, untagged_addr, tag);
+
+ /* Tag shadow memory.
+ NOTE: require using `untagged_addr` here for libhwasan API. */
+ gimple_build (&stmts, loc, as_combined_fn (BUILT_IN_HWASAN_TAG_MEM),
+ void_type_node, untagged_addr, tag, new_size);
+
+ /* Insert the built up code sequence into the original instruction stream
+ the iterator points to. */
+ gsi_insert_seq_before (iter, stmts, GSI_SAME_STMT);
+
+ /* Finally, replace old alloca ptr with NEW_ALLOCA. */
+ replace_call_with_value (iter, addr);
+ return;
+ }
+
+ tree last_alloca = get_last_alloca_addr ();
+ const HOST_WIDE_INT redzone_mask = ASAN_RED_ZONE_SIZE - 1;
+
+
/* If ALIGN > ASAN_RED_ZONE_SIZE, we embed left redzone into first ALIGN
bytes of allocated space. Otherwise, align alloca to ASAN_RED_ZONE_SIZE
manually. */
@@ -842,6 +974,31 @@ get_mem_refs_of_builtin_call (gcall *call,
break;
case BUILT_IN_STRLEN:
+ /* Special case strlen here since its length is taken from its return
+ value.
+
+ The approach taken by the sanitizers is to check a memory access
+ before it's taken. For ASAN strlen is intercepted by libasan, so no
+ check is inserted by the compiler.
+
+ This function still returns `true` and provides a length to the rest
+ of the ASAN pass in order to record what areas have been checked,
+ avoiding superfluous checks later on.
+
+ HWASAN does not intercept any of these internal functions.
+ This means that checks for memory accesses must be inserted by the
+ compiler.
+ strlen is a special case, because we can tell the length from the
+ return of the function, but that is not known until after the function
+ has returned.
+
+ Hence we can't check the memory access before it happens.
+ We could check the memory access after it has already happened, but
+ for now we choose to just ignore `strlen` calls.
+ This decision was simply made because that means the special case is
+ limited to this one case of this one function. */
+ if (hwasan_sanitize_p ())
+ return false;
source0 = gimple_call_arg (call, 0);
len = gimple_call_lhs (call);
break;
@@ -1411,6 +1568,156 @@ asan_redzone_buffer::flush_if_full (void)
flush_redzone_payload ();
}
+
+/* HWAddressSanitizer (hwasan) is a probabilistic method for detecting
+ out-of-bounds and use-after-free bugs.
+ Read more:
+ http://code.google.com/p/address-sanitizer/
+
+ Similar to AddressSanitizer (asan) it consists of two parts: the
+ instrumentation module in this file, and a run-time library.
+
+ The instrumentation module adds a run-time check before every memory insn in
+ the same manner as asan (see the block comment for AddressSanitizer above).
+ Currently, hwasan only adds out-of-line instrumentation, where each check is
+ implemented as a function call to the run-time library. Hence a check for a
+ load of N bytes from address X would be implemented with a function call to
+ __hwasan_loadN(X), and checking a store of N bytes from address X would be
+ implemented with a function call to __hwasan_storeN(X).
+
+ The main difference between hwasan and asan is in the information stored to
+ help this checking. Both sanitizers use a shadow memory area which stores
+ data recording the state of main memory at a corresponding address.
+
+ For hwasan, each 16 byte granule in main memory has a corresponding 1 byte
+ in shadow memory. This shadow address can be calculated with equation:
+ (addr >> log_2(HWASAN_TAG_GRANULE_SIZE))
+ + __hwasan_shadow_memory_dynamic_address;
+ The conversion between real and shadow memory for asan is given in the block
+ comment at the top of this file.
+ The description of how this shadow memory is laid out for asan is in the
+ block comment at the top of this file, here we describe how this shadow
+ memory is used for hwasan.
+
+ For hwasan, each variable is assigned a byte-sized 'tag'. The extent of
+ the shadow memory for that variable is filled with the assigned tag, and
+ every pointer referencing that variable has its top byte set to the same
+ tag. The run-time library redefines malloc so that every allocation returns
+ a tagged pointer and tags the corresponding shadow memory with the same tag.
+
+ On each pointer dereference the tag found in the pointer is compared to the
+ tag found in the shadow memory corresponding to the accessed memory address.
+ If these tags are found to differ then this memory access is judged to be
+ invalid and a report is generated.
+
+ This method of bug detection is not perfect -- it can not catch every bad
+ access -- but catches them probabilistically instead. There is always the
+ possibility that an invalid memory access will happen to access memory
+ tagged with the same tag as the pointer that this access used.
+ The chances of this are approx. 0.4% for any two uncorrelated objects.
+
+ Random tag generation can mitigate this problem by decreasing the
+ probability that an invalid access will be missed in the same manner over
+ multiple runs. i.e. if two objects are tagged the same in one run of the
+ binary they are unlikely to be tagged the same in the next run.
+ Both heap and stack allocated objects have random tags by default.
+
+ [16 byte granule implications]
+ Since the shadow memory only has a resolution on real memory of 16 bytes,
+ invalid accesses that are within the same 16 byte granule as a valid
+ address will not be caught.
+
+ There is a "short-granule" feature in the runtime library which does catch
+ such accesses, but this feature is not implemented for stack objects (since
+ stack objects are allocated and tagged by compiler instrumentation, and
+ this feature has not yet been implemented in GCC instrumentation).
+
+ Another outcome of this 16 byte resolution is that each tagged object must
+ be 16 byte aligned. If two objects were to share any 16 byte granule in
+ memory, then they both would have to be given the same tag, and invalid
+ accesses to one using a pointer to the other would be undetectable.
+
+ [Compiler instrumentation]
+ Compiler instrumentation ensures that two adjacent buffers on the stack are
+ given different tags, this means an access to one buffer using a pointer
+ generated from the other (e.g. through buffer overrun) will have mismatched
+ tags and be caught by hwasan.
+
+ We don't randomly tag every object on the stack, since that would require
+ keeping many registers to record each tag. Instead we randomly generate a
+ tag for each function frame, and each new stack object uses a tag offset
+ from that frame tag.
+ i.e. each object is tagged as RFT + offset, where RFT is the "random frame
+ tag" generated for this frame.
+ This means that randomisation does not peturb the difference between tags
+ on tagged stack objects within a frame, but this is mitigated by the fact
+ that objects with the same tag within a frame are very far apart
+ (approx. 2^HWASAN_TAG_SIZE objects apart).
+
+ As a demonstration, using the same example program as in the asan block
+ comment above:
+
+ int
+ foo ()
+ {
+ char a[23] = {0};
+ int b[2] = {0};
+
+ a[5] = 1;
+ b[1] = 2;
+
+ return a[5] + b[1];
+ }
+
+ On AArch64 the stack will be ordered as follows for the above function:
+
+ Slot 1/ [24 bytes for variable 'a']
+ Slot 2/ [8 bytes padding for alignment]
+ Slot 3/ [8 bytes for variable 'b']
+ Slot 4/ [8 bytes padding for alignment]
+
+ (The padding is there to ensure 16 byte alignment as described in the 16
+ byte granule implications).
+
+ While the shadow memory will be ordered as follows:
+
+ - 2 bytes (representing 32 bytes in real memory) tagged with RFT + 1.
+ - 1 byte (representing 16 bytes in real memory) tagged with RFT + 2.
+
+ And any pointer to "a" will have the tag RFT + 1, and any pointer to "b"
+ will have the tag RFT + 2.
+
+ [Top Byte Ignore requirements]
+ Hwasan requires the ability to store an 8 bit tag in every pointer. There
+ is no instrumentation done to remove this tag from pointers before
+ dereferencing, which means the hardware must ignore this tag during memory
+ accesses.
+
+ Architectures where this feature is available should indicate this using
+ the TARGET_MEMTAG_CAN_TAG_ADDRESSES hook.
+
+ [Stack requires cleanup on unwinding]
+ During normal operation of a hwasan sanitized program more space in the
+ shadow memory becomes tagged as the stack grows. As the stack shrinks this
+ shadow memory space must become untagged. If it is not untagged then when
+ the stack grows again (during other function calls later on in the program)
+ objects on the stack that are usually not tagged (e.g. parameters passed on
+ the stack) can be placed in memory whose shadow space is tagged with
+ something else, and accesses can cause false positive reports.
+
+ Hence we place untagging code on every epilogue of functions which tag some
+ stack objects.
+
+ Moreover, the run-time library intercepts longjmp & setjmp to untag when
+ the stack is unwound this way.
+
+ C++ exceptions are not yet handled, which means this sanitizer can not
+ handle C++ code that throws exceptions -- it will give false positives
+ after an exception has been thrown. The implementation that the hwasan
+ library has for handling these relies on the frame pointer being after any
+ local variables. This is not generally the case for GCC. */
+
+
/* Returns whether we are tagging pointers and checking those tags on memory
access. */
bool
@@ -1433,6 +1740,27 @@ hwasan_sanitize_allocas_p (void)
return (hwasan_sanitize_stack_p () && param_hwasan_protect_allocas);
}
+/* Should we instrument reads? */
+bool
+hwasan_instrument_reads (void)
+{
+ return (hwasan_sanitize_p () && param_hwasan_instrument_reads);
+}
+
+/* Should we instrument writes? */
+bool
+hwasan_instrument_writes (void)
+{
+ return (hwasan_sanitize_p () && param_hwasan_instrument_writes);
+}
+
+/* Should we instrument builtin calls? */
+bool
+hwasan_memintrin (void)
+{
+ return (hwasan_sanitize_p () && param_hwasan_instrument_mem_intrinsics);
+}
+
/* Insert code to protect stack vars. The prologue sequence should be emitted
directly, epilogue sequence returned. BASE is the register holding the
stack base, against which OFFSETS array offsets are relative to, OFFSETS
@@ -1928,6 +2256,8 @@ static tree
report_error_func (bool is_store, bool recover_p, HOST_WIDE_INT size_in_bytes,
int *nargs)
{
+ gcc_assert (!hwasan_sanitize_p ());
+
static enum built_in_function report[2][2][6]
= { { { BUILT_IN_ASAN_REPORT_LOAD1, BUILT_IN_ASAN_REPORT_LOAD2,
BUILT_IN_ASAN_REPORT_LOAD4, BUILT_IN_ASAN_REPORT_LOAD8,
@@ -2220,6 +2550,7 @@ build_check_stmt (location_t loc, tree base, tree len,
gimple *g;
gcc_assert (!(size_in_bytes > 0 && !is_non_zero_len));
+ gcc_assert (size_in_bytes == -1 || size_in_bytes >= 1);
gsi = *iter;
@@ -2264,7 +2595,11 @@ build_check_stmt (location_t loc, tree base, tree len,
if (is_scalar_access)
flags |= ASAN_CHECK_SCALAR_ACCESS;
- g = gimple_build_call_internal (IFN_ASAN_CHECK, 4,
+ enum internal_fn fn = hwasan_sanitize_p ()
+ ? IFN_HWASAN_CHECK
+ : IFN_ASAN_CHECK;
+
+ g = gimple_build_call_internal (fn, 4,
build_int_cst (integer_type_node, flags),
base, len,
build_int_cst (integer_type_node,
@@ -2288,9 +2623,9 @@ static void
instrument_derefs (gimple_stmt_iterator *iter, tree t,
location_t location, bool is_store)
{
- if (is_store && !param_asan_instrument_writes)
+ if (is_store && !(asan_instrument_writes () || hwasan_instrument_writes ()))
return;
- if (!is_store && !param_asan_instrument_reads)
+ if (!is_store && !(asan_instrument_reads () || hwasan_instrument_reads ()))
return;
tree type, base;
@@ -2351,7 +2686,8 @@ instrument_derefs (gimple_stmt_iterator *iter, tree t,
{
if (DECL_THREAD_LOCAL_P (inner))
return;
- if (!param_asan_globals && is_global_var (inner))
+ if ((hwasan_sanitize_p () || !param_asan_globals)
+ && is_global_var (inner))
return;
if (!TREE_STATIC (inner))
{
@@ -2444,7 +2780,7 @@ instrument_mem_region_access (tree base, tree len,
static bool
instrument_builtin_call (gimple_stmt_iterator *iter)
{
- if (!param_asan_memintrin)
+ if (!(asan_memintrin () || hwasan_memintrin ()))
return false;
bool iter_advanced_p = false;
@@ -2580,10 +2916,27 @@ maybe_instrument_call (gimple_stmt_iterator *iter)
break;
}
}
- tree decl = builtin_decl_implicit (BUILT_IN_ASAN_HANDLE_NO_RETURN);
- gimple *g = gimple_build_call (decl, 0);
- gimple_set_location (g, gimple_location (stmt));
- gsi_insert_before (iter, g, GSI_SAME_STMT);
+ /* If a function does not return, then we must handle clearing up the
+ shadow stack accordingly. For ASAN we can simply set the entire stack
+ to "valid" for accesses by setting the shadow space to 0 and all
+ accesses will pass checks. That means that some bad accesses may be
+ missed, but we will not report any false positives.
+
+ This is not possible for HWASAN. Since there is no "always valid" tag
+ we can not set any space to "always valid". If we were to clear the
+ entire shadow stack then code resuming from `longjmp` or a caught
+ exception would trigger false positives when correctly accessing
+ variables on the stack. Hence we need to handle things like
+ `longjmp`, thread exit, and exceptions in a different way. These
+ problems must be handled externally to the compiler, e.g. in the
+ language runtime. */
+ if (! hwasan_sanitize_p ())
+ {
+ tree decl = builtin_decl_implicit (BUILT_IN_ASAN_HANDLE_NO_RETURN);
+ gimple *g = gimple_build_call (decl, 0);
+ gimple_set_location (g, gimple_location (stmt));
+ gsi_insert_before (iter, g, GSI_SAME_STMT);
+ }
}
bool instrumented = false;
@@ -2982,6 +3335,9 @@ initialize_sanitizer_builtins (void)
= build_function_type_list (void_type_node, uint64_type_node,
ptr_type_node, NULL_TREE);
+ tree BT_FN_PTR_CONST_PTR_UINT8
+ = build_function_type_list (ptr_type_node, const_ptr_type_node,
+ unsigned_char_type_node, NULL_TREE);
tree BT_FN_VOID_PTR_UINT8_PTRMODE
= build_function_type_list (void_type_node, ptr_type_node,
unsigned_char_type_node,
@@ -3305,6 +3661,43 @@ asan_expand_mark_ifn (gimple_stmt_iterator *iter)
gcc_checking_assert (TREE_CODE (decl) == VAR_DECL);
+ if (hwasan_sanitize_p ())
+ {
+ gcc_assert (param_hwasan_instrument_stack);
+ gimple_seq stmts = NULL;
+ /* Here we swap ASAN_MARK calls for HWASAN_MARK.
+ This is because we are using the approach of using ASAN_MARK as a
+ synonym until here.
+ That approach means we don't yet have to duplicate all the special
+ cases for ASAN_MARK and ASAN_POISON with the exact same handling but
+ called HWASAN_MARK etc.
+
+ N.b. __asan_poison_stack_memory (which implements ASAN_MARK for ASAN)
+ rounds the size up to its shadow memory granularity, while
+ __hwasan_tag_memory (which implements the same for HWASAN) does not.
+ Hence we emit HWASAN_MARK with an aligned size unlike ASAN_MARK. */
+ tree len = gimple_call_arg (g, 2);
+ tree new_len;
+ /* If the length is a simple SHWI, then we can calculate the rounded up
+ length directly. Otherwise we need to emit gimple to do this
+ rounding at runtime. */
+ if (tree_fits_shwi_p (len))
+ {
+ unsigned HOST_WIDE_INT size_in_bytes = tree_to_shwi (len);
+ uint8_t tg_mask = HWASAN_TAG_GRANULE_SIZE - 1;
+ size_in_bytes = (size_in_bytes + tg_mask) & ~tg_mask;
+ new_len = build_int_cst (pointer_sized_int_node, size_in_bytes);
+ }
+ else
+ new_len = hwasan_emit_round_up (&stmts, loc, len,
+ HWASAN_TAG_GRANULE_SIZE);
+ gimple_build (&stmts, loc, as_combined_fn (IFN_HWASAN_MARK),
+ void_type_node, gimple_call_arg (g, 0),
+ base, new_len);
+ gsi_replace_with_seq (iter, stmts, true);
+ return false;
+ }
+
if (is_poison)
{
if (asan_handled_variables == NULL)
@@ -3379,6 +3772,7 @@ asan_expand_mark_ifn (gimple_stmt_iterator *iter)
bool
asan_expand_check_ifn (gimple_stmt_iterator *iter, bool use_calls)
{
+ gcc_assert (!hwasan_sanitize_p ());
gimple *g = gsi_stmt (*iter);
location_t loc = gimple_location (g);
bool recover_p;
@@ -3652,11 +4046,61 @@ asan_expand_poison_ifn (gimple_stmt_iterator *iter,
int nargs;
bool store_p = gimple_call_internal_p (use, IFN_ASAN_POISON_USE);
- tree fun = report_error_func (store_p, recover_p, tree_to_uhwi (size),
- &nargs);
-
- gcall *call = gimple_build_call (fun, 1,
- build_fold_addr_expr (shadow_var));
+ gcall *call;
+ if (hwasan_sanitize_p ())
+ {
+ tree fun = builtin_decl_implicit (BUILT_IN_HWASAN_TAG_MISMATCH4);
+ /* NOTE: hwasan has no __hwasan_report_* functions like asan does.
+ We use __hwasan_tag_mismatch4 with arguments that tell it the
+ size of access and load to report all tag mismatches.
+
+ The arguments to this function are:
+ Address of invalid access.
+ Bitfield containing information about the access
+ (access_info)
+ Pointer to a frame of registers
+ (for use in printing the contents of registers in a dump)
+ Not used yet -- to be used by inline instrumentation.
+ Size of access.
+
+ The access_info bitfield encodes the following pieces of
+ information:
+ - Is this a store or load?
+ access_info & 0x10 => store
+ - Should the program continue after reporting the error?
+ access_info & 0x20 => recover
+ - What size access is this (not used here since we can always
+ pass the size in the last argument)
+
+ if (access_info & 0xf == 0xf)
+ size is taken from last argument.
+ else
+ size == 1 << (access_info & 0xf)
+
+ The last argument contains the size of the access iff the
+ access_info size indicator is 0xf (we always use this argument
+ rather than storing the size in the access_info bitfield).
+
+ See the function definition `__hwasan_tag_mismatch4` in
+ libsanitizer/hwasan for the full definition.
+ */
+ unsigned access_info = (0x20 * recover_p)
+ + (0x10 * store_p)
+ + (0xf);
+ call = gimple_build_call (fun, 4,
+ build_fold_addr_expr (shadow_var),
+ build_int_cst (pointer_sized_int_node,
+ access_info),
+ build_int_cst (pointer_sized_int_node, 0),
+ size);
+ }
+ else
+ {
+ tree fun = report_error_func (store_p, recover_p, tree_to_uhwi (size),
+ &nargs);
+ call = gimple_build_call (fun, 1,
+ build_fold_addr_expr (shadow_var));
+ }
gimple_set_location (call, gimple_location (use));
gimple *call_to_insert = call;
@@ -3704,6 +4148,12 @@ asan_expand_poison_ifn (gimple_stmt_iterator *iter,
static unsigned int
asan_instrument (void)
{
+ if (hwasan_sanitize_p ())
+ {
+ transform_statements ();
+ return 0;
+ }
+
if (shadow_ptr_types[0] == NULL_TREE)
asan_init_shadow_ptr_types ();
transform_statements ();
@@ -3741,7 +4191,7 @@ public:
/* opt_pass methods: */
opt_pass * clone () { return new pass_asan (m_ctxt); }
- virtual bool gate (function *) { return gate_asan (); }
+ virtual bool gate (function *) { return gate_asan () || gate_hwasan (); }
virtual unsigned int execute (function *) { return asan_instrument (); }
}; // class pass_asan
@@ -3777,7 +4227,10 @@ public:
{}
/* opt_pass methods: */
- virtual bool gate (function *) { return !optimize && gate_asan (); }
+ virtual bool gate (function *)
+ {
+ return !optimize && (gate_asan () || gate_hwasan ());
+ }
virtual unsigned int execute (function *) { return asan_instrument (); }
}; // class pass_asan_O0
@@ -3790,6 +4243,8 @@ make_pass_asan_O0 (gcc::context *ctxt)
return new pass_asan_O0 (ctxt);
}
+/* HWASAN */
+
/* For stack tagging:
Return the offset from the frame base tag that the "next" expanded object
@@ -4138,4 +4593,138 @@ hwasan_truncate_to_tag_size (rtx tag, rtx target)
return tag;
}
+/* Construct a function tree for __hwasan_{load,store}{1,2,4,8,16,_n}.
+ IS_STORE is either 1 (for a store) or 0 (for a load). */
+static combined_fn
+hwasan_check_func (bool is_store, bool recover_p, HOST_WIDE_INT size_in_bytes,
+ int *nargs)
+{
+ static enum built_in_function check[2][2][6]
+ = { { { BUILT_IN_HWASAN_LOAD1, BUILT_IN_HWASAN_LOAD2,
+ BUILT_IN_HWASAN_LOAD4, BUILT_IN_HWASAN_LOAD8,
+ BUILT_IN_HWASAN_LOAD16, BUILT_IN_HWASAN_LOADN },
+ { BUILT_IN_HWASAN_STORE1, BUILT_IN_HWASAN_STORE2,
+ BUILT_IN_HWASAN_STORE4, BUILT_IN_HWASAN_STORE8,
+ BUILT_IN_HWASAN_STORE16, BUILT_IN_HWASAN_STOREN } },
+ { { BUILT_IN_HWASAN_LOAD1_NOABORT,
+ BUILT_IN_HWASAN_LOAD2_NOABORT,
+ BUILT_IN_HWASAN_LOAD4_NOABORT,
+ BUILT_IN_HWASAN_LOAD8_NOABORT,
+ BUILT_IN_HWASAN_LOAD16_NOABORT,
+ BUILT_IN_HWASAN_LOADN_NOABORT },
+ { BUILT_IN_HWASAN_STORE1_NOABORT,
+ BUILT_IN_HWASAN_STORE2_NOABORT,
+ BUILT_IN_HWASAN_STORE4_NOABORT,
+ BUILT_IN_HWASAN_STORE8_NOABORT,
+ BUILT_IN_HWASAN_STORE16_NOABORT,
+ BUILT_IN_HWASAN_STOREN_NOABORT } } };
+ if (size_in_bytes == -1)
+ {
+ *nargs = 2;
+ return as_combined_fn (check[recover_p][is_store][5]);
+ }
+ *nargs = 1;
+ int size_log2 = exact_log2 (size_in_bytes);
+ gcc_assert (size_log2 >= 0 && size_log2 <= 5);
+ return as_combined_fn (check[recover_p][is_store][size_log2]);
+}
+
+/* Expand the HWASAN_{LOAD,STORE} builtins. */
+bool
+hwasan_expand_check_ifn (gimple_stmt_iterator *iter, bool)
+{
+ gimple *g = gsi_stmt (*iter);
+ location_t loc = gimple_location (g);
+ bool recover_p;
+ if (flag_sanitize & SANITIZE_USER_HWADDRESS)
+ recover_p = (flag_sanitize_recover & SANITIZE_USER_HWADDRESS) != 0;
+ else
+ recover_p = (flag_sanitize_recover & SANITIZE_KERNEL_HWADDRESS) != 0;
+
+ HOST_WIDE_INT flags = tree_to_shwi (gimple_call_arg (g, 0));
+ gcc_assert (flags < ASAN_CHECK_LAST);
+ bool is_scalar_access = (flags & ASAN_CHECK_SCALAR_ACCESS) != 0;
+ bool is_store = (flags & ASAN_CHECK_STORE) != 0;
+ bool is_non_zero_len = (flags & ASAN_CHECK_NON_ZERO_LEN) != 0;
+
+ tree base = gimple_call_arg (g, 1);
+ tree len = gimple_call_arg (g, 2);
+
+ /* `align` is unused for HWASAN_CHECK, but we pass the argument anyway
+ since that way the arguments match ASAN_CHECK. */
+ /* HOST_WIDE_INT align = tree_to_shwi (gimple_call_arg (g, 3)); */
+
+ unsigned HOST_WIDE_INT size_in_bytes
+ = is_scalar_access ? tree_to_shwi (len) : -1;
+
+ gimple_stmt_iterator gsi = *iter;
+
+ if (!is_non_zero_len)
+ {
+ /* So, the length of the memory area to hwasan-protect is
+ non-constant. Let's guard the generated instrumentation code
+ like:
+
+ if (len != 0)
+ {
+ // hwasan instrumentation code goes here.
+ }
+ // falltrough instructions, starting with *ITER. */
+
+ g = gimple_build_cond (NE_EXPR,
+ len,
+ build_int_cst (TREE_TYPE (len), 0),
+ NULL_TREE, NULL_TREE);
+ gimple_set_location (g, loc);
+
+ basic_block then_bb, fallthrough_bb;
+ insert_if_then_before_iter (as_a <gcond *> (g), iter,
+ /*then_more_likely_p=*/true,
+ &then_bb, &fallthrough_bb);
+ /* Note that fallthrough_bb starts with the statement that was
+ pointed to by ITER. */
+
+ /* The 'then block' of the 'if (len != 0) condition is where
+ we'll generate the hwasan instrumentation code now. */
+ gsi = gsi_last_bb (then_bb);
+ }
+
+ gimple_seq stmts = NULL;
+ tree base_addr = gimple_build (&stmts, loc, NOP_EXPR,
+ pointer_sized_int_node, base);
+
+ int nargs = 0;
+ combined_fn fn = hwasan_check_func (is_store, recover_p, size_in_bytes,
&nargs);
+ if (nargs == 1)
+ gimple_build (&stmts, loc, fn, void_type_node, base_addr);
+ else
+ {
+ gcc_assert (nargs == 2);
+ tree sz_arg = gimple_build (&stmts, loc, NOP_EXPR,
+ pointer_sized_int_node, len);
+ gimple_build (&stmts, loc, fn, void_type_node, base_addr, sz_arg);
+ }
+
+ gsi_insert_seq_after (&gsi, stmts, GSI_NEW_STMT);
+ gsi_remove (iter, true);
+ *iter = gsi;
+ return false;
+}
+
+/* For stack tagging:
+
+ Dummy: the HWASAN_MARK internal function should only ever be in the code
+ after the sanopt pass. */
+bool
+hwasan_expand_mark_ifn (gimple_stmt_iterator *)
+{
+ gcc_unreachable ();
+}
+
+bool
+gate_hwasan ()
+{
+ return hwasan_sanitize_p ();
+}
+
#include "gt-asan.h"
diff --git a/gcc/builtin-types.def b/gcc/builtin-types.def
index
1ad6657da45cc4976532e1b8bc233f67d8da9ccf..770f3575cee2c6a4a5cfb16354bd8ba2d65f574e
100644
--- a/gcc/builtin-types.def
+++ b/gcc/builtin-types.def
@@ -503,6 +503,7 @@ DEF_FUNCTION_TYPE_2 (BT_FN_INT_FEXCEPT_T_PTR_INT, BT_INT,
BT_FEXCEPT_T_PTR,
BT_INT)
DEF_FUNCTION_TYPE_2 (BT_FN_INT_CONST_FEXCEPT_T_PTR_INT, BT_INT,
BT_CONST_FEXCEPT_T_PTR, BT_INT)
+DEF_FUNCTION_TYPE_2 (BT_FN_PTR_CONST_PTR_UINT8, BT_PTR, BT_CONST_PTR, BT_UINT8)
DEF_POINTER_TYPE (BT_PTR_FN_VOID_PTR_PTR, BT_FN_VOID_PTR_PTR)
diff --git a/gcc/gimple-pretty-print.c b/gcc/gimple-pretty-print.c
index
d97a231e7e8333c1b32a39d03851b974dc55c1f7..075d6e5208a392956436bf92cb88a9e3378e5216
100644
--- a/gcc/gimple-pretty-print.c
+++ b/gcc/gimple-pretty-print.c
@@ -753,6 +753,7 @@ dump_gimple_call_args (pretty_printer *buffer, const gcall
*gs,
limit = ARRAY_SIZE (reduction_args);
break;
+ case IFN_HWASAN_MARK:
case IFN_ASAN_MARK:
#define DEF(X) #X
static const char *const asan_mark_args[] = {IFN_ASAN_MARK_FLAGS};
diff --git a/gcc/gimplify.c b/gcc/gimplify.c
index
2566ec7f0af8dc844901ad0bda2a73fdb4383b4d..7b7c5e348893b7249638ff9c5a04d16af7498698
100644
--- a/gcc/gimplify.c
+++ b/gcc/gimplify.c
@@ -1237,8 +1237,11 @@ asan_poison_variable (tree decl, bool poison,
gimple_stmt_iterator *it,
/* It's necessary to have all stack variables aligned to ASAN granularity
bytes. */
- if (DECL_ALIGN_UNIT (decl) <= ASAN_SHADOW_GRANULARITY)
- SET_DECL_ALIGN (decl, BITS_PER_UNIT * ASAN_SHADOW_GRANULARITY);
+ gcc_assert (!hwasan_sanitize_p () || hwasan_sanitize_stack_p ());
+ unsigned shadow_granularity
+ = hwasan_sanitize_p () ? HWASAN_TAG_GRANULE_SIZE : ASAN_SHADOW_GRANULARITY;
+ if (DECL_ALIGN_UNIT (decl) <= shadow_granularity)
+ SET_DECL_ALIGN (decl, BITS_PER_UNIT * shadow_granularity);
HOST_WIDE_INT flags = poison ? ASAN_MARK_POISON : ASAN_MARK_UNPOISON;
@@ -15365,7 +15368,7 @@ gimplify_function_tree (tree fndecl)
if necessary. */
cfun->curr_properties |= PROP_gimple_lva;
- if (asan_sanitize_use_after_scope () && sanitize_flags_p (SANITIZE_ADDRESS))
+ if (asan_sanitize_use_after_scope ())
asan_poisoned_variables = new hash_set<tree> ();
bind = gimplify_body (fndecl, true);
if (asan_poisoned_variables)
diff --git a/gcc/internal-fn.c b/gcc/internal-fn.c
index
792d2ca568a4a11c3e3ad8abf6443f173b0650a7..5adf1680ef918ccbee0d4af2ba0f0b097470484a
100644
--- a/gcc/internal-fn.c
+++ b/gcc/internal-fn.c
@@ -469,6 +469,139 @@ expand_UBSAN_OBJECT_SIZE (internal_fn, gcall *)
/* This should get expanded in the sanopt pass. */
static void
+expand_HWASAN_CHECK (internal_fn, gcall *)
+{
+ gcc_unreachable ();
+}
+
+/* For hwasan stack tagging:
+ Clear tags on the dynamically allocated space.
+ For use after an object dynamically allocated on the stack goes out of
+ scope. */
+static void
+expand_HWASAN_ALLOCA_UNPOISON (internal_fn, gcall *gc)
+{
+ tree restored_position = gimple_call_arg (gc, 0);
+ rtx restored_rtx = expand_expr (restored_position, NULL_RTX, VOIDmode,
+ EXPAND_NORMAL);
+ rtx func = init_one_libfunc ("__hwasan_tag_memory");
+ rtx off = expand_simple_binop (ptr_mode, MINUS, restored_rtx,
+ stack_pointer_rtx, NULL_RTX, 0,
+ OPTAB_WIDEN);
+ rtx dynamic = convert_memory_address (ptr_mode, virtual_stack_dynamic_rtx);
+ emit_library_call_value (func, NULL_RTX, LCT_NORMAL, VOIDmode,
+ dynamic, ptr_mode,
+ HWASAN_STACK_BACKGROUND, QImode,
+ off, ptr_mode);
+}
+
+/* For hwasan stack tagging:
+ Return a tag to be used for a dynamic allocation. */
+static void
+expand_HWASAN_CHOOSE_TAG (internal_fn, gcall *gc)
+{
+ tree tag = gimple_call_lhs (gc);
+ rtx target = expand_expr (tag, NULL_RTX, VOIDmode, EXPAND_NORMAL);
+ machine_mode mode = GET_MODE (target);
+ gcc_assert (mode == QImode);
+
+ rtx base_tag = targetm.memtag.extract_tag (hwasan_frame_base (), NULL_RTX);
+ gcc_assert (base_tag);
+ rtx tag_offset = gen_int_mode (hwasan_current_frame_tag (), QImode);
+ rtx chosen_tag = expand_simple_binop (QImode, PLUS, base_tag, tag_offset,
+ target, /* unsignedp = */1,
+ OPTAB_WIDEN);
+ chosen_tag = hwasan_truncate_to_tag_size (chosen_tag, target);
+
+ /* Really need to put the tag into the `target` RTX. */
+ if (chosen_tag != target)
+ {
+ rtx temp = chosen_tag;
+ gcc_assert (GET_MODE (chosen_tag) == mode);
+ emit_move_insn (target, temp);
+ }
+
+ hwasan_increment_frame_tag ();
+}
+
+/* For hwasan stack tagging:
+ Tag a region of space in the shadow stack according to the base pointer of
+ an object on the stack. N.b. the length provided in the internal call is
+ required to be aligned to HWASAN_TAG_GRANULE_SIZE. */
+static void
+expand_HWASAN_MARK (internal_fn, gcall *gc)
+{
+ HOST_WIDE_INT flag = tree_to_shwi (gimple_call_arg (gc, 0));
+ bool is_poison = ((asan_mark_flags)flag) == ASAN_MARK_POISON;
+
+ tree base = gimple_call_arg (gc, 1);
+ gcc_checking_assert (TREE_CODE (base) == ADDR_EXPR);
+ /* base is a pointer argument, hence in ptr_mode.
+ We convert to Pmode for use in the targetm.memtag.extract_tag and
+ targetm.memtag.untagged_pointer hooks.
+ We then convert the result to ptr_mode for the emit_library_call.
+
+ This conversion is not for functionality so much as for the code to be
+ correctly formed. If ptr_mode is smaller than Pmode then the top byte of
+ a Pmode value will be truncated in C code losing the tag values, and
+ HWASAN will not work. */
+ rtx base_rtx = convert_memory_address (Pmode, expand_normal (base));
+
+ rtx tag = is_poison ? HWASAN_STACK_BACKGROUND
+ : targetm.memtag.extract_tag (base_rtx, NULL_RTX);
+ rtx address = targetm.memtag.untagged_pointer (base_rtx, NULL_RTX);
+ address = convert_memory_address (ptr_mode, address);
+
+ tree len = gimple_call_arg (gc, 2);
+ rtx r_len = expand_normal (len);
+
+ rtx func = init_one_libfunc ("__hwasan_tag_memory");
+ emit_library_call (func, LCT_NORMAL, VOIDmode, address, ptr_mode,
+ tag, QImode, r_len, ptr_mode);
+}
+
+/* For hwasan stack tagging:
+ Store a tag into a pointer. */
+static void
+expand_HWASAN_SET_TAG (internal_fn, gcall *gc)
+{
+ tree g_target = gimple_call_lhs (gc);
+ tree g_ptr = gimple_call_arg (gc, 0);
+ tree g_tag = gimple_call_arg (gc, 1);
+
+ rtx ptr = convert_memory_address (Pmode, expand_normal (g_ptr));
+ rtx tag = expand_expr (g_tag, NULL_RTX, QImode, EXPAND_NORMAL);
+ rtx target = expand_normal (g_target);
+ machine_mode mode = GET_MODE (target);
+
+ rtx untagged = targetm.memtag.untagged_pointer (ptr, target);
+ rtx tagged_value = targetm.memtag.set_tag (untagged, tag, target);
+ /* Really need to put the return value into the `target` RTX, since that's
+ the return value of the function.
+ `target` will be in ptr_mode, while `tagged_value` will be in Pmode.
+ These can be different. When they are different we try to truncate the
+ Pmode value into ptr_mode. This will usually lose the tag, but since such
+ a difference between ptr_mode and Pmode will already cause problems
+ wherever the HWASAN library returns a pointer losing the tag here does not
+ introduce new incompatibilities.
+
+ We do this mainly so that compiling for such a target with ptr_mode and
+ Pmode sizes being different doesn't ICE, even if the resulting binary is
+ not usable. */
+ if (tagged_value != target)
+ {
+ rtx temp = tagged_value;
+ machine_mode ret_mode = GET_MODE (tagged_value);
+ if (ret_mode != mode)
+ temp = simplify_gen_unary (TRUNCATE, mode, tagged_value, ret_mode);
+
+ emit_move_insn (target, temp);
+ }
+}
+
+/* This should get expanded in the sanopt pass. */
+
+static void
expand_ASAN_CHECK (internal_fn, gcall *)
{
gcc_unreachable ();
diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def
index
310d37aa53819791b5df1683afca831f08e5892a..91a7bfea3eecad747320afcd11e05eedb9f7fcbf
100644
--- a/gcc/internal-fn.def
+++ b/gcc/internal-fn.def
@@ -321,6 +321,13 @@ DEF_INTERNAL_FN (UBSAN_PTR, ECF_LEAF | ECF_NOTHROW, ". R .
")
DEF_INTERNAL_FN (UBSAN_OBJECT_SIZE, ECF_LEAF | ECF_NOTHROW, NULL)
DEF_INTERNAL_FN (ABNORMAL_DISPATCHER, ECF_NORETURN, NULL)
DEF_INTERNAL_FN (BUILTIN_EXPECT, ECF_CONST | ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (HWASAN_ALLOCA_UNPOISON, ECF_LEAF | ECF_NOTHROW, ". R ")
+DEF_INTERNAL_FN (HWASAN_CHOOSE_TAG, ECF_LEAF | ECF_NOTHROW, ". ")
+DEF_INTERNAL_FN (HWASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW,
+ ". . R . . ")
+DEF_INTERNAL_FN (HWASAN_MARK, ECF_LEAF | ECF_NOTHROW, NULL)
+DEF_INTERNAL_FN (HWASAN_SET_TAG,
+ ECF_TM_PURE | ECF_PURE | ECF_LEAF | ECF_NOTHROW, ". R R ")
DEF_INTERNAL_FN (ASAN_CHECK, ECF_TM_PURE | ECF_LEAF | ECF_NOTHROW,
". . R . . ")
DEF_INTERNAL_FN (ASAN_MARK, ECF_LEAF | ECF_NOTHROW, NULL)
diff --git a/gcc/sanitizer.def b/gcc/sanitizer.def
index
4f854fb994229fd4ed91d3b5cff7c7acff9a55bc..f02731e8f2b5c7e78000d768e7c472658883ccb4
100644
--- a/gcc/sanitizer.def
+++ b/gcc/sanitizer.def
@@ -183,6 +183,61 @@ DEF_SANITIZER_BUILTIN(BUILT_IN_ASAN_POINTER_SUBTRACT,
"__sanitizer_ptr_sub",
/* Hardware Address Sanitizer. */
DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_INIT, "__hwasan_init",
BT_FN_VOID, ATTR_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD1, "__hwasan_load1",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD2, "__hwasan_load2",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD4, "__hwasan_load4",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD8, "__hwasan_load8",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD16, "__hwasan_load16",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOADN, "__hwasan_loadN",
+ BT_FN_VOID_PTR_PTRMODE, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE1, "__hwasan_store1",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE2, "__hwasan_store2",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE4, "__hwasan_store4",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE8, "__hwasan_store8",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE16, "__hwasan_store16",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STOREN, "__hwasan_storeN",
+ BT_FN_VOID_PTR_PTRMODE, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD1_NOABORT, "__hwasan_load1_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD2_NOABORT, "__hwasan_load2_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD4_NOABORT, "__hwasan_load4_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD8_NOABORT, "__hwasan_load8_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOAD16_NOABORT,
"__hwasan_load16_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_LOADN_NOABORT, "__hwasan_loadN_noabort",
+ BT_FN_VOID_PTR_PTRMODE, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE1_NOABORT,
"__hwasan_store1_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE2_NOABORT,
"__hwasan_store2_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE4_NOABORT,
"__hwasan_store4_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE8_NOABORT,
"__hwasan_store8_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STORE16_NOABORT,
+ "__hwasan_store16_noabort",
+ BT_FN_VOID_PTR, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_STOREN_NOABORT,
"__hwasan_storeN_noabort",
+ BT_FN_VOID_PTR_PTRMODE, ATTR_TMPURE_NOTHROW_LEAF_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_TAG_MISMATCH4, "__hwasan_tag_mismatch4",
+ BT_FN_VOID_PTR, ATTR_NOTHROW_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_HANDLE_LONGJMP,
"__hwasan_handle_longjmp",
+ BT_FN_VOID_CONST_PTR, ATTR_NOTHROW_LIST)
+DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_TAG_PTR, "__hwasan_tag_pointer",
+ BT_FN_PTR_CONST_PTR_UINT8, ATTR_TMPURE_NOTHROW_LEAF_LIST)
DEF_SANITIZER_BUILTIN(BUILT_IN_HWASAN_TAG_MEM, "__hwasan_tag_memory",
BT_FN_VOID_PTR_UINT8_PTRMODE, ATTR_NOTHROW_LIST)
diff --git a/gcc/sanopt.c b/gcc/sanopt.c
index
6c3bce923785f8a2dcd324edc0ce7ceb5f633a0c..965ab367c299f9315cddeb968d1c75086c8762b4
100644
--- a/gcc/sanopt.c
+++ b/gcc/sanopt.c
@@ -776,7 +776,8 @@ sanopt_optimize_walker (basic_block bb, class sanopt_ctx
*ctx)
basic_block son;
gimple_stmt_iterator gsi;
sanopt_info *info = (sanopt_info *) bb->aux;
- bool asan_check_optimize = (flag_sanitize & SANITIZE_ADDRESS) != 0;
+ bool asan_check_optimize
+ = ((flag_sanitize & (SANITIZE_ADDRESS | SANITIZE_HWADDRESS)) != 0);
for (gsi = gsi_start_bb (bb); !gsi_end_p (gsi);)
{
@@ -806,6 +807,7 @@ sanopt_optimize_walker (basic_block bb, class sanopt_ctx
*ctx)
if (asan_check_optimize
&& gimple_call_builtin_p (stmt, BUILT_IN_ASAN_BEFORE_DYNAMIC_INIT))
{
+ gcc_assert (!hwasan_sanitize_p ());
use_operand_p use;
gimple *use_stmt;
if (single_imm_use (gimple_vdef (stmt), &use, &use_stmt))
@@ -834,6 +836,7 @@ sanopt_optimize_walker (basic_block bb, class sanopt_ctx
*ctx)
case IFN_UBSAN_PTR:
remove = maybe_optimize_ubsan_ptr_ifn (ctx, stmt);
break;
+ case IFN_HWASAN_CHECK:
case IFN_ASAN_CHECK:
if (asan_check_optimize)
remove = maybe_optimize_asan_check_ifn (ctx, stmt);
@@ -1262,6 +1265,10 @@ sanitize_rewrite_addressable_params (function *fun)
unsigned int
pass_sanopt::execute (function *fun)
{
+ /* n.b. ASAN_MARK is used for both HWASAN and ASAN.
+ asan_num_accesses is hence used to count either HWASAN_CHECK or ASAN_CHECK
+ stuff. This is fine because you can only have one of these active at a
+ time. */
basic_block bb;
int asan_num_accesses = 0;
bool contains_asan_mark = false;
@@ -1269,10 +1276,10 @@ pass_sanopt::execute (function *fun)
/* Try to remove redundant checks. */
if (optimize
&& (flag_sanitize
- & (SANITIZE_NULL | SANITIZE_ALIGNMENT
+ & (SANITIZE_NULL | SANITIZE_ALIGNMENT | SANITIZE_HWADDRESS
| SANITIZE_ADDRESS | SANITIZE_VPTR | SANITIZE_POINTER_OVERFLOW)))
asan_num_accesses = sanopt_optimize (fun, &contains_asan_mark);
- else if (flag_sanitize & SANITIZE_ADDRESS)
+ else if (flag_sanitize & (SANITIZE_ADDRESS | SANITIZE_HWADDRESS))
{
gimple_stmt_iterator gsi;
FOR_EACH_BB_FN (bb, fun)
@@ -1292,7 +1299,7 @@ pass_sanopt::execute (function *fun)
sanitize_asan_mark_poison ();
}
- if (asan_sanitize_stack_p ())
+ if (asan_sanitize_stack_p () || hwasan_sanitize_stack_p ())
sanitize_rewrite_addressable_params (fun);
bool use_calls = param_asan_instrumentation_with_call_threshold < INT_MAX
@@ -1334,6 +1341,9 @@ pass_sanopt::execute (function *fun)
case IFN_UBSAN_VPTR:
no_next = ubsan_expand_vptr_ifn (&gsi);
break;
+ case IFN_HWASAN_CHECK:
+ no_next = hwasan_expand_check_ifn (&gsi, use_calls);
+ break;
case IFN_ASAN_CHECK:
no_next = asan_expand_check_ifn (&gsi, use_calls);
break;
@@ -1345,6 +1355,9 @@ pass_sanopt::execute (function *fun)
&need_commit_edge_insert,
shadow_vars_mapping);
break;
+ case IFN_HWASAN_MARK:
+ no_next = hwasan_expand_mark_ifn (&gsi);
+ break;
default:
break;
}
diff --git a/gcc/toplev.c b/gcc/toplev.c
index
9938b6afbd4fa22898dbc3c29b92061a71810b08..cb4ae77fccb44769732f9677a65c96be6321c9fa
100644
--- a/gcc/toplev.c
+++ b/gcc/toplev.c
@@ -512,7 +512,7 @@ compile_file (void)
if (flag_sanitize & SANITIZE_THREAD)
tsan_finish_file ();
- if (flag_sanitize & SANITIZE_HWADDRESS)
+ if (gate_hwasan ())
hwasan_finish_file ();
omp_finish_file ();
hwasan6-v6.patch.gz
Description: application/gzip
