From: Kyrylo Tkachov <[email protected]>
Postreload GCSE looks for an earlier load or store of the same memory in
each predecessor of the block of a load, or along a chain of single
predecessors. It then adds copies or loads on the incoming edges and
deletes the load. It does not see occurrences before a join, and it
gives up if the block uses the destination of the load before the load.
For example:
bb2: (set (mem:V4SI (reg:DI sp)) (reg:V4SI v0))
bb3, bb4: no change to v0 or to the slot
bb5: (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3)))
(set (reg:V4SI v1) (mem:V4SI (reg:DI sp)))
The load in bb5 can become:
(set (reg:V4SI v1) (reg:V4SI v0))
Compute which pairs of a memory expression and a hard register that
holds its value are available at the start of each block, with
compute_available. An available occurrence whose register does not
change later in its block generates its pair. A block kills a pair if
the expression is not transparent through the block or if the block
sets the register. compute_hash_table records the registers that each
block sets, and treats partial call clobbers as full clobbers. Blocks
with abnormal predecessors start with no pairs.
In the scan for partially redundant loads, first try to replace each
load with a copy from a register that holds the value at the start of
the block and that the block does not change before the load. The load
keeps its destination, so the occurrences in the hash table stay valid.
Also scan cold blocks, but there only replace loads, and only if the
copy is no larger than the load.
The pairs are only computed when transparency is. The bitmaps take
(4 * blocks + FIRST_PSEUDO_REGISTER) * pairs bits. If that exceeds
max-gcse-memory, the analysis is skipped with a note in the dump file.
Building the kill sets takes time linear in the size of the
transparency bitmaps and the new bitmaps. compute_available needs all
blocks to be reachable, so delete unreachable blocks first, as PRE and
cprop do.
On the motivating testcase from dav1d at -O3 -fno-ivopts, this patch removes
27% of the
SVE vector loads (ldr z) that remain after patch 2. With patch 2, that
is 33% of the loads in base, all of them reloads of spill slots. At -O3
it forwards two scalar reloads.
Compile time, as instruction counts of the aarch64 cc1 on an x86_64 host:
- diamonds-N, generated RTL with one stack slot that is reloaded after
each of N joins: v1 of this patch walked the CFG for each load, and
the count grew from +48% over base at N = 1024 to +804% at N = 12288.
This version stays within 1% of base.
- slots-N, with about N pairs in 3N blocks: within 0.3% of base up to
N = 4096, and +1.4% at N = 8192, where the new bitmaps need 97 MB,
less than the default max-gcse-memory. The existing transparency
computation takes most of the time for these inputs.
- At -O3 the series adds 0.05% to 0.08% for combine.ii, dwarf2out.ii,
fold-const.ii and sqlite3.i.
Bootstrapped and tested on aarch64-none-linux-gnu and x86_64-linux.
Ok for trunk?
Thanks,
Kyrill
Cc: Jeff Law <[email protected]>
gcc/ChangeLog:
* postreload-gcse.cc: Include cfgcleanup.h and lcm.h. Update the
overview comment.
(stats): Add loads_forwarded.
(struct expr): Add first_fact and num_facts.
(struct occr): Add reg_available_p.
(regs_set_in_block, modified_regs, fact_regs, fact_avin): New
variables.
(reg_changed_after_insn_p): Declare.
(alloc_mem, free_mem): Allocate and free modified_regs.
(insert_expr_in_table): Set reg_available_p.
(record_last_reg_set_info, record_last_reg_set_info_regno): Add the
register to regs_set_in_block.
(reset_opr_set_tables): Clear regs_set_in_block.
(compute_hash_table): Record regs_set_in_block in modified_regs.
(lookup_load_in_table): New function, split out of...
(eliminate_partially_redundant_loads): ...here. Try
replace_fully_redundant_load first. In cold blocks, only do that.
(get_fact, compute_fact_availability): New functions.
(replace_fully_redundant_load): Likewise.
(gcse_after_reload_main): Delete unreachable blocks. Call
compute_fact_availability. Free fact_avin and fact_regs. Report
the number of forwarded loads.
gcc/testsuite/ChangeLog:
* gcc.dg/rtl/aarch64/postreload-gcse-available.c: New test.
* gcc.dg/rtl/aarch64/postreload-gcse-clobber.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-gcse-forward.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-gcse-kill.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-gcse-memory.c: Likewise.
* gcc.target/aarch64/sve/spill-gcse.c: Likewise.
* gcc.target/aarch64/sve/spill-gcse-cold.c: Likewise.
Signed-off-by: Kyrylo Tkachov <[email protected]>
---
Changes since v1:
- Replace the per-load backward walks over the CFG with an availability
problem that compute_available solves (Jeff).
- Replace loads in the existing scan for partially redundant loads.
- Respect max-gcse-memory.
- More tests: kills, abnormal edges, call clobbers, register pairs,
max-gcse-memory, cold blocks.
gcc/postreload-gcse.cc | 304 +++++++++++++--
.../rtl/aarch64/postreload-gcse-available.c | 126 +++++++
.../rtl/aarch64/postreload-gcse-clobber.c | 132 +++++++
.../rtl/aarch64/postreload-gcse-forward.c | 349 ++++++++++++++++++
.../gcc.dg/rtl/aarch64/postreload-gcse-kill.c | 284 ++++++++++++++
.../rtl/aarch64/postreload-gcse-memory.c | 117 ++++++
.../gcc.target/aarch64/sve/spill-gcse-cold.c | 33 ++
.../gcc.target/aarch64/sve/spill-gcse.c | 29 ++
8 files changed, 1343 insertions(+), 31 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-available.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-clobber.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-forward.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-kill.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-memory.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/spill-gcse-cold.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/spill-gcse.c
diff --git a/gcc/postreload-gcse.cc b/gcc/postreload-gcse.cc
index 2203c60ae..4c1a6ba27 100644
--- a/gcc/postreload-gcse.cc
+++ b/gcc/postreload-gcse.cc
@@ -33,6 +33,7 @@ along with GCC; see the file COPYING3. If not see
#include "recog.h"
#include "cfgrtl.h"
+#include "cfgcleanup.h"
#include "profile.h"
#include "expr.h"
#include "tree-pass.h"
@@ -40,6 +41,7 @@ along with GCC; see the file COPYING3. If not see
#include "intl.h"
#include "gcse-common.h"
#include "gcse.h"
+#include "lcm.h"
#include "regs.h"
#include "function-abi.h"
@@ -61,7 +63,11 @@ along with GCC; see the file COPYING3. If not see
change until the end of the basic block add this memory expression to
the hash table.
2. Perform Redundancy elimination:
+ Compute where the holding register of each available memory
+ expression still contains the memory value.
For each load instruction do the following:
+ if the same register holds the value of the load on all incoming
+ paths, replace the load with a copy from that register. Otherwise
perform partial redundancy elimination, check if it's worth adding
loads to make the load fully redundant. If so add loads and
register copies and delete the load.
@@ -81,6 +87,7 @@ static struct
int moves_inserted;
int copies_inserted;
int insns_deleted;
+ int loads_forwarded;
} stats;
/* We need to keep a hash table of expressions. The table entries are of
@@ -99,6 +106,12 @@ struct expr
/* Index in the transparent bitmaps. */
unsigned int bitmap_index;
+ /* The availability facts of this expression, one for each distinct
+ holding register, have the indices FIRST_FACT to
+ FIRST_FACT + NUM_FACTS - 1. */
+ unsigned int first_fact;
+ unsigned int num_facts;
+
/* List of available occurrence in basic blocks in the function. */
struct occr *avail_occr;
};
@@ -165,6 +178,9 @@ struct occr
rtx_insn *insn;
/* Nonzero if this [anticipatable] occurrence has been deleted. */
char deleted_p;
+ /* True if the loaded or stored register of INSN keeps its value until
+ the end of the block. */
+ bool reg_available_p;
};
static struct obstack occr_obstack;
@@ -191,6 +207,9 @@ static struct obstack unoccr_obstack;
if a reg was modified since the start of a basic block. */
static int *reg_avail_info;
+/* The hard registers that REG_AVAIL_INFO records as set. */
+static HARD_REG_SET regs_set_in_block;
+
/* A list of insns that may modify memory within the current basic block. */
struct modifies_mem
{
@@ -229,6 +248,18 @@ static vec<modify_pair> *canon_modify_mem_list;
/* Vector of simple bitmaps indexed by block number. Each component sbitmap
indicates which expressions are transparent through the block. */
static sbitmap *transp;
+
+/* Vector indexed by block number of the hard registers that the block
+ modifies, including partial call clobbers. */
+static HARD_REG_SET *modified_regs;
+
+/* For each availability fact, the hard register that holds the value of
+ the expression. */
+static vec<rtx> fact_regs;
+
+/* Vector indexed by block number of the availability facts at the start of
+ the block, or null if they are not known. */
+static sbitmap *fact_avin;
/* Helpers for memory allocation/freeing. */
@@ -236,6 +267,7 @@ static void alloc_mem (void);
static void free_mem (void);
/* Support for hash table construction and transformations. */
+static bool reg_changed_after_insn_p (rtx, int);
static bool oprs_unchanged_p (rtx, rtx_insn *, bool);
static void record_last_reg_set_info (rtx_insn *, rtx);
static void record_last_reg_set_info_regno (rtx_insn *, int);
@@ -324,6 +356,7 @@ alloc_mem (void)
canon_modify_mem_list
= (vec_modify_pair_heap *) xcalloc (last_basic_block_for_fn (cfun),
sizeof (vec_modify_pair_heap));
+ modified_regs = XNEWVEC (HARD_REG_SET, last_basic_block_for_fn (cfun));
}
/* Free memory allocated by alloc_mem. */
@@ -354,6 +387,7 @@ free_mem (void)
free (reg_avail_info);
free (modify_mem_list);
free (canon_modify_mem_list);
+ free (modified_regs);
}
@@ -425,6 +459,11 @@ insert_expr_in_table (rtx x, rtx_insn *insn)
avail_occr->deleted_p = 0;
cur_expr->avail_occr = avail_occr;
}
+
+ /* REG_AVAIL_INFO describes the whole block at this point. */
+ avail_occr->reg_available_p
+ = !reg_changed_after_insn_p (get_avail_load_store_reg (insn),
+ INSN_CUID (insn));
}
@@ -679,7 +718,10 @@ record_last_reg_set_info (rtx_insn *insn, rtx reg)
regno = REGNO (reg);
end_regno = END_REGNO (reg);
do
- reg_avail_info[regno] = INSN_CUID (insn);
+ {
+ reg_avail_info[regno] = INSN_CUID (insn);
+ SET_HARD_REG_BIT (regs_set_in_block, regno);
+ }
while (++regno < end_regno);
}
@@ -687,6 +729,7 @@ static inline void
record_last_reg_set_info_regno (rtx_insn *insn, int regno)
{
reg_avail_info[regno] = INSN_CUID (insn);
+ SET_HARD_REG_BIT (regs_set_in_block, regno);
}
@@ -747,6 +790,7 @@ static void
reset_opr_set_tables (void)
{
memset (reg_avail_info, 0, FIRST_PSEUDO_REGISTER * sizeof (int));
+ CLEAR_HARD_REG_SET (regs_set_in_block);
obstack_free (&modifies_mem_obstack, modifies_mem_obstack_bottom);
modifies_mem_list = NULL;
}
@@ -851,7 +895,8 @@ hash_scan_set (rtx_insn *insn)
blocks. Basically you should think of this hash table as the
representation of AVAIL_OUT. This is the set of expressions that
is generated in a basic block and not killed before the end of the
- same basic block. Notice that this is really a local computation. */
+ same basic block. Notice that this is really a local computation.
+ Also record in MODIFIED_REGS the hard registers that each block sets. */
static void
compute_hash_table (void)
@@ -874,6 +919,8 @@ compute_hash_table (void)
record_opr_changes (insn);
}
+ modified_regs[bb->index] = regs_set_in_block;
+
/* The next pass actually builds the hash table. */
FOR_BB_INSNS (bb, insn)
if (INSN_P (insn) && GET_CODE (PATTERN (insn)) == SET)
@@ -1002,6 +1049,204 @@ compute_expr_transp (expr **slot, FILE *dump_file
ATTRIBUTE_UNUSED)
return 1;
}
+/* Return the hash table entry for the memory that INSN loads, if INSN is
+ a load whose memory operands are unchanged since the start of its block.
+ Return null otherwise. */
+
+static struct expr *
+lookup_load_in_table (rtx_insn *insn)
+{
+ /* Is it a load - of the form (set (reg) (mem))? */
+ if (!NONJUMP_INSN_P (insn)
+ || GET_CODE (PATTERN (insn)) != SET
+ || !REG_P (SET_DEST (PATTERN (insn)))
+ || !MEM_P (SET_SRC (PATTERN (insn))))
+ return NULL;
+
+ rtx src = SET_SRC (PATTERN (insn));
+ if (MEM_VOLATILE_P (src)
+ || GET_MODE (src) == BLKmode
+ || !general_operand (src, GET_MODE (src))
+ /* Are the operands unchanged since the start of the block? */
+ || !oprs_unchanged_p (src, insn, false)
+ || (cfun->can_throw_non_call_exceptions && may_trap_p (src))
+ || side_effects_p (src))
+ return NULL;
+
+ /* Is the expression recorded? */
+ return lookup_expr_in_table (src);
+}
+
+/* Return the index of the availability fact for EXPR held in REG, adding
+ REG to FACT_REGS if necessary. */
+
+static unsigned int
+get_fact (struct expr *expr, rtx reg)
+{
+ unsigned int i = expr->first_fact;
+ while (i < fact_regs.length () && REGNO (fact_regs[i]) != REGNO (reg))
+ i++;
+ if (i == fact_regs.length ())
+ fact_regs.safe_push (reg);
+ return i;
+}
+
+/* Compute FACT_AVIN, where each fact is a pair of an expression and a hard
+ register that holds its value. The available occurrences whose register
+ survives to the end of their block generate the facts. A block kills a
+ fact if the expression is not transparent through it or if it sets the
+ register.
+
+ The facts describe the instructions before the redundancy elimination.
+ For each load that eliminate_partially_redundant_load removes, it adds
+ instructions only to the incoming edges of the load's block. They only
+ set the destination of the load, which the block does not use or set
+ before the load. The facts at the start of a block therefore remain
+ valid for the loads that the elimination processes later. */
+
+static void
+compute_fact_availability (void)
+{
+ /* Number the distinct holding registers of each expression. */
+ unsigned int n_exprs = expr_table->elements ();
+ auto_vec<struct expr *> exprs;
+ exprs.safe_grow_cleared (n_exprs);
+ for (struct expr *expr : *expr_table)
+ {
+ exprs[expr->bitmap_index] = expr;
+ expr->first_fact = fact_regs.length ();
+ for (struct occr *occr = expr->avail_occr; occr; occr = occr->next)
+ if (occr->reg_available_p)
+ get_fact (expr, get_avail_load_store_reg (occr->insn));
+ expr->num_facts = fact_regs.length () - expr->first_fact;
+ }
+
+ unsigned int n_facts = fact_regs.length ();
+ if (n_facts == 0)
+ return;
+
+ /* Use the GCSE memory limit for the bitmap vectors of the problem, four
+ indexed by block and one indexed by hard register. */
+ int n_blocks = last_basic_block_for_fn (cfun);
+ unsigned HOST_WIDE_INT memory_request
+ = ((4 * (unsigned HOST_WIDE_INT) n_blocks + FIRST_PSEUDO_REGISTER)
+ * SBITMAP_SET_SIZE (n_facts) * sizeof (SBITMAP_ELT_TYPE));
+ if (memory_request / 1024 > (unsigned HOST_WIDE_INT) param_max_gcse_memory)
+ {
+ if (dump_file)
+ fprintf (dump_file, "not computing the availability of %u values in "
+ "registers: needs " HOST_WIDE_INT_PRINT_UNSIGNED "kB, more "
+ "than max-gcse-memory\n", n_facts, memory_request / 1024);
+ return;
+ }
+
+ sbitmap *avloc = sbitmap_vector_alloc (n_blocks, n_facts);
+ sbitmap *kill = sbitmap_vector_alloc (n_blocks, n_facts);
+ sbitmap *avout = sbitmap_vector_alloc (n_blocks, n_facts);
+ fact_avin = sbitmap_vector_alloc (n_blocks, n_facts);
+ bitmap_vector_clear (avloc, n_blocks);
+ bitmap_vector_clear (kill, n_blocks);
+
+ /* Record the facts held in each hard register. */
+ sbitmap *reg_facts = sbitmap_vector_alloc (FIRST_PSEUDO_REGISTER, n_facts);
+ bitmap_vector_clear (reg_facts, FIRST_PSEUDO_REGISTER);
+ HARD_REG_SET fact_hard_regs;
+ CLEAR_HARD_REG_SET (fact_hard_regs);
+ for (unsigned int i = 0; i < n_facts; i++)
+ for (unsigned int regno = REGNO (fact_regs[i]);
+ regno < END_REGNO (fact_regs[i]); regno++)
+ {
+ bitmap_set_bit (reg_facts[regno], i);
+ SET_HARD_REG_BIT (fact_hard_regs, regno);
+ }
+
+ auto_sbitmap has_facts (n_exprs);
+ bitmap_clear (has_facts);
+ for (struct expr *expr : *expr_table)
+ {
+ if (expr->num_facts)
+ bitmap_set_bit (has_facts, expr->bitmap_index);
+ for (struct occr *occr = expr->avail_occr; occr; occr = occr->next)
+ if (occr->reg_available_p)
+ {
+ unsigned int i = get_fact (expr,
+ get_avail_load_store_reg (occr->insn));
+ gcc_checking_assert (i < expr->first_fact + expr->num_facts);
+ bitmap_set_bit (avloc[BLOCK_FOR_INSN (occr->insn)->index], i);
+ }
+ }
+
+ /* Registers need not keep their values on abnormal edges, for example
+ on those to a setjmp receiver. */
+ auto_sbitmap opaque (n_exprs);
+ basic_block bb;
+ FOR_EACH_BB_FN (bb, cfun)
+ {
+ sbitmap bb_kill = kill[bb->index];
+ if (bb_has_abnormal_pred (bb))
+ {
+ bitmap_ones (bb_kill);
+ continue;
+ }
+
+ unsigned int i;
+ sbitmap_iterator sbi;
+ bitmap_and_compl (opaque, has_facts, transp[bb->index]);
+ EXECUTE_IF_SET_IN_BITMAP (opaque, 0, i, sbi)
+ bitmap_set_range (bb_kill, exprs[i]->first_fact, exprs[i]->num_facts);
+
+ hard_reg_set_iterator hrsi;
+ HARD_REG_SET regs = modified_regs[bb->index] & fact_hard_regs;
+ EXECUTE_IF_SET_IN_HARD_REG_SET (regs, 0, i, hrsi)
+ bitmap_ior (bb_kill, bb_kill, reg_facts[i]);
+ }
+
+ compute_available (avloc, kill, avout, fact_avin);
+ sbitmap_vector_free (reg_facts);
+ sbitmap_vector_free (avloc);
+ sbitmap_vector_free (kill);
+ sbitmap_vector_free (avout);
+}
+
+/* Try to replace the source of INSN, a load in BB of the memory in EXPR,
+ with a register that holds the same value. AVIN contains the facts that
+ are available at the start of BB. The register must not have changed
+ since the start of BB. Keep INSN and its destination, so that the
+ occurrences of EXPR remain valid. Return true on success. */
+
+static bool
+replace_fully_redundant_load (basic_block bb, rtx_insn *insn,
+ struct expr *expr, sbitmap avin)
+{
+ rtx set = PATTERN (insn);
+ for (unsigned int i = expr->first_fact;
+ i < expr->first_fact + expr->num_facts; i++)
+ {
+ rtx reg = fact_regs[i];
+ if (!bitmap_bit_p (avin, i) || reg_changed_after_insn_p (reg, 0))
+ continue;
+
+ /* In blocks optimized for size, require a copy that is no larger
+ than the load. */
+ if (optimize_bb_for_size_p (bb)
+ && (set_rtx_cost (gen_rtx_SET (SET_DEST (set), reg), false)
+ > set_rtx_cost (set, false)))
+ continue;
+
+ if (validate_change (insn, &SET_SRC (set), copy_rtx (reg), 0))
+ {
+ stats.loads_forwarded++;
+ if (dump_file)
+ {
+ fprintf (dump_file, "forwarding load from available register: ");
+ print_rtl_single (dump_file, insn);
+ }
+ return true;
+ }
+ }
+ return false;
+}
+
/* This handles the case where several stores feed a partially redundant
load. It checks if the redundancy elimination is possible and if it's
worth it.
@@ -1257,8 +1502,15 @@ eliminate_partially_redundant_loads (void)
if (! bb_has_well_behaved_predecessors (bb))
continue;
- /* Do not try anything on cold basic blocks. */
- if (optimize_bb_for_size_p (bb))
+ /* Registers need not keep their values on abnormal edges, for
+ example on EH edges from instructions that can trap. */
+ sbitmap avin = NULL;
+ if (fact_avin && !bb_has_abnormal_pred (bb))
+ avin = fact_avin[bb->index];
+
+ /* In cold basic blocks, only try to replace loads with copies. */
+ bool cold_p = optimize_bb_for_size_p (bb);
+ if (cold_p && (!avin || bitmap_empty_p (avin)))
continue;
/* Reset the table of things changed since the start of the current
@@ -1269,33 +1521,12 @@ eliminate_partially_redundant_loads (void)
any loads in it that we can record. */
FOR_BB_INSNS (bb, insn)
{
- /* Is it a load - of the form (set (reg) (mem))? */
- if (NONJUMP_INSN_P (insn)
- && GET_CODE (PATTERN (insn)) == SET
- && REG_P (SET_DEST (PATTERN (insn)))
- && MEM_P (SET_SRC (PATTERN (insn))))
- {
- rtx pat = PATTERN (insn);
- rtx src = SET_SRC (pat);
- struct expr *expr;
-
- if (!MEM_VOLATILE_P (src)
- && GET_MODE (src) != BLKmode
- && general_operand (src, GET_MODE (src))
- /* Are the operands unchanged since the start of the
- block? */
- && oprs_unchanged_p (src, insn, false)
- && !(cfun->can_throw_non_call_exceptions && may_trap_p (src))
- && !side_effects_p (src)
- /* Is the expression recorded? */
- && (expr = lookup_expr_in_table (src)) != NULL)
- {
- /* We now have a load (insn) and an available memory at
- its BB start (expr). Try to remove the loads if it is
- redundant. */
- eliminate_partially_redundant_load (bb, insn, expr);
- }
- }
+ /* If INSN loads an expression that is available at the end of
+ some block, try to remove the load if it is redundant. */
+ if (struct expr *expr = lookup_load_in_table (insn))
+ if (!(avin && replace_fully_redundant_load (bb, insn, expr, avin))
+ && !cold_p)
+ eliminate_partially_redundant_load (bb, insn, expr);
/* Keep track of everything modified by this insn, so that we
know what has been modified since the start of the current
@@ -1351,6 +1582,9 @@ delete_redundant_insns (void)
static void
gcse_after_reload_main (rtx f ATTRIBUTE_UNUSED)
{
+ /* compute_available requires all blocks to be reachable. */
+ delete_unreachable_blocks ();
+
/* Disable computing transparentness if it is too expensive. */
bool do_transp
= !gcse_or_cprop_is_too_expensive (_("using simple load CSE after register
"
@@ -1384,6 +1618,7 @@ gcse_after_reload_main (rtx f ATTRIBUTE_UNUSED)
expr_table->elements ());
bitmap_vector_ones (transp, last_basic_block_for_fn (cfun));
expr_table->traverse <FILE *, compute_expr_transp> (dump_file);
+ compute_fact_availability ();
}
else
transp = NULL;
@@ -1391,16 +1626,23 @@ gcse_after_reload_main (rtx f ATTRIBUTE_UNUSED)
delete_redundant_insns ();
if (do_transp)
sbitmap_vector_free (transp);
+ if (fact_avin)
+ sbitmap_vector_free (fact_avin);
+ fact_avin = NULL;
+ fact_regs.release ();
if (dump_file)
{
fprintf (dump_file, "GCSE AFTER RELOAD stats:\n");
+ fprintf (dump_file, "loads forwarded: %d\n", stats.loads_forwarded);
fprintf (dump_file, "copies inserted: %d\n", stats.copies_inserted);
fprintf (dump_file, "moves inserted: %d\n", stats.moves_inserted);
fprintf (dump_file, "insns deleted: %d\n", stats.insns_deleted);
fprintf (dump_file, "\n\n");
}
+ statistics_counter_event (cfun, "loads forwarded",
+ stats.loads_forwarded);
statistics_counter_event (cfun, "copies inserted",
stats.copies_inserted);
statistics_counter_event (cfun, "moves inserted",
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-available.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-available.c
new file mode 100644
index 000000000..3088cfd96
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-available.c
@@ -0,0 +1,126 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fgcse-after-reload -fdump-rtl-gcse2" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+/* { dg-final { scan-rtl-dump-times "forwarding load from available register:"
1 "gcse2" } } */
+
+/* Each arm writes a different value to the stack, but Q0 holds that value
+ on both incoming paths. Neither store dominates the join. */
+
+/*
+** same_register:
+** ...
+** str q0, \[x1\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+same_register (void)
+{
+(function "same_register"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cinsn 27 (set (reg:V4SI v0) (plus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cinsn 24 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 1000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (cinsn 25 (set (reg:V4SI v0) (minus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cinsn 26 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 1001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* The expression is available on both paths, but in different registers.
+ The join uses the old Q1, so edge copies cannot replace the load either. */
+
+/*
+** different_registers:
+** ...
+** ldr q[0-9]+, \[sp\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+different_registers (void)
+{
+(function "different_registers"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cinsn 27 (set (reg:V4SI v0) (plus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cinsn 24 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 2000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (cinsn 25 (set (reg:V4SI v0) (minus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cinsn 26 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v3)))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 2001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-clobber.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-clobber.c
new file mode 100644
index 000000000..5973f16de
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-clobber.c
@@ -0,0 +1,132 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fgcse-after-reload -fdump-rtl-gcse2" } */
+/* { dg-final { scan-rtl-dump-times "forwarding load from available register:"
2 "gcse2" } } */
+/* { dg-final { scan-rtl-dump-not "generating" "gcse2" } } */
+
+/* A call to a const function on one path preserves X21 but clobbers the
+ upper half of V8. The join block uses the old destinations before the
+ reloads, so copies on the incoming edges are not valid. */
+
+void __RTL (startwith ("ira"))
+call (void)
+{
+(function "call"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -48))))
+ (cinsn 3 (set (mem:V16QI (plus:DI (reg:DI sp) (const_int 32)) [0 S16
A128])
+ (reg:V16QI v8)))
+ (cinsn 4 (set (mem:DI (reg:DI sp) [0 S8 A64]) (reg:DI x21)))
+ (cinsn 5 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 6 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 20) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 7 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (ccall_insn/u 8 (parallel
+ [(set (reg:DI x0)
+ (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32])
+ (const_int 0)))
+ (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI x30))]) (nil) (nil) 0)
+ (cinsn 9 (set (reg:DI x4) (reg:DI x0)))
+ (cjump_insn 10 (set (pc) (label_ref 30)))
+ (edge-to 5)
+ )
+ (cbarrier 11)
+ (block 4
+ (edge-from 2)
+ (clabel 20 1000)
+ (cnote 21 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 30 1001)
+ (cnote 31 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (set (reg:V16QI v2) (plus:V16QI (reg:V16QI v1) (reg:V16QI
v3))))
+ (cinsn 33 (set (reg:V16QI v1) (mem:V16QI (plus:DI (reg:DI sp) (const_int
32)) [0 S16 A128])))
+ (cinsn 34 (set (reg:DI x3) (plus:DI (reg:DI x22) (const_int 1))))
+ (cinsn 35 (set (reg:DI x22) (mem:DI (reg:DI sp) [0 S8 A64])))
+ (cinsn 36 (set (mem:V16QI (reg:DI x1) [0 S16 A128]) (reg:V16QI v1)))
+ (cinsn 37 (set (mem:V16QI (reg:DI x2) [0 S16 A128]) (reg:V16QI v2)))
+ (cinsn 38 (set (mem:DI (reg:DI x6) [0 S8 A64]) (reg:DI x22)))
+ (cinsn 39 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x3)))
+ (cinsn 40 (set (mem:DI (reg:DI x8) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 41 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 48))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* A write to X1 on one path changes the register pair X0:X1. The value
+ of the other slot is still available in X2:X3. The join block uses the
+ old destinations before the reloads. */
+
+void __RTL (startwith ("ira"))
+pair (void)
+{
+(function "pair"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:TI (reg:DI sp) [0 S16 A128]) (reg:TI x0)))
+ (cinsn 4 (set (mem:TI (plus:DI (reg:DI sp) (const_int 16)) [0 S16 A128])
+ (reg:TI x2)))
+ (cinsn 5 (set (reg:CC cc) (compare:CC (reg:SI x4) (const_int 0))))
+ (cjump_insn 6 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 20) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 7 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 8 (set (reg:DI x1) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 9 (set (pc) (label_ref 30)))
+ (edge-to 5)
+ )
+ (cbarrier 10)
+ (block 4
+ (edge-from 2)
+ (clabel 20 2000)
+ (cnote 21 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:DI x16) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 30 2001)
+ (cnote 31 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (set (reg:DI x12) (plus:DI (reg:DI x6) (reg:DI x7))))
+ (cinsn 33 (set (reg:TI x6) (mem:TI (reg:DI sp) [0 S16 A128])))
+ (cinsn 34 (set (reg:DI x13) (plus:DI (reg:DI x8) (const_int 1))))
+ (cinsn 35 (set (reg:TI x8) (mem:TI (plus:DI (reg:DI sp) (const_int 16))
[0 S16 A128])))
+ (cinsn 36 (set (mem:TI (reg:DI x10) [0 S16 A128]) (reg:TI x6)))
+ (cinsn 37 (set (mem:TI (reg:DI x11) [0 S16 A128]) (reg:TI x8)))
+ (cinsn 38 (set (mem:DI (reg:DI x14) [0 S8 A64]) (reg:DI x12)))
+ (cinsn 39 (set (mem:DI (reg:DI x15) [0 S8 A64]) (reg:DI x13)))
+ (cinsn 40 (set (mem:DI (reg:DI x17) [0 S8 A64]) (reg:DI x1)))
+ (cinsn 41 (set (mem:DI (reg:DI x20) [0 S8 A64]) (reg:DI x16)))
+ (cinsn 42 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* { dg-final { scan-assembler {\tldr\tq[0-9]+, \[sp, 32\]} } } */
+/* { dg-final { scan-assembler-not {\tldr\tx22, \[sp\]} } } */
+/* { dg-final { scan-assembler {\tldp\tx6, x7, \[sp\]} } } */
+/* { dg-final { scan-assembler-not {\tldp\tx8, x9, \[sp, 16\]} } } */
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-forward.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-forward.c
new file mode 100644
index 000000000..5a281bd55
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-forward.c
@@ -0,0 +1,349 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fgcse-after-reload -fdump-rtl-gcse2" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+/* { dg-final { scan-rtl-dump-times "forwarding load from available register:"
2 "gcse2" } } */
+
+/* Q0 holds the stack value across a diamond. The join uses the old Q1
+ before reloading it, so a copy on the incoming edges is not valid. */
+
+/*
+** join_forward:
+** ...
+** str q0, \[x1\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+join_forward (void)
+{
+(function "join_forward"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 1000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 1001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* A write to D0 also changes the value of Q0. */
+
+/*
+** source_clobber:
+** ...
+** ldr q[0-9]+, \[sp\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+source_clobber (void)
+{
+(function "source_clobber"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 2000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 12 (set (reg:DI v0) (reg:DI v4)))
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 2001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* A partial store on one incoming path invalidates the stack value. */
+
+/*
+** alias_store:
+** ...
+** ldr q[0-9]+, \[sp\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+alias_store (void)
+{
+(function "alias_store"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 3000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 12 (set (mem:SI (reg:DI sp) [0 S4 A32]) (reg:SI x6)))
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 3001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* Q0 holds the value of [sp] on both paths, but only one path stores
+ to [sp, 16]. */
+
+/*
+** different_address:
+** ...
+** ldr q[0-9]+, \[sp, 16\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+different_address (void)
+{
+(function "different_address"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 4000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 12 (set (mem:V4SI (plus:DI (reg:DI sp) (const_int 16)) [0 S16
A128])
+ (reg:V4SI v5)))
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 4001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (plus:DI (reg:DI sp) (const_int
16)) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* The loop can overwrite the stored value before the reload. */
+
+/*
+** loop_store:
+** ...
+** ldr q[0-9]+, \[sp\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+loop_store (void)
+{
+(function "loop_store"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (edge-to 3 (flags "FALLTHRU"))
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (edge-from 4)
+ (clabel 4 5000)
+ (cnote 5 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 6 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 7 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 14) (pc))))
+ (edge-to 4 (flags "FALLTHRU"))
+ (edge-to 5)
+ )
+ (block 4
+ (edge-from 3 (flags "FALLTHRU"))
+ (cnote 8 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 9 (set (mem:SI (reg:DI sp) [0 S4 A32]) (reg:SI x6)))
+ (cinsn 10 (set (reg:SI x0) (plus:SI (reg:SI x0) (const_int -1))))
+ (cjump_insn 11 (set (pc) (label_ref 4)))
+ (edge-to 3)
+ )
+ (cbarrier 12)
+ (block 5
+ (edge-from 3)
+ (clabel 14 5001)
+ (cnote 15 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 16 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 17 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 18 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* The entry store anchors the value across a loop that preserves Q0 and
+ the stack slot. The exit uses Q1 before the reload. */
+
+/*
+** loop_forward:
+** ...
+** str q0, \[x1\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+loop_forward (void)
+{
+(function "loop_forward"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (edge-to 3 (flags "FALLTHRU"))
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (edge-from 4)
+ (clabel 4 6000)
+ (cnote 5 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 6 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 7 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 14) (pc))))
+ (edge-to 4 (flags "FALLTHRU"))
+ (edge-to 5)
+ )
+ (block 4
+ (edge-from 3 (flags "FALLTHRU"))
+ (cnote 8 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 9 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 1))))
+ (cinsn 10 (set (reg:SI x0) (plus:SI (reg:SI x0) (const_int -1))))
+ (cjump_insn 11 (set (pc) (label_ref 4)))
+ (edge-to 3)
+ )
+ (cbarrier 12)
+ (block 5
+ (edge-from 3)
+ (clabel 14 6001)
+ (cnote 15 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 19 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 16 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 17 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 18 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-kill.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-kill.c
new file mode 100644
index 000000000..d265e955e
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-kill.c
@@ -0,0 +1,284 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fgcse-after-reload -fdump-rtl-gcse2" } */
+/* { dg-final { scan-rtl-dump-not "forwarding load from available register:"
"gcse2" } } */
+
+/* Each reload follows a use of its old destination. None of the reloads
+ can become a copy of the register that held the value. */
+
+/* X19 holds the value of the slot at the end of block 3, but the only
+ predecessor of block 5 is an abnormal edge. */
+void __RTL (startwith ("ira"))
+abnormal_only (void)
+{
+(function "abnormal_only"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -16))))
+ (cinsn 3 (set (mem:DI (reg:DI sp) [0 S8 A64]) (reg:DI x19)))
+ (edge-to 3 (flags "FALLTHRU"))
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (ccall_insn/u 7 (parallel
+ [(set (reg:DI x0)
+ (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32])
+ (const_int 0)))
+ (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI x30))]) (nil) (nil) 0)
+ (edge-to 4 (flags "FALLTHRU"))
+ (edge-to 5 (flags "ABNORMAL | ABNORMAL_CALL"))
+ )
+ (block 4
+ (edge-from 3 (flags "FALLTHRU"))
+ (cnote 8 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 9 (set (mem:DI (reg:DI x8) [0 S8 A64]) (reg:DI x0)))
+ (cjump_insn 10 (set (pc) (label_ref 30)))
+ (edge-to 6)
+ )
+ (cbarrier 11)
+ (block 5
+ (edge-from 3 (flags "ABNORMAL | ABNORMAL_CALL"))
+ (clabel 20 1000)
+ (cnote 21 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:DI x3) (plus:DI (reg:DI x22) (const_int 1))))
+ (cinsn 23 (set (reg:DI x22) (mem:DI (reg:DI sp) [0 S8 A64])))
+ (cinsn 24 (set (mem:DI (reg:DI x6) [0 S8 A64]) (reg:DI x22)))
+ (cinsn 25 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x3)))
+ (edge-to 6 (flags "FALLTHRU"))
+ )
+ (block 6
+ (edge-from 4)
+ (edge-from 5 (flags "FALLTHRU"))
+ (clabel 30 1001)
+ (cnote 31 [bb 6] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 16))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* Both arms store Q0, but one arm then changes Q0. */
+void __RTL (startwith ("ira"))
+changed_after_store (void)
+{
+(function "changed_after_store"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 24 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 27 (set (reg:V4SI v0) (plus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 2000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 26 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 2001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* Q0 holds the value on both paths, but the join block changes Q0
+ before the load. */
+void __RTL (startwith ("ira"))
+inblock_reg (void)
+{
+(function "inblock_reg"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 3000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 3001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 24 (set (reg:V4SI v0) (plus:V4SI (reg:V4SI v0) (reg:V4SI v4))))
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* The join block stores to the slot before the load. */
+void __RTL (startwith ("ira"))
+inblock_mem (void)
+{
+(function "inblock_mem"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -32))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 10) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 15)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 10 4000)
+ (cnote 11 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:DI x4) (plus:DI (reg:DI x5) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 15 4001)
+ (cnote 16 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 24 (set (mem:SI (reg:DI sp) [0 S4 A32]) (reg:SI x6)))
+ (cinsn 17 (set (reg:V4SI v2) (plus:V4SI (reg:V4SI v1) (reg:V4SI v3))))
+ (cinsn 18 (set (reg:V4SI v1) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 19 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (cinsn 20 (set (mem:V4SI (reg:DI x2) [0 S16 A128]) (reg:V4SI v2)))
+ (cinsn 21 (set (mem:V4SI (reg:DI x3) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 22 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 23 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 32))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* X19 holds the value of the slot on all paths into block 5, but one of
+ them is an abnormal edge. Block 6 has only a normal predecessor. */
+void __RTL (startwith ("ira"))
+abnsucc (void)
+{
+(function "abnsucc"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -16))))
+ (cinsn 3 (set (mem:DI (reg:DI sp) [0 S8 A64]) (reg:DI x19)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x1) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 20) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 5)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (ccall_insn/u 7 (parallel
+ [(set (reg:DI x0)
+ (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32])
+ (const_int 0)))
+ (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI x30))]) (nil) (nil) 0)
+ (edge-to 4 (flags "FALLTHRU"))
+ (edge-to 5 (flags "ABNORMAL | ABNORMAL_CALL"))
+ )
+ (block 4
+ (edge-from 3 (flags "FALLTHRU"))
+ (cnote 8 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 9 (set (reg:DI x4) (plus:DI (reg:DI x0) (const_int 1))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 2)
+ (edge-from 4 (flags "FALLTHRU"))
+ (edge-from 3 (flags "ABNORMAL | ABNORMAL_CALL"))
+ (clabel 20 5000)
+ (cnote 21 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 22 (set (reg:CC cc) (compare:CC (reg:SI x2) (const_int 0))))
+ (cjump_insn 23 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 30) (pc))))
+ (edge-to 6 (flags "FALLTHRU"))
+ (edge-to 7)
+ )
+ (block 6
+ (edge-from 5 (flags "FALLTHRU"))
+ (cnote 24 [bb 6] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 25 (set (reg:DI x3) (plus:DI (reg:DI x22) (const_int 1))))
+ (cinsn 26 (set (reg:DI x22) (mem:DI (reg:DI sp) [0 S8 A64])))
+ (cinsn 27 (set (mem:DI (reg:DI x6) [0 S8 A64]) (reg:DI x22)))
+ (cinsn 28 (set (mem:DI (reg:DI x7) [0 S8 A64]) (reg:DI x3)))
+ (edge-to 7 (flags "FALLTHRU"))
+ )
+ (block 7
+ (edge-from 5)
+ (edge-from 6 (flags "FALLTHRU"))
+ (clabel 30 5001)
+ (cnote 31 [bb 7] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 32 (set (mem:DI (reg:DI x8) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 33 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 16))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-memory.c
b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-memory.c
new file mode 100644
index 000000000..7ac4eff86
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-gcse-memory.c
@@ -0,0 +1,117 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -fgcse-after-reload --param max-gcse-memory=0
-fdump-rtl-gcse2" } */
+/* { dg-final { scan-rtl-dump "not computing the availability of" "gcse2" } }
*/
+/* { dg-final { scan-rtl-dump-not "forwarding load from available register:"
"gcse2" } } */
+
+/* V0 holds the slot across three diamonds, and each join reloads it after
+ a use of the old destination. The availability bitmaps need more memory
+ than max-gcse-memory allows, so none of the reloads becomes a copy. */
+
+void __RTL (startwith ("ira"))
+memory_limit (void)
+{
+(function "memory_limit"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int -16))))
+ (cinsn 3 (set (mem:V4SI (reg:DI sp) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 4 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 5 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 1004) (pc))))
+ (edge-to 3 (flags "FALLTHRU"))
+ (edge-to 4)
+ )
+ (block 3
+ (edge-from 2 (flags "FALLTHRU"))
+ (cnote 6 [bb 3] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 7 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 1))))
+ (cjump_insn 8 (set (pc) (label_ref 1005)))
+ (edge-to 5)
+ )
+ (cbarrier 9)
+ (block 4
+ (edge-from 2)
+ (clabel 1004 1004)
+ (cnote 10 [bb 4] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 11 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 2))))
+ (edge-to 5 (flags "FALLTHRU"))
+ )
+ (block 5
+ (edge-from 3)
+ (edge-from 4 (flags "FALLTHRU"))
+ (clabel 1005 1005)
+ (cnote 12 [bb 5] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 13 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 14 (set (reg:V4SI v31) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 15 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 16 (set (reg:V4SI v31) (plus:V4SI (reg:V4SI v31) (reg:V4SI v29))))
+ (cinsn 17 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 18 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 1007) (pc))))
+ (edge-to 6 (flags "FALLTHRU"))
+ (edge-to 7)
+ )
+ (block 6
+ (edge-from 5 (flags "FALLTHRU"))
+ (cnote 19 [bb 6] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 20 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 1))))
+ (cjump_insn 21 (set (pc) (label_ref 1008)))
+ (edge-to 8)
+ )
+ (cbarrier 22)
+ (block 7
+ (edge-from 5)
+ (clabel 1007 1007)
+ (cnote 23 [bb 7] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 24 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 2))))
+ (edge-to 8 (flags "FALLTHRU"))
+ )
+ (block 8
+ (edge-from 6)
+ (edge-from 7 (flags "FALLTHRU"))
+ (clabel 1008 1008)
+ (cnote 25 [bb 8] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 26 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 27 (set (reg:V4SI v31) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 28 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 29 (set (reg:V4SI v31) (plus:V4SI (reg:V4SI v31) (reg:V4SI v29))))
+ (cinsn 30 (set (reg:CC cc) (compare:CC (reg:SI x0) (const_int 0))))
+ (cjump_insn 31 (set (pc) (if_then_else (eq (reg:CC cc) (const_int 0))
+ (label_ref 1010) (pc))))
+ (edge-to 9 (flags "FALLTHRU"))
+ (edge-to 10)
+ )
+ (block 9
+ (edge-from 8 (flags "FALLTHRU"))
+ (cnote 32 [bb 9] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 33 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 1))))
+ (cjump_insn 34 (set (pc) (label_ref 1011)))
+ (edge-to 11)
+ )
+ (cbarrier 35)
+ (block 10
+ (edge-from 8)
+ (clabel 1010 1010)
+ (cnote 36 [bb 10] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 37 (set (reg:DI x4) (plus:DI (reg:DI x4) (const_int 2))))
+ (edge-to 11 (flags "FALLTHRU"))
+ )
+ (block 11
+ (edge-from 9)
+ (edge-from 10 (flags "FALLTHRU"))
+ (clabel 1011 1011)
+ (cnote 38 [bb 11] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 39 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 40 (set (reg:V4SI v31) (mem:V4SI (reg:DI sp) [0 S16 A128])))
+ (cinsn 41 (set (reg:V4SI v30) (plus:V4SI (reg:V4SI v30) (reg:V4SI v31))))
+ (cinsn 42 (set (reg:V4SI v31) (plus:V4SI (reg:V4SI v31) (reg:V4SI v29))))
+ (cinsn 43 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v30)))
+ (cinsn 44 (set (mem:DI (reg:DI x2) [0 S8 A64]) (reg:DI x4)))
+ (cinsn 45 (set (reg:DI sp) (plus:DI (reg:DI sp) (const_int 16))))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse-cold.c
b/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse-cold.c
new file mode 100644
index 000000000..bfc12eb74
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse-cold.c
@@ -0,0 +1,33 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fgcse-after-reload -msve-vector-bits=256
-fdump-rtl-gcse2" } */
+
+/* As spill-gcse.c, but the reload is in a cold block. The copy is no
+ larger than the load, so replace the load there too. */
+
+#include <arm_sve.h>
+
+typedef svuint8_t vec __attribute__ ((arm_sve_vector_bits (256)));
+
+void g (void) __attribute__ ((cold));
+
+void
+f (vec *a, int *q, int c, int d)
+{
+ svbool_t pg = svptrue_b8 ();
+ vec x = svadd_x (pg, a[0], a[1]);
+ a[2] = x;
+ if (c)
+ *q = 0;
+ if (d)
+ {
+ a[3] = svmul_x (pg, x, x);
+ asm volatile ("" ::: "z0", "z1", "z2", "z3", "z4", "z5", "z6", "z7",
+ "z8", "z9", "z10", "z11", "z12", "z13", "z14", "z15",
+ "z16", "z17", "z18", "z19", "z20", "z21", "z22", "z23",
+ "z24", "z25", "z26", "z27", "z28", "z29", "z30", "z31");
+ a[4] = x;
+ g ();
+ }
+}
+
+/* { dg-final { scan-rtl-dump {forwarding load from available register:
[^\n]*\(reg(/v)?:VNx16QI} "gcse2" } } */
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse.c
b/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse.c
new file mode 100644
index 000000000..b3d4a8596
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/spill-gcse.c
@@ -0,0 +1,29 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -fgcse-after-reload -msve-vector-bits=256
-fdump-rtl-gcse2" } */
+
+/* X is live across the asm, so it is spilled. The spill slot is written
+ before the branch and X is reloaded after the join, but the register
+ that was stored still holds X there. Copy that register instead of
+ loading X again. */
+
+#include <arm_sve.h>
+
+typedef svuint8_t vec __attribute__ ((arm_sve_vector_bits (256)));
+
+void
+f (vec *a, int *q, int c)
+{
+ svbool_t pg = svptrue_b8 ();
+ vec x = svadd_x (pg, a[0], a[1]);
+ a[2] = x;
+ if (c)
+ *q = 0;
+ a[3] = svmul_x (pg, x, x);
+ asm volatile ("" ::: "z0", "z1", "z2", "z3", "z4", "z5", "z6", "z7",
+ "z8", "z9", "z10", "z11", "z12", "z13", "z14", "z15",
+ "z16", "z17", "z18", "z19", "z20", "z21", "z22", "z23",
+ "z24", "z25", "z26", "z27", "z28", "z29", "z30", "z31");
+ a[4] = x;
+}
+
+/* { dg-final { scan-rtl-dump {forwarding load from available register:
[^\n]*\(reg(/v)?:VNx16QI} "gcse2" } } */
--
2.43.0
