From: Kyrylo Tkachov <[email protected]> cselib looks up memory values only in the mode of the access. A value that is stored or loaded in one mode and then loaded in another mode of the same size therefore misses a register that already holds it. For example, SVE spill code can contain:
(set (mem:VNx4SI (plus:DI (reg:DI sp) (const_int K))) (reg:VNx4SI v7)) ... (set (reg:VNx2DI v23) (mem:VNx2DI (plus:DI (reg:DI sp) (const_int K)))) where the load can become: (set (reg:VNx2DI v23) (reg:VNx2DI v7)) Scalar integer and floating-point modes have the same problem. DSE forwards stored values across mode changes, but only before register allocation, because it needs a new pseudo. Add cselib_lookup_mem_other_modes, which returns the values recorded at the address of a MEM in other modes of the same size. Split the filters and the address lookup out of cselib_lookup_mem, so that both functions use them. Like cselib_lookup_mem, ignore floating-point modes under -ffloat-store. In reload_cse_simplify_set, view the hard registers that hold these values in the mode of the load with simplify_subreg, which checks REG_CAN_CHANGE_MODE_P. Add them to the existing candidates, so that the existing cost comparison, LOAD_EXTEND_OP handling and instruction validation apply. As in LRA and regcprop, this relies on REG_CAN_CHANGE_MODE_P to allow a mode change only if the register then holds the bytes that memory would hold in the new mode. The motivating code is a dav1d loop filter compiled with -march=armv9-a+sve2 -msve-vector-bits=256. A testcase derived from it is given in gcc.target/aarch64/sve/spill-mode-change.c. At -O3 this patch removes 12% of its SVE vector loads (ldr z), and 8% with -fno-ivopts. All the removed loads are reloads of spill slots. The instruction count of the aarch64 cc1 for -O2 compiles of combine.ii, dwarf2out.ii, fold-const.ii and sqlite3.i changes by -0.01% to +0.01%. Bootstrapped and tested on aarch64-none-linux-gnu and x86_64-linux. Ok for trunk? Thanks, Kyrill Cc: Jeff Law <[email protected]> gcc/ChangeLog: * cselib.cc (cselib_lookup_mem_addr): New function, split out of... (cselib_lookup_mem): ...here. (cselib_lookup_mem_other_modes): New function. * cselib.h (cselib_lookup_mem_other_modes): Declare. * postreload.cc (reload_cse_add_mem_regs): New function. (reload_cse_simplify_set): Also consider registers that hold the loaded value in another mode of the same size. gcc/testsuite/ChangeLog: * gcc.dg/rtl/aarch64/postreload-mode-forward.c: New test. * gcc.dg/rtl/aarch64/postreload-mode-forward-be.c: Likewise. * gcc.dg/rtl/aarch64/postreload-mode-forward-call.c: Likewise. * gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c: Likewise. * gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c: Likewise. * gcc.target/aarch64/sve/spill-mode-change.c: Likewise. * gcc.target/i386/addr-space-10.c: Likewise. Signed-off-by: Kyrylo Tkachov <[email protected]> --- Changes since v1: - Handle all modes, not only vector modes (Jeff). - Add a cselib query instead of exporting elt_list and copying the address lookup of cselib_lookup_mem. - Add the candidates to the existing loop in reload_cse_simplify_set, so that the cost check and the LOAD_EXTEND_OP handling apply. - Apply -ffloat-store to the stored mode too. - More tests: scalar modes, calls, -ffloat-store, big-endian, a mode change that the target rejects, x86 address spaces. gcc/cselib.cc | 64 ++++- gcc/cselib.h | 1 + gcc/postreload.cc | 34 ++- .../rtl/aarch64/postreload-mode-forward-be.c | 29 +++ .../aarch64/postreload-mode-forward-call.c | 70 ++++++ .../aarch64/postreload-mode-forward-extend.c | 62 +++++ .../postreload-mode-forward-float-store.c | 102 ++++++++ .../rtl/aarch64/postreload-mode-forward.c | 228 ++++++++++++++++++ .../aarch64/sve/spill-mode-change.c | 95 ++++++++ gcc/testsuite/gcc.target/i386/addr-space-10.c | 39 +++ 10 files changed, 708 insertions(+), 16 deletions(-) create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c create mode 100644 gcc/testsuite/gcc.target/i386/addr-space-10.c diff --git a/gcc/cselib.cc b/gcc/cselib.cc index 5567caa47..020fecb2d 100644 --- a/gcc/cselib.cc +++ b/gcc/cselib.cc @@ -1765,6 +1765,30 @@ add_mem_for_addr (cselib_val *addr_elt, cselib_val *mem_elt, rtx x) } } +/* Return the canonical value for the address of X, which is a MEM rtx. + If CREATE, make a new value for the address if we haven't seen it + before. Return null if cselib does not record X or if the address + has no value. */ + +static cselib_val * +cselib_lookup_mem_addr (rtx x, int create) +{ + machine_mode mode = GET_MODE (x); + + if (MEM_VOLATILE_P (x) || mode == BLKmode + || !cselib_record_memory + || (FLOAT_MODE_P (mode) && flag_float_store)) + return 0; + + machine_mode addr_mode = GET_MODE (XEXP (x, 0)); + if (addr_mode == VOIDmode) + addr_mode = Pmode; + + /* Look up the value for the address. */ + cselib_val *addr = cselib_lookup (XEXP (x, 0), addr_mode, create, mode); + return addr ? canonical_cselib_val (addr) : 0; +} + /* Return true if V has a memory location at address ADDR in address space AS. */ @@ -1790,25 +1814,13 @@ static cselib_val * cselib_lookup_mem (rtx x, int create) { machine_mode mode = GET_MODE (x); - machine_mode addr_mode; cselib_val **slot; cselib_val *addr; cselib_val *mem_elt; - if (MEM_VOLATILE_P (x) || mode == BLKmode - || !cselib_record_memory - || (FLOAT_MODE_P (mode) && flag_float_store)) - return 0; - - addr_mode = GET_MODE (XEXP (x, 0)); - if (addr_mode == VOIDmode) - addr_mode = Pmode; - - /* Look up the value for the address. */ - addr = cselib_lookup (XEXP (x, 0), addr_mode, create, mode); + addr = cselib_lookup_mem_addr (x, create); if (! addr) return 0; - addr = canonical_cselib_val (addr); /* Find a value that describes a value of our mode at that address. */ for (elt_list *l = addr->addr_list; l; l = l->next) @@ -1829,6 +1841,32 @@ cselib_lookup_mem (rtx x, int create) return mem_elt; } +/* Add to VALUES the values that describe the contents of X, which is a MEM + rtx, in other modes of the same size. Like cselib_lookup_mem, ignore + floating-point modes if -ffloat-store. */ + +void +cselib_lookup_mem_other_modes (rtx x, vec<cselib_val *> &values) +{ + cselib_val *addr = cselib_lookup_mem_addr (x, 0); + if (!addr) + return; + + machine_mode mode = GET_MODE (x); + for (elt_list *l = addr->addr_list; l; l = l->next) + { + machine_mode val_mode = GET_MODE (l->elt->val_rtx); + if (val_mode != mode + && known_eq (GET_MODE_SIZE (val_mode), GET_MODE_SIZE (mode)) + && !(FLOAT_MODE_P (val_mode) && flag_float_store) + && cselib_has_mem_loc_p (l->elt, addr, MEM_ADDR_SPACE (x))) + { + promote_debug_loc (l->elt->locs); + values.safe_push (l->elt); + } + } +} + /* Search through the possible substitutions in P. We prefer a non reg substitution because this allows us to expand the tree further. If we find, just a reg, take the lowest regno. There may be several diff --git a/gcc/cselib.h b/gcc/cselib.h index 6493303f7..156b57d79 100644 --- a/gcc/cselib.h +++ b/gcc/cselib.h @@ -80,6 +80,7 @@ extern cselib_val *cselib_lookup (rtx, machine_mode, int, machine_mode); extern cselib_val *cselib_lookup_from_insn (rtx, machine_mode, int, machine_mode, rtx_insn *); +extern void cselib_lookup_mem_other_modes (rtx, vec<cselib_val *> &); extern void cselib_init (int); extern void cselib_clear_table (void); extern void cselib_finish (void); diff --git a/gcc/postreload.cc b/gcc/postreload.cc index 475b58896..0e837d91b 100644 --- a/gcc/postreload.cc +++ b/gcc/postreload.cc @@ -355,6 +355,29 @@ reload_cse_regs_1 (void) cleanup_cfg (0); } +/* Add to LOCS the hard registers that hold the contents of MEM, a memory + reference, in other modes of the same size. Express each register + in the mode of MEM. */ + +static void +reload_cse_add_mem_regs (rtx mem, vec<rtx> &locs) +{ + auto_vec<cselib_val *, 4> values; + cselib_lookup_mem_other_modes (mem, values); + for (cselib_val *v : values) + for (elt_loc_list *l = v->locs; l; l = l->next) + if (REG_P (l->loc) + /* The target must allow the bitcast in the register. */ + && REG_CAN_CHANGE_MODE_P (REGNO (l->loc), GET_MODE (l->loc), + GET_MODE (mem))) + { + rtx reg = simplify_subreg (GET_MODE (mem), l->loc, + GET_MODE (l->loc), 0); + if (reg && REG_P (reg)) + locs.safe_push (reg); + } +} + /* Try to simplify a single SET instruction. SET is the set pattern. INSN is the instruction it came from. This function only handles one case: if we set a register to a value @@ -394,7 +417,13 @@ reload_cse_simplify_set (rtx set, rtx_insn *insn) return 0; val = cselib_lookup (src, GET_MODE (SET_DEST (set)), 0, VOIDmode); - if (! val) + auto_vec<rtx, 8> locs; + if (val) + for (l = val->locs; l; l = l->next) + locs.safe_push (l->loc); + if (MEM_P (src)) + reload_cse_add_mem_regs (src, locs); + if (locs.is_empty ()) return 0; /* If memory loads are cheaper than register copies, don't change them. */ @@ -406,9 +435,8 @@ reload_cse_simplify_set (rtx set, rtx_insn *insn) else old_cost = set_src_cost (src, GET_MODE (SET_DEST (set)), speed); - for (l = val->locs; l; l = l->next) + for (rtx this_rtx : locs) { - rtx this_rtx = l->loc; int this_cost; if (CONSTANT_P (this_rtx) && ! references_value_p (this_rtx)) diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c new file mode 100644 index 000000000..d7a65d51a --- /dev/null +++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c @@ -0,0 +1,29 @@ +/* { dg-do compile { target aarch64*-*-* } } */ +/* { dg-options "-O2 -mbig-endian" } */ +/* { dg-final { check-function-bodies "**" "" "" } } */ + +/* On big-endian targets, Advanced SIMD registers still allow a change + between vector modes of the same size. */ + +/* +** forward_advsimd: +** str q0, \[x0\] +** str q0, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +forward_advsimd (void) +{ +(function "forward_advsimd" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0))) + (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c new file mode 100644 index 000000000..585479df4 --- /dev/null +++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c @@ -0,0 +1,70 @@ +/* { dg-do compile { target aarch64*-*-* } } */ +/* { dg-options "-O2" } */ +/* { dg-final { check-function-bodies "**" "" "" } } */ + +/* A call to a const function preserves the low 64 bits of V8 but not + the full 128-bit register. */ + +/* +** preserved: +** ... +** str d8, \[x19\] +** bl f +** str d8, \[x20\] +** ... +*/ +void __RTL (startwith ("ira")) +preserved (void) +{ +(function "preserved" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V8QI (reg:DI x19) [0 S8 A64]) (reg:V8QI v8))) + (ccall_insn/u 3 (parallel + [(set (reg:DI x0) + (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32]) + (const_int 0))) + (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI) + (clobber (reg:DI x30))]) (nil) (nil) 0) + (cinsn 4 (set (reg:V4HI v1) (mem:V4HI (reg:DI x19) [0 S8 A64]))) + (cinsn 5 (set (mem:V4HI (reg:DI x20) [0 S8 A64]) (reg:V4HI v1))) + (cinsn 6 (set (mem:DI (reg:DI x21) [0 S8 A64]) (reg:DI x0))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** clobbered: +** ... +** str q8, \[x19\] +** bl f +** ldr q[0-9]+, \[x19\] +** ... +*/ +void __RTL (startwith ("ira")) +clobbered (void) +{ +(function "clobbered" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V16QI (reg:DI x19) [0 S16 A128]) (reg:V16QI v8))) + (ccall_insn/u 3 (parallel + [(set (reg:DI x0) + (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32]) + (const_int 0))) + (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI) + (clobber (reg:DI x30))]) (nil) (nil) 0) + (cinsn 4 (set (reg:V8HI v1) (mem:V8HI (reg:DI x19) [0 S16 A128]))) + (cinsn 5 (set (mem:V8HI (reg:DI x20) [0 S16 A128]) (reg:V8HI v1))) + (cinsn 6 (set (mem:DI (reg:DI x21) [0 S8 A64]) (reg:DI x0))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c new file mode 100644 index 000000000..c630c74a1 --- /dev/null +++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c @@ -0,0 +1,62 @@ +/* { dg-do compile { target aarch64*-*-* } } */ +/* { dg-options "-O2 -mtune=xgene1 -fdump-rtl-postreload" } */ + +/* AArch64 loads of narrow integers zero-extend to 64 bits (LOAD_EXTEND_OP). + A register that replaces such a load must keep the extension. xgene1 + makes loads more expensive than extensions. */ + +void __RTL (startwith ("ira")) +sf_to_si (void) +{ +(function "sf_to_si" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:SF (reg:DI x0) [0 S4 A32]) (reg:SF v0))) + (cinsn 3 (set (reg:SI x3) (mem:SI (reg:DI x0) [0 S4 A32]))) + (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +void __RTL (startwith ("ira")) +hf_to_hi (void) +{ +(function "hf_to_hi" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:HF (reg:DI x0) [0 S2 A16]) (reg:HF v1))) + (cinsn 3 (set (reg:HI x3) (mem:HI (reg:DI x0) [0 S2 A16]))) + (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +void __RTL (startwith ("ira")) +gpr_sf_to_si (void) +{ +(function "gpr_sf_to_si" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:SF (reg:DI x0) [0 S4 A32]) (reg:SF x2))) + (cinsn 3 (set (reg:SI x3) (mem:SI (reg:DI x0) [0 S4 A32]))) + (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:SI 32 v0\)\)\)} "postreload" } } */ +/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:HI 33 v1\)\)\)} "postreload" } } */ +/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:SI 2 x2\)\)\)} "postreload" } } */ +/* { dg-final { scan-assembler-not {\tldrh?\tw3} } } */ diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c new file mode 100644 index 000000000..f57cd523f --- /dev/null +++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c @@ -0,0 +1,102 @@ +/* { dg-do compile { target aarch64*-*-* } } */ +/* { dg-options "-O2 -ffloat-store" } */ +/* { dg-final { check-function-bodies "**" "" "" } } */ + +/* -ffloat-store prevents forwarding into and out of floating-point modes, + including vector ones. */ + +/* +** int_to_float: +** str x2, \[x0\] +** ldr x3, \[x0\] +** str x3, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +int_to_float (void) +{ +(function "int_to_float" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:DI (reg:DI x0) [0 S8 A64]) (reg:DI x2))) + (cinsn 3 (set (reg:DF x3) (mem:DF (reg:DI x0) [0 S8 A64]))) + (cinsn 4 (set (mem:DF (reg:DI x1) [0 S8 A64]) (reg:DF x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** float_to_int: +** str x2, \[x0\] +** ldr x3, \[x0\] +** str x3, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +float_to_int (void) +{ +(function "float_to_int" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:DF (reg:DI x0) [0 S8 A64]) (reg:DF x2))) + (cinsn 3 (set (reg:DI x3) (mem:DI (reg:DI x0) [0 S8 A64]))) + (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** int_to_float_vector: +** str q0, \[x0\] +** ldr q1, \[x0\] +** str q1, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +int_to_float_vector (void) +{ +(function "int_to_float_vector" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0))) + (cinsn 3 (set (reg:V4SF v1) (mem:V4SF (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V4SF (reg:DI x1) [0 S16 A128]) (reg:V4SF v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** float_to_int_vector: +** str q0, \[x0\] +** ldr q1, \[x0\] +** str q1, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +float_to_int_vector (void) +{ +(function "float_to_int_vector" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SF (reg:DI x0) [0 S16 A128]) (reg:V4SF v0))) + (cinsn 3 (set (reg:V4SI v1) (mem:V4SI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c new file mode 100644 index 000000000..93db5e4c9 --- /dev/null +++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c @@ -0,0 +1,228 @@ +/* { dg-do compile { target aarch64*-*-* } } */ +/* { dg-options "-O2 -mtune=generic-armv9-a -fno-schedule-insns2" } */ +/* { dg-final { check-function-bodies "**" "" "" } } */ + +/* Postreload replaces a load with a register that holds the loaded bytes + in another mode of the same size. generic-armv9-a makes moves between + general and FP registers cheaper than loads. */ + +/* +** int_to_float: +** str x2, \[x0\] +** str x2, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +int_to_float (void) +{ +(function "int_to_float" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:DI (reg:DI x0) [0 S8 A64]) (reg:DI x2))) + (cinsn 3 (set (reg:DF v0) (mem:DF (reg:DI x0) [0 S8 A64]))) + (cinsn 4 (set (mem:DF (reg:DI x1) [0 S8 A64]) (reg:DF v0))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** float_to_int: +** str x2, \[x0\] +** str x2, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +float_to_int (void) +{ +(function "float_to_int" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:DF (reg:DI x0) [0 S8 A64]) (reg:DF x2))) + (cinsn 3 (set (reg:DI x3) (mem:DI (reg:DI x0) [0 S8 A64]))) + (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** vector_elements: +** str q0, \[x0\] +** str q0, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +vector_elements (void) +{ +(function "vector_elements" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0))) + (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** scalar_to_vector: +** str q0, \[x0\] +** str q0, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +scalar_to_vector (void) +{ +(function "scalar_to_vector" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:TI (reg:DI x0) [0 S16 A128]) (reg:TI v0))) + (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** gpr_pair: +** stp x2, x3, \[x0\] +** mov x4, x2 +** mov x5, x3 +** stp x4, x5, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +gpr_pair (void) +{ +(function "gpr_pair" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:TI (reg:DI x0) [0 S16 A128]) (reg:TI x2))) + (cinsn 3 (set (reg:TF x4) (mem:TF (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:TF (reg:DI x1) [0 S16 A128]) (reg:TF x4))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* The load also takes the value from an earlier load in another mode. */ + +/* +** load_to_load: +** ldr q0, \[x0\] +** str q0, \[x1\] +** str q0, \[x2\] +** ret +*/ +void __RTL (startwith ("ira")) +load_to_load (void) +{ +(function "load_to_load" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (reg:TI v0) (mem:TI (reg:DI x0) [0 S16 A128]))) + (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:TI (reg:DI x1) [0 S16 A128]) (reg:TI v0))) + (cinsn 5 (set (mem:V2DI (reg:DI x2) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** clobbered: +** str q0, \[x0\] +** ldr q1, \[x0\] +** str q1, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +clobbered (void) +{ +(function "clobbered" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0))) + (cinsn 3 (set (reg:V4SI v0) (reg:V4SI v2))) + (cinsn 4 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 5 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* +** alias_store: +** str q0, \[x0\] +** str w2, \[x0\] +** ldr q1, \[x0\] +** str q1, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +alias_store (void) +{ +(function "alias_store" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0))) + (cinsn 3 (set (mem:SI (reg:DI x0) [0 S4 A32]) (reg:SI x2))) + (cinsn 4 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128]))) + (cinsn 5 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} + +/* V2x8QI occupies two D registers, so Q0 does not hold the V16QI value. */ + +/* +** partial_struct: +** stp d0, d1, \[x0\] +** ldr q2, \[x0\] +** str q2, \[x1\] +** ret +*/ +void __RTL (startwith ("ira")) +partial_struct (void) +{ +(function "partial_struct" + (insn-chain + (block 2 + (edge-from entry (flags "FALLTHRU")) + (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK) + (cinsn 2 (set (mem:V2x8QI (reg:DI x0) [0 S16 A64]) (reg:V2x8QI v0))) + (cinsn 3 (set (reg:V16QI v2) (mem:V16QI (reg:DI x0) [0 S16 A128]))) + (cinsn 4 (set (mem:V16QI (reg:DI x1) [0 S16 A128]) (reg:V16QI v2))) + (edge-to exit (flags "FALLTHRU")) + ) + ) +) +} diff --git a/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c b/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c new file mode 100644 index 000000000..0d10c047d --- /dev/null +++ b/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c @@ -0,0 +1,95 @@ +/* { dg-do compile } */ +/* { dg-options "-O3 -march=armv9-a -mtune=generic-armv9-a -msve-vector-bits=256 -fno-ivopts" } */ +/* { dg-require-effective-target aarch64_little_endian } */ + +/* Reuse a stored vector across a change in element size. */ + +#include <arm_sve.h> + +typedef svuint8_t v8 __attribute__ ((arm_sve_vector_bits (256))); + +static inline void +f (v8 *a) +{ + for (int i = 0; i < 16; i += 8) + for (int j = 0; j < 4; j++) + { + svuint32_t x = svtrn1_u32 (svreinterpret_u32 (a[i + j]), + svreinterpret_u32 (a[i + j + 4])); + svuint32_t y = svtrn2_u32 (svreinterpret_u32 (a[j]), + svreinterpret_u32 (a[i + j])); + a[i + j] = svreinterpret_u8 (x); + a[i + j + 4] = svreinterpret_u8 (y); + } + for (int i = 0; i < 8; i++) + { + v8 x = svreinterpret_u8 (svtrn1_u64 (svreinterpret_u64 (a[i]), + svreinterpret_u64 (a[i + 8]))); + a[i + 8] = svreinterpret_u8 (svtrn2_u64 (svreinterpret_u64 (a[i]), + svreinterpret_u64 (a[i + 8]))); + a[i] = x; + } +} + +void +g (uint8_t *a, long b, uint32_t *c, uint64_t *d, uint8_t *e) +{ + unsigned h = c[10]; + svbool_t p = svptrue_pat_b8 (SV_VL32); + svbool_t q = svptrue_pat_b8 (SV_VL16); + svbool_t r = svptrue_pat_b32 (SV_VL8); + v8 t = svld1_u8 (p, e); +#define T(x) svtbl_u8 (svuzp1_u8 (svuzp1_u8 (svreinterpret_u8 (x), \ + svdup_u8 (0)), svdup_u8 (0)), t) + uint32_t k = d[1]; + for (int i = 0; i < 32; i += 8) + { + v8 v0 = svreinterpret_u8 (svinsr_n_u64 (svdup_u64 (0), 0)); + svuint32_t v1 = svunpklo_u32 (svunpklo_u16 (v0)); + svuint32_t v2 = svadd_n_u32_x (r, svindex_u32 (0, 1), i); + svuint32_t v3 = svlsl_u32_x (r, svdup_u32 (1), v2); + svbool_t p0 = svand_z ( + r, svcmpne_n_u32 (r, svand_u32_x (r, svdup_u32 (h), v3), 0), + svcmpne_n_u32 (r, v1, 0)); + svuint32_t v4 = svsel_u32 (p0, svdup_u32 (1), svdup_u32 (0)); + svbool_t p1 = svcmpne_n_u8 (p, T (v4), 0); + if (svptest_any (p, p1)) + continue; + uint8_t *s = a + b, *u = s + 7; + v8 v[16]; + for (int j = 0; j < 16; j++) + { + v[j] = svsplice_u8 (q, svld1_u8 (q, s), svld1_u8 (q, u)); + u += b; + } + f (v); + svuint32_t v5 = svmax_n_u32_x (r, svdup_u32 (k), 1); + svuint32_t v6 = svlsr_n_u32_x (r, v1, 4); + svbool_t p2 = svcmpne_n_u32 ( + r, svand_u32_x (r, svdup_u32 (c[1]), v3), 0); + svuint32_t v7 = svsel_u32 (p2, svdup_u32 (8), svdup_u32 (4)); + v8 v15 = T (v5), v9 = T (v6), v10 = T (v7); + svbool_t p3 = svcmple_u8 (p, v[7], v15); + v8 v11 = svabd_u8_x (p, v[6], v[7]); + v11 = svqadd_u8 (v11, svlsr_n_u8_x ( + p, svabd_u8_x (p, v[9], v[8]), 1)); + svbool_t p4 = svcmple_u8 (p, v11, v15); + svbool_t p5 = svand_z (p, svand_z (p, p3, p4), p1); + if (svptest_any (p, p5)) + continue; + svbool_t p6 = svcmpgt_u8 (p, v[7], v9); + svbool_t p7 = svcmpge_n_u8 (p, v10, 6); + v8 v12 = svdup_u8 (128); + svint8_t v13 = svqsub_s8_m (p6, svreinterpret_s8 (v12), svdup_s8 (0)); + v8 v14 = sveor_u8_x (p, svreinterpret_u8 (v13), v12); + v[8] = svsel_u8 (p7, v14, v[8]); + for (int j = 0; j < 16; j++) + { + svst1_u8 (q, s, v[j]); + s += b; + } + } +} + +/* { dg-final { scan-assembler-not {\tstr\tz[0-9]+, (\[sp, #[0-9]+, mul vl\])\n\tldr\tz[0-9]+, \1} } } */ +/* { dg-final { scan-assembler {\tstr\tz[0-9]+, \[sp, #[0-9]+, mul vl\]} } } */ diff --git a/gcc/testsuite/gcc.target/i386/addr-space-10.c b/gcc/testsuite/gcc.target/i386/addr-space-10.c new file mode 100644 index 000000000..229a4bb03 --- /dev/null +++ b/gcc/testsuite/gcc.target/i386/addr-space-10.c @@ -0,0 +1,39 @@ +/* { dg-do compile { target { ! ia32 } } } */ +/* { dg-options "-O2 -fno-dse" } */ + +/* *P and *Q are different memory locations with the same address. The + stored values are also in generic memory locations. A load from Q in + another mode of the same size must not reuse those values. */ + +typedef int v4si __attribute__ ((vector_size (16))); +typedef float v4sf __attribute__ ((vector_size (16))); + +static const long long tabl[4] = { 1, 2, 3, 4 }; +static const int tabi[4] = { 1, 2, 3, 4 }; +static const v4si tabv[2] = { { 1, 2, 3, 4 }, { 5, 6, 7, 8 } }; + +double +f (int i, long long __seg_gs *p) +{ + double *q = (double *) (__UINTPTR_TYPE__) p; + *p = tabl[i]; + return *q; +} + +float +g (int i, int __seg_gs *p) +{ + float *q = (float *) (__UINTPTR_TYPE__) p; + *p = tabi[i]; + return *q; +} + +v4sf +h (int i, v4si __seg_gs *p) +{ + v4sf *q = (v4sf *) (__UINTPTR_TYPE__) p; + *p = tabv[i]; + return *q; +} + +/* { dg-final { scan-assembler-times {\t[a-z]+\t\(%[a-z0-9]+\), %} 3 } } */ -- 2.43.0
