These patches have all the bits that are not themselves the SIMT
implementation, but are required by it.
Three main categories:
1. Configuration changes. Offload dimensions, hooks, new procedure calling
conventions, etc.
In SIMT mode, the EXEC register is now a fixed register, and considered live on
entry. Parameters are always passed as vectors, both in registers, and on the
stack. Callee-save registers are stored/restored using a different instruction
sequence that preserves EXEC.
2. Machine description regularization.
The SIMT conversion pass requires that the scalar instructions can be
transformed into a corresponding vector instruction, which means that they need
to have the same RTL shape. The patch edits a number of instructions that were
different for no reason, and adds instructions that were missing. Where that's
not possible there are new attributes describing how to find the right
transformation. (Almost certainly there's more inconsistencies to find.)
3. New insns, macros, etc. Some are explicitly for SIMT. Some are just useful
tools for the SIMT passes to use.
SIMT support includes "omp_simt_enter", "omp_simt_exit", etc. The
"omp_simt_xchg_bfly" is a placeholder.
Utility features include, "extract_first", "SCALAR_HARDREG_P",
"VECTOR_HARDREG_P", etc.
The "__builtin_simt_enter" and "__builtin_simt_exit" are intended for use in
the testsuite.
gcc/ChangeLog:
* config/gcn/constraints.md (Sa): New register constraint for
AFP_REGS.
* config/gcn/predicates.md (gcn_conditional_register_operand): Accept
DImode condition registers as well as BImode.
* config/gcn/gcn-builtins.def (SIMT_ENTER, SIMT_EXIT): New builtins.
* config/gcn/gcn.h: Include gomp-constants.h.
(SCALAR_HARDREG_P, VECTOR_HARDREG_P): New macros.
(struct machine_function): Add vectors_enabled.
* config/gcn/gcn.cc (gcn_init_machine_status): Initialize
vectors_enabled.
(gcn_option_override): Disable variable tracking in SIMT mode.
(gcn_conditional_register_usage): Make EXEC fixed in SIMT mode.
(gcn_function_arg): Pass all arguments in vector registers in SIMT
mode.
(gcn_function_arg_advance): Likewise.
(move_callee_saved_registers_simt): New function.
(move_callee_saved_registers): Use it in SIMT mode.
(gcn_expand_prologue): Mark the addcsi3_scalar_zero insn frame
related, instead of the addsi3_scalar_carry insn, and add a
prologue_use of the stack pointer high part. Initialize EXEC in
kernels in SIMT mode.
(gcn_expand_epilogue): Add a use of EXEC in SIMT mode.
(gcn_expand_builtin_1): Expand GCN_BUILTIN_SIMT_ENTER and
GCN_BUILTIN_SIMT_EXIT.
(gcn_simt_vf): New function.
(gcn_extra_live_on_entry): New function.
(gcn_md_asm_adjust): New function.
(gcn_md_reorg): Skip the VCC liveness pass in SIMT mode.
(GCN_DEFAULT_VECTORS): Rename to ...
(GCN_DEFAULT_SIMD_VECTORS): ... this, and add
GCN_DEFAULT_SIMT_VECTORS.
(gcn_goacc_validate_dims): Honour the OpenACC vector length, up to 64
lanes, in SIMT mode.
(gcn_fork_join): Emit fork/join insns in SIMT mode.
(print_operand_address): Assert that the implied zero offset in v1 is
not used in SIMT mode.
(print_operand): Handle EXEC_REG, and invert the condition for VCCZ
and EXECZ.
(TARGET_EXTRA_LIVE_ON_ENTRY, TARGET_MD_ASM_ADJUST, TARGET_SIMT_VF):
Define.
* config/gcn/gcn.md (LANE_ID_REG): New constant.
(UNSPECV_SIMT_FORK, UNSPECV_SIMT_JOIN, UNSPECV_SIMT_ENTER)
(UNSPECV_SIMT_EXIT, UNSPECV_SIMT_EXEC_BARRIER): New unspecv values.
(UNSPEC_SIMT_XCHG_BFLY, UNSPEC_SIMT_VOTE_ANY, UNSPEC_VSUBREG_WRITE)
(UNSPEC_VSUBREG_READ, UNSPEC_ENSURE_SCALAR): New unspec values.
(simt, simt_clobber, simt_const, simt_special): New attributes.
(enabled): Take the simt attribute into account.
(*movbi): Add the simt attribute, and use v_readfirstlane_b32 in SIMT
mode.
(*mov<mode>_insn): Likewise, for SISF and QIHI, plus a v_mov_b32
alternative for scalar-to-vector moves in SIMT mode.
(*mov<mode>_insn): Rename to ...
(mov<mode>_insn): ... this, for DIDF.
(*movti_insn): Rename to ...
(movti_insn): ... this.
(cjump): Add the missing attributes, and set simt_special.
(*cjump_vcc): New insn.
(gcn_simple_call, gcn_indirect_call, gcn_call_value)
(gcn_call_value_indirect, movdi_symbol, cstoresi4)
(cstoredi4_vec_and_scalar, cstoredi4_vector, cstore<mode>4, addsi3)
(adddi3, subsi3, mulsi3, <su>mulsi3_highpart_reg)
(<su>mulsi3_highpart_imm, <su>mulsidi3_imm, muldi3, <expander>si2)
(<expander>di2, <expander>si3, <expander>di3, one_cmpldi2): Set the
new SIMT attributes.
(subdi3): Turn into a define_expand, and clobber a scratch rather
than SCC.
(*subdi3_insn): New insn, split out of subdi3.
(@cselect<mode>): New insn.
(oacc_fork): Expand to UNSPECV_SIMT_FORK in SIMT mode.
(oacc_join): Expand to UNSPECV_SIMT_JOIN in SIMT mode, and keep the
wavefront barrier for the worker dimension.
(*oacc_fork_insn, *oacc_join_insn, omp_simt_enter, omp_simt_exit)
(omp_simt_lane, omp_simt_xchg_bfly, @omp_simt_xchg_bfly<mode>)
(omp_simt_vote_any, @ensure_scalar<mode>, exec_barrier, saveexec)
(setexec, setexec_<expander>, setexec_<expander>_with_save): New
patterns.
(execop): New code iterator.
* config/gcn/gcn-valu.md (V64_MOV, V64_MOV_ALT, VSUBREG_ALL)
(VSUBREG_ALT, V64SIDI, V64SIDITI, V64DITI): New mode iterators.
(vec_duplicate<mode><exec>): Disable in SIMT mode until vectors are
enabled, so that the middle-end does not use vectors for memcpy.
(vec_duplicate<mode>): Likewise, for V_2REG and V_4REG.
(@extract_first_<mode>): New patterns for V_1REG and V_2REG.
(add<mode>3, sub<mode>3): Clobber a scratch rather than VCC.
(<su>mul<vnsi><mode>3): New pattern.
(<expander><mode>2<exec>): Add popcount_extra_op to the output
template.
(div_scale<mode><exec_vcc>): Set simt_special.
(vec_cmp<mode>di): Rename to ...
(@vec_cmp<mode>di): ... this, and relax operand 3 to
gcn_alu_operand.
(@vsubreg_write<mode><mode>, @vsubreg_read<mode><mode>)
(*vsubreg_write<mode><mode>_insn, *vsubreg_read<mode><mode>_insn)
(*vsubreg_read<mode>di_insn, omp_simt_xchg_bfly<mode>): New patterns.
---
gcc/config/gcn/constraints.md | 3 +
gcc/config/gcn/gcn-builtins.def | 7 +
gcc/config/gcn/gcn-valu.md | 189 +++++-
gcc/config/gcn/gcn.cc | 435 ++++++++++++--
gcc/config/gcn/gcn.h | 8 +
gcc/config/gcn/gcn.md | 565 ++++++++++++++----
gcc/config/gcn/predicates.md | 21 +-
.../gcc.target/gcn/avgpr-spill-int.c | 5 +
.../gcc.target/gcn/avgpr-spill-short.c | 9 +-
gcc/testsuite/gcc.target/gcn/simt-asm-g-1.c | 38 ++
gcc/testsuite/gcc.target/gcn/simt-asm-r-1.c | 36 ++
.../gcc.target/gcn/simt-avgpr-spill-double.c | 9 +
.../gcc.target/gcn/simt-avgpr-spill-int.c | 154 +++++
.../gcc.target/gcn/simt-avgpr-spill-long.c | 9 +
.../gcc.target/gcn/simt-avgpr-spill-run.c | 34 ++
.../gcc.target/gcn/simt-avgpr-spill-short.c | 9 +
16 files changed, 1359 insertions(+), 172 deletions(-)
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-asm-g-1.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-asm-r-1.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-double.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-int.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-long.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-run.c
create mode 100644 gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-short.c
diff --git a/gcc/config/gcn/constraints.md b/gcc/config/gcn/constraints.md
index e2fa688b373..7aa149af10c 100644
--- a/gcc/config/gcn/constraints.md
+++ b/gcc/config/gcn/constraints.md
@@ -95,6 +95,9 @@ (define_register_constraint "Sm" "SGPR_MEM_SRC_REGS"
(define_register_constraint "Sv" "SGPR_VOP_SRC_REGS"
"registers usable as a source of VOP3A instruction")
+(define_register_constraint "Sa" "AFP_REGS"
+ "virtual scalar registers")
+
(define_register_constraint "ca" "ALL_CONDITIONAL_REGS"
"SCC VCCZ or EXECZ")
diff --git a/gcc/config/gcn/gcn-builtins.def b/gcc/config/gcn/gcn-builtins.def
index b1f4b1292d7..e732cacfee3 100644
--- a/gcc/config/gcn/gcn-builtins.def
+++ b/gcc/config/gcn/gcn-builtins.def
@@ -158,6 +158,13 @@ DEF_BUILTIN (ACC_SINGLE_COPY_END, -1, "single_copy_end",
B_INSN,
DEF_BUILTIN (ACC_BARRIER, -1, "acc_barrier", B_INSN, _A1 (GCN_BTI_VOID),
gcn_expand_builtin_1)
+/* SIMT testing. */
+
+DEF_BUILTIN (SIMT_ENTER, -1, "simt_enter", B_INSN,
+ _A3 (GCN_BTI_INT, GCN_BTI_INT, GCN_BTI_INT), gcn_expand_builtin_1)
+DEF_BUILTIN (SIMT_EXIT, -1, "simt_exit", B_INSN,
+ _A2 (GCN_BTI_VOID, GCN_BTI_INT), gcn_expand_builtin_1)
+
/* Kernel inputs. */
DEF_BUILTIN (FIRST_CALL_THIS_THREAD_P, -1, "first_call_this_thread_p", B_INSN,
diff --git a/gcc/config/gcn/gcn-valu.md b/gcc/config/gcn/gcn-valu.md
index ef94da86745..f3f7708615c 100644
--- a/gcc/config/gcn/gcn-valu.md
+++ b/gcc/config/gcn/gcn-valu.md
@@ -194,6 +194,10 @@ (define_mode_iterator V_MOV_ALT
V16QI V16HI V16HF V16SI V16SF V16DI V16DF V16TI
V32QI V32HI V32HF V32SI V32SF V32DI V32DF V32TI
V64QI V64HI V64HF V64SI V64SF V64DI V64DF V64TI])
+(define_mode_iterator V64_MOV
+ [V64QI V64HI V64HF V64SI V64SF V64DI V64DF V64TI])
+(define_mode_iterator V64_MOV_ALT
+ [V64QI V64HI V64HF V64SI V64SF V64DI V64DF V64TI])
(define_mode_attr scalar_mode
[(QI "qi") (HI "hi") (SI "si") (TI "ti")
@@ -902,11 +906,15 @@ (define_insn "*vec_set<mode>_1"
(set_attr "exec" "none")
(set_attr "laneselect" "write")])
+; Note that the vec_duplicate patterns are used by the middle-end to determine
+; if vectors can be used to optimize memcpy. We use these conditions to
+; disable the patterns at expand time, and renabled them later.
+
(define_insn "vec_duplicate<mode><exec>"
[(set (match_operand:V_1REG 0 "register_operand" "=v")
(vec_duplicate:V_1REG
(match_operand:<SCALAR_MODE> 1 "gcn_alu_operand" "SvB")))]
- ""
+ "!flag_simt_mode || (cfun && cfun->machine &&
cfun->machine->vectors_enabled)"
"v_mov_b32\t%0, %1"
[(set_attr "type" "vop3a")
(set_attr "length" "8")])
@@ -915,7 +923,7 @@ (define_insn "vec_duplicate<mode><exec>"
[(set (match_operand:V_2REG 0 "register_operand" "= v")
(vec_duplicate:V_2REG
(match_operand:<SCALAR_MODE> 1 "gcn_alu_operand" "SvDB")))]
- ""
+ "!flag_simt_mode || (cfun && cfun->machine &&
cfun->machine->vectors_enabled)"
"v_mov_b32\t%L0, %L1\;v_mov_b32\t%H0, %H1"
[(set_attr "type" "vop3a")
(set_attr "length" "16")])
@@ -924,7 +932,7 @@ (define_insn "vec_duplicate<mode><exec>"
[(set (match_operand:V_4REG 0 "register_operand" "= v")
(vec_duplicate:V_4REG
(match_operand:<SCALAR_MODE> 1 "gcn_alu_operand" "SvDB")))]
- ""
+ "!flag_simt_mode || (cfun && cfun->machine &&
cfun->machine->vectors_enabled)"
"v_mov_b32\t%L0, %L1\;v_mov_b32\t%H0, %H1\;v_mov_b32\t%J0,
%J1\;v_mov_b32\t%K0, %K1"
[(set_attr "type" "mult")
(set_attr "length" "32")])
@@ -1044,6 +1052,38 @@ (define_expand "vec_extract<V_MOV:mode><V_MOV_ALT:mode>"
DONE;
})
+(define_insn "@extract_first_<mode>"
+ [(set (match_operand:<SCALAR_MODE> 0 "register_operand" "=SD")
+ (vec_select:<SCALAR_MODE>
+ (match_operand:V_1REG 1 "register_operand" "v")
+ (parallel [(ctz:SI (reg:DI EXEC_REG))])))]
+ ""
+ "v_readfirstlane_b32\t%0, %1"
+ [(set_attr "type" "vop1")
+ (set_attr "length" "1")])
+
+(define_insn_and_split "@extract_first_<mode>"
+ [(set (match_operand:<SCALAR_MODE> 0 "register_operand" "=SD")
+ (vec_select:<SCALAR_MODE>
+ (match_operand:V_2REG 1 "register_operand" "v")
+ (parallel [(ctz:SI (reg:DI EXEC_REG))])))]
+ ""
+ "#"
+ "reload_completed"
+ [(const_int 0)]
+ {
+ emit_insn (gen_extract_first_<vnsi> (gcn_operand_part (<SCALAR_MODE>mode,
+ operands[0], 0),
+ gcn_operand_part (<MODE>mode,
+ operands[1], 0)));
+ emit_insn (gen_extract_first_<vnsi> (gcn_operand_part (<SCALAR_MODE>mode,
+ operands[0], 1),
+ gcn_operand_part (<MODE>mode,
+ operands[1], 1)));
+ DONE;
+ }
+ [(set_attr "type" "mult")])
+
(define_expand "extract_last_<mode>"
[(match_operand:<SCALAR_MODE> 0 "register_operand")
(match_operand:DI 1 "gcn_alu_operand")
@@ -1430,7 +1470,7 @@ (define_insn_and_split "add<mode>3"
(plus:V_DI
(match_operand:V_DI 1 "register_operand" "%vDb")
(match_operand:V_DI 2 "gcn_alu_operand" " vDb")))
- (clobber (reg:DI VCC_REG))]
+ (clobber (match_scratch:DI 3 "= cV"))]
""
"#"
"gcn_can_split_p (<MODE>mode, operands[0])
@@ -1438,7 +1478,7 @@ (define_insn_and_split "add<mode>3"
&& gcn_can_split_p (<MODE>mode, operands[2])"
[(const_int 0)]
{
- rtx vcc = gen_rtx_REG (DImode, VCC_REG);
+ rtx vcc = operands[3];
emit_insn (gen_add<vnsi>3_vcc
(gcn_operand_part (<MODE>mode, operands[0], 0),
gcn_operand_part (<MODE>mode, operands[1], 0),
@@ -1572,7 +1612,7 @@ (define_insn_and_split "sub<mode>3"
(minus:V_DI
(match_operand:V_DI 1 "gcn_alu_operand" "vDb, v")
(match_operand:V_DI 2 "gcn_alu_operand" " v,vDb")))
- (clobber (reg:DI VCC_REG))]
+ (clobber (match_scratch:DI 3 "=cV, cV"))]
""
"#"
"gcn_can_split_p (<MODE>mode, operands[0])
@@ -2058,7 +2098,7 @@ (define_insn "subr<mode>3_ds_scalar"
;; {{{ ALU special case: mult
(define_insn "<su>mul<mode>3_highpart<exec>"
- [(set (match_operand:V_SI 0 "register_operand" "= v")
+ [(set (match_operand:V_SI 0 "register_operand" "= v")
(truncate:V_SI
(lshiftrt:<VnDI>
(mult:<VnDI>
@@ -2358,6 +2398,28 @@ (define_insn_and_split "mul<mode>3_zext_exec"
DONE;
})
+(define_insn_and_split "<su>mul<vnsi><mode>3"
+ [(set (match_operand:V_DI 0 "register_operand" "=&v")
+ (mult:V_DI
+ (any_extend:V_DI
+ (match_operand:<VnSI> 1 "gcn_alu_operand" " v"))
+ (any_extend:V_DI
+ (match_operand:<VnSI> 2 "gcn_alu_operand" " vA"))))]
+ ""
+ "#"
+ "reload_completed"
+ [(const_int 0)]
+ {
+ rtx out_lo = gcn_operand_part (<MODE>mode, operands[0], 0);
+ rtx out_hi = gcn_operand_part (<MODE>mode, operands[0], 1);
+ rtx left = operands[1];
+ rtx right = operands[2];
+
+ emit_insn (gen_mul<vnsi>3 (out_lo, left, right));
+ emit_insn (gen_<su>mul<vnsi>3_highpart (out_hi, left, right));
+ DONE;
+ })
+
(define_insn_and_split "mul<mode>3_zext_dup2"
[(set (match_operand:V_DI 0 "register_operand" "= &v")
(mult:V_DI
@@ -2623,7 +2685,7 @@ (define_insn "<expander><mode>2<exec>"
(bitunop:V_INT_1REG
(match_operand:V_INT_1REG 1 "gcn_valu_src0_operand" "vSvB")))]
""
- "v_<mnemonic>0\t%0, %1"
+ "v_<mnemonic>0\t%0, %1<popcount_extra_op>"
[(set_attr "type" "vop1")
(set_attr "length" "8")])
@@ -3466,7 +3528,8 @@ (define_insn "div_scale<mode><exec_vcc>"
""
"v_div_scale%i0\t%0, %4, %3, %1, %2"
[(set_attr "type" "vop3b")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_special" "set_vcc")])
;; v_div_fmas is "FMA and Scale" that uses the VCC output from v_div_scale
;; to conditionally scale the output of the whole division operation.
@@ -3800,11 +3863,11 @@ (define_insn_and_split "<convop><mode><vndi>2_exec"
;; }}}
;; {{{ Vector comparison/merge
-(define_insn "vec_cmp<mode>di"
+(define_insn "@vec_cmp<mode>di"
[(set (match_operand:DI 0 "register_operand" "=cV,cV, e, e,Sg,Sg, e,
e")
(match_operator:DI 1 "gcn_fp_compare_operator"
[(match_operand:V_noQI 2 "gcn_alu_operand" "vSv, B,vSv, B, v,vA,vSv,
B")
- (match_operand:V_noQI 3 "gcn_vop3_operand" " v, v, v, v,vA, v, v,
v")]))
+ (match_operand:V_noQI 3 "gcn_alu_operand" " v, v, v, v,vA, v, v,
v")]))
(clobber (match_scratch:DI 4 "= X, X, cV,cV,
X, X, X, X"))]
""
"@
@@ -4525,3 +4588,107 @@ (define_expand "vec_series<mode>"
})
;; }}}
+;; {{{ SIMT
+
+(define_mode_iterator VSUBREG_ALL
+ [DI V64QI V64HI V64HF V64SI V64SF V64DI V64DF V64TI])
+(define_mode_iterator VSUBREG_ALT
+ [DI V64QI V64HI V64HF V64SI V64SF V64DI V64DF V64TI])
+(define_mode_iterator V64SIDI [V64SI V64DI])
+(define_mode_iterator V64SIDITI [V64SI V64DI V64TI])
+(define_mode_iterator V64DITI [V64DI V64TI])
+
+(define_expand "@vsubreg_write<VSUBREG_ALL:mode><VSUBREG_ALT:mode>"
+ [(set (match_operand:VSUBREG_ALL 0 "register_operand")
+ (unspec:VSUBREG_ALL [(match_operand:VSUBREG_ALT 1 "gcn_alu_operand")
+ (match_operand:SI 2 "immediate_operand")
+ (match_dup 0)]
+ UNSPEC_VSUBREG_WRITE))]
+ "")
+
+(define_expand "@vsubreg_read<VSUBREG_ALL:mode><VSUBREG_ALT:mode>"
+ [(set (match_operand:VSUBREG_ALL 0 "register_operand")
+ (unspec:VSUBREG_ALL [(match_operand:VSUBREG_ALT 1 "gcn_alu_operand")
+ (match_operand:SI 2 "immediate_operand")]
+ UNSPEC_VSUBREG_READ))]
+ "")
+
+(define_insn_and_split "*vsubreg_write<V64_MOV:mode><V64_MOV_ALT:mode>_insn"
+ [(set (match_operand:V64_MOV 0 "register_operand" "=v")
+ (unspec:V64_MOV [(match_operand:V64_MOV_ALT 1 "gcn_alu_operand" " v")
+ (match_operand:SI 2 "immediate_operand" " A")
+ (match_operand:V64_MOV 3 "register_operand" " 0")]
+ UNSPEC_VSUBREG_WRITE))]
+ ""
+ "#"
+ "reload_completed"
+ [(set (match_dup 3) (match_dup 4))]
+ {
+ /* (subreg:outermode (reg:innermode)) */
+ machine_mode outermode = <V64_MOV_ALT:MODE>mode;
+ machine_mode innermode = <V64_MOV:MODE>mode;
+
+ if (GET_MODE_SIZE (innermode) < GET_MODE_SIZE (outermode))
+ /* Paradoxical subregs require a truncate. */
+ operands[4] = gen_rtx_TRUNCATE (outermode, operands[1]);
+ else
+ operands[4] = operands[1];
+ operands[3] = gen_rtx_REG (outermode,
+ REGNO (operands[0]) + INTVAL (operands[2]));
+ })
+
+(define_insn_and_split "*vsubreg_read<V64_MOV:mode><V64_MOV_ALT:mode>_insn"
+ [(set (match_operand:V64_MOV 0 "register_operand" "=v")
+ (unspec:V64_MOV [(match_operand:V64_MOV_ALT 1 "gcn_alu_operand" " v")
+ (match_operand:SI 2 "immediate_operand" " A")]
+ UNSPEC_VSUBREG_READ))]
+ ""
+ "#"
+ "reload_completed"
+ [(set (match_dup 0) (match_dup 3))]
+ {
+ /* (subreg:outermode (reg:innermode)) */
+ machine_mode innermode = <V64_MOV_ALT:MODE>mode;
+ machine_mode outermode = <V64_MOV:MODE>mode;
+
+ if (GET_MODE_SIZE (innermode) < GET_MODE_SIZE (outermode))
+ {
+ /* Paradoxical subreg require an extend. */
+ /* FIXME: When can this be a no-op conversion? */
+ operands[3] = gen_rtx_ZERO_EXTEND (outermode, operands[1]);
+ }
+ else
+ operands[3] = gen_rtx_REG (outermode,
+ REGNO (operands[1]) + INTVAL (operands[2]));
+ })
+
+(define_insn_and_split "*vsubreg_read<V64_MOV:mode>di_insn"
+ [(set (match_operand:V64_MOV 0 "register_operand" "= v")
+ (unspec:V64_MOV [(match_operand:DI 1 "gcn_alu_operand" "SScV")
+ (match_operand:SI 2 "immediate_operand" " A")]
+ UNSPEC_VSUBREG_READ))]
+ ""
+ "#"
+ "reload_completed"
+ [(parallel
+ [(set (match_dup 0)
+ (vec_merge:V64_MOV
+ (match_dup 3)
+ (match_dup 4)
+ (match_dup 1)))
+ (clobber (scratch:V64DI))])]
+ {
+ operands[3] = gcn_vec_constant (<MODE>mode, 1);
+ operands[4] = gcn_vec_constant (<MODE>mode, 0);
+ })
+
+(define_insn "omp_simt_xchg_bfly<mode>"
+ [(set (match_operand:V_MOV 0 "register_operand" "=v")
+ (unspec:V_MOV
+ [(match_operand:V_MOV 1 "register_operand" "v")
+ (match_operand:SI 2 "nonmemory_operand" "Sg")]
+ UNSPEC_SIMT_XCHG_BFLY))]
+ "flag_simt_mode"
+ "; %0 = simt_xchg_bfly %1 %2")
+
+;; }}}
diff --git a/gcc/config/gcn/gcn.cc b/gcc/config/gcn/gcn.cc
index 346a9a8021d..8b8342e3468 100644
--- a/gcc/config/gcn/gcn.cc
+++ b/gcc/config/gcn/gcn.cc
@@ -125,6 +125,10 @@ gcn_init_machine_status (void)
//if (TARGET_GCN3)
f->use_flat_addressing = true;
+ /* Vector modes start disabled, in SIMT mode, and will be reenabled later
+ during the simt_vect pass. */
+ f->vectors_enabled = !flag_simt_mode;
+
return f;
}
@@ -189,6 +193,10 @@ gcn_option_override (void)
to be set to '2', so I'll leave it default for now.
SET_OPTION_IF_UNSET (&global_options, &global_options_set,
param_vect_partial_vector_usage, 1); */
+
+ /* FIXME: do transformations to unbreak debug info in simt_vect! */
+ if (flag_simt_mode)
+ flag_var_tracking = false;
}
/* }}} */
@@ -2616,6 +2624,13 @@ gcn_secondary_reload (bool in_p, rtx x, reg_class_t
rclass,
static void
gcn_conditional_register_usage (void)
{
+ if (flag_simt_mode)
+ {
+ /* In SIMT mode the EXEC mask is never used as a constraint. */
+ fixed_regs[EXEC_LO_REG] = 1;
+ fixed_regs[EXEC_HI_REG] = 1;
+ }
+
/* Some architectures have a register allocation granularity that does not
permit use of the full register count. */
for (int i = 256 - (256 % TARGET_VGPR_GRANULARITY);
@@ -2883,10 +2898,9 @@ gcn_function_arg (cumulative_args_t cum_v, const
function_arg_info &arg)
if (targetm.calls.must_pass_in_stack (arg))
return 0;
- int first_reg = (VECTOR_MODE_P (arg.mode)
- ? FIRST_VPARM_REG : FIRST_PARM_REG);
- int cum_num = (VECTOR_MODE_P (arg.mode)
- ? cum->vnum : cum->num);
+ bool use_vector_args = (flag_simt_mode || VECTOR_MODE_P (arg.mode));
+ int first_reg = (use_vector_args ? FIRST_VPARM_REG : FIRST_PARM_REG);
+ int cum_num = (use_vector_args ? cum->vnum : cum->num);
int reg_num = first_reg + cum_num;
int num_regs = num_arg_regs (arg);
if (num_regs > 0)
@@ -2944,10 +2958,9 @@ gcn_function_arg_advance (cumulative_args_t cum_v,
if (!arg.named)
return;
- int first_reg = (VECTOR_MODE_P (arg.mode)
- ? FIRST_VPARM_REG : FIRST_PARM_REG);
- int *cum_num = (VECTOR_MODE_P (arg.mode)
- ? &cum->vnum : &cum->num);
+ bool use_vector_args = (flag_simt_mode || VECTOR_MODE_P (arg.mode));
+ int first_reg = (use_vector_args ? FIRST_VPARM_REG : FIRST_PARM_REG);
+ int *cum_num = (use_vector_args ? &cum->vnum : &cum->num);
int num_regs = num_arg_regs (arg);
if (num_regs > 0)
while ((first_reg + *cum_num) % num_regs != 0)
@@ -3228,6 +3241,138 @@ gcn_compute_frame_offsets (void)
return offsets;
}
+/* Insert code into the prologue or epilogue to store or load any
+ callee-save register to/from the stack, in a way that doesn't break
+ SIMT code.
+
+ NOTE: in SIMT mode the EXEC is not touched, so it's only valid to clobber
+ *active* lanes of caller-save registers, and only necessary to save active
+ lanes of callee-save register.
+
+ Helper function for gcn_expand_prologue and gcn_expand_epilogue. */
+
+static void
+move_callee_saved_registers_simt (rtx sp, machine_function *offsets,
+ bool prologue)
+{
+ int regno, offset = 0;
+
+ start_sequence ();
+
+ /* Save/Restore scalars.
+ TODO: coalese consecutive registers into combined stores. */
+ rtx sdata = gen_rtx_REG (SImode, VGPR_REGNO (3));
+ rtx sp_plus_zero = NULL;
+ for (regno = 0; regno < FIRST_VGPR_REG; regno++)
+ if ((df_regs_ever_live_p (regno) && !call_used_or_fixed_reg_p (regno))
+ || ((regno & ~1) == LINK_REGNUM && offsets->lr_needs_saving)
+ || ((regno & ~1) == HARD_FRAME_POINTER_REGNUM
+ && offsets->need_frame_pointer))
+ {
+ if (!sp_plus_zero)
+ {
+ rtx v0 = gen_rtx_REG (SImode, VGPR_REGNO (0));
+ emit_move_insn (v0, const0_rtx);
+ sp_plus_zero = gen_rtx_PLUS (DImode, sp, v0);
+ }
+
+ rtx reg = gen_rtx_REG (SImode, regno);
+ rtx mem = gen_rtx_MEM (SImode,
+ gen_rtx_PLUS (DImode, sp_plus_zero,
+ gen_rtx_CONST_INT (VOIDmode,
+ offset)));
+ set_mem_addr_space (mem, ADDR_SPACE_GLOBAL);
+
+ if (prologue)
+ {
+ emit_move_insn (sdata, reg);
+ emit_move_insn (mem, sdata);
+ }
+ else
+ {
+ emit_move_insn (sdata, mem);
+ emit_move_insn (reg, sdata);
+ }
+
+ offset += 4;
+
+ /* TODO: CFI meta-data. */
+ }
+
+ rtx move_scalars = end_sequence ();
+ start_sequence ();
+
+ /* Prepare a ladder reg suitable for storing V64SImode values. Note
+ that this only clobbers the active lanes of a caller-save register. */
+ rtx _0_1_2_3 = gen_rtx_REG (V64SImode, VGPR_REGNO (1));
+ rtx _0_4_8_12 = gen_rtx_REG (V64SImode, VGPR_REGNO (3));
+ emit_insn (gen_ashlv64si3 (_0_4_8_12, _0_1_2_3, GEN_INT (2)));
+
+ /* Save/Restore vectors. */
+ rtx vsp = gen_rtx_VEC_DUPLICATE (V64DImode, sp);
+ int raw_offset = offset;
+ for (regno = FIRST_VGPR_REG; regno < FIRST_PSEUDO_REGISTER; regno++)
+ if (df_regs_ever_live_p (regno) && !call_used_or_fixed_reg_p (regno))
+ {
+ rtx reg = gen_rtx_REG (V64SImode, regno);
+ rtx addr = gen_rtx_PLUS (V64DImode,
+ gen_rtx_PLUS (V64DImode, vsp, _0_4_8_12),
+ gcn_vec_constant (V64DImode, offset));
+ rtx mem = gen_rtx_MEM (V64SImode, addr);
+ set_mem_addr_space (mem, ADDR_SPACE_GLOBAL);
+
+ if (prologue)
+ emit_move_insn (mem, reg);
+ else
+ emit_move_insn (reg, mem);
+
+ /* TODO: CFI meta-data. */
+
+ offset += 4*64;
+ raw_offset += 4*64;
+
+ if (offset > (TARGET_11BIT_GLOBAL_OFFSET ? 1<<11 : 1<<12))
+ {
+ /* We ran out of bits in the offset field, so reset the base.
+ Don't edit the SP itself, because the caller will do that. */
+ rtx old_sp = XEXP (vsp, 0);
+ rtx new_sp = old_sp;
+ if (old_sp == sp)
+ {
+ /* The frame pointer should be dead here. */
+ new_sp = gen_rtx_REG (DImode, HARD_FRAME_POINTER_REGNUM);
+ vsp = gen_rtx_VEC_DUPLICATE (V64DImode, new_sp);
+ }
+ rtx scc = gen_rtx_REG (BImode, SCC_REG);
+ rtx new_lo = gcn_operand_part (DImode, new_sp, 0);
+ rtx old_lo = gcn_operand_part (DImode, old_sp, 0);
+ rtx new_hi = gcn_operand_part (DImode, new_sp, 1);
+ rtx old_hi = gcn_operand_part (DImode, old_sp, 1);
+ emit_insn (gen_addsi3_scalar_carry (new_lo, old_lo,
+ GEN_INT (offset), scc));
+ emit_insn (gen_addcsi3_scalar_zero (new_hi, old_hi, scc));
+ offset = 0;
+ }
+ }
+
+ rtx move_vectors = end_sequence ();
+
+ if (prologue)
+ {
+ emit_insn (move_scalars);
+ emit_insn (move_vectors);
+ }
+ else
+ {
+ emit_insn (move_vectors);
+ emit_insn (move_scalars);
+ }
+
+ /* This happens when a new register becomes "live" after reload.
+ Check your splitters! */
+ gcc_assert (raw_offset <= offsets->callee_saves);
+}
+
/* Insert code into the prologue or epilogue to store or load any
callee-save register to/from the stack.
@@ -3237,6 +3382,12 @@ static void
move_callee_saved_registers (rtx sp, machine_function *offsets,
bool prologue)
{
+ if (flag_simt_mode)
+ {
+ move_callee_saved_registers_simt (sp, offsets, prologue);
+ return;
+ }
+
int regno, offset, saved_scalars;
rtx exec = gen_rtx_REG (DImode, EXEC_REG);
rtx vcc = gen_rtx_REG (DImode, VCC_LO_REG);
@@ -3495,8 +3646,8 @@ gcn_expand_prologue ()
the note. */
rtx scc = gen_rtx_REG (BImode, SCC_REG);
rtx adjustment = gen_int_mode (sp_adjust, SImode);
- rtx insn = emit_insn (gen_addsi3_scalar_carry (sp_lo, sp_lo,
- adjustment, scc));
+ emit_insn (gen_addsi3_scalar_carry (sp_lo, sp_lo, adjustment, scc));
+ rtx insn = emit_insn (gen_addcsi3_scalar_zero (sp_hi, sp_hi, scc));
if (!offsets->need_frame_pointer)
{
RTX_FRAME_RELATED_P (insn) = 1;
@@ -3505,7 +3656,7 @@ gcn_expand_prologue ()
gen_rtx_PLUS (DImode, sp,
adjustment)));
}
- emit_insn (gen_addcsi3_scalar_zero (sp_hi, sp_hi, scc));
+ emit_insn (gen_prologue_use (sp_hi));
}
if (offsets->need_frame_pointer)
@@ -3539,6 +3690,14 @@ gcn_expand_prologue ()
}
else
{
+ /* Initialize the registers unmasked. */
+ if (flag_simt_mode && !cfun->machine->normal_function)
+ {
+ rtx exec = gen_rtx_REG (DImode, EXEC_REG);
+ emit_move_insn (exec, constm1_rtx);
+ emit_insn (gen_prologue_use (exec));
+ }
+
if (TARGET_PACKED_WORK_ITEMS)
{
/* v0 contains the X, Y and Z dimensions all in one.
@@ -3672,6 +3831,14 @@ gcn_expand_prologue ()
emit_insn (gen_prologue_use (gen_rtx_REG (DImode, FLAT_SCRATCH_REG)));
}
+ /* Kernels always enter in single-threaded mode. */
+ if (flag_simt_mode && !cfun->machine->normal_function)
+ {
+ rtx exec = gen_rtx_REG (DImode, EXEC_REG);
+ emit_move_insn (exec, const1_rtx);
+ emit_insn (gen_prologue_use (exec));
+ }
+
/* Ensure that the scheduler doesn't do anything unexpected. */
emit_insn (gen_blockage ());
@@ -3730,6 +3897,9 @@ gcn_expand_epilogue (void)
/* Similar for frame pointer. */
if (offsets->need_frame_pointer)
emit_use (gen_rtx_REG (DImode, HARD_FRAME_POINTER_REGNUM));
+
+ if (flag_simt_mode)
+ emit_use (gen_rtx_REG (DImode, EXEC_REG));
}
else if (flag_openmp)
{
@@ -5108,6 +5278,30 @@ gcn_expand_builtin_1 (tree exp, rtx target, rtx
/*subtarget */ ,
}
return result;
}
+ case GCN_BUILTIN_SIMT_ENTER:
+ {
+ if (!REG_P (target))
+ target = gen_reg_rtx (SImode);
+ rtx arg1 = expand_expr (CALL_EXPR_ARG (exp, 0), NULL_RTX, mode,
+ EXPAND_NORMAL);
+ rtx arg2 = expand_expr (CALL_EXPR_ARG (exp, 1), NULL_RTX, mode,
+ EXPAND_NORMAL);
+ if (MEM_P (arg1))
+ arg1 = force_reg (SImode, arg1);
+ if (MEM_P (arg2))
+ arg2 = force_reg (SImode, arg2);
+ emit_insn (gen_omp_simt_enter (target, arg1, arg2));
+ return target;
+ }
+ case GCN_BUILTIN_SIMT_EXIT:
+ {
+ rtx arg1 = expand_expr (CALL_EXPR_ARG (exp, 0), NULL_RTX, mode,
+ EXPAND_NORMAL);
+ if (MEM_P (arg1))
+ arg1 = force_reg (SImode, arg1);
+ emit_insn (gen_omp_simt_exit (arg1));
+ return target;
+ }
default:
gcc_unreachable ();
}
@@ -5978,6 +6172,23 @@ gcn_prefer_gather_scatter (machine_mode ARG_UNUSED
(mode),
/* }}} */
/* {{{ SIMT. */
+/* Implement TARGET_SIMT_VF. */
+
+static int
+gcn_simt_vf (void)
+{
+ switch (param_offload_gpu_mode)
+ {
+ case OFFLOAD_GPU_MODE_AUTO:
+ case OFFLOAD_GPU_MODE_SIMD:
+ return 1;
+ case OFFLOAD_GPU_MODE_SIMT_DIRECTED:
+ case OFFLOAD_GPU_MODE_SIMT:
+ return 64;
+ }
+ return 1;
+}
+
/* Implement TARGET_STACK_VARIABLE_PADDING. */
poly_uint64
@@ -6006,6 +6217,95 @@ gcn_simt_stack_variable_offset (const_tree type)
return result;
}
+/* Implement TARGET_EXTRA_LIVE_ON_ENTRY. */
+
+void
+gcn_extra_live_on_entry (bitmap regs)
+{
+ if (!flag_simt_mode)
+ return;
+
+ bitmap_set_bit (regs, EXEC_LO_REG);
+ bitmap_set_bit (regs, EXEC_HI_REG);
+}
+
+/* Implement TARGET_MD_ASM_ADJUST.
+
+ Rewrite the generic register constraints, "r" and "g", in SIMT mode.
+ These work fine in SIMD mode (GENERAL_REGS maps to SGPRS), but fail in
+ SIMT mode because they should probably use VGPRs for that.
+
+ In SIMT mode:
+ - "r" becomes "Sg".
+ - "g" becomes "vmi". */
+
+static rtx_insn *
+gcn_md_asm_adjust (vec<rtx> & /*outputs*/, vec<rtx> & /*inputs*/,
+ vec<machine_mode> & /*input_modes*/,
+ vec<const char *> &constraints, vec<rtx> & /*uses*/,
+ vec<rtx> & /*clobbers*/, HARD_REG_SET & /*clobbered_regs*/,
+ location_t /*loc*/)
+{
+ if (!flag_simt_mode)
+ return NULL;
+
+ const char *const g_replacement = "vmi";
+
+ for (auto &constraint : constraints)
+ {
+ /* Scan the constraints. */
+ size_t growth = 0;
+ bool found = false;
+ for (const char *p = constraint; *p; p += CONSTRAINT_LEN (*p, p))
+ {
+ /* Take care not to interpret the second part of "Sg" as "g". */
+ if (CONSTRAINT_LEN (*p, p) != 1)
+ continue;
+ if (*p == 'r')
+ found = true;
+ else if (*p == 'g')
+ {
+ found = true;
+ growth += strlen (g_replacement) - 1;
+ }
+ }
+ if (!found)
+ continue;
+
+ /* Generate the new constraint string. */
+ char *modified = (char *) ggc_alloc_atomic (strlen (constraint)
+ + growth + 1);
+ char *q = modified;
+ for (const char *p = constraint; *p; )
+ {
+ size_t len = CONSTRAINT_LEN (*p, p);
+ if (len == 1 && *p == 'r')
+ *q++ = 'v', p++;
+ else if (len == 1 && *p == 'g')
+ {
+ strcpy (q, g_replacement);
+ q += strlen (g_replacement);
+ p++;
+ }
+ else
+ {
+ memcpy (q, p, len);
+ q += len;
+ p += len;
+ }
+ }
+ *q = '\0';
+
+ if (dump_file)
+ fprintf (dump_file,
+ "gcn_md_asm_adjust: SIMT mode, so modify the constraint "
+ "\"%s\" to \"%s\".\n", constraint, modified);
+ constraint = modified;
+ }
+
+ return NULL;
+}
+
/* }}} */
/* {{{ md_reorg pass. */
@@ -6162,6 +6462,10 @@ gcn_md_reorg (void)
df_insn_rescan_all ();
}
+ if (!flag_simt_mode)
+ {
+ /* FIXME: break this out. */
+
df_live_add_problem ();
df_live_set_all_dirty ();
df_analyze ();
@@ -6361,6 +6665,7 @@ gcn_md_reorg (void)
}
CLEAR_REG_SET (&live);
+ } /* !flag_simt_mode */
/* "Manually Inserted Wait States (NOPs)."
@@ -6698,7 +7003,8 @@ gcn_md_reorg (void)
#define GCN_DEFAULT_GANGS 0 /* Choose at runtime. */
#define GCN_DEFAULT_WORKERS 0 /* Choose at runtime. */
-#define GCN_DEFAULT_VECTORS 1 /* Use autovectorization only, for now. */
+#define GCN_DEFAULT_SIMD_VECTORS 1 /* Use autovectorization only. */
+#define GCN_DEFAULT_SIMT_VECTORS 64 /* Full vectors. */
/* Implement TARGET_GOACC_VALIDATE_DIMS.
@@ -6711,24 +7017,42 @@ gcn_goacc_validate_dims (tree decl, int dims[], int
fn_level,
{
bool changed = false;
const int max_workers = 16;
+ const int max_simt = 64;
- /* The vector size must appear to be 64, to the user, unless this is a
- SEQ routine. The real, internal value is always 1, which means use
- autovectorization, but the user should not see that. */
- if (fn_level <= GOMP_DIM_VECTOR && fn_level >= -1
- && dims[GOMP_DIM_VECTOR] >= 0)
- {
- if (fn_level < 0 && dims[GOMP_DIM_VECTOR] >= 0
- && dims[GOMP_DIM_VECTOR] != 64)
- warning_at (decl ? DECL_SOURCE_LOCATION (decl) : UNKNOWN_LOCATION,
- OPT_Wopenacc_dims,
- (dims[GOMP_DIM_VECTOR]
- ? G_("using %<vector_length (64)%>, ignoring %d")
- : G_("using %<vector_length (64)%>, "
- "ignoring runtime setting")),
- dims[GOMP_DIM_VECTOR]);
- dims[GOMP_DIM_VECTOR] = 1;
- changed = true;
+ /* In SIMD mode, the vector size must appear to be 64, to the user, unless
+ this is a SEQ routine. The real, internal value is always 1, which means
+ use autovectorization, but the user should not see that.
+ In SIMT mode, the vector size works as OpenACC intends. */
+ if (flag_simt_mode)
+ {
+ if (dims[GOMP_DIM_VECTOR] > max_simt)
+ {
+ warning_at (decl ? DECL_SOURCE_LOCATION (decl) : UNKNOWN_LOCATION,
+ OPT_Wopenacc_dims,
+ "using %<num_vector (%d)%>, ignoring %d",
+ max_simt, dims[GOMP_DIM_VECTOR]);
+ dims[GOMP_DIM_VECTOR] = max_simt;
+ changed = true;
+ }
+ }
+ else
+ {
+ if (fn_level <= GOMP_DIM_VECTOR && fn_level >= -1
+ && dims[GOMP_DIM_VECTOR] >= 0)
+ {
+ if (fn_level < 0 && dims[GOMP_DIM_VECTOR] >= 0
+ && dims[GOMP_DIM_VECTOR] != 64)
+ warning_at (decl ? DECL_SOURCE_LOCATION (decl) : UNKNOWN_LOCATION,
+ OPT_Wopenacc_dims,
+ (dims[GOMP_DIM_VECTOR]
+ ? G_("using %<vector_length (64)%>, ignoring %d")
+ : G_("using %<vector_length (64)%>, "
+ "ignoring runtime setting")),
+ dims[GOMP_DIM_VECTOR]);
+
+ dims[GOMP_DIM_VECTOR] = 1;
+ changed = true;
+ }
}
/* Check the num workers is not too large. */
@@ -6745,7 +7069,10 @@ gcn_goacc_validate_dims (tree decl, int dims[], int
fn_level,
/* Set global defaults. */
if (!decl)
{
- dims[GOMP_DIM_VECTOR] = GCN_DEFAULT_VECTORS;
+ if (!flag_simt_mode)
+ dims[GOMP_DIM_VECTOR] = GCN_DEFAULT_SIMD_VECTORS;
+ else if (dims[GOMP_DIM_VECTOR] < 0)
+ dims[GOMP_DIM_VECTOR] = GCN_DEFAULT_SIMT_VECTORS;
if (dims[GOMP_DIM_WORKER] < 0)
dims[GOMP_DIM_WORKER] = GCN_DEFAULT_WORKERS;
if (dims[GOMP_DIM_GANG] < 0)
@@ -6813,12 +7140,24 @@ gcn_oacc_dim_pos (int dim)
static bool
gcn_fork_join (gcall *call, const int dims[], bool is_fork)
{
- tree arg = gimple_call_arg (call, 2);
- unsigned axis = TREE_INT_CST_LOW (arg);
+ switch (param_offload_gpu_mode)
+ {
+ case OFFLOAD_GPU_MODE_AUTO:
+ case OFFLOAD_GPU_MODE_SIMD:
+ {
+ tree arg = gimple_call_arg (call, 2);
+ unsigned axis = TREE_INT_CST_LOW (arg);
- if (!is_fork && axis == GOMP_DIM_WORKER && dims[axis] != 1)
- return true;
+ if (!is_fork && axis == GOMP_DIM_WORKER && dims[axis] != 1)
+ return true;
+ break;
+ }
+ case OFFLOAD_GPU_MODE_SIMT_DIRECTED:
+ /* FIXME do something different. */
+ case OFFLOAD_GPU_MODE_SIMT:
+ return true;
+ }
return false;
}
@@ -7428,9 +7767,16 @@ print_operand_address (FILE *file, rtx mem)
/* The assembler requires a 64-bit VGPR pair here, even though
the offset should be only 32-bit. */
if (vgpr_offset == NULL_RTX)
- /* In this case, the vector offset is zero, so we use the first
- lane of v1, which is initialized to zero. */
- fprintf (file, "v1");
+ {
+ /* In this case, the vector offset is zero, so we use the
first
+ lane of v1, which is initialized to zero. */
+ fprintf (file, "v1");
+
+ /* The above is not valid for SIMT where scalars are
+ duplicated across all lanes, and lane 0 might not even
+ be enabled. */
+ gcc_assert (!flag_simt_mode);
+ }
else if (REG_P (vgpr_offset)
&& VGPR_REGNO_P (REGNO (vgpr_offset)))
fprintf (file, "v%d", REGNO (vgpr_offset) - FIRST_VGPR_REG);
@@ -7843,8 +8189,11 @@ print_operand (FILE *file, rtx x, int code)
}
switch (REGNO (XEXP (x, 0)))
{
- case VCC_REG:
case VCCZ_REG:
+ /* VCCZ is 1 if EXEC is 0. */
+ invert = !invert;
+ /* Fall through. */
+ case VCC_REG:
s = "_vcc";
break;
case SCC_REG:
@@ -7853,6 +8202,10 @@ print_operand (FILE *file, rtx x, int code)
s = "_scc";
break;
case EXECZ_REG:
+ /* EXECZ is 1 if EXEC is 0. */
+ invert = !invert;
+ /* Fall through. */
+ case EXEC_REG:
s = "_exec";
break;
default:
@@ -8280,6 +8633,8 @@ gcn_dwarf_register_span (rtx rtl)
#define TARGET_EXPAND_BUILTIN gcn_expand_builtin
#undef TARGET_EXPAND_DIVMOD_LIBFUNC
#define TARGET_EXPAND_DIVMOD_LIBFUNC gcn_expand_divmod_libfunc
+#undef TARGET_EXTRA_LIVE_ON_ENTRY
+#define TARGET_EXTRA_LIVE_ON_ENTRY gcn_extra_live_on_entry
#undef TARGET_FRAME_POINTER_REQUIRED
#define TARGET_FRAME_POINTER_REQUIRED gcn_frame_pointer_rqd
#undef TARGET_FUNCTION_ARG
@@ -8326,6 +8681,8 @@ gcn_dwarf_register_span (rtx rtl)
#define TARGET_LIBC_HAS_FUNCTION gcn_libc_has_function
#undef TARGET_MACHINE_DEPENDENT_REORG
#define TARGET_MACHINE_DEPENDENT_REORG gcn_md_reorg
+#undef TARGET_MD_ASM_ADJUST
+#define TARGET_MD_ASM_ADJUST gcn_md_asm_adjust
#undef TARGET_MEMORY_MOVE_COST
#define TARGET_MEMORY_MOVE_COST gcn_memory_move_cost
#undef TARGET_MODES_TIEABLE_P
@@ -8356,6 +8713,8 @@ gcn_dwarf_register_span (rtx rtl)
gcn_simd_clone_compute_vecsize_and_simdlen
#undef TARGET_SIMD_CLONE_USABLE
#define TARGET_SIMD_CLONE_USABLE gcn_simd_clone_usable
+#undef TARGET_SIMT_VF
+#define TARGET_SIMT_VF gcn_simt_vf
#undef TARGET_SMALL_REGISTER_CLASSES_FOR_MODE_P
#define TARGET_SMALL_REGISTER_CLASSES_FOR_MODE_P \
gcn_small_register_classes_for_mode_p
diff --git a/gcc/config/gcn/gcn.h b/gcc/config/gcn/gcn.h
index 162a0434999..630f8bf491a 100644
--- a/gcc/config/gcn/gcn.h
+++ b/gcc/config/gcn/gcn.h
@@ -16,6 +16,9 @@
#include "config/gcn/gcn-opts.h"
+/* Ensure these are available in the machine description. */
+#include "gomp-constants.h"
+
extern const struct gcn_device_def {
enum processor_type id;
const char *name;
@@ -231,6 +234,10 @@ STATIC_ASSERT (LAST_AVGPR_REG + 1 - FIRST_AVGPR_REG ==
256);
#define VGPR_REGNO_P(N) ((N) >= FIRST_VGPR_REG && (N) <=
LAST_VGPR_REG)
#define AVGPR_REGNO_P(N) ((N) >= FIRST_AVGPR_REG && (N) <=
LAST_AVGPR_REG)
#define SSRC_REGNO_P(N) ((N) <= PRIVATE_LIMT_REG && (N) !=
VCCZ_REG)
+#define SCALAR_HARDREG_P(N) ((N) < FIRST_VGPR_REG \
+ || ((N) > LAST_AVGPR_REG \
+ && (N) < FIRST_PSEUDO_REGISTER))
+#define VECTOR_HARDREG_P(N) ((N) >= FIRST_VGPR_REG && (N) <=
LAST_AVGPR_REG)
#define SDST_REGNO_P(N) ((N) <= EXEC_HI_REG && (N) != VCCZ_REG)
#define CC_REG_P(X) (REG_P (X) && CC_REGNO_P (REGNO (X)))
#define CC_REGNO_P(X) ((X) == SCC_REG || (X) == VCC_REG)
@@ -793,6 +800,7 @@ struct GTY(()) machine_function
unsigned HOST_WIDE_INT reduction_limit;
bool use_flat_addressing;
+ bool vectors_enabled;
};
#endif
diff --git a/gcc/config/gcn/gcn.md b/gcc/config/gcn/gcn.md
index 0efc4f6a012..bebd53c8d18 100644
--- a/gcc/config/gcn/gcn.md
+++ b/gcc/config/gcn/gcn.md
@@ -47,7 +47,9 @@ (define_constants
(EXEC_LO_REG 126)
(EXEC_HI_REG 127)
(EXECZ_REG 128)
- (SCC_REG 129)])
+ (SCC_REG 129)
+ (LANE_ID_REG 161) ; v1
+ ])
(define_constants
[(SP_REGNUM 16)
@@ -60,7 +62,10 @@ (define_c_enum "unspecv" [
UNSPECV_KERNEL_RETURN
UNSPECV_BARRIER
UNSPECV_ATOMIC
- UNSPECV_ICACHE_INV])
+ UNSPECV_ICACHE_INV
+ UNSPECV_SIMT_FORK UNSPECV_SIMT_JOIN
+ UNSPECV_SIMT_ENTER UNSPECV_SIMT_EXIT
+ UNSPECV_SIMT_EXEC_BARRIER])
(define_c_enum "unspec" [
UNSPEC_ADDPTR
@@ -87,7 +92,11 @@ (define_c_enum "unspec" [
UNSPEC_FLOOR UNSPEC_CEIL UNSPEC_SIN UNSPEC_COS UNSPEC_EXP2 UNSPEC_LOG2
UNSPEC_LDEXP UNSPEC_FREXP_EXP UNSPEC_FREXP_MANT
UNSPEC_DIV_SCALE UNSPEC_DIV_FMAS UNSPEC_DIV_FIXUP
- UNSPEC_PAIR])
+ UNSPEC_PAIR
+ UNSPEC_SIMT_XCHG_BFLY
+ UNSPEC_SIMT_VOTE_ANY
+ UNSPEC_VSUBREG_WRITE UNSPEC_VSUBREG_READ
+ UNSPEC_ENSURE_SCALAR])
;; }}}
;; {{{ Attributes
@@ -290,6 +299,7 @@ (define_attr "cdna" "any,cdna2" (const_string "any"))
(define_attr "rdna" "any,no,yes,n3,3p" (const_string "any"))
(define_attr "xnack" "na,off,on" (const_string "na"))
+(define_attr "simt" "any,simd,simt" (const_string "any"))
(define_attr "enabled" ""
(cond [
@@ -313,6 +323,12 @@ (define_attr "enabled" ""
(const_int 0)
(and (eq_attr "xnack" "on")
(eq (symbol_ref "TARGET_XNACK") (const_int 0)))
+ (const_int 0)
+ (and (eq_attr "simt" "simd")
+ (ne (symbol_ref "flag_simt_mode") (const_int 0)))
+ (const_int 0)
+ (and (eq_attr "simt" "simt")
+ (ne (symbol_ref "flag_simt_mode") (const_int 1)))
(const_int 0)]
(const_int 1)))
@@ -346,6 +362,45 @@ (define_attr "vccwait" "" (const_int 0))
(define_attr "transop" "yes,no" (const_string "no"))
+;; Attributes that control the simt-vect pass.
+; simt_clobber takes a decimal encoding that describes how to handle each
+; clobber in the pattern, concatenated into a single integer:
+; - 0: unset (default; causes as ICE in simt-vect)
+; - 1: keep as-is
+; - 2: delete
+; - 3: transform scalar to vector
+; E.g. "231" deletes the first clobber, transforms the second, and does
+; nothing to the third clobber (a fourth clobber would ICE).
+
+(define_attr "simt_clobber" "" (const_int 0))
+
+; simt_const controls what mode const_vector is assigned, when transformed.
+; - default: use the same mode as the set destination
+; - si2: use V64SI unconditionally for second input (e.g. for shifts)
+; - sextdi2: use (sign_extend:V64DI (const_vector:V64SI))
+; for constants in the second input (e.g. for *_imm)
+; - uextdi2: likewise, but with zero_extend
+; - cstore: look inside (set (r) (compare (MODE)))
+
+(define_attr "simt_const" "default,si2,sextdi2,uextdi2,cstore"
+ (const_string "default"))
+
+; simt_special instructs the simt_vect pass to treat the conversion as
+; a special case.
+;
+; - default: no special treatment
+; - set_vcc: prevents [(set (...) (...)) (set (vcc:DI) (op:DI ...))]
+; getting broken by a DI -> V64DI conversion.
+; - add_di_clobber: Add an additional clobber:DI
+; - add_vdi_clobber: Likewise, but clobber:V64DI
+; - add_vsi_di_clobber: Likewise, two clobbers (V64SI and DI)
+; - symbols_ok: prevents breaking out symbol_ref and label_ref
+; - scalar_shiftop: keep the shift amount operand scalar
+
+(define_attr "simt_special" "default,set_vcc,add_di_clobber,add_vdi_clobber,
+ add_vsi_di_clobber,symbols_ok,scalar_shiftop"
+ (const_string "default"))
+
;; }}}
;; {{{ Iterators useful across the wole machine description
@@ -545,22 +600,23 @@ (define_insn "*movbi"
[(set (match_operand:BI 0 "nonimmediate_operand")
(match_operand:BI 1 "gcn_load_operand"))]
"gcn_valid_move_p (BImode, operands[0], operands[1])"
- {@ [cons: =0,1 ;attrs:
type,exec,length,vcmp,xnack,laneselect,flatmemaccess]
- [Sg ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ] s_mov_b32\t%0 ,%1
- [v ,vSvA;vop1 ,* ,4 ,* ,* ,* ,* ] v_mov_b32\t%0 ,%1
- [Sg ,v ;vop3a,none,4 ,* ,* ,read,* ] v_readlane_b32\t%0 ,%1 ,0
- [cs ,SS ;sopk ,* ,4 ,* ,* ,* ,* ] s_cmpk_lg_u32\t%1 ,0
- [cV ,v ;vopc ,* ,4 ,vcmp,* ,* ,* ] v_cmp_ne_u32\tvcc ,0 ,%1
- [cV ,SS ;mult ,* ,8 ,* ,* ,* ,* ] s_mov_b32\tvcc_lo
,%1\;s_mov_b32\tvcc_hi ,0
- [Sm ,RS ;smem ,* ,12,* ,off,* ,* ] s_load_dword\t%0
,%A1\;s_waitcnt\tlgkmcnt(0)
- [&Sm,RS ;smem ,* ,12,* ,on ,* ,* ] ^
- [RS ,Sm ;smem ,* ,12,* ,* ,* ,* ] s_store_dword\t%1 ,%A0
- [v ,RF ;flat ,* ,12,* ,off,* ,load ] flat_load_dword\t%0
,%A1%O1%g1\;s_waitcnt\t0
- [&v ,RF ;flat ,* ,12,* ,on ,* ,load ] ^
- [RF ,v ;flat ,* ,12,* ,* ,* ,store] flat_store_dword\t%A0 ,%1%O0%g0
- [v ,RM ;flat ,* ,12,* ,off,* ,load ] global_load_dword\t%0
,%A1%O1%g1\;s_waitcnt\tvmcnt(0)
- [&v ,RM ;flat ,* ,12,* ,on ,* ,load ] ^
- [RM ,v ;flat ,* ,12,* ,* ,* ,store] global_store_dword\t%A0,%1%O0%g0
+ {@ [cons: =0,1 ;attrs:
type,exec,length,vcmp,xnack,laneselect,simt,flatmemaccess]
+ [Sg ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ,* ] s_mov_b32\t%0 ,%1
+ [v ,vSvA;vop1 ,* ,4 ,* ,* ,* ,* ,* ] v_mov_b32\t%0 ,%1
+ [Sg ,v ;vop3a,none,4 ,* ,* ,read,simd,* ] v_readlane_b32\t%0 ,%1 ,0
+ [Sg ,v ;vop3a,none,4 ,* ,* ,* ,simt,* ] v_readfirstlane_b32\t%0 ,%1
+ [cs ,SS ;sopk ,* ,4 ,* ,* ,* ,* ,* ] s_cmpk_lg_u32\t%1 ,0
+ [cV ,v ;vopc ,* ,4 ,vcmp,* ,* ,* ,* ] v_cmp_ne_u32\tvcc ,0 ,%1
+ [cV ,SS ;mult ,* ,8 ,* ,* ,* ,* ,* ] s_mov_b32\tvcc_lo
,%1\;s_mov_b32\tvcc_hi ,0
+ [Sm ,RS ;smem ,* ,12,* ,off,* ,* ,* ] s_load_dword\t%0
,%A1\;s_waitcnt\tlgkmcnt(0)
+ [&Sm,RS ;smem ,* ,12,* ,on ,* ,* ,* ] ^
+ [RS ,Sm ;smem ,* ,12,* ,* ,* ,* ,* ] s_store_dword\t%1 ,%A0
+ [v ,RF ;flat ,* ,12,* ,off,* ,* ,load ] flat_load_dword\t%0
,%A1%O1%g1\;s_waitcnt\t0
+ [&v ,RF ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [RF ,v ;flat ,* ,12,* ,* ,* ,* ,store] flat_store_dword\t%A0
,%1%O0%g0
+ [v ,RM ;flat ,* ,12,* ,off,* ,* ,load ] global_load_dword\t%0
,%A1%O1%g1\;s_waitcnt\tvmcnt(0)
+ [&v ,RM ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [RM ,v ;flat ,* ,12,* ,* ,* ,* ,store]
global_store_dword\t%A0,%1%O0%g0
})
; 32bit move pattern
@@ -569,38 +625,40 @@ (define_insn "*mov<mode>_insn"
[(set (match_operand:SISF 0 "nonimmediate_operand")
(match_operand:SISF 1 "gcn_load_operand"))]
"!(MEM_P (operands[0]) && MEM_P (operands[1]))"
- {@ [cons: =0, 1; attrs: type, exec, length, cdna, xnack, laneselect,
flatmemaccess]
- [SD ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ] s_mov_b32\t%0, %1
- [SD ,J ;sopk ,* ,4 ,* ,* ,* ,* ] s_movk_i32\t%0, %1
- [SD ,B ;sop1 ,* ,8 ,* ,* ,* ,* ] s_mov_b32\t%0, %1
- [SD ,RB ;smem ,* ,12,* ,off,* ,* ] s_buffer_load%s0\t%0,
s[0:3], %1\;s_waitcnt\tlgkmcnt(0)
- [&SD ,RB ;smem ,* ,12,* ,on ,* ,* ] ^
- [RB ,Sm ;smem ,* ,12,* ,* ,* ,* ] s_buffer_store%s1\t%1,
s[0:3], %0
- [Sm ,RS ;smem ,* ,12,* ,off,* ,* ] s_load_dword\t%0,
%A1\;s_waitcnt\tlgkmcnt(0)
- [&Sm ,RS ;smem ,* ,12,* ,on ,* ,* ] ^
- [RS ,Sm ;smem ,* ,12,* ,* ,* ,* ] s_store_dword\t%1, %A0
- [v ,v ;vop1 ,* ,4 ,* ,* ,* ,* ] v_mov_b32\t%0, %1
- [Sg ,v ;vop3a,none,8 ,* ,* ,read ,* ] v_readlane_b32\t%0, %1, 0
- [v ,Sv ;vop3a,none,8 ,* ,* ,write,* ] v_writelane_b32\t%0, %1, 0
- [v ,^a ;vop3p_mai,*,8,* ,* ,* ,* ] v_accvgpr_read_b32\t%0, %1
- [a ,v ;vop3p_mai,*,8,* ,* ,* ,* ] v_accvgpr_write_b32\t%0, %1
- [a ,a ;vop1 ,* ,4,cdna2,* ,* ,* ] v_accvgpr_mov_b32\t%0, %1
- [v ,RF ;flat ,* ,12,* ,off,* ,load ] flat_load_dword\t%0,
%A1%O1%g1\;s_waitcnt\t0
- [&v ,RF ;flat ,* ,12,* ,on ,* ,load ] ^
- [^a ,RF ;flat ,* ,12,cdna2,off,* ,load ] ^
- [&^a ,RF ;flat ,* ,12,cdna2,on ,* ,load ] ^
- [RF ,v ;flat ,* ,12,* ,* ,* ,store] flat_store_dword\t%A0,
%1%O0%g0
- [RF ,a ;flat ,* ,12,cdna2,* ,* ,store] ^
- [v ,B ;vop1 ,* ,8 ,* ,* ,* ,* ] v_mov_b32\t%0, %1
- [RLRG,v ;ds ,* ,12,* ,* ,* ,* ] ds_write_b32\t%A0,
%1%O0\;s_waitcnt\tlgkmcnt(0)
- [v ,RLRG;ds ,* ,12,* ,* ,* ,* ] ds_read_b32\t%0,
%A1%O1\;s_waitcnt\tlgkmcnt(0)
- [SD ,Y ;sop1 ,* ,8 ,* ,* ,* ,* ] s_mov_b32\t%0, %1
- [v ,RM ;flat ,* ,12,* ,off,* ,load ] global_load_dword\t%0,
%A1%O1%g1\;s_waitcnt\tvmcnt(0)
- [&v ,RM ;flat ,* ,12,* ,on ,* ,load ] ^
- [^a ,RM ;flat ,* ,12,cdna2,off,* ,load ] ^
- [&^a ,RM ;flat ,* ,12,cdna2,on ,* ,load ] ^
- [RM ,v ;flat ,* ,12,* ,* ,* ,store] global_store_dword\t%A0,
%1%O0%g0
- [RM ,a ;flat ,* ,12,cdna2,* ,* ,store] ^
+ {@ [cons: =0,1 ;attrs: type,exec,length,cdna
,xnack,laneselect,simt,flatmemaccess]
+ [SD ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ,* ] s_mov_b32\t%0, %1
+ [SD ,J ;sopk ,* ,4 ,* ,* ,* ,* ,* ] s_movk_i32\t%0, %1
+ [SD ,B ;sop1 ,* ,8 ,* ,* ,* ,* ,* ] s_mov_b32\t%0, %1
+ [SD ,RB ;smem ,* ,12,* ,off,* ,* ,* ]
s_buffer_load%s0\t%0, s[0:3], %1\;s_waitcnt\tlgkmcnt(0)
+ [&SD ,RB ;smem ,* ,12,* ,on ,* ,* ,* ] ^
+ [RB ,Sm ;smem ,* ,12,* ,* ,* ,* ,* ]
s_buffer_store%s1\t%1, s[0:3], %0
+ [Sm ,RS ;smem ,* ,12,* ,off,* ,* ,* ] s_load_dword\t%0,
%A1\;s_waitcnt\tlgkmcnt(0)
+ [&Sm ,RS ;smem ,* ,12,* ,on ,* ,* ,* ] ^
+ [RS ,Sm ;smem ,* ,12,* ,* ,* ,* ,* ] s_store_dword\t%1,
%A0
+ [v ,v ;vop1 ,* ,4 ,* ,* ,* ,* ,* ] v_mov_b32\t%0, %1
+ [Sg ,v ;vop3a ,none,8 ,* ,* ,read ,simd,* ] v_readlane_b32\t%0,
%1, 0
+ [v ,Sv ;vop3a ,none,8 ,* ,* ,write,simd,* ]
v_writelane_b32\t%0, %1, 0
+ [Sg ,v ;vop3a ,none,8 ,* ,* ,* ,simt,* ]
v_readfirstlane_b32\t%0, %1
+ [v ,Sv ;vop3a ,none,8 ,* ,* ,* ,simt,* ] v_mov_b32\t%0, %1
+ [v ,^a ;vop3p_mai,* ,8 ,* ,* ,* ,* ,* ]
v_accvgpr_read_b32\t%0, %1
+ [a ,v ;vop3p_mai,* ,8 ,* ,* ,* ,* ,* ]
v_accvgpr_write_b32\t%0, %1
+ [a ,a ;vop1 ,* ,4 ,cdna2,* ,* ,* ,* ]
v_accvgpr_mov_b32\t%0, %1
+ [v ,RF ;flat ,* ,12,* ,off,* ,* ,load ]
flat_load_dword\t%0, %A1%O1%g1\;s_waitcnt\t0
+ [&v ,RF ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [^a ,RF ;flat ,* ,12,cdna2,off,* ,* ,load ] ^
+ [&^a ,RF ;flat ,* ,12,cdna2,on ,* ,* ,load ] ^
+ [RF ,v ;flat ,* ,12,* ,* ,* ,* ,store]
flat_store_dword\t%A0, %1%O0%g0
+ [RF ,a ;flat ,* ,12,cdna2,* ,* ,* ,store] ^
+ [v ,B ;vop1 ,* ,8 ,* ,* ,* ,* ,* ] v_mov_b32\t%0, %1
+ [RLRG,v ;ds ,* ,12,* ,* ,* ,* ,* ] ds_write_b32\t%A0,
%1%O0\;s_waitcnt\tlgkmcnt(0)
+ [v ,RLRG;ds ,* ,12,* ,* ,* ,* ,* ] ds_read_b32\t%0,
%A1%O1\;s_waitcnt\tlgkmcnt(0)
+ [SD ,Y ;sop1 ,* ,8 ,* ,* ,* ,* ,* ] s_mov_b32\t%0, %1
+ [v ,RM ;flat ,* ,12,* ,off,* ,* ,load ]
global_load_dword\t%0, %A1%O1%g1\;s_waitcnt\tvmcnt(0)
+ [&v ,RM ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [^a ,RM ;flat ,* ,12,cdna2,off,* ,* ,load ] ^
+ [&^a ,RM ;flat ,* ,12,cdna2,on ,* ,* ,load ] ^
+ [RM ,v ;flat ,* ,12,* ,* ,* ,* ,store]
global_store_dword\t%A0, %1%O0%g0
+ [RM ,a ;flat ,* ,12,cdna2,* ,* ,* ,store] ^
})
; 8/16bit move pattern
@@ -610,36 +668,38 @@ (define_insn "*mov<mode>_insn"
[(set (match_operand:QIHI 0 "nonimmediate_operand")
(match_operand:QIHI 1 "gcn_load_operand"))]
"gcn_valid_move_p (<MODE>mode, operands[0], operands[1])"
- {@ [cons: =0, 1; attrs: type, exec, length, cdna, xnack, laneselect,
flatmemaccess]
- [SD ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ] s_mov_b32\t%0, %1
- [SD ,J ;sopk ,* ,4 ,* ,* ,* ,* ] s_movk_i32\t%0, %1
- [SD ,B ;sop1 ,* ,8 ,* ,* ,* ,* ] s_mov_b32\t%0, %1
- [v ,v ;vop1 ,* ,4 ,* ,* ,* ,* ] v_mov_b32\t%0, %1
- [Sg ,v ;vop3a,none,4 ,* ,* ,read ,* ] v_readlane_b32\t%0, %1, 0
- [v ,Sv ;vop3a,none,4 ,* ,* ,write,* ] v_writelane_b32\t%0, %1, 0
- [v ,^a ;vop3p_mai,*,8,* ,* ,* ,* ] v_accvgpr_read_b32\t%0, %1
- [a ,v ;vop3p_mai,*,8,* ,* ,* ,* ] v_accvgpr_write_b32\t%0, %1
- [a ,a ;vop1 ,* ,8,cdna2,* ,* ,* ] v_accvgpr_mov_b32\t%0, %1
- [v ,RF ;flat ,* ,12,* ,off,* ,load ] flat_load%o1\t%0,
%A1%O1%g1\;s_waitcnt\t0
- [&v ,RF ;flat ,* ,12,* ,on ,* ,load ] ^
- [^a ,RF ;flat ,* ,12,cdna2,off,* ,load ] ^
- [&^a ,RF ;flat ,* ,12,cdna2,on ,* ,load ] ^
- [RF ,v ;flat ,* ,12,* ,* ,* ,store] flat_store%s0\t%A0, %1%O0%g0
- [RF ,a ;flat ,* ,12,cdna2,* ,* ,store] ^
- [v ,B ;vop1 ,* ,8 ,* ,* ,* ,* ] v_mov_b32\t%0, %1
- [RLRG,v ;ds ,* ,12,* ,* ,* ,* ] ds_write%b0\t%A0,
%1%O0\;s_waitcnt\tlgkmcnt(0)
- [v ,RLRG;ds ,* ,12,* ,* ,* ,* ] ds_read%u1\t%0,
%A1%O1\;s_waitcnt\tlgkmcnt(0)
- [v ,RM ;flat ,* ,12,* ,off,* ,load ] global_load%o1\t%0,
%A1%O1%g1\;s_waitcnt\tvmcnt(0)
- [&v ,RM ;flat ,* ,12,* ,on ,* ,load ] ^
- [^a ,RM ;flat ,* ,12,cdna2,off,* ,load ] ^
- [&^a ,RM ;flat ,* ,12,cdna2,on ,* ,load ] ^
- [RM ,v ;flat ,* ,12,* ,* ,* ,store] global_store%s0\t%A0,
%1%O0%g0
- [RM ,a ;flat ,* ,12,cdna2,* ,* ,store] ^
+ {@ [cons: =0,1 ;attrs: type,exec,length,cdna
,xnack,laneselect,simt,flatmemaccess]
+ [SD ,SSA ;sop1 ,* ,4 ,* ,* ,* ,* ,* ] s_mov_b32\t%0, %1
+ [SD ,J ;sopk ,* ,4 ,* ,* ,* ,* ,* ] s_movk_i32\t%0, %1
+ [SD ,B ;sop1 ,* ,8 ,* ,* ,* ,* ,* ] s_mov_b32\t%0, %1
+ [v ,v ;vop1 ,* ,4 ,* ,* ,* ,* ,* ] v_mov_b32\t%0, %1
+ [Sg ,v ;vop3a ,none,4 ,* ,* ,read ,simd,* ] v_readlane_b32\t%0,
%1, 0
+ [v ,Sv ;vop3a ,none,4 ,* ,* ,write,simd,* ]
v_writelane_b32\t%0, %1, 0
+ [Sg ,v ;vop3a ,none,4 ,* ,* ,* ,simt,* ]
v_readfirstlane_b32\t%0, %1
+ [v ,Sv ;vop3a ,none,4 ,* ,* ,* ,simt,* ] v_mov_b32\t%0, %1
+ [v ,^a ;vop3p_mai,* ,8 ,* ,* ,* ,* ,* ]
v_accvgpr_read_b32\t%0, %1
+ [a ,v ;vop3p_mai,* ,8 ,* ,* ,* ,* ,* ]
v_accvgpr_write_b32\t%0, %1
+ [a ,a ;vop1 ,* ,8 ,cdna2,* ,* ,* ,* ]
v_accvgpr_mov_b32\t%0, %1
+ [v ,RF ;flat ,* ,12,* ,off,* ,* ,load ] flat_load%o1\t%0,
%A1%O1%g1\;s_waitcnt\t0
+ [&v ,RF ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [^a ,RF ;flat ,* ,12,cdna2,off,* ,* ,load ] ^
+ [&^a ,RF ;flat ,* ,12,cdna2,on ,* ,* ,load ] ^
+ [RF ,v ;flat ,* ,12,* ,* ,* ,* ,store] flat_store%s0\t%A0,
%1%O0%g0
+ [RF ,a ;flat ,* ,12,cdna2,* ,* ,* ,store] ^
+ [v ,B ;vop1 ,* ,8 ,* ,* ,* ,* ,* ] v_mov_b32\t%0, %1
+ [RLRG,v ;ds ,* ,12,* ,* ,* ,* ,* ] ds_write%b0\t%A0,
%1%O0\;s_waitcnt\tlgkmcnt(0)
+ [v ,RLRG;ds ,* ,12,* ,* ,* ,* ,* ] ds_read%u1\t%0,
%A1%O1\;s_waitcnt\tlgkmcnt(0)
+ [v ,RM ;flat ,* ,12,* ,off,* ,* ,load ] global_load%o1\t%0,
%A1%O1%g1\;s_waitcnt\tvmcnt(0)
+ [&v ,RM ;flat ,* ,12,* ,on ,* ,* ,load ] ^
+ [^a ,RM ;flat ,* ,12,cdna2,off,* ,* ,load ] ^
+ [&^a ,RM ;flat ,* ,12,cdna2,on ,* ,* ,load ] ^
+ [RM ,v ;flat ,* ,12,* ,* ,* ,* ,store]
global_store%s0\t%A0, %1%O0%g0
+ [RM ,a ;flat ,* ,12,cdna2,* ,* ,* ,store] ^
})
; 64bit move pattern
-(define_insn_and_split "*mov<mode>_insn"
+(define_insn_and_split "mov<mode>_insn"
[(set (match_operand:DIDF 0 "nonimmediate_operand")
(match_operand:DIDF 1 "general_operand"))]
"GET_CODE(operands[1]) != SYMBOL_REF
@@ -705,7 +765,7 @@ (define_insn_and_split "*mov<mode>_insn"
; 128-bit move.
-(define_insn_and_split "*movti_insn"
+(define_insn_and_split "movti_insn"
[(set (match_operand:TI 0 "nonimmediate_operand")
(match_operand:TI 1 "general_operand" ))]
"!(MEM_P (operands[0]) && MEM_P (operands[1]))"
@@ -866,6 +926,42 @@ (define_insn "cjump"
".Lskip%=:";
}
}
+ [(set_attr "type" "sopp")
+ (set (attr "length")
+ (if_then_else (and (ge (minus (match_dup 0) (pc))
+ (const_int -131072))
+ (lt (minus (match_dup 0) (pc))
+ (const_int 131072)))
+ (const_int 4)
+ (const_int 36)))
+ (set_attr "simt_special" "symbols_ok")])
+
+(define_insn "*cjump_vcc"
+ [(set (pc)
+ (if_then_else
+ (match_operator:DI 1 "gcn_conditional_operator"
+ [(match_operand:DI 2 "gcn_conditional_register_operand" "cV")
+ (const_int 0)])
+ (label_ref (match_operand 0))
+ (pc)))]
+ ""
+ {
+ if (get_attr_length (insn) == 4)
+ return "s_cbranch%C1\t%0";
+ else
+ {
+ /* !!! This sequence clobbers EXEC_SAVE_REG and CC_SAVE_REG but
+ restores SCC. */
+ return "s_cbranch%c1\t.Lskip%=\;"
+ "s_mov_b32\ts22, scc\;"
+ "s_getpc_b64\ts[20:21]\;"
+ "s_add_u32\ts20, s20, %0@rel32@lo+4\;"
+ "s_addc_u32\ts21, s21, %0@rel32@hi+4\;"
+ "s_cmpk_lg_u32\ts22, 0\;"
+ "s_setpc_b64\ts[20:21]\n"
+ ".Lskip%=:";
+ }
+ }
[(set_attr "type" "sopp")
(set (attr "length")
(if_then_else (and (ge (minus (match_dup 0) (pc))
@@ -908,7 +1004,8 @@ (define_insn "gcn_simple_call"
s_getpc_b64\t%2\;s_add_u32\t%L2, %L2, %0@rel32@lo+4\;s_addc_u32\t%H2, %H2,
%0@rel32@hi+4\;s_swappc_b64\ts[18:19], %2
s_swappc_b64\ts[18:19], %0"
[(set_attr "type" "mult,sop1")
- (set_attr "length" "24,4")])
+ (set_attr "length" "24,4")
+ (set_attr "simt_clobber" "11")])
(define_insn "movdi_symbol"
[(set (match_operand:DI 0 "nonimmediate_operand" "=Sg")
@@ -931,7 +1028,9 @@ (define_insn "movdi_symbol"
"s_addc_u32\t%H0, %H0, %1@rel32@hi+4";
}
[(set_attr "type" "mult")
- (set_attr "length" "32")])
+ (set_attr "length" "32")
+ ; The conversion changes this to a straight move, and inserts another
instance.
+ (set_attr "simt_clobber" "2")])
(define_insn "movdi_symbol_save_scc"
[(set (match_operand:DI 0 "nonimmediate_operand" "=Sg")
@@ -972,7 +1071,8 @@ (define_insn "gcn_indirect_call"
""
"s_swappc_b64\ts[18:19], %0"
[(set_attr "type" "sop1")
- (set_attr "length" "4")])
+ (set_attr "length" "4")
+ (set_attr "simt_clobber" "11")])
(define_expand "call_value"
[(parallel [(set (match_operand 0 "")
@@ -994,7 +1094,9 @@ (define_insn "gcn_call_value"
s_getpc_b64\t%3\;s_add_u32\t%L3, %L3, %1@rel32@lo+4\;s_addc_u32\t%H3, %H3,
%1@rel32@hi+4\;s_swappc_b64\ts[18:19], %3
s_swappc_b64\ts[18:19], %1"
[(set_attr "type" "sop1")
- (set_attr "length" "24")])
+ (set_attr "length" "24")
+ (set_attr "simt_clobber" "11")
+ (set_attr "simt_special" "symbols_ok")])
(define_insn "gcn_call_value_indirect"
[(set (match_operand 0 "register_operand" "=Sgv")
@@ -1005,7 +1107,8 @@ (define_insn "gcn_call_value_indirect"
""
"s_swappc_b64\ts[18:19], %1"
[(set_attr "type" "sop1")
- (set_attr "length" "4")])
+ (set_attr "length" "4")
+ (set_attr "simt_clobber" "11")])
; GCN does not have an instruction to clear only part of the instruction
; cache, so the operands are ignored.
@@ -1046,7 +1149,9 @@ (define_insn "cstoresi4"
s_cmp%D1\t%2, %3
s_cmp%D1\t%2, %3"
[(set_attr "type" "sopc,sopk,sopk,sopk")
- (set_attr "length" "4,4,8,8")])
+ (set_attr "length" "4,4,8,8")
+ (set_attr "simt_special" "add_di_clobber")
+ (set_attr "simt_const" "cstore")])
(define_expand "cbranchsi4"
[(match_operator 0 "gcn_compare_operator"
@@ -1067,8 +1172,8 @@ (define_expand "cbranchsi4"
(define_expand "cstoredi4"
[(set (match_operand:BI 0 "gcn_conditional_register_operand")
(match_operator:BI 1 "gcn_compare_operator"
- [(match_operand:DI 2 "gcn_alu_operand")
- (match_operand:DI 3 "gcn_alu_operand")]))]
+ [(match_operand:DI 2 "gcn_alu_operand")
+ (match_operand:DI 3 "gcn_alu_operand")]))]
""
{})
@@ -1083,7 +1188,9 @@ (define_insn "cstoredi4_vec_and_scalar"
v_cmp%E1\tvcc, %2, %3"
[(set_attr "type" "sopc,vopc")
(set_attr "vcmp" "vcmp")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_special" "add_di_clobber")
+ (set_attr "simt_const" "cstore")])
(define_insn "cstoredi4_vector"
[(set (match_operand:BI 0 "gcn_conditional_register_operand" "= cV")
@@ -1094,7 +1201,9 @@ (define_insn "cstoredi4_vector"
"v_cmp%E1\tvcc, %2, %3"
[(set_attr "type" "vopc")
(set_attr "vcmp" "vcmp")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_special" "add_di_clobber")
+ (set_attr "simt_const" "cstore")])
(define_expand "cbranchdi4"
[(match_operator 0 "gcn_compare_operator"
@@ -1121,7 +1230,9 @@ (define_insn "cstore<mode>4"
"v_cmp%E1\tvcc, %2, %3"
[(set_attr "type" "vopc")
(set_attr "vcmp" "vcmp")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_special" "add_di_clobber")
+ (set_attr "simt_const" "cstore")])
(define_expand "cbranch<mode>4"
[(match_operator 0 "gcn_fp_compare_operator"
@@ -1137,6 +1248,22 @@ (define_expand "cbranch<mode>4"
DONE;
})
+(define_insn "@cselect<mode>"
+ [(set (match_operand:SIDI 0 "register_operand")
+ (if_then_else:SIDI
+ (ne:BI (match_operand:BI 1 "gcn_conditional_register_operand")
+ (const_int 0))
+ (match_operand:SIDI 2 "gcn_alu_operand")
+ (match_operand:SIDI 3 "gcn_alu_operand")))]
+ ""
+ {@ [cons: =0, 1, 2, 3; attrs: length]
+ [SDcV, cs, ASS, ASS; 4] s_cselect%b0\t%0, %2, %3
+ [SDcV, cs, B , ASS; 8] ^
+ [SDcV, cs, ASS, B ; 8] ^
+ }
+ [(set_attr "type" "sop2")
+ (set_attr "simt_special" "add_vdi_clobber")])
+
;; }}}
;; {{{ ALU special cases: Plus
@@ -1152,7 +1279,8 @@ (define_insn "addsi3"
s_add_i32\t%0, %1, %2
{v_add_u32|v_add_nc_u32}\t%0, %2, %1"
[(set_attr "type" "sop2,sopk,sop2,vop2")
- (set_attr "length" "4,4,8,8")])
+ (set_attr "length" "4,4,8,8")
+ (set_attr "simt_clobber" "2")])
(define_expand "addsi3_scc"
[(parallel [(set (match_operand:SI 0 "register_operand")
@@ -1205,7 +1333,8 @@ (define_insn_and_split "adddi3"
DONE;
}
[(set_attr "type" "mult,vmult")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "21")])
(define_expand "adddi3_scc"
[(parallel [(set (match_operand:DI 0 "register_operand")
@@ -1346,14 +1475,31 @@ (define_insn "subsi3"
{v_subrev_u32|v_subrev_nc_u32}\t%0, %2, %1
{v_sub_u32|v_sub_nc_u32}\t%0, %1, %2"
[(set_attr "type" "sop2,sop2,vop2,vop2")
- (set_attr "length" "4,8,8,8")])
+ (set_attr "length" "4,8,8,8")
+ (set_attr "simt_clobber" "2")])
-(define_insn_and_split "subdi3"
+(define_expand "subdi3"
+ [(parallel [
+ (set (match_operand:DI 0 "register_operand")
+ (minus:DI
+ (match_operand:DI 1 "gcn_alu_operand")
+ (match_operand:DI 2 "gcn_alu_operand")))
+ (clobber (match_dup 3))])]
+ ""
+ {
+ if (reload_completed)
+ operands[3] = gen_rtx_REG (BImode, SCC_REG);
+ else
+ operands[3] = gen_rtx_SCRATCH (BImode);
+ }
+)
+
+(define_insn_and_split "*subdi3_insn"
[(set (match_operand:DI 0 "register_operand" "=Sg, Sg")
(minus:DI
(match_operand:DI 1 "gcn_alu_operand" "SgA,SgB")
(match_operand:DI 2 "gcn_alu_operand" "SgB,SgA")))
- (clobber (reg:BI SCC_REG))]
+ (clobber (match_scratch:BI 3 "=cs, cs"))]
""
"#"
"reload_completed"
@@ -1375,7 +1521,8 @@ (define_insn_and_split "subdi3"
gcn_operand_part (DImode, operands[1], 1)));
DONE;
}
- [(set_attr "length" "8")])
+ [(set_attr "length" "8")
+ (set_attr "simt_clobber" "3")])
(define_insn "subsi3_scalar_carry"
[(set (match_operand:SI 0 "register_operand" "=Sg, Sg")
@@ -1452,7 +1599,8 @@ (define_insn "mulsi3"
s_mul_i32\t%0, %1, %2
v_mul_lo_u32\t%0, %1, %2"
[(set_attr "type" "sop2,sopk,sop2,vop3a")
- (set_attr "length" "4,4,8,4")])
+ (set_attr "length" "4,4,8,4")
+ (set_attr "simt_clobber" "2")])
(define_code_iterator any_extend [sign_extend zero_extend])
(define_code_attr sgnsuffix [(sign_extend "%i") (zero_extend "%u")])
@@ -1484,7 +1632,7 @@ (define_expand "<su>mulsi3_highpart"
})
(define_insn "<su>mulsi3_highpart_reg"
- [(set (match_operand:SI 0 "register_operand" "=Sg, v")
+ [(set (match_operand:SI 0 "register_operand" "=Sg, v")
(truncate:SI
(lshiftrt:DI
(mult:DI
@@ -1498,7 +1646,8 @@ (define_insn "<su>mulsi3_highpart_reg"
s_mul_hi<sgnsuffix>0\t%0, %1, %2
v_mul_hi<sgnsuffix>0\t%0, %2, %1"
[(set_attr "type" "sop2,vop3a")
- (set_attr "length" "4,8")])
+ (set_attr "length" "4,8")
+ (set_attr "simt_special" "scalar_shiftop")])
(define_insn "<su>mulsi3_highpart_imm"
[(set (match_operand:SI 0 "register_operand" "=Sg,Sg,v")
@@ -1515,7 +1664,9 @@ (define_insn "<su>mulsi3_highpart_imm"
s_mul_hi<sgnsuffix>0\t%0, %1, %2
v_mul_hi<sgnsuffix>0\t%0, %2, %1"
[(set_attr "type" "sop2,sop2,vop3a")
- (set_attr "length" "4,8,8")])
+ (set_attr "length" "4,8,8")
+ (set_attr "simt_const" "<su>extdi2")
+ (set_attr "simt_special" "scalar_shiftop")])
(define_expand "<su>mulsidi3"
[(set (match_operand:DI 0 "register_operand" "")
@@ -1567,7 +1718,8 @@ (define_insn_and_split "<su>mulsidi3_imm"
emit_insn (gen_mulsi3 (dstlo, operands[1], operands[2]));
emit_insn (gen_<su>mulsi3_highpart (dsthi, operands[1], operands[2]));
DONE;
- })
+ }
+ [(set_attr "simt_const" "<su>extdi2")])
(define_insn_and_split "muldi3"
[(set (match_operand:DI 0 "register_operand" "=&Sg,&Sg,?&v,?&v")
@@ -1598,7 +1750,8 @@ (define_insn_and_split "muldi3"
add = gen_rtx_PARALLEL (VOIDmode, gen_rtvec (2, add, clob));
emit_insn (add);
DONE;
- })
+ }
+ [(set_attr "simt_clobber" "32")])
(define_insn "<u>mulhisi3"
[(set (match_operand:SI 0 "register_operand" "=v")
@@ -1636,7 +1789,8 @@ (define_insn "<expander>si2"
s_<s_mnemonic>0\t%0, %1
v_<mnemonic>0\t%0, %1<popcount_extra_op>"
[(set_attr "type" "sop1,vop1")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "2")])
(define_code_iterator countzeros [clz ctz])
@@ -1658,7 +1812,8 @@ (define_insn "<expander>di2"
""
"s_<s_mnemonic>1\t%0, %1"
[(set_attr "type" "sop1")
- (set_attr "length" "4,8")])
+ (set_attr "length" "4,8")
+ (set_attr "simt_special" "add_vsi_di_clobber")])
(define_insn "gcn_flbit<mode>_int"
[(set (match_operand:SI 0 "register_operand" "=Sg,Sg")
@@ -1716,7 +1871,8 @@ (define_insn "<expander>si3"
v_<mnemonic>0\t%0, %1, %2
ds_<mnemonic>0\t%A0, %2%O0"
[(set_attr "type" "sop2,vop2,ds")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "2")])
(define_insn "<expander>si3"
[(set (match_operand:SI 0 "register_operand" "=Sg, Sg, ?v")
@@ -1730,7 +1886,8 @@ (define_insn "<expander>si3"
s_<mnemonic>0\t%0, %1, %2
v_<revmnemonic>0\t%0, %2, %1"
[(set_attr "type" "sop2,sop2,vop2")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "2")])
(define_expand "<expander>si3_scc"
[(parallel [(set (match_operand:SI 0 "gcn_valu_dst_operand")
@@ -1765,6 +1922,7 @@ (define_insn_and_split "one_cmpldi2"
operands[5] = gcn_operand_part (DImode, operands[0], 1);
operands[6] = gcn_operand_part (DImode, operands[1], 1);
}
+ [(set_attr "simt_clobber" "2")]
)
(define_code_iterator vec_and_scalar64_com [and ior xor])
@@ -1795,7 +1953,8 @@ (define_insn_and_split "<expander>di3"
operands[9] = gcn_operand_part (DImode, operands[2], 1);
}
[(set_attr "type" "sop2,vop2")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "2")])
(define_insn "<expander>di3"
[(set (match_operand:DI 0 "register_operand" "=Sg, Sg, ?v")
@@ -1809,7 +1968,9 @@ (define_insn "<expander>di3"
s_<mnemonic>0\t%0, %1, %2
v_<revmnemonic>0\t%0, %2, %1"
[(set_attr "type" "sop2,sop2,vop2")
- (set_attr "length" "8")])
+ (set_attr "length" "8")
+ (set_attr "simt_clobber" "2")
+ (set_attr "simt_const" "si2")])
;; }}}
;; {{{ ALU: generic 128-bit binop
@@ -2472,26 +2633,200 @@ (define_insn "*gcn_wavefront_barrier"
[(set_attr "type" "sopp")])
(define_expand "oacc_fork"
- [(set (match_operand:SI 0 "")
- (match_operand:SI 1 ""))
- (use (match_operand:SI 2 ""))]
+ [(set (match_operand:SI 0 "register_operand")
+ (unspec_volatile [(match_operand:SI 1 "")
+ (match_operand:SI 2 "")]
+ UNSPECV_SIMT_FORK))]
""
{
/* We need to have oacc_fork/oacc_join named patterns as a pair,
but the fork isn't actually used. */
- gcc_unreachable ();
+ if (!flag_simt_mode)
+ {
+ gcc_unreachable ();
+ DONE;
+ }
})
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_insn "*oacc_fork_insn"
+ [(set (match_operand:SI 0 "register_operand")
+ (unspec_volatile [(match_operand:SI 1 "")
+ (match_operand:SI 2 "")]
+ UNSPECV_SIMT_FORK))]
+ ""
+ "; simt_fork %0 %1 %2 ")
+
(define_expand "oacc_join"
- [(set (match_operand:SI 0 "")
- (match_operand:SI 1 ""))
- (use (match_operand:SI 2 ""))]
+ [(unspec_volatile [(match_operand:SI 0 "")
+ (match_operand:SI 1 "")
+ (match_operand:SI 2 "")]
+ UNSPECV_SIMT_JOIN)]
""
{
- emit_insn (gen_gcn_wavefront_barrier ());
+ int dim = INTVAL (operands[2]);
+ if (dim == GOMP_DIM_WORKER)
+ {
+ emit_insn (gen_gcn_wavefront_barrier ());
+ DONE;
+ }
+ else if (!flag_simt_mode)
+ {
+ gcc_unreachable ();
+ DONE;
+ }
+ /* else use default expansion. */
+ })
+
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_insn "*oacc_join_insn"
+ [(unspec_volatile [(match_operand:SI 0 "")
+ (match_operand:SI 1 "")
+ (match_operand:SI 2 "")]
+ UNSPECV_SIMT_JOIN)]
+ ""
+ "; simt_join %0 %1 %2 ")
+
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_insn "omp_simt_enter"
+ [(set (match_operand 0 "register_operand" "=r")
+ (unspec_volatile
+ [(match_operand 1 "nonmemory_operand" "ri")
+ (match_operand 2 "const_int_operand" "n")]
+ UNSPECV_SIMT_ENTER))
+ (set (reg:DI EXEC_REG) (const_int -1))
+ (set (reg:BI EXECZ_REG) (const_int 0))]
+ ""
+ "s_mov_b64 exec, -1; simt enter %0 %1 %2")
+
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_insn "omp_simt_exit"
+ [(unspec_volatile [(match_operand 0 "register_operand" "r")]
+ UNSPECV_SIMT_EXIT)
+ (set (reg:DI EXEC_REG) (const_int 1))
+ (set (reg:BI EXECZ_REG) (const_int 0))]
+ ""
+ "s_mov_b64 exec, 1; simt exit %0")
+
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_expand "omp_simt_lane"
+ [(set (match_operand:SI 0 "register_operand" "")
+ (reg:SI LANE_ID_REG))]
+ "flag_simt_mode")
+
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_expand "omp_simt_xchg_bfly"
+ [(match_operand 0 "register_operand")
+ (match_operand 1 "register_operand")
+ (match_operand:SI 2 "nonmemory_operand")]
+ "flag_simt_mode"
+ {
+ emit_insn (gen_omp_simt_xchg_bfly (GET_MODE (operands[0]), operands[0],
+ operands[1], operands[2]));
DONE;
})
+(define_insn "@omp_simt_xchg_bfly<mode>"
+ [(set (match_operand:MOV_MODE 0 "register_operand" "=r")
+ (unspec:MOV_MODE
+ [(match_operand:MOV_MODE 1 "register_operand" "r")
+ (match_operand:SI 2 "nonmemory_operand" "ri")]
+ UNSPEC_SIMT_XCHG_BFLY))]
+ "flag_simt_mode"
+ "; %0 = simt_xchg_bfly %1 %2")
+
+;; Implement IFN_GOMP_SIMT_VOTE_ANY:
+;; set operand 0 to zero iff all lanes supply zero in operand 1
+; This pseudo instruction should be replaced by the gcn_simt pass
+(define_insn "omp_simt_vote_any"
+ [(set (match_operand:SI 0 "register_operand" "")
+ (unspec:SI [(match_operand:SI 1 "register_operand" "")]
+ UNSPEC_SIMT_VOTE_ANY))]
+ ""
+ "; %0 = simt_vote_all %1")
+
+;; Ensure that scalar values are visible to all lanes at the SIMT_ENTER
+;; transition point. This could be a vec_duplicate, or just ensuring it's
+;; in a scalar register.
+
+(define_insn_and_split "@ensure_scalar<mode>"
+ [(set (match_operand:MOV_MODE 0 "register_operand")
+ (unspec:MOV_MODE [(match_operand:MOV_MODE 1 "register_operand")]
+ UNSPEC_ENSURE_SCALAR))
+ (clobber (match_scratch:SI 2))]
+ "flag_simt_mode"
+ {@ [cons: =0, 1, =2; attrs: type, length]
+ [SSSa,0,X ;sop1,0] ; live scalar in %0
+ [v ,v,Sg;mult,*] #
+ }
+ "reload_completed && gcn_vgpr_register_operand (operands[0], <MODE>mode)"
+ [(const_int 0)]
+ {
+ int partcount = GET_MODE_SIZE (<MODE>mode) / 4;
+ for (int p = 0; p < partcount; p++)
+ {
+ emit_move_insn (operands[2],
+ gcn_operand_part (<MODE>mode, operands[1], p));
+ emit_move_insn (gcn_operand_part (<MODE>mode, operands[0], p),
+ operands[2]);
+ }
+ DONE;
+ })
+
+;; SIMT EXEC handling instructions that won't get optimized away
+
+(define_insn "exec_barrier"
+ [(unspec_volatile [(const_int 0)] UNSPECV_SIMT_EXEC_BARRIER)]
+ "flag_simt_mode"
+ "; exec barrier"
+ [(set_attr "length" "0")])
+
+(define_expand "saveexec"
+ [(set (match_operand:DI 0 "register_operand")
+ (reg:DI EXEC_REG))]
+ "flag_simt_mode")
+
+(define_insn "setexec"
+ [(set (reg:DI EXEC_REG)
+ (match_operand:DI 0 "register_operand" "Sg"))
+ (set (reg:BI EXECZ_REG)
+ (eq:BI (match_dup 0) (const_int 0)))
+ (unspec_volatile [(const_int 0)] UNSPECV_SIMT_EXEC_BARRIER)] ; HACK HACK
HACK
+ "flag_simt_mode"
+ "s_mov_b64\texec, %0"
+ [(set_attr "type" "sop1")
+ (set_attr "length" "4")])
+
+(define_code_iterator execop [and ior xor])
+
+(define_insn "setexec_<expander>"
+ [(set (reg:DI EXEC_REG)
+ (execop:DI (match_operand:DI 0 "register_operand" "Sg")
+ (reg:DI EXEC_REG)))
+ (set (reg:BI EXECZ_REG)
+ (eq:BI (execop:DI (match_dup 0) (reg:DI EXEC_REG)) (const_int 0)))
+ (set (reg:BI SCC_REG)
+ (ne:BI (execop:DI (match_dup 0) (reg:DI EXEC_REG)) (const_int 0)))]
+ "flag_simt_mode"
+ "s_<bare_mnemonic>_b64\texec, exec, %0"
+ [(set_attr "type" "sop2")
+ (set_attr "length" "4")])
+
+(define_insn "setexec_<expander>_with_save"
+ [(set (match_operand:DI 0 "register_operand" "=Sg")
+ (reg:DI EXEC_REG))
+ (set (reg:DI EXEC_REG)
+ (execop:DI (match_operand:DI 1 "register_operand" "Sg")
+ (reg:DI EXEC_REG)))
+ (set (reg:BI EXECZ_REG)
+ (eq:BI (execop:DI (match_dup 1) (reg:DI EXEC_REG)) (const_int 0)))
+ (set (reg:BI SCC_REG)
+ (ne:BI (execop:DI (match_dup 1) (reg:DI EXEC_REG)) (const_int 0)))]
+ "flag_simt_mode"
+ "s_<bare_mnemonic>_saveexec_b64\t%0, %1"
+ [(set_attr "type" "sop1")
+ (set_attr "length" "4")])
+
;; }}}
(include "gcn-valu.md")
diff --git a/gcc/config/gcn/predicates.md b/gcc/config/gcn/predicates.md
index 721ded682af..7f365b7847f 100644
--- a/gcc/config/gcn/predicates.md
+++ b/gcc/config/gcn/predicates.md
@@ -24,14 +24,23 @@ (define_predicate "gcn_conditional_register_operand"
if (GET_CODE (op) == SUBREG)
op = SUBREG_REG (op);
- if (!REG_P (op) || GET_MODE (op) != BImode)
+ if (!REG_P (op))
return 0;
- return REGNO (op) == VCCZ_REG
- || REGNO (op) == VCC_REG /* Implied VCCZ. */
- || REGNO (op) == SCC_REG
- || REGNO (op) == EXECZ_REG
- || REGNO (op) >= FIRST_PSEUDO_REGISTER;
+ if (GET_MODE (op) == DImode
+ && (REGNO (op) == VCC_REG /* Implied VCCZ. */
+ || REGNO (op) >= FIRST_PSEUDO_REGISTER))
+ return 1;
+
+ if (GET_MODE (op) == BImode
+ && (REGNO (op) == VCCZ_REG
+ || REGNO (op) == VCC_REG /* Implied VCCZ. */
+ || REGNO (op) == SCC_REG
+ || REGNO (op) == EXECZ_REG
+ || REGNO (op) >= FIRST_PSEUDO_REGISTER))
+ return 1;
+
+ return 0;
})
(define_predicate "gcn_ssrc_register_operand"
diff --git a/gcc/testsuite/gcc.target/gcn/avgpr-spill-int.c
b/gcc/testsuite/gcc.target/gcn/avgpr-spill-int.c
index 462d67417c3..2d033231ccc 100644
--- a/gcc/testsuite/gcc.target/gcn/avgpr-spill-int.c
+++ b/gcc/testsuite/gcc.target/gcn/avgpr-spill-int.c
@@ -2,6 +2,11 @@
/* { dg-additional-options "-march=gfx908 -O1" } */
/* { dg-final { scan-assembler "accvgpr" } } */
+/* What spills here, in the vector variants, is the 64-bit per-lane
+ addresses: the array is larger than the immediate offset of a flat
+ access, so each element needs an address of its own. The scalar variants
+ (see sgpr-spill-int.c) spill the 50 values themselves. */
+
typedef int v32si __attribute__ ((__vector_size__ (sizeof(int)*32)));
#ifndef TYPE
#define TYPE v32si
diff --git a/gcc/testsuite/gcc.target/gcn/avgpr-spill-short.c
b/gcc/testsuite/gcc.target/gcn/avgpr-spill-short.c
index 71032e7674f..24e2c3751ca 100644
--- a/gcc/testsuite/gcc.target/gcn/avgpr-spill-short.c
+++ b/gcc/testsuite/gcc.target/gcn/avgpr-spill-short.c
@@ -2,7 +2,12 @@
/* { dg-additional-options "-march=gfx908 -O1" } */
/* { dg-final { scan-assembler "accvgpr" } } */
-typedef short v32hi __attribute__ ((__vector_size__ (sizeof(short)*32)));
-#define TYPE v32hi
+/* Use 64-lane vectors so that the array is larger than the 4096-byte
+ immediate offset of a flat access; with 32-lane shorts the array is only
+ 3200 bytes, every element is reachable from one base address, and there
+ is nothing to spill. */
+
+typedef short v64hi __attribute__ ((__vector_size__ (sizeof(short)*64)));
+#define TYPE v64hi
#include "avgpr-spill-int.c"
diff --git a/gcc/testsuite/gcc.target/gcn/simt-asm-g-1.c
b/gcc/testsuite/gcc.target/gcn/simt-asm-g-1.c
new file mode 100644
index 00000000000..825aea29166
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-asm-g-1.c
@@ -0,0 +1,38 @@
+/* Ensure that a "g" asm operand can hold a per-lane value in SIMT mode.
+
+ "g" is "r" plus memory plus immediates, and its register half means
+ GENERAL_REGS, which on this target is the scalar registers. A 64-lane
+ value cannot satisfy that, so unless the constraint is rewritten to
+ "vmi" (see gcn_md_asm_adjust) this fails to compile with "inconsistent
+ operand constraints in an 'asm'", which makes inline asm that uses "g"
+ -- in the libraries, or in user code -- unusable in SIMT mode.
+
+ See simt-asm-r-1.c for the same thing with "r". */
+
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-O1" } */
+
+#ifndef __SIMT__
+#error "This test requires -msimt"
+#endif
+
+#define TID __builtin_gcn_dim_pos (2)
+
+/* Global variables aren't thread-local. */
+int a[64];
+
+void
+f (void)
+{
+ int A, x;
+
+ A = __builtin_gcn_simt_enter (1, 2);
+ x = a[TID];
+ __asm__ volatile ("; use %0" :: "g"(x));
+ a[TID] = x + 1;
+ __builtin_gcn_simt_exit (A);
+}
+
+/* The value is per-lane, so it has to end up in a VGPR. */
+/* { dg-final { scan-assembler "; use v\[0-9\]+" } } */
diff --git a/gcc/testsuite/gcc.target/gcn/simt-asm-r-1.c
b/gcc/testsuite/gcc.target/gcn/simt-asm-r-1.c
new file mode 100644
index 00000000000..b77deb887cf
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-asm-r-1.c
@@ -0,0 +1,36 @@
+/* Ensure that an "r" asm operand can hold a per-lane value in SIMT mode.
+
+ The generic "r" constraint means GENERAL_REGS, which on this target is
+ the scalar registers, so a 64-lane value cannot satisfy it: unless the
+ constraint is rewritten to "v" (see gcn_md_asm_adjust) this fails to
+ compile with "inconsistent operand constraints in an 'asm'", which makes
+ inline asm that uses "r" -- in the libraries, or in user code --
+ unusable in SIMT mode. */
+
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-O1" } */
+
+#ifndef __SIMT__
+#error "This test requires -msimt"
+#endif
+
+#define TID __builtin_gcn_dim_pos (2)
+
+/* Global variables aren't thread-local. */
+int a[64];
+
+void
+f (void)
+{
+ int A, x;
+
+ A = __builtin_gcn_simt_enter (1, 2);
+ x = a[TID];
+ __asm__ volatile ("; use %0" :: "r"(x));
+ a[TID] = x + 1;
+ __builtin_gcn_simt_exit (A);
+}
+
+/* The value is per-lane, so it has to end up in a VGPR. */
+/* { dg-final { scan-assembler "; use v\[0-9\]+" } } */
diff --git a/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-double.c
b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-double.c
new file mode 100644
index 00000000000..ac6f7c1a621
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-double.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-march=gfx908 -O1" } */
+/* { dg-final { scan-assembler "accvgpr" } } */
+/* { dg-final { scan-assembler-not "v_readlane" } } */
+
+#define TYPE double
+
+#include "simt-avgpr-spill-int.c"
diff --git a/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-int.c
b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-int.c
new file mode 100644
index 00000000000..f10f6d5d82f
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-int.c
@@ -0,0 +1,154 @@
+/* Check that spills inside a SIMT region go to the AVGPRs.
+
+ In SIMT mode every value is a 64-lane vector, so the 50 values kept live
+ below need 50 VGPRs, and a non-kernel function may only use 24 VGPRs (see
+ MAX_NORMAL_VGPR_COUNT). The spills have to preserve all 64 lanes, so on
+ devices with AVGPRs they should go there; in particular they must not go
+ through v_readlane into an SGPR, which would keep lane 0 only.
+
+ Everything is indexed by the lane ID, so that the values really are
+ per-lane and the simt_vect pass converts them to 64-lane vectors: values
+ that are the same in every lane stay scalar, and 50 scalars fit in the
+ SGPRs without spilling anything. The loads are volatile so that the
+ values have to be spilled rather than simply reloaded.
+
+ This is the SIMT-mode counterpart of avgpr-spill-int.c, which uses vector
+ extensions to create the same pressure in SIMD mode. */
+
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-march=gfx908 -O1" } */
+/* { dg-final { scan-assembler "accvgpr" } } */
+/* { dg-final { scan-assembler-not "v_readlane" } } */
+
+#ifndef __SIMT__
+#error "This test requires -msimt"
+#endif
+
+#define TID __builtin_gcn_dim_pos (2)
+
+#ifndef TYPE
+#define TYPE int
+#endif
+#define N 50
+
+/* Global variables aren't thread-local. */
+volatile TYPE in[64 + N];
+TYPE res[N][64];
+
+void __attribute__((noinline,noclone))
+f (void)
+{
+ int A;
+ TYPE v0, v1, v2, v3, v4, v5, v6, v7, v8, v9;
+ TYPE v10, v11, v12, v13, v14, v15, v16, v17, v18, v19;
+ TYPE v20, v21, v22, v23, v24, v25, v26, v27, v28, v29;
+ TYPE v30, v31, v32, v33, v34, v35, v36, v37, v38, v39;
+ TYPE v40, v41, v42, v43, v44, v45, v46, v47, v48, v49;
+
+ A = __builtin_gcn_simt_enter (1, 2);
+
+ v0 = in[TID + 0];
+ v1 = in[TID + 1];
+ v2 = in[TID + 2];
+ v3 = in[TID + 3];
+ v4 = in[TID + 4];
+ v5 = in[TID + 5];
+ v6 = in[TID + 6];
+ v7 = in[TID + 7];
+ v8 = in[TID + 8];
+ v9 = in[TID + 9];
+ v10 = in[TID + 10];
+ v11 = in[TID + 11];
+ v12 = in[TID + 12];
+ v13 = in[TID + 13];
+ v14 = in[TID + 14];
+ v15 = in[TID + 15];
+ v16 = in[TID + 16];
+ v17 = in[TID + 17];
+ v18 = in[TID + 18];
+ v19 = in[TID + 19];
+ v20 = in[TID + 20];
+ v21 = in[TID + 21];
+ v22 = in[TID + 22];
+ v23 = in[TID + 23];
+ v24 = in[TID + 24];
+ v25 = in[TID + 25];
+ v26 = in[TID + 26];
+ v27 = in[TID + 27];
+ v28 = in[TID + 28];
+ v29 = in[TID + 29];
+ v30 = in[TID + 30];
+ v31 = in[TID + 31];
+ v32 = in[TID + 32];
+ v33 = in[TID + 33];
+ v34 = in[TID + 34];
+ v35 = in[TID + 35];
+ v36 = in[TID + 36];
+ v37 = in[TID + 37];
+ v38 = in[TID + 38];
+ v39 = in[TID + 39];
+ v40 = in[TID + 40];
+ v41 = in[TID + 41];
+ v42 = in[TID + 42];
+ v43 = in[TID + 43];
+ v44 = in[TID + 44];
+ v45 = in[TID + 45];
+ v46 = in[TID + 46];
+ v47 = in[TID + 47];
+ v48 = in[TID + 48];
+ v49 = in[TID + 49];
+
+ res[0][TID] = v0;
+ res[1][TID] = v1;
+ res[2][TID] = v2;
+ res[3][TID] = v3;
+ res[4][TID] = v4;
+ res[5][TID] = v5;
+ res[6][TID] = v6;
+ res[7][TID] = v7;
+ res[8][TID] = v8;
+ res[9][TID] = v9;
+ res[10][TID] = v10;
+ res[11][TID] = v11;
+ res[12][TID] = v12;
+ res[13][TID] = v13;
+ res[14][TID] = v14;
+ res[15][TID] = v15;
+ res[16][TID] = v16;
+ res[17][TID] = v17;
+ res[18][TID] = v18;
+ res[19][TID] = v19;
+ res[20][TID] = v20;
+ res[21][TID] = v21;
+ res[22][TID] = v22;
+ res[23][TID] = v23;
+ res[24][TID] = v24;
+ res[25][TID] = v25;
+ res[26][TID] = v26;
+ res[27][TID] = v27;
+ res[28][TID] = v28;
+ res[29][TID] = v29;
+ res[30][TID] = v30;
+ res[31][TID] = v31;
+ res[32][TID] = v32;
+ res[33][TID] = v33;
+ res[34][TID] = v34;
+ res[35][TID] = v35;
+ res[36][TID] = v36;
+ res[37][TID] = v37;
+ res[38][TID] = v38;
+ res[39][TID] = v39;
+ res[40][TID] = v40;
+ res[41][TID] = v41;
+ res[42][TID] = v42;
+ res[43][TID] = v43;
+ res[44][TID] = v44;
+ res[45][TID] = v45;
+ res[46][TID] = v46;
+ res[47][TID] = v47;
+ res[48][TID] = v48;
+ res[49][TID] = v49;
+
+ __builtin_gcn_simt_exit (A);
+}
diff --git a/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-long.c
b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-long.c
new file mode 100644
index 00000000000..e94ba21dc0b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-long.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-march=gfx908 -O1" } */
+/* { dg-final { scan-assembler "accvgpr" } } */
+/* { dg-final { scan-assembler-not "v_readlane" } } */
+
+#define TYPE long
+
+#include "simt-avgpr-spill-int.c"
diff --git a/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-run.c
b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-run.c
new file mode 100644
index 00000000000..5b86ed9a001
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-run.c
@@ -0,0 +1,34 @@
+/* Ensure that the values spilled by simt-avgpr-spill-int.c keep their
+ per-lane values. A spill that kept lane 0 only would return the wrong
+ value in every lane but the first.
+
+ No -march is given here, so that the test runs on the GPU that is
+ present; where that has no AVGPRs the spills go to memory instead, which
+ has to preserve all 64 lanes just the same. */
+
+/* { dg-do run } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-O1" } */
+
+#include "simt-avgpr-spill-int.c"
+
+/* Note: main enters in single-threaded mode. */
+int
+main (void)
+{
+ int i, j;
+
+ for (i = 0; i < 64 + N; i++)
+ in[i] = i * 3 + 1;
+
+ f ();
+
+ /* Use single-threading to check the results because broken multithread
+ might also have broken checking. */
+ for (j = 0; j < N; j++)
+ for (i = 0; i < 64; i++)
+ if (res[j][i] != (TYPE) ((i + j) * 3 + 1))
+ __builtin_exit (1);
+
+ return 0;
+}
diff --git a/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-short.c
b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-short.c
new file mode 100644
index 00000000000..a129323afc1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/gcn/simt-avgpr-spill-short.c
@@ -0,0 +1,9 @@
+/* { dg-do compile } */
+/* { dg-require-effective-target amdgcn_simt } */
+/* { dg-additional-options "-march=gfx908 -O1" } */
+/* { dg-final { scan-assembler "accvgpr" } } */
+/* { dg-final { scan-assembler-not "v_readlane" } } */
+
+#define TYPE short
+
+#include "simt-avgpr-spill-int.c"
--
2.54.0
