https://gcc.gnu.org/g:36ac20bbf9aa7b8dc6c8d9f357dc4d3f7ff63550
commit 36ac20bbf9aa7b8dc6c8d9f357dc4d3f7ff63550 Author: Michael Meissner <[email protected]> Date: Wed Jul 1 10:47:34 2026 -0400 Add dense math register support. This patch is a modification of the V6 patches that I sent out on April 21st, 2026. In particular, I made the changes in relation to the comments posted in February that I didn't fully address previously. Here is comment from February: * https://gcc.gnu.org/pipermail/gcc-patches/2026-February/708071.html Here is my reply: * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/715248.html Here are the V6 patches posted on April 21st, 2026: * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713352.html * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713353.html * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713354.html * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713356.html * https://gcc.gnu.org/pipermail/gcc-patches/2026-April/713357.html There are 7 patches in this patch set: Patch #1 adds the wD constraint and the accumulator_operand predicate. Patch #2 switches mma.md to use the wD constraint and accumulator_operand predicate. Patch #3 adds the -mdense-math option, but in this patch, -mdense-math is not implemented. Patch #4 adds support for 512-bit dense math registers. Patch #5 adds support for 1,024-bit dense math registers. Patch #6 is an optional patch that changes the name of the MMA instructions from the original name used in the power10/power11 time line to a new alternate name that has 'dm' (for dense math) in the instruction name. Note, this is a new patch for the V7 patch set. Patch #7 clones the mma builtin tests to test the code generation of MMA instructions if -mcpu=future is used. Note, this is a new patch for the V7 patch set. If patch #6 is not applied, this patch will need to be modified. The following is the description of dense math registes from previous versions of the patches. The Dense Math Facility (dmf) is designed to be an extension to the ISA 3.1 (i.e. power10/power11) MMA facility. Now, since these are future patches, the Dense Math Facility might appear in future PowerPC machines or maybe it won't be used in real hardware. One of the concepts of the DMF system is the accumulators used in the MMA and the DMF extensions will become separate registers, rather than being overlaid over the traditional floating point registers (i.e. VSX registers 0..31). In addition to being separate registers, the dense math accumulators are now logically 1,024 biits instead of 512. The way the Dense Math registers and instructions are designed, existing power10/power11 MMA instructions that operate on 512 bits will work with Dense Math. In ISA 3.1, each of the 8 accumulators are overlaid over 4 adjacent FPR registers, and the compiler must not touch the 4 adjacent FPRs while the MMA accumulator is used. In the Dense Math system, the accumulator is a separate register. When -mcpu=power11 or -mcpu=power10 is used, the GCC compiler will not allocate the appropriate FPR (VSX) reigsters when generating MMA instructions. If a function compiled for Power10/Power11 is run on a system with Dense Math support enabled, the effect is a bunch of the FPR registers will not be allocated because the compiler assumes the accumulaters are there. After these patches are applied, if the user compiles the code with -mcpu=future, the compiler can allocate up to 32 more vector registers, because the Dense Math accumulators are separate registers. In fact two of the MMA tests (mma-double-test.c and mma-single-test.c) do about 20 less spills of floating point values to the stack, since the compiler can allocate those FPR vector registers for other purposes. These 5 patches will allow GCC to allocate these registers if the -mcpu=future option is used. 1: The first patch adds a new constraint (%wD) that can be used by code generating MMA instructions. If the user used -mcpu=power10 or -mcpu=power11, %wD will act like %d and insist the register be VSX registers 0..31. If the user used -mcpu=future, the new separate dense math accumulators will be used. 2: This patch just adds the -mdense-math option, but it does not add support for dense math registers until patch #3. 3: This patch adds the support for the current MMA 512-bit instructions to use separate accumulators instead of overlaid VSX registers. 4: This patch adds support for an extension to MMA where the accumulators grow to 1,024 bits instead of 512 bits. 5: This patch is an optional patch that adds comments to the various MMA insn that explain what MMA instructions are generated by the particular insn. This patch is the foundation for the Dense Math support. It is expected other patches may be added to this to support potential new features added to the Dense Math Facility. I have built bootstrap little endian compilers on power10 systems, and big endian compiler on power9 systems. There were no regression in the tests. Can I add the patches to the GCC trunk after the -mcpu=future patch is applied and GCC 17 has opened up? 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/constraints.md (wD): New constraint. * config/rs6000/predicates.md (accumulator_operand): New predicate. * config/rs6000/rs6000.cc (rs6000_debug_reg_global): Print the register class for the 'wD' constraint. (rs6000_init_hard_regno_mode_ok): Set up the 'wD' register constraint class. * config/rs6000/rs6000.h (enum r6000_reg_class_enum): Add element for the 'wD' constraint. * doc/md.texi (PowerPC constraints): Document the 'wD' constraint. 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/mma.md (mma_<vv>): Use the wD constraint and accumulator_operand predicate for all MMA instructions taking accumulator operands. (mma_<avv>): Likewise. (mma_<pv>"): Likewise. (mma_<apv>): Likewise. (mma_<vvi4i4i8>): Likewise. (mma_<avvi4i4i8>): Likewise. (mma_<vvi4i4i2>"): Likewise. (mma_<avvi4i4i2>): Likewise. (mma_<vvi4i4>): Likewise. (mma_<avvi4i4>): Likewise. (mma_<pvi4i2>): Likewise. (mma_<apvi4i2>): Likewise. (mma_<vvi4i4i4>): Likewise. (mma_<avvi4i4i4>): Likewise. 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/rs6000-c.cc (rs6000_define_or_undefine_macro): Define __MMA_DENSE_MATH__ if we have MMA that uses dense math register accumulators. Define __MMA_NO_DENSE_MATH__ if we have MMA but we are using ISA 3.1 where the accumulators are overlaid over VSX registers 0..32. Define __DENSE_MATH__ if we have dense math registers. * config/rs6000/rs6000.cc (rs6000_option_override_internal): Do not allow -mdense-math unless -mcpu=future is used. (rs6000_opt_masks): Add -mdense-math support. * config/rs6000/rs6000.h (TARGET_MMA_DENSE_MATH): New macro. (TARGET_MMA_NO_DENSE_MATH): Likewise. * config/rs6000/rs6000.opt (-mdense-math): New option. * doc/invoke.texi (RS/6000 and PowerPC Options): Add -mdense-math. 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/mma.md (movoo): Allow -mdense-math -mno-mma. (movxo): Convert to being a define_expand that can handle both the original MMA support without dense math registers, and adding dense math support. Allow -mdense-math -mno-mma. (movxo_nodm): Rename original movxo insn, and restrict this insn to when we do not have dense math registers. (movxo_dm): New define_insn_and_split for dense math registers. (vsx_assemble_pair): Allow -mdense-math -mno-mma. (vsx_disassemble_pair): Likewise. (mma_assemble_acc): Likewise. (mma_disassemble_acc): Likewise. (mma_<acc>): Allow built-ins to be used if -mdense-math. (mma_xxsetaccz): Convert into a define_expand to handle both non-dense math and dense math registers. (mma_xxsetaccz_nodm): Rename from mma_xxsetaccz and limit code to non dense math systems. (mma_xxsetaccz_dm): New insn for direct math register support. * config/rs6000/predicates.md (dmf_register_operand): New predicate. (accumulator_operand): Add support for dense math registers. * config/rs6000/rs6000-builtin.cc (rs6000_gimple_fold_mma_builtin): Do not issue xxmfacc (deprime) instruction if we have dense math registers. * config/rs6000/rs6000-cpus.def (FUTURE_MASKS_SERVER): Add -mdense-math. (POWERPC_MASKS): Likewise. * config/rs6000/rs6000.cc (enum rs6000_reg_type): Add dense math register support. (enum rs6000_reload_reg_typ): Likewise. (LAST_RELOAD_REG_CLASS): Likewise. (reload_reg_map): Likewise. (rs6000_reg_names): Likewise. (alt_reg_names): Likewise. (rs6000_hard_regno_nregs_internal): Likewise. (rs6000_hard_regno_mode_ok_uncached): Likewise. (rs6000_debug_reg_global): Likewise. (rs6000_setup_reg_addr_masks): Likewise. (rs6000_init_hard_regno_mode_ok): Likewise. (rs6000_secondary_reload_memory): Likewise. (rs6000_secondary_reload_simple_move): Likewise. (rs6000_preferred_reload_class): Likewise. (rs6000_secondary_reload_class): Likewise. (print_operand): Likewise. (rs6000_dmf_register_move_cost): New helper function. (rs6000_register_move_cost): Add dense math register support. (rs6000_memory_move_cost): Likewise. (rs6000_compute_pressure_classes): Likewise. (rs6000_debugger_regno): Likewise. (rs6000_opt_masks): Likewise. (rs6000_split_multireg_move): Likewise. * config/rs6000/rs6000.h (UNITS_PER_DMF_WORD): New macro. (FIRST_PSEUDO_REGISTER): Add dense math register support. (FIXED_REGISTERS): Likewise. (CALL_REALLY_USED_REGISTERS): Likewise. (REG_ALLOC_ORDER): Likewise. (DMF_REGNO_P): New macro. (enum reg_class): Add dense math register support. (REG_CLASS_NAMES): Likewise. (REGISTER_NAMES): Likewise. (ADDITIONAL_REGISTER_NAMES): Likewise. * config/rs6000/rs6000.md (FIRST_DMF_REGNO): New constant. (LAST_DMF_REGNO): Likewise. 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/mma.md (UNSPEC_DMF_INSERT512_UPPER): New unspec. (UNSPEC_DMF_INSERT512_LOWER): Likewise. (UNSPEC_DMF_EXTRACT512): Likewise. (UNSPEC_DMF_RELOAD_FROM_MEMORY): Likewise. (UNSPEC_DMF_RELOAD_TO_MEMORY): Likewise. (movtdo): New define_expand and define_insn_and_split to implement 1,024 bit dense math registers. (movtdo_insert512_upper): New insn. (movtdo_insert512_lower): Likewise. (movtdo_extract512): Likewise. (reload_tdo_from_memory): Likewise. (reload_tdo_to_memory): Likewise. * config/rs6000/rs6000-builtin.cc (rs6000_type_string): Add dense math register support. (rs6000_init_builtins): Add support for __dm1024 keyword. * config/rs6000/rs6000-call.cc (rs6000_return_in_memory): Add support for TDOmode. (rs6000_function_arg): Likewise. * config/rs6000/rs6000-modes.def (TDOmode): New mode. * config/rs6000/rs6000.cc (rs6000_hard_regno_mode_ok_uncached): Add support for TDOmode. (rs6000_hard_regno_mode_ok): Likewise. (rs6000_modes_tieable_p): Likewise. (rs6000_debug_reg_global): Likewise. (rs6000_setup_reg_addr_masks): Likewise. (rs6000_init_hard_regno_mode_ok): Add support for TDOmode. Setup reload hooks for dense math TDO reload mode. (reg_offset_addressing_ok_p): Add support for TDOmode. (rs6000_emit_move): Likewise. (rs6000_secondary_reload_simple_move): Likewise. (rs6000_preferred_reload_class): Likewise. (rs6000_mangle_type): Add mangling for __dm1024 type. (rs6000_dmf_register_move_cost): Add support for TDOmode. (rs6000_split_multireg_move): Likewise. (rs6000_invalid_conversion): Likewise. * config/rs6000/rs6000.h (VECTOR_ALIGNMENT_P): Add TDOmode. (enum rs6000_builtin_type_index): Add dense math register type nodes. (dm1024_type_node): Likewise. (ptr_dm1024_type_node): Likewise. gcc/testsuite/ * gcc.target/powerpc/dm-1024bit.c: New test. 2026-07-01 Michael Meissner <[email protected]> gcc/ * config/rs6000/mma.md (vvi4i4i8): Eliminate using the 'pm' prefix here, so we can emit pmdm* on dense math systems. (avvi4i4i8): Likewise. (vvi4i4i2): Likewise. (avvi4i4i2): Likewise. (vvi4i4): Likewise. (avvi4i4): Likewise. (pvi4i2): Likewise. (apvi4i2): Likewise. (vvi4i4i4): Likewise. (mma_<vv>): If -mdesne-math, emit 'dmxv*' form of the instruction instead of 'xv*'. (mma_<avv>): Likewise. (mma_<pv>): Likewise. (mma_<apv>): Likewise. (mma_pm<vvi4i4i8>): If -mdense-math, emit 'pmdm*' instead of 'pm*'. (mma_pm<avvi4i4i8>): Likewise. (mma_pm<vvi4i4i2>): Likewise. (mma_pm<avvi4i4i2>): Likewise. (mma_pm<vvi4i4>): Likewise. (mma_pm<avvi4i4>): Likewise. (mma_pm<pvi4i2>): Likewise. (mma_pm<apvi4i2>): Likewise. (mma_pm<vvi4i4i4>): Likewise. (mma_pm<avvi4i4i4>): Likewise. * config/rs6000/rs6000.cc (print_operand): For %!, print 'dm' if -mdense-math. * config/rs6000/rs6000.h (PRINT_OPERAND_PUNCT_VALID_P): Allow %!. 2026-07-01 Michael Meissner <[email protected]> gcc/testsuite/ * gcc.target/powerpc/dm-builtin-1.c: New test. * gcc.target/powerpc/dm-builtin-10-pair.c: Likewise. * gcc.target/powerpc/dm-builtin-10-quad.c: Likewise. * gcc.target/powerpc/dm-builtin-2.c: Likewise. * gcc.target/powerpc/dm-builtin-3.c: Likewise. * gcc.target/powerpc/dm-builtin-4.c: Likewise. * gcc.target/powerpc/dm-builtin-5.c: Likewise. * gcc.target/powerpc/dm-builtin-6.c: Likewise. * gcc.target/powerpc/dm-builtin-7.c: Likewise. * gcc.target/powerpc/dm-builtin-8.c: Likewise. * gcc.target/powerpc/dm-builtin-9.c: Likewise. Diff: --- gcc/config/rs6000/constraints.md | 3 + gcc/config/rs6000/mma.md | 468 ++++++++++++++++----- gcc/config/rs6000/predicates.md | 41 ++ gcc/config/rs6000/rs6000-builtin.cc | 30 +- gcc/config/rs6000/rs6000-c.cc | 22 +- gcc/config/rs6000/rs6000-call.cc | 10 +- gcc/config/rs6000/rs6000-cpus.def | 2 + gcc/config/rs6000/rs6000-modes.def | 4 + gcc/config/rs6000/rs6000.cc | 338 ++++++++++++--- gcc/config/rs6000/rs6000.h | 52 ++- gcc/config/rs6000/rs6000.md | 2 + gcc/config/rs6000/rs6000.opt | 4 + gcc/doc/invoke.texi | 7 + gcc/doc/md.texi | 5 + gcc/testsuite/gcc.target/powerpc/dm-builtin-1.c | 313 ++++++++++++++ .../gcc.target/powerpc/dm-builtin-10-pair.c | 21 + .../gcc.target/powerpc/dm-builtin-10-quad.c | 23 + gcc/testsuite/gcc.target/powerpc/dm-builtin-2.c | 72 ++++ gcc/testsuite/gcc.target/powerpc/dm-builtin-3.c | 31 ++ gcc/testsuite/gcc.target/powerpc/dm-builtin-4.c | 73 ++++ gcc/testsuite/gcc.target/powerpc/dm-builtin-5.c | 47 +++ gcc/testsuite/gcc.target/powerpc/dm-builtin-6.c | 21 + gcc/testsuite/gcc.target/powerpc/dm-builtin-7.c | 26 ++ gcc/testsuite/gcc.target/powerpc/dm-builtin-8.c | 27 ++ gcc/testsuite/gcc.target/powerpc/dm-builtin-9.c | 28 ++ 25 files changed, 1495 insertions(+), 175 deletions(-) diff --git a/gcc/config/rs6000/constraints.md b/gcc/config/rs6000/constraints.md index d0ed47faab84..0d1cde5bd4de 100644 --- a/gcc/config/rs6000/constraints.md +++ b/gcc/config/rs6000/constraints.md @@ -107,6 +107,9 @@ (match_test "TARGET_P8_VECTOR") (match_operand 0 "s5bit_cint_operand"))) +(define_register_constraint "wD" "rs6000_constraints[RS6000_CONSTRAINT_wD]" + "Accumulator register.") + (define_constraint "wE" "@internal Vector constant that can be loaded with the XXSPLTIB instruction." (match_test "xxspltib_constant_nosplit (op, mode)")) diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md index 1103f1fc0375..16a7bb26e0e9 100644 --- a/gcc/config/rs6000/mma.md +++ b/gcc/config/rs6000/mma.md @@ -91,6 +91,12 @@ UNSPEC_MMA_XVI8GER4SPP UNSPEC_MMA_XXMFACC UNSPEC_MMA_XXMTACC + UNSPEC_MMA_DMSETDMRZ + UNSPEC_DMF_INSERT512_UPPER + UNSPEC_DMF_INSERT512_LOWER + UNSPEC_DMF_EXTRACT512 + UNSPEC_DMF_RELOAD_FROM_MEMORY + UNSPEC_DMF_RELOAD_TO_MEMORY ]) (define_c_enum "unspecv" @@ -224,44 +230,46 @@ (UNSPEC_MMA_XVF64GERNP "xvf64gernp") (UNSPEC_MMA_XVF64GERNN "xvf64gernn")]) -(define_int_attr vvi4i4i8 [(UNSPEC_MMA_PMXVI4GER8 "pmxvi4ger8")]) +;; The follwoing instructions do not have the 'pm' prefix, so that on dense +;; math systems, we can change the pm to pmdm. +(define_int_attr vvi4i4i8 [(UNSPEC_MMA_PMXVI4GER8 "xvi4ger8")]) -(define_int_attr avvi4i4i8 [(UNSPEC_MMA_PMXVI4GER8PP "pmxvi4ger8pp")]) +(define_int_attr avvi4i4i8 [(UNSPEC_MMA_PMXVI4GER8PP "xvi4ger8pp")]) -(define_int_attr vvi4i4i2 [(UNSPEC_MMA_PMXVI16GER2 "pmxvi16ger2") - (UNSPEC_MMA_PMXVI16GER2S "pmxvi16ger2s") - (UNSPEC_MMA_PMXVF16GER2 "pmxvf16ger2") - (UNSPEC_MMA_PMXVBF16GER2 "pmxvbf16ger2")]) +(define_int_attr vvi4i4i2 [(UNSPEC_MMA_PMXVI16GER2 "xvi16ger2") + (UNSPEC_MMA_PMXVI16GER2S "xvi16ger2s") + (UNSPEC_MMA_PMXVF16GER2 "xvf16ger2") + (UNSPEC_MMA_PMXVBF16GER2 "xvbf16ger2")]) -(define_int_attr avvi4i4i2 [(UNSPEC_MMA_PMXVI16GER2PP "pmxvi16ger2pp") - (UNSPEC_MMA_PMXVI16GER2SPP "pmxvi16ger2spp") - (UNSPEC_MMA_PMXVF16GER2PP "pmxvf16ger2pp") - (UNSPEC_MMA_PMXVF16GER2PN "pmxvf16ger2pn") - (UNSPEC_MMA_PMXVF16GER2NP "pmxvf16ger2np") - (UNSPEC_MMA_PMXVF16GER2NN "pmxvf16ger2nn") - (UNSPEC_MMA_PMXVBF16GER2PP "pmxvbf16ger2pp") - (UNSPEC_MMA_PMXVBF16GER2PN "pmxvbf16ger2pn") - (UNSPEC_MMA_PMXVBF16GER2NP "pmxvbf16ger2np") - (UNSPEC_MMA_PMXVBF16GER2NN "pmxvbf16ger2nn")]) +(define_int_attr avvi4i4i2 [(UNSPEC_MMA_PMXVI16GER2PP "xvi16ger2pp") + (UNSPEC_MMA_PMXVI16GER2SPP "xvi16ger2spp") + (UNSPEC_MMA_PMXVF16GER2PP "xvf16ger2pp") + (UNSPEC_MMA_PMXVF16GER2PN "xvf16ger2pn") + (UNSPEC_MMA_PMXVF16GER2NP "xvf16ger2np") + (UNSPEC_MMA_PMXVF16GER2NN "xvf16ger2nn") + (UNSPEC_MMA_PMXVBF16GER2PP "xvbf16ger2pp") + (UNSPEC_MMA_PMXVBF16GER2PN "xvbf16ger2pn") + (UNSPEC_MMA_PMXVBF16GER2NP "xvbf16ger2np") + (UNSPEC_MMA_PMXVBF16GER2NN "xvbf16ger2nn")]) -(define_int_attr vvi4i4 [(UNSPEC_MMA_PMXVF32GER "pmxvf32ger")]) +(define_int_attr vvi4i4 [(UNSPEC_MMA_PMXVF32GER "xvf32ger")]) -(define_int_attr avvi4i4 [(UNSPEC_MMA_PMXVF32GERPP "pmxvf32gerpp") - (UNSPEC_MMA_PMXVF32GERPN "pmxvf32gerpn") - (UNSPEC_MMA_PMXVF32GERNP "pmxvf32gernp") - (UNSPEC_MMA_PMXVF32GERNN "pmxvf32gernn")]) +(define_int_attr avvi4i4 [(UNSPEC_MMA_PMXVF32GERPP "xvf32gerpp") + (UNSPEC_MMA_PMXVF32GERPN "xvf32gerpn") + (UNSPEC_MMA_PMXVF32GERNP "xvf32gernp") + (UNSPEC_MMA_PMXVF32GERNN "xvf32gernn")]) -(define_int_attr pvi4i2 [(UNSPEC_MMA_PMXVF64GER "pmxvf64ger")]) +(define_int_attr pvi4i2 [(UNSPEC_MMA_PMXVF64GER "xvf64ger")]) -(define_int_attr apvi4i2 [(UNSPEC_MMA_PMXVF64GERPP "pmxvf64gerpp") - (UNSPEC_MMA_PMXVF64GERPN "pmxvf64gerpn") - (UNSPEC_MMA_PMXVF64GERNP "pmxvf64gernp") - (UNSPEC_MMA_PMXVF64GERNN "pmxvf64gernn")]) +(define_int_attr apvi4i2 [(UNSPEC_MMA_PMXVF64GERPP "xvf64gerpp") + (UNSPEC_MMA_PMXVF64GERPN "xvf64gerpn") + (UNSPEC_MMA_PMXVF64GERNP "xvf64gernp") + (UNSPEC_MMA_PMXVF64GERNN "xvf64gernn")]) -(define_int_attr vvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4 "pmxvi8ger4")]) +(define_int_attr vvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4 "xvi8ger4")]) -(define_int_attr avvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4PP "pmxvi8ger4pp") - (UNSPEC_MMA_PMXVI8GER4SPP "pmxvi8ger4spp")]) +(define_int_attr avvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4PP "xvi8ger4pp") + (UNSPEC_MMA_PMXVI8GER4SPP "xvi8ger4spp")]) ;; Vector pair support. OOmode can only live in VSRs. @@ -270,7 +278,7 @@ (match_operand:OO 1 "input_operand"))] "" { - if (TARGET_MMA) + if (TARGET_MMA || TARGET_DENSE_MATH) { rs6000_emit_move (operands[0], operands[1], OOmode); DONE; @@ -295,7 +303,7 @@ (define_insn_and_split "*movoo" [(set (match_operand:OO 0 "nonimmediate_operand" "=wa,ZwO,wa") (match_operand:OO 1 "input_operand" "ZwO,wa,wa"))] - "TARGET_MMA + "(TARGET_MMA || TARGET_DENSE_MATH) && (gpc_reg_operand (operands[0], OOmode) || gpc_reg_operand (operands[1], OOmode))" "@ @@ -314,13 +322,13 @@ (set_attr "length" "*,*,8")]) -;; Vector quad support. XOmode can only live in FPRs. +;; Vector quad support. (define_expand "movxo" [(set (match_operand:XO 0 "nonimmediate_operand") (match_operand:XO 1 "input_operand"))] "" { - if (TARGET_MMA) + if (TARGET_MMA || TARGET_DENSE_MATH) { rs6000_emit_move (operands[0], operands[1], XOmode); DONE; @@ -339,10 +347,13 @@ gcc_assert (false); }) -(define_insn_and_split "*movxo" +;; If we do not have dense math registers, XOmode can only live in FPR +;; registers (0..31). + +(define_insn_and_split "*movxo_nodm" [(set (match_operand:XO 0 "nonimmediate_operand" "=d,ZwO,d") (match_operand:XO 1 "input_operand" "ZwO,d,d"))] - "TARGET_MMA + "TARGET_MMA_NO_DENSE_MATH && (gpc_reg_operand (operands[0], XOmode) || gpc_reg_operand (operands[1], XOmode))" "@ @@ -359,11 +370,39 @@ (set_attr "length" "*,*,16") (set_attr "max_prefixed_insns" "2,2,*")]) +;; If dense math registers are available, XOmode can live in either VSX +;; registers (0..63) or dense math registers. + +(define_insn_and_split "*movxo_dm" + [(set (match_operand:XO 0 "nonimmediate_operand" "=wa,ZwO,wa,wD,wD,wa") + (match_operand:XO 1 "input_operand" "ZwO,wa, wa,wa,wD,wD"))] + "TARGET_DENSE_MATH + && (gpc_reg_operand (operands[0], XOmode) + || gpc_reg_operand (operands[1], XOmode))" + "@ + # + # + # + dmxxinstdmr512 %0,%1,%Y1,0 + dmmr %0,%1 + dmxxextfdmr512 %0,%Y0,%1,0" + "&& reload_completed + && !dmf_register_operand (operands[0], XOmode) + && !dmf_register_operand (operands[1], XOmode)" + [(const_int 0)] +{ + rs6000_split_multireg_move (operands[0], operands[1]); + DONE; +} + [(set_attr "type" "vecload,vecstore,veclogical,mma,mma,mma") + (set_attr "length" "*,*,16,*,*,*") + (set_attr "max_prefixed_insns" "2,2,*,*,*,*")]) + (define_expand "vsx_assemble_pair" [(match_operand:OO 0 "vsx_register_operand") (match_operand:V16QI 1 "mma_assemble_input_operand") (match_operand:V16QI 2 "mma_assemble_input_operand")] - "TARGET_MMA" + "TARGET_MMA || TARGET_DENSE_MATH" { rtx src = gen_rtx_UNSPEC (OOmode, gen_rtvec (2, operands[1], operands[2]), @@ -380,7 +419,7 @@ (unspec:OO [(match_operand:V16QI 1 "mma_assemble_input_operand" "mwa") (match_operand:V16QI 2 "mma_assemble_input_operand" "mwa")] UNSPEC_VSX_ASSEMBLE))] - "TARGET_MMA" + "TARGET_MMA || TARGET_DENSE_MATH" "#" "&& reload_completed" [(const_int 0)] @@ -396,7 +435,7 @@ [(match_operand:V16QI 0 "mma_disassemble_output_operand") (match_operand:OO 1 "vsx_register_operand") (match_operand 2 "const_0_to_1_operand")] - "TARGET_MMA" + "TARGET_MMA || TARGET_DENSE_MATH" { rtx src; int regoff = INTVAL (operands[2]); @@ -412,7 +451,7 @@ (unspec:V16QI [(match_operand:OO 1 "vsx_register_operand" "wa") (match_operand 2 "const_0_to_1_operand")] UNSPEC_MMA_EXTRACT))] - "TARGET_MMA + "(TARGET_MMA || TARGET_DENSE_MATH) && vsx_register_operand (operands[1], OOmode)" "#" "&& reload_completed" @@ -431,7 +470,7 @@ (match_operand:V16QI 2 "mma_assemble_input_operand") (match_operand:V16QI 3 "mma_assemble_input_operand") (match_operand:V16QI 4 "mma_assemble_input_operand")] - "TARGET_MMA" + "TARGET_MMA || TARGET_DENSE_MATH" { rtx src = gen_rtx_UNSPEC_VOLATILE (XOmode, gen_rtvec (4, operands[1], operands[2], @@ -452,7 +491,7 @@ (match_operand:V16QI 3 "mma_assemble_input_operand" "mwa") (match_operand:V16QI 4 "mma_assemble_input_operand" "mwa")] UNSPECV_MMA_ASSEMBLE))] - "TARGET_MMA + "(TARGET_MMA || TARGET_DENSE_MATH) && fpr_reg_operand (operands[0], XOmode)" "#" "&& reload_completed" @@ -470,7 +509,7 @@ [(match_operand:V16QI 0 "mma_disassemble_output_operand") (match_operand:XO 1 "fpr_reg_operand") (match_operand 2 "const_0_to_3_operand")] - "TARGET_MMA" + "TARGET_MMA || TARGET_DENSE_MATH" { rtx src; int regoff = INTVAL (operands[2]); @@ -486,7 +525,7 @@ (unspec:V16QI [(match_operand:XO 1 "fpr_reg_operand" "d") (match_operand 2 "const_0_to_3_operand")] UNSPEC_MMA_EXTRACT))] - "TARGET_MMA + "(TARGET_MMA || TARGET_DENSE_MATH) && fpr_reg_operand (operands[1], XOmode)" "#" "&& reload_completed" @@ -499,69 +538,136 @@ DONE; }) -;; MMA instructions that do not use their accumulators as an input, still -;; must not allow their vector operands to overlap the registers used by -;; the accumulator. We enforce this by marking the output as early clobber. +;; If dense math registers are not available, MMA instructions that do +;; not use their accumulators that overlap with FPR registers as an +;; input, still must not allow their vector operands to overlap the +;; registers used by the accumulator. We enforce this by marking the +;; output as early clobber. The prime and de-prime instructions are +;; not needed on systems with dense math registers. -(define_insn "mma_<acc>" +;; If we have dense math registers, xxmfacc/xxmtacc become normal moves. + +(define_expand "mma_<acc>" [(set (match_operand:XO 0 "fpr_reg_operand" "=&d") (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0")] MMA_ACC))] "TARGET_MMA" +{ + if (TARGET_DENSE_MATH) + { + emit_move_insn (operands[0], operands[1]); + DONE; + } +}) + +(define_insn "*mma_<acc>" + [(set (match_operand:XO 0 "fpr_reg_operand" "=&d") + (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0")] + MMA_ACC))] + "TARGET_MMA_NO_DENSE_MATH" "<acc> %A0" [(set_attr "type" "mma")]) ;; We can't have integer constants in XOmode so we wrap this in an -;; UNSPEC_VOLATILE. +;; UNSPEC_VOLATILE. If we have dense math registers, we can just use a +;; normal UNSPEC instead of UNSPEC_VOLATILE. An UNSPEC_VOLATILE limits +;; optimization since insns can't be moved before the UNSPEC_VOLATILE. + +;; Instructions: xxmtacc and xxmfacc + +(define_expand "mma_xxsetaccz" + [(set (match_operand:XO 0 "accumulator_operand") + (unspec_volatile:XO [(const_int 0)] + UNSPECV_MMA_XXSETACCZ))] + "TARGET_MMA || TARGET_DENSE_MATH" +{ + if (TARGET_DENSE_MATH) + { + emit_insn (gen_mma_xxsetaccz_dm (operands[0])); + DONE; + } +}) -(define_insn "mma_xxsetaccz" +;; Clear accumulator without dense math registers +(define_insn "*mma_xxsetaccz_nodm" [(set (match_operand:XO 0 "fpr_reg_operand" "=d") (unspec_volatile:XO [(const_int 0)] UNSPECV_MMA_XXSETACCZ))] - "TARGET_MMA" + "TARGET_MMA_NO_DENSE_MATH" "xxsetaccz %A0" [(set_attr "type" "mma")]) +;; Clear accumulator when dense math registers are available. +(define_insn "mma_xxsetaccz_dm" + [(set (match_operand:XO 0 "accumulator_operand" "=wD") + (unspec [(const_int 0)] + UNSPEC_MMA_DMSETDMRZ))] + "TARGET_DENSE_MATH" + "dmsetdmrz %A0" + [(set_attr "type" "mma")]) + + +;; MMA operations below. If dense math registers are available, these +;; operations will use the 8 accumultors which are separate registers. +;; If dense math registers are not available, these operations will use +;; accumulators that are overlaid on top of the FPR registers. + +;; Instructions: +;; xvi4ger8 xvi8ger4 xvi16ger2 xvi16ger2s xvf16ger2 +;; xvbf16ger2 xvf32ger + (define_insn "mma_<vv>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")] MMA_VV))] "TARGET_MMA" - "<vv> %A0,%x1,%x2" + "%!<vv> %A0,%x1,%x2" [(set_attr "type" "mma")]) +;; Instructions: +;; xvi4ger8pp xvi8ger4pp xvi8ger4spp xvi16ger2pp xvi16ger2spp +;; xvf16ger2pp xvf16ger2pn xvf16ger2np xvf16ger2nn xvbf16ger2pp +;; xvbf16ger2pn xvbf16ger2np xvbf16ger2nn xvf32gerpp xvf32gerpn +;; xvf32gernp xvf32gernn + (define_insn "mma_<avv>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")] MMA_AVV))] "TARGET_MMA" - "<avv> %A0,%x2,%x3" + "%!<avv> %A0,%x2,%x3" [(set_attr "type" "mma")]) +;; Instruction: xvf64ger + (define_insn "mma_<pv>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa")] MMA_PV))] "TARGET_MMA" - "<pv> %A0,%x1,%x2" + "%!<pv> %A0,%x1,%x2" [(set_attr "type" "mma")]) +;; Instructions: xvf64gerpp xvf64gerpn xvf64gernp xvf64gernn + (define_insn "mma_<apv>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:OO 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa")] MMA_APV))] "TARGET_MMA" - "<apv> %A0,%x2,%x3" + "%!<apv> %A0,%x2,%x3" [(set_attr "type" "mma")]) -(define_insn "mma_<vvi4i4i8>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") +;; Instruction: pmxvi4ger8 + +(define_insn "mma_pm<vvi4i4i8>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:SI 3 "const_0_to_15_operand" "n,n") @@ -569,13 +675,15 @@ (match_operand:SI 5 "u8bit_cint_operand" "n,n")] MMA_VVI4I4I8))] "TARGET_MMA" - "<vvi4i4i8> %A0,%x1,%x2,%3,%4,%5" + "pm%!<vvi4i4i8> %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<avvi4i4i8>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") +;; Instruction: pmxvi4ger8pp + +(define_insn "mma_pm<avvi4i4i8>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") (match_operand:SI 4 "const_0_to_15_operand" "n,n") @@ -583,12 +691,15 @@ (match_operand:SI 6 "u8bit_cint_operand" "n,n")] MMA_AVVI4I4I8))] "TARGET_MMA" - "<avvi4i4i8> %A0,%x2,%x3,%4,%5,%6" + "pm%!<avvi4i4i8> %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<vvi4i4i2>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") +;; Instructions: +;; pmxvi16ger2 pmxvi16ger2s pmxvf16ger2 pmxvbf16ger2 + +(define_insn "mma_pm<vvi4i4i2>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:SI 3 "const_0_to_15_operand" "n,n") @@ -596,13 +707,18 @@ (match_operand:SI 5 "const_0_to_3_operand" "n,n")] MMA_VVI4I4I2))] "TARGET_MMA" - "<vvi4i4i2> %A0,%x1,%x2,%3,%4,%5" + "pm%!<vvi4i4i2> %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<avvi4i4i2>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") +;; Instructions: +;; pmxvi16ger2pp pmxvi16ger2spp pmxvf16ger2pp pmxvf16ger2pn +;; pmxvf16ger2np pmxvf16ger2nn pmxvbf16ger2pp pmxvbf16ger2pn +;; pmxvbf16ger2np pmxvbf16ger2nn + +(define_insn "mma_pm<avvi4i4i2>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") (match_operand:SI 4 "const_0_to_15_operand" "n,n") @@ -610,62 +726,72 @@ (match_operand:SI 6 "const_0_to_3_operand" "n,n")] MMA_AVVI4I4I2))] "TARGET_MMA" - "<avvi4i4i2> %A0,%x2,%x3,%4,%5,%6" + "pm%!<avvi4i4i2> %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<vvi4i4>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") +;; Instruction: pmxvf32ger + +(define_insn "mma_pm<vvi4i4>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:SI 3 "const_0_to_15_operand" "n,n") (match_operand:SI 4 "const_0_to_15_operand" "n,n")] MMA_VVI4I4))] "TARGET_MMA" - "<vvi4i4> %A0,%x1,%x2,%3,%4" + "pm%!<vvi4i4> %A0,%x1,%x2,%3,%4" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<avvi4i4>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") +;; Instructions: pmxvf32gerpp pmxvf32gerpn pmxvf32gernp pmxvf32gernn + +(define_insn "mma_pm<avvi4i4>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") (match_operand:SI 4 "const_0_to_15_operand" "n,n") (match_operand:SI 5 "const_0_to_15_operand" "n,n")] MMA_AVVI4I4))] "TARGET_MMA" - "<avvi4i4> %A0,%x2,%x3,%4,%5" + "pm%!<avvi4i4> %A0,%x2,%x3,%4,%5" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<pvi4i2>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") +;; Instruction: pmxvf64ger + +(define_insn "mma_pm<pvi4i2>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:OO 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:SI 3 "const_0_to_15_operand" "n,n") (match_operand:SI 4 "const_0_to_3_operand" "n,n")] MMA_PVI4I2))] "TARGET_MMA" - "<pvi4i2> %A0,%x1,%x2,%3,%4" + "pm%!<pvi4i2> %A0,%x1,%x2,%3,%4" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<apvi4i2>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") +;; Instructions: pmxvf64gerpp pmxvf64gerpn pmxvf64gernp pmxvf64gernn + +(define_insn "mma_pm<apvi4i2>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:OO 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") (match_operand:SI 4 "const_0_to_15_operand" "n,n") (match_operand:SI 5 "const_0_to_3_operand" "n,n")] MMA_APVI4I2))] "TARGET_MMA" - "<apvi4i2> %A0,%x2,%x3,%4,%5" + "pm%!<apvi4i2> %A0,%x2,%x3,%4,%5" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<vvi4i4i4>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") +;; Instruction: pmxvi8ger4 + +(define_insn "mma_pm<vvi4i4i4>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") (unspec:XO [(match_operand:V16QI 1 "vsx_register_operand" "v,?wa") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:SI 3 "const_0_to_15_operand" "n,n") @@ -673,13 +799,15 @@ (match_operand:SI 5 "const_0_to_15_operand" "n,n")] MMA_VVI4I4I4))] "TARGET_MMA" - "<vvi4i4i4> %A0,%x1,%x2,%3,%4,%5" + "pm%!<vvi4i4i4> %A0,%x1,%x2,%3,%4,%5" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) -(define_insn "mma_<avvi4i4i4>" - [(set (match_operand:XO 0 "fpr_reg_operand" "=&d,&d") - (unspec:XO [(match_operand:XO 1 "fpr_reg_operand" "0,0") +;; Instructions: pmxvi8ger4pp pmxvi8ger4spp + +(define_insn "mma_pm<avvi4i4i4>" + [(set (match_operand:XO 0 "accumulator_operand" "=&wD,&wD") + (unspec:XO [(match_operand:XO 1 "accumulator_operand" "0,0") (match_operand:V16QI 2 "vsx_register_operand" "v,?wa") (match_operand:V16QI 3 "vsx_register_operand" "v,?wa") (match_operand:SI 4 "const_0_to_15_operand" "n,n") @@ -687,6 +815,156 @@ (match_operand:SI 6 "const_0_to_15_operand" "n,n")] MMA_AVVI4I4I4))] "TARGET_MMA" - "<avvi4i4i4> %A0,%x2,%x3,%4,%5,%6" + "pm%!<avvi4i4i4> %A0,%x2,%x3,%4,%5,%6" [(set_attr "type" "mma") (set_attr "prefixed" "yes")]) + +;; TDOmode (__dmf keyword for 1,024 bit registers). +(define_expand "movtdo" + [(set (match_operand:TDO 0 "nonimmediate_operand") + (match_operand:TDO 1 "input_operand"))] + "TARGET_DENSE_MATH" +{ + rs6000_emit_move (operands[0], operands[1], TDOmode); + DONE; +}) + +(define_insn_and_split "*movtdo" + [(set (match_operand:TDO 0 "nonimmediate_operand" "=wa,m,wa,wD,wD,wa") + (match_operand:TDO 1 "input_operand" "m,wa,wa,wa,wD,wD"))] + "TARGET_DENSE_MATH + && (gpc_reg_operand (operands[0], TDOmode) + || gpc_reg_operand (operands[1], TDOmode))" + "@ + # + # + # + # + dmmr %0,%1 + #" + "&& reload_completed + && (!dmf_register_operand (operands[0], TDOmode) + || !dmf_register_operand (operands[1], TDOmode))" + [(const_int 0)] +{ + rtx op0 = operands[0]; + rtx op1 = operands[1]; + + if (REG_P (op0) && REG_P (op1)) + { + int regno0 = REGNO (op0); + int regno1 = REGNO (op1); + + if (DMF_REGNO_P (regno0) && VSX_REGNO_P (regno1)) + { + rtx op1_upper = gen_rtx_REG (XOmode, regno1); + rtx op1_lower = gen_rtx_REG (XOmode, regno1 + 4); + emit_insn (gen_movtdo_insert512_upper (op0, op1_upper)); + emit_insn (gen_movtdo_insert512_lower (op0, op0, op1_lower)); + DONE; + } + + else if (VSX_REGNO_P (regno0) && DMF_REGNO_P (regno1)) + { + rtx op0_upper = gen_rtx_REG (XOmode, regno0); + rtx op0_lower = gen_rtx_REG (XOmode, regno0 + 4); + emit_insn (gen_movtdo_extract512 (op0_upper, op1, const0_rtx)); + emit_insn (gen_movtdo_extract512 (op0_lower, op1, const1_rtx)); + DONE; + } + + else + gcc_assert (VSX_REGNO_P (regno0) && VSX_REGNO_P (regno1)); + } + + rs6000_split_multireg_move (operands[0], operands[1]); + DONE; +} + [(set_attr "type" "vecload,vecstore,vecmove,vecmove,vecmove,vecmove") + (set_attr "length" "*,*,32,8,*,8") + (set_attr "max_prefixed_insns" "4,4,*,*,*,*")]) + +;; Move from VSX registers to dense math registers via two insert 512 bit +;; instructions. +(define_insn "movtdo_insert512_upper" + [(set (match_operand:TDO 0 "dmf_register_operand" "=wD") + (unspec:TDO [(match_operand:XO 1 "vsx_register_operand" "wa")] + UNSPEC_DMF_INSERT512_UPPER))] + "TARGET_DENSE_MATH" + "dmxxinstdmr512 %0,%1,%Y1,0" + [(set_attr "type" "mma")]) + +(define_insn "movtdo_insert512_lower" + [(set (match_operand:TDO 0 "dmf_register_operand" "=wD") + (unspec:TDO [(match_operand:TDO 1 "dmf_register_operand" "0") + (match_operand:XO 2 "vsx_register_operand" "wa")] + UNSPEC_DMF_INSERT512_LOWER))] + "TARGET_DENSE_MATH" + "dmxxinstdmr512 %0,%2,%Y2,1" + [(set_attr "type" "mma")]) + +;; Move from dense math registers to VSX registers via two extract 512 bit +;; instructions. +(define_insn "movtdo_extract512" + [(set (match_operand:XO 0 "vsx_register_operand" "=wa") + (unspec:XO [(match_operand:TDO 1 "dmf_register_operand" "wD") + (match_operand 2 "const_0_to_1_operand" "n")] + UNSPEC_DMF_EXTRACT512))] + "TARGET_DENSE_MATH" + "dmxxextfdmr512 %0,%Y0,%1,%2" + [(set_attr "type" "mma")]) + +;; Reload dense math registers from memory. +(define_insn_and_split "reload_tdo_from_memory" + [(set (match_operand:TDO 0 "dmf_register_operand" "=wD") + (unspec:TDO [(match_operand:TDO 1 "memory_operand" "m")] + UNSPEC_DMF_RELOAD_FROM_MEMORY)) + (clobber (match_operand:XO 2 "vsx_register_operand" "=wa"))] + "TARGET_DENSE_MATH" + "#" + "&& reload_completed" + [(const_int 0)] +{ + rtx dest = operands[0]; + rtx src = operands[1]; + rtx tmp = operands[2]; + rtx mem_upper = adjust_address (src, XOmode, BYTES_BIG_ENDIAN ? 0 : 64); + rtx mem_lower = adjust_address (src, XOmode, BYTES_BIG_ENDIAN ? 64 : 0); + + emit_move_insn (tmp, mem_upper); + emit_insn (gen_movtdo_insert512_upper (dest, tmp)); + + emit_move_insn (tmp, mem_lower); + emit_insn (gen_movtdo_insert512_lower (dest, dest, tmp)); + DONE; +} + [(set_attr "length" "16") + (set_attr "max_prefixed_insns" "2") + (set_attr "type" "vecload")]) + +;; Reload dense math registers to memory +(define_insn_and_split "reload_tdo_to_memory" + [(set (match_operand:TDO 0 "memory_operand" "=m") + (unspec:TDO [(match_operand:TDO 1 "dmf_register_operand" "wD")] + UNSPEC_DMF_RELOAD_TO_MEMORY)) + (clobber (match_operand:XO 2 "vsx_register_operand" "=wa"))] + "TARGET_DENSE_MATH" + "#" + "&& reload_completed" + [(const_int 0)] +{ + rtx dest = operands[0]; + rtx src = operands[1]; + rtx tmp = operands[2]; + rtx mem_upper = adjust_address (dest, XOmode, BYTES_BIG_ENDIAN ? 0 : 64); + rtx mem_lower = adjust_address (dest, XOmode, BYTES_BIG_ENDIAN ? 64 : 0); + + emit_insn (gen_movtdo_extract512 (tmp, src, const0_rtx)); + emit_move_insn (mem_upper, tmp); + + emit_insn (gen_movtdo_extract512 (tmp, src, const1_rtx)); + emit_move_insn (mem_lower, tmp); + DONE; +} + [(set_attr "length" "16") + (set_attr "max_prefixed_insns" "2")]) diff --git a/gcc/config/rs6000/predicates.md b/gcc/config/rs6000/predicates.md index 4162c22f8f68..7f8d316648cc 100644 --- a/gcc/config/rs6000/predicates.md +++ b/gcc/config/rs6000/predicates.md @@ -186,6 +186,47 @@ return VLOGICAL_REGNO_P (REGNO (op)); }) +;; Return 1 if op is a dense math register +(define_predicate "dmf_register_operand" + (match_operand 0 "register_operand") +{ + if (!TARGET_DENSE_MATH) + return 0; + + if (!REG_P (op)) + return 0; + + if (!HARD_REGISTER_P (op)) + return 1; + + return DMF_REGNO_P (REGNO (op)); +}) + +;; Return 1 if op is an accumulator. +;; +;; On power10 and power11 systems, the accumulators overlap with the +;; FPRs and the register must be divisible by 4. +;; +;; On systems with dense math registers, the accumulators are separate +;; registers and do not overlap with the FPR registers. +(define_predicate "accumulator_operand" + (match_operand 0 "register_operand") +{ + if (SUBREG_P (op)) + op = SUBREG_REG (op); + + if (!REG_P (op)) + return 0; + + if (!HARD_REGISTER_P (op)) + return 1; + + int r = REGNO (op); + return (TARGET_DENSE_MATH + ? DMF_REGNO_P (r) + : FP_REGNO_P (r) && (r & 3) == 0); +}) + ;; Return 1 if op is the carry register. (define_predicate "ca_operand" (match_operand 0 "register_operand") diff --git a/gcc/config/rs6000/rs6000-builtin.cc b/gcc/config/rs6000/rs6000-builtin.cc index e3aab2b6ef16..dd959296320d 100644 --- a/gcc/config/rs6000/rs6000-builtin.cc +++ b/gcc/config/rs6000/rs6000-builtin.cc @@ -512,6 +512,8 @@ const char *rs6000_type_string (tree type_node) return "__vector_pair"; else if (type_node == vector_quad_type_node) return "__vector_quad"; + else if (type_node == dm1024_type_node) + return "__dm1024"; return "unknown"; } @@ -813,6 +815,21 @@ rs6000_init_builtins (void) t = build_qualified_type (vector_quad_type_node, TYPE_QUAL_CONST); ptr_vector_quad_type_node = build_pointer_type (t); + /* For TDOmode (1,024 bit dense math accumulators), don't use an alignment of + 1,024, use 512. TDOmode loads and stores are always broken up into 2 + vector pair loads or stores. In addition, we don't have support for + aligning the stack to 1,024 bits. */ + dm1024_type_node = make_node (OPAQUE_TYPE); + SET_TYPE_MODE (dm1024_type_node, TDOmode); + TYPE_SIZE (dm1024_type_node) = bitsize_int (GET_MODE_BITSIZE (TDOmode)); + TYPE_PRECISION (dm1024_type_node) = GET_MODE_BITSIZE (TDOmode); + TYPE_SIZE_UNIT (dm1024_type_node) = size_int (GET_MODE_SIZE (TDOmode)); + SET_TYPE_ALIGN (dm1024_type_node, 512); + TYPE_USER_ALIGN (dm1024_type_node) = 0; + lang_hooks.types.register_builtin_type (dm1024_type_node, "__dm1024"); + t = build_qualified_type (dm1024_type_node, TYPE_QUAL_CONST); + ptr_dm1024_type_node = build_pointer_type (t); + tdecl = add_builtin_type ("__bool char", bool_char_type_node); TYPE_NAME (bool_char_type_node) = tdecl; @@ -1156,9 +1173,16 @@ rs6000_gimple_fold_mma_builtin (gimple_stmt_iterator *gsi, return true; } - /* If we're disassembling an accumulator into a different type, we need - to emit a xxmfacc instruction now, since we cannot do it later. */ - if (fncode == RS6000_BIF_DISASSEMBLE_ACC) + /* On a system without dense math registers, if we're disassembling an + accumulator into a different type, we need to emit a xxmfacc + instruction now, since we cannot do it later. + + On a dense math system, we use the explicit moves (dmxxinstdmr512 and + dmxxextfdmr512) to move values between VSX registers and the separate + dense math accumulators. This means, we don't have to sprinkle + xxmfacc's into the code to make sure the overlaid accumulators are + ready to issue MMA instructions. */ + if (fncode == RS6000_BIF_DISASSEMBLE_ACC && !TARGET_DENSE_MATH) { new_decl = rs6000_builtin_decls[RS6000_BIF_XXMFACC_INTERNAL]; new_call = gimple_build_call (new_decl, 1, src); diff --git a/gcc/config/rs6000/rs6000-c.cc b/gcc/config/rs6000/rs6000-c.cc index 3cbdb6fb2ba1..36bcc40ca571 100644 --- a/gcc/config/rs6000/rs6000-c.cc +++ b/gcc/config/rs6000/rs6000-c.cc @@ -587,9 +587,27 @@ rs6000_target_modify_macros (bool define_p, HOST_WIDE_INT flags) if (rs6000_cpu == PROCESSOR_CELL) rs6000_define_or_undefine_macro (define_p, "__PPU__"); - /* Tell the user if we support the MMA instructions. */ + /* Tell the user if we support the MMA instructions. Also tell them if we + have MMA with ISA 3.1 that uses accumulators overlaid over VSX registers + 0..31 or if we have support with separate dense math accumulators. */ if ((flags & OPTION_MASK_MMA) != 0) - rs6000_define_or_undefine_macro (define_p, "__MMA__"); + { + rs6000_define_or_undefine_macro (define_p, "__MMA__"); + if ((flags & OPTION_MASK_DENSE_MATH) != 0) + { + rs6000_define_or_undefine_macro (define_p, "__MMA_DENSE_MATH__"); + rs6000_define_or_undefine_macro (false, "__MMA_NO_DENSE_MATH__"); + } + else + { + rs6000_define_or_undefine_macro (false, "__MMA_DENSE_MATH__"); + rs6000_define_or_undefine_macro (define_p, "__MMA_NO_DENSE_MATH__"); + } + } + /* Tell the user if we support the dense math registers for use with MMA and + cryptography. */ + if ((flags & OPTION_MASK_DENSE_MATH) != 0) + rs6000_define_or_undefine_macro (define_p, "__DENSE_MATH__"); /* Whether pc-relative code is being generated. */ if ((flags & OPTION_MASK_PCREL) != 0) rs6000_define_or_undefine_macro (define_p, "__PCREL__"); diff --git a/gcc/config/rs6000/rs6000-call.cc b/gcc/config/rs6000/rs6000-call.cc index b9b791bfe8ae..e6e908355445 100644 --- a/gcc/config/rs6000/rs6000-call.cc +++ b/gcc/config/rs6000/rs6000-call.cc @@ -437,14 +437,15 @@ rs6000_return_in_memory (const_tree type, const_tree fntype ATTRIBUTE_UNUSED) if (cfun && !cfun->machine->mma_return_type_error && TREE_TYPE (cfun->decl) == fntype - && (TYPE_MODE (type) == OOmode || TYPE_MODE (type) == XOmode)) + && OPAQUE_MODE_P (TYPE_MODE (type))) { /* Record we have now handled function CFUN, so the next time we are called, we do not re-report the same error. */ cfun->machine->mma_return_type_error = true; if (TYPE_CANONICAL (type) != NULL_TREE) type = TYPE_CANONICAL (type); - error ("invalid use of MMA type %qs as a function return value", + error ("invalid use of %s type %qs as a function return value", + (TYPE_MODE (type) == TDOmode) ? "dense math" : "MMA", IDENTIFIER_POINTER (DECL_NAME (TYPE_NAME (type)))); } @@ -1632,11 +1633,12 @@ rs6000_function_arg (cumulative_args_t cum_v, const function_arg_info &arg) int n_elts; /* We do not allow MMA types being used as function arguments. */ - if (mode == OOmode || mode == XOmode) + if (OPAQUE_MODE_P (mode)) { if (TYPE_CANONICAL (type) != NULL_TREE) type = TYPE_CANONICAL (type); - error ("invalid use of MMA operand of type %qs as a function parameter", + error ("invalid use of %s operand of type %qs as a function parameter", + (mode == TDOmode) ? "dense math" : "MMA", IDENTIFIER_POINTER (DECL_NAME (TYPE_NAME (type)))); return NULL_RTX; } diff --git a/gcc/config/rs6000/rs6000-cpus.def b/gcc/config/rs6000/rs6000-cpus.def index a110860acce6..38e6bc880b25 100644 --- a/gcc/config/rs6000/rs6000-cpus.def +++ b/gcc/config/rs6000/rs6000-cpus.def @@ -85,6 +85,7 @@ /* -mcpu=future flags. */ #define FUTURE_MASKS_SERVER (POWER11_MASKS_SERVER \ + | OPTION_MASK_DENSE_MATH \ | OPTION_MASK_FUTURE) /* Flags that need to be turned off if -mno-vsx. */ @@ -117,6 +118,7 @@ #define POWERPC_MASKS (OPTION_MASK_ALTIVEC \ | OPTION_MASK_CMPB \ | OPTION_MASK_CRYPTO \ + | OPTION_MASK_DENSE_MATH \ | OPTION_MASK_DFP \ | OPTION_MASK_DLMZB \ | OPTION_MASK_EFFICIENT_UNALIGNED_VSX \ diff --git a/gcc/config/rs6000/rs6000-modes.def b/gcc/config/rs6000/rs6000-modes.def index 7140b634c414..6fca027949d6 100644 --- a/gcc/config/rs6000/rs6000-modes.def +++ b/gcc/config/rs6000/rs6000-modes.def @@ -79,3 +79,7 @@ PARTIAL_INT_MODE (TI, 128, PTI); /* Modes used by __vector_pair and __vector_quad. */ OPAQUE_MODE (OO, 32); OPAQUE_MODE (XO, 64); + +/* Mode used by __dmf. */ +OPAQUE_MODE (TDO, 128); + diff --git a/gcc/config/rs6000/rs6000.cc b/gcc/config/rs6000/rs6000.cc index d8669d9ffce4..2da66de42196 100644 --- a/gcc/config/rs6000/rs6000.cc +++ b/gcc/config/rs6000/rs6000.cc @@ -292,7 +292,8 @@ enum rs6000_reg_type { ALTIVEC_REG_TYPE, FPR_REG_TYPE, SPR_REG_TYPE, - CR_REG_TYPE + CR_REG_TYPE, + DMF_REG_TYPE }; /* Map register class to register type. */ @@ -306,22 +307,24 @@ static enum rs6000_reg_type reg_class_to_reg_type[N_REG_CLASSES]; /* Register classes we care about in secondary reload or go if legitimate - address. We only need to worry about GPR, FPR, and Altivec registers here, - along an ANY field that is the OR of the 3 register classes. */ + address. We only need to worry about GPR, FPR, Altivec, and dense math + registers here, along an ANY field that is the OR of the 4 register + classes. */ enum rs6000_reload_reg_type { RELOAD_REG_GPR, /* General purpose registers. */ RELOAD_REG_FPR, /* Traditional floating point regs. */ RELOAD_REG_VMX, /* Altivec (VMX) registers. */ - RELOAD_REG_ANY, /* OR of GPR, FPR, Altivec masks. */ + RELOAD_REG_DMR, /* Dense math registers. */ + RELOAD_REG_ANY, /* OR of GPR/FPR/VMX/DMR masks. */ N_RELOAD_REG }; -/* For setting up register classes, loop through the 3 register classes mapping +/* For setting up register classes, loop through the 4 register classes mapping into real registers, and skip the ANY class, which is just an OR of the bits. */ #define FIRST_RELOAD_REG_CLASS RELOAD_REG_GPR -#define LAST_RELOAD_REG_CLASS RELOAD_REG_VMX +#define LAST_RELOAD_REG_CLASS RELOAD_REG_DMR /* Map reload register type to a register in the register class. */ struct reload_reg_map_type { @@ -333,6 +336,7 @@ static const struct reload_reg_map_type reload_reg_map[N_RELOAD_REG] = { { "Gpr", FIRST_GPR_REGNO }, /* RELOAD_REG_GPR. */ { "Fpr", FIRST_FPR_REGNO }, /* RELOAD_REG_FPR. */ { "VMX", FIRST_ALTIVEC_REGNO }, /* RELOAD_REG_VMX. */ + { "Dmr", FIRST_DMF_REGNO }, /* RELOAD_REG_DMR. */ { "Any", -1 }, /* RELOAD_REG_ANY. */ }; @@ -1226,6 +1230,8 @@ char rs6000_reg_names[][8] = "0", "1", "2", "3", "4", "5", "6", "7", /* vrsave vscr sfp */ "vrsave", "vscr", "sfp", + /* dense math registers. */ + "0", "1", "2", "3", "4", "5", "6", "7", }; #ifdef TARGET_REGNAMES @@ -1252,6 +1258,8 @@ static const char alt_reg_names[][8] = "%cr0", "%cr1", "%cr2", "%cr3", "%cr4", "%cr5", "%cr6", "%cr7", /* vrsave vscr sfp */ "vrsave", "vscr", "sfp", + /* dense math registers. */ + "%dmr0", "%dmr1", "%dmr2", "%dmr3", "%dmr4", "%dmr5", "%dmr6", "%dmr7", }; #endif @@ -1842,6 +1850,9 @@ rs6000_hard_regno_nregs_internal (int regno, machine_mode mode) else if (ALTIVEC_REGNO_P (regno)) reg_size = UNITS_PER_ALTIVEC_WORD; + else if (DMF_REGNO_P (regno)) + reg_size = UNITS_PER_DMF_WORD; + else reg_size = UNITS_PER_WORD; @@ -1863,9 +1874,47 @@ rs6000_hard_regno_mode_ok_uncached (int regno, machine_mode mode) if (mode == OOmode) return (TARGET_MMA && VSX_REGNO_P (regno) && (regno & 1) == 0); - /* MMA accumulator modes need FPR registers divisible by 4. */ + /* On ISA 3.1 (power10), MMA accumulator modes need FPR registers divisible + by 4. + + If dense math registers are enabled, we can allow all VSX registers plus + the dense math registers. VSX registers are used to load and store the + registers as the accumulator registers do not have load and store + instructions. Because we just use the VSX registers for load/store + operations, we just need to make sure load vector pair and store vector + pair instructions can be used. */ if (mode == XOmode) - return (TARGET_MMA && FP_REGNO_P (regno) && (regno & 3) == 0); + { + if (!TARGET_DENSE_MATH) + return (FP_REGNO_P (regno) && (regno & 3) == 0); + + else if (DMF_REGNO_P (regno)) + return 1; + + else + return (VSX_REGNO_P (regno) + && VSX_REGNO_P (last_regno) + && (regno & 1) == 0); + } + + if (mode == TDOmode) + { + if (!TARGET_DENSE_MATH) + return 0; + + if (DMF_REGNO_P (regno)) + return 1; + + else + return (VSX_REGNO_P (regno) + && VSX_REGNO_P (last_regno) + && (regno & 1) == 0); + } + + /* No other types other than XOmode or TDOmode can go in dense math + registers. */ + if (DMF_REGNO_P (regno)) + return 0; /* PTImode can only go in GPRs. Quad word memory operations require even/odd register combinations, and use PTImode where we need to deal with quad @@ -1971,9 +2020,11 @@ rs6000_hard_regno_mode_ok (unsigned int regno, machine_mode mode) GPR registers, and TImode can go in any GPR as well as VSX registers (PR 57744). - Similarly, don't allow OOmode (vector pair, restricted to even VSX - registers) or XOmode (vector quad, restricted to FPR registers divisible - by 4) to tie with other modes. + Similarly, don't allow OOmode (vector pair), XOmode (vector quad), or + TDOmode (dense math register) to pair with anything else. Vector pairs are + restricted to even/odd VSX registers. Without dense math, vector quads are + limited to FPR registers divisible by 4. With dense math, vector quads are + limited to even VSX registers or dense math registers. Altivec/VSX vector tests were moved ahead of scalar float mode, so that IEEE 128-bit floating point on VSX systems ties with other vectors. */ @@ -1982,7 +2033,8 @@ static bool rs6000_modes_tieable_p (machine_mode mode1, machine_mode mode2) { if (mode1 == PTImode || mode1 == OOmode || mode1 == XOmode - || mode2 == PTImode || mode2 == OOmode || mode2 == XOmode) + || mode1 == TDOmode || mode2 == PTImode || mode2 == OOmode + || mode2 == XOmode || mode2 == TDOmode) return mode1 == mode2; if (ALTIVEC_OR_VSX_VECTOR_MODE (mode1)) @@ -2273,6 +2325,7 @@ rs6000_debug_reg_global (void) V4DFmode, OOmode, XOmode, + TDOmode, CCmode, CCUNSmode, CCEQmode, @@ -2308,6 +2361,7 @@ rs6000_debug_reg_global (void) rs6000_debug_reg_print (FIRST_ALTIVEC_REGNO, LAST_ALTIVEC_REGNO, "vs"); + rs6000_debug_reg_print (FIRST_DMF_REGNO, LAST_DMF_REGNO, "dense_math"); rs6000_debug_reg_print (LR_REGNO, LR_REGNO, "lr"); rs6000_debug_reg_print (CTR_REGNO, CTR_REGNO, "ctr"); rs6000_debug_reg_print (CR0_REGNO, CR7_REGNO, "cr"); @@ -2328,6 +2382,7 @@ rs6000_debug_reg_global (void) "wr reg_class = %s\n" "wx reg_class = %s\n" "wA reg_class = %s\n" + "wD reg_class = %s\n" "\n", reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_d]], reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_v]], @@ -2335,7 +2390,8 @@ rs6000_debug_reg_global (void) reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_we]], reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wr]], reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wx]], - reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]]); + reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wA]], + reg_class_names[rs6000_constraints[RS6000_CONSTRAINT_wD]]); nl = "\n"; for (m = 0; m < NUM_MACHINE_MODES; ++m) @@ -2632,6 +2688,21 @@ rs6000_setup_reg_addr_masks (void) addr_mask = 0; reg = reload_reg_map[rc].reg; + /* Special case dense math registers. */ + if (rc == RELOAD_REG_DMR) + { + if (TARGET_DENSE_MATH && (m2 == XOmode || m2 == TDOmode)) + { + addr_mask = RELOAD_REG_VALID; + reg_addr[m].addr_mask[rc] = addr_mask; + any_addr_mask |= addr_mask; + } + else + reg_addr[m].addr_mask[rc] = 0; + + continue; + } + /* Can mode values go in the GPR/FPR/Altivec registers? */ if (reg >= 0 && rs6000_hard_regno_mode_ok_p[m][reg]) { @@ -2727,10 +2798,10 @@ rs6000_setup_reg_addr_masks (void) /* Vector pairs can do both indexed and offset loads if the instructions are enabled, otherwise they can only do offset loads - since it will be broken into two vector moves. Vector quads can - only do offset loads. */ + since it will be broken into two vector moves. Vector quads and + dense math types can only do offset loads. */ else if ((addr_mask != 0) && TARGET_MMA - && (m2 == OOmode || m2 == XOmode)) + && (m2 == OOmode || m2 == XOmode || m2 == TDOmode)) { addr_mask |= RELOAD_REG_OFFSET; if (rc == RELOAD_REG_FPR || rc == RELOAD_REG_VMX) @@ -2782,6 +2853,9 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) for (r = CR1_REGNO; r <= CR7_REGNO; ++r) rs6000_regno_regclass[r] = CR_REGS; + for (r = FIRST_DMF_REGNO; r <= LAST_DMF_REGNO; ++r) + rs6000_regno_regclass[r] = DMF_REGS; + rs6000_regno_regclass[LR_REGNO] = LINK_REGS; rs6000_regno_regclass[CTR_REGNO] = CTR_REGS; rs6000_regno_regclass[CA_REGNO] = NO_REGS; @@ -2806,6 +2880,7 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) reg_class_to_reg_type[(int)LINK_OR_CTR_REGS] = SPR_REG_TYPE; reg_class_to_reg_type[(int)CR_REGS] = CR_REG_TYPE; reg_class_to_reg_type[(int)CR0_REGS] = CR_REG_TYPE; + reg_class_to_reg_type[(int)DMF_REGS] = DMF_REG_TYPE; if (TARGET_VSX) { @@ -2954,6 +3029,14 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) rs6000_vector_align[XOmode] = 512; } + /* Add support for 1,024 bit dense math registers. */ + if (TARGET_DENSE_MATH) + { + rs6000_vector_unit[TDOmode] = VECTOR_NONE; + rs6000_vector_mem[TDOmode] = VECTOR_VSX; + rs6000_vector_align[TDOmode] = 512; + } + /* Register class constraints for the constraints that depend on compile switches. When the VSX code was added, different constraints were added based on the type (DFmode, V2DFmode, V4SFmode). For the vector types, all @@ -2992,6 +3075,13 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) if (TARGET_DIRECT_MOVE_128) rs6000_constraints[RS6000_CONSTRAINT_we] = VSX_REGS; + /* Support for the accumulator registers, either FPR registers (aka original + mma) or dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH) + rs6000_constraints[RS6000_CONSTRAINT_wD] = FLOAT_REGS; + else if (TARGET_MMA_DENSE_MATH) + rs6000_constraints[RS6000_CONSTRAINT_wD] = DMF_REGS; + /* Set up the reload helper and direct move functions. */ if (TARGET_VSX || TARGET_ALTIVEC) { @@ -3160,6 +3250,12 @@ rs6000_init_hard_regno_mode_ok (bool global_init_p) } } + if (TARGET_DENSE_MATH) + { + reg_addr[TDOmode].reload_load = CODE_FOR_reload_tdo_from_memory; + reg_addr[TDOmode].reload_store = CODE_FOR_reload_tdo_to_memory; + } + /* Precalculate HARD_REGNO_NREGS. */ for (r = 0; HARD_REGISTER_NUM_P (r); ++r) for (m = 0; m < NUM_MACHINE_MODES; ++m) @@ -4405,6 +4501,15 @@ rs6000_option_override_internal (bool global_init_p) if (!TARGET_PCREL && TARGET_PCREL_OPT) rs6000_isa_flags &= ~OPTION_MASK_PCREL_OPT; + /* Turn off dense math register support on non-future systems. */ + if (TARGET_DENSE_MATH && !TARGET_FUTURE) + { + if ((rs6000_isa_flags_explicit & OPTION_MASK_DENSE_MATH) != 0) + error ("%qs requires %qs", "-mdense-math", "-mcpu=future"); + + rs6000_isa_flags &= ~OPTION_MASK_DENSE_MATH; + } + if (TARGET_DEBUG_REG || TARGET_DEBUG_TARGET) rs6000_print_isa_options (stderr, 0, "after subtarget", rs6000_isa_flags); @@ -8665,12 +8770,15 @@ reg_offset_addressing_ok_p (machine_mode mode) return mode_supports_dq_form (mode); break; - /* The vector pair/quad types support offset addressing if the - underlying vectors support offset addressing. */ + /* The vector pair/quad types and the dense math types support offset + addressing if the underlying vectors support offset addressing. */ case E_OOmode: case E_XOmode: return TARGET_MMA; + case E_TDOmode: + return TARGET_DENSE_MATH; + case E_SDmode: /* If we can do direct load/stores of SDmode, restrict it to reg+reg addressing for the LFIWZX and STFIWX instructions. */ @@ -11224,6 +11332,12 @@ rs6000_emit_move (rtx dest, rtx source, machine_mode mode) (mode == OOmode) ? "__vector_pair" : "__vector_quad"); break; + case E_TDOmode: + if (CONST_INT_P (operands[1])) + error ("%qs is an opaque type, and you cannot set it to constants", + "__dm1024"); + break; + case E_SImode: case E_DImode: /* Use default pattern for address of ELF small data */ @@ -12352,6 +12466,11 @@ rs6000_secondary_reload_memory (rtx addr, addr_mask = (reg_addr[mode].addr_mask[RELOAD_REG_VMX] & ~RELOAD_REG_AND_M16); + /* Dense math registers use VSX registers for memory operations, and need to + generate some extra instructions. */ + else if (rclass == DMF_REGS) + return 2; + /* If the register allocator hasn't made up its mind yet on the register class to use, settle on defaults to use. */ else if (rclass == NO_REGS) @@ -12680,6 +12799,13 @@ rs6000_secondary_reload_simple_move (enum rs6000_reg_type to_type, || (to_type == SPR_REG_TYPE && from_type == GPR_REG_TYPE))) return true; + /* We can transfer between VSX registers and dense math registers without + needing extra registers. */ + if (TARGET_DENSE_MATH && (mode == XOmode || mode == TDOmode) + && ((to_type == DMF_REG_TYPE && from_type == VSX_REG_TYPE) + || (to_type == VSX_REG_TYPE && from_type == DMF_REG_TYPE))) + return true; + return false; } @@ -13374,6 +13500,10 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass) machine_mode mode = GET_MODE (x); bool is_constant = CONSTANT_P (x); + /* Dense math registers can't be loaded or stored. */ + if (rclass == DMF_REGS) + return NO_REGS; + /* If a mode can't go in FPR/ALTIVEC/VSX registers, don't return a preferred reload class for it. */ if ((rclass == ALTIVEC_REGS || rclass == VSX_REGS) @@ -13470,7 +13600,10 @@ rs6000_preferred_reload_class (rtx x, enum reg_class rclass) return VSX_REGS; if (mode == XOmode) - return FLOAT_REGS; + return TARGET_DENSE_MATH ? VSX_REGS : FLOAT_REGS; + + if (mode == TDOmode) + return VSX_REGS; if (GET_MODE_CLASS (mode) == MODE_INT) return GENERAL_REGS; @@ -13595,6 +13728,11 @@ rs6000_secondary_reload_class (enum reg_class rclass, machine_mode mode, else regno = -1; + /* Dense math registers don't have loads or stores. We have to go through + the VSX registers to load XOmode (vector quad). */ + if (TARGET_DENSE_MATH && rclass == DMF_REGS) + return VSX_REGS; + /* If we have VSX register moves, prefer moving scalar values between Altivec registers and GPR by going via an FPR (and then via memory) instead of reloading the secondary memory address for Altivec moves. */ @@ -14126,8 +14264,14 @@ print_operand (FILE *file, rtx x, int code) output_operand. */ case 'A': - /* Write the MMA accumulator number associated with VSX register X. */ - if (!REG_P (x) || !FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0) + /* Write the MMA accumulator number associated with VSX register X. On + dense math systems, only allow dense math accumulators, not + accumulators overlapping with the FPR registers. */ + if (!REG_P (x)) + output_operand_lossage ("invalid %%A value"); + else if (TARGET_DENSE_MATH && DMF_REGNO_P (REGNO (x))) + fprintf (file, "%d", REGNO (x) - FIRST_DMF_REGNO); + else if (!FP_REGNO_P (REGNO (x)) || (REGNO (x) % 4) != 0) output_operand_lossage ("invalid %%A value"); else fprintf (file, "%d", (REGNO (x) - FIRST_FPR_REGNO) / 4); @@ -14689,6 +14833,12 @@ print_operand (FILE *file, rtx x, int code) "local dynamic TLS references"); return; + /* Print out 'dm' if dense math registers are available. */ + case '!': + if (TARGET_DENSE_MATH) + fputs ("dm", file); + return; + default: output_operand_lossage ("invalid %%xn code"); } @@ -20636,6 +20786,8 @@ rs6000_mangle_type (const_tree type) return "u13__vector_pair"; if (type == vector_quad_type_node) return "u13__vector_quad"; + if (type == dm1024_type_node) + return "u8__dm1024"; /* For all other types, use the default mangling. */ return NULL; @@ -22748,6 +22900,35 @@ rs6000_debug_address_cost (rtx x, machine_mode mode, } +/* Subroutine to determine the move cost of dense math registers. If we are + moving to/from VSX_REGISTER registers, the cost is either 1 move (for + 512-bit accumulators) or 2 moves (for 1,024 dense math registers). If we are + moving to anything else like GPR registers, make the cost very high. */ + +static int +rs6000_dmf_register_move_cost (machine_mode mode, reg_class_t rclass) +{ + const int reg_move_base = 2; + HARD_REG_SET vsx_set = (reg_class_contents[rclass] + & reg_class_contents[VSX_REGS]); + + if (TARGET_DENSE_MATH && !hard_reg_set_empty_p (vsx_set)) + { + /* __vector_quad (i.e. XOmode) is tranfered in 1 instruction. */ + if (mode == XOmode) + return reg_move_base; + + /* __dm1024 (i.e. TDOmode) is transferred in 2 instructions. */ + else if (mode == TDOmode) + return reg_move_base * 2; + + else + return reg_move_base * 2 * hard_regno_nregs (FIRST_DMF_REGNO, mode); + } + + return 1000 * 2 * hard_regno_nregs (FIRST_DMF_REGNO, mode); +} + /* A C expression returning the cost of moving data from a register of class CLASS1 to one of CLASS2. */ @@ -22761,17 +22942,28 @@ rs6000_register_move_cost (machine_mode mode, if (TARGET_DEBUG_COST) dbg_cost_ctrl++; + HARD_REG_SET to_vsx, from_vsx; + to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS]; + from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS]; + + /* Special case dense math registers, that can only move to/from VSX registers. */ + if (from == DMF_REGS && to == DMF_REGS) + ret = 2 * hard_regno_nregs (FIRST_DMF_REGNO, mode); + + else if (from == DMF_REGS) + ret = rs6000_dmf_register_move_cost (mode, to); + + else if (to == DMF_REGS) + ret = rs6000_dmf_register_move_cost (mode, from); + /* If we have VSX, we can easily move between FPR or Altivec registers, otherwise we can only easily move within classes. Do this first so we give best-case answers for union classes containing both gprs and vsx regs. */ - HARD_REG_SET to_vsx, from_vsx; - to_vsx = reg_class_contents[to] & reg_class_contents[VSX_REGS]; - from_vsx = reg_class_contents[from] & reg_class_contents[VSX_REGS]; - if (!hard_reg_set_empty_p (to_vsx) - && !hard_reg_set_empty_p (from_vsx) - && (TARGET_VSX - || hard_reg_set_intersect_p (to_vsx, from_vsx))) + else if (!hard_reg_set_empty_p (to_vsx) + && !hard_reg_set_empty_p (from_vsx) + && (TARGET_VSX + || hard_reg_set_intersect_p (to_vsx, from_vsx))) { int reg = FIRST_FPR_REGNO; if (TARGET_VSX @@ -22867,6 +23059,9 @@ rs6000_memory_move_cost (machine_mode mode, reg_class_t rclass, ret = 4 * hard_regno_nregs (32, mode); else if (reg_classes_intersect_p (rclass, ALTIVEC_REGS)) ret = 4 * hard_regno_nregs (FIRST_ALTIVEC_REGNO, mode); + else if (reg_classes_intersect_p (rclass, DMF_REGS)) + ret = (rs6000_dmf_register_move_cost (mode, VSX_REGS) + + rs6000_memory_move_cost (mode, VSX_REGS, false)); else ret = 4 + rs6000_register_move_cost (mode, rclass, GENERAL_REGS); @@ -24075,6 +24270,8 @@ rs6000_compute_pressure_classes (enum reg_class *pressure_classes) if (TARGET_HARD_FLOAT) pressure_classes[n++] = FLOAT_REGS; } + if (TARGET_DENSE_MATH) + pressure_classes[n++] = DMF_REGS; pressure_classes[n++] = CR_REGS; pressure_classes[n++] = SPECIAL_REGS; @@ -24239,6 +24436,10 @@ rs6000_debugger_regno (unsigned int regno, unsigned int format) return 67; if (regno == 64) return 64; + /* XXX: This is a guess. The GCC register number for FIRST_DMF_REGNO is 111, + but the frame pointer regnum uses that. */ + if (DMF_REGNO_P (regno)) + return regno - FIRST_DMF_REGNO + 112; gcc_unreachable (); } @@ -24460,6 +24661,7 @@ static struct rs6000_opt_mask const rs6000_opt_masks[] = false, true }, { "cmpb", OPTION_MASK_CMPB, false, true }, { "crypto", OPTION_MASK_CRYPTO, false, true }, + { "dense-math", OPTION_MASK_DENSE_MATH, false, true }, { "direct-move", 0, false, true }, { "dlmzb", OPTION_MASK_DLMZB, false, true }, { "efficient-unaligned-vsx", OPTION_MASK_EFFICIENT_UNALIGNED_VSX, @@ -27406,9 +27608,10 @@ rs6000_split_multireg_move (rtx dst, rtx src) mode = GET_MODE (dst); nregs = hard_regno_nregs (reg, mode); - /* If we have a vector quad register for MMA, and this is a load or store, - see if we can use vector paired load/stores. */ - if (mode == XOmode && TARGET_MMA + /* If we have a vector quad register for MMA or dense math register + and this is a load or store, see if we can use vector paired + load/stores. */ + if ((mode == XOmode || mode == TDOmode) && TARGET_MMA && (MEM_P (dst) || MEM_P (src))) { reg_mode = OOmode; @@ -27416,7 +27619,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) } /* If we have a vector pair/quad mode, split it into two/four separate vectors. */ - else if (mode == OOmode || mode == XOmode) + else if (mode == OOmode || mode == XOmode || mode == TDOmode) reg_mode = V1TImode; else if (FP_REGNO_P (reg)) reg_mode = DECIMAL_FLOAT_MODE_P (mode) ? DDmode : @@ -27462,13 +27665,13 @@ rs6000_split_multireg_move (rtx dst, rtx src) return; } - /* The __vector_pair and __vector_quad modes are multi-register - modes, so if we have to load or store the registers, we have to be - careful to properly swap them if we're in little endian mode - below. This means the last register gets the first memory - location. We also need to be careful of using the right register - numbers if we are splitting XO to OO. */ - if (mode == OOmode || mode == XOmode) + /* The __vector_pair, __vector_quad, and __dm1024 modes are multi-register + modes, so if we have to load or store the registers, we have to be careful + to properly swap them if we're in little endian mode below. This means + the last register gets the first memory location. We also need to be + careful of using the right register numbers if we are splitting XO to + OO. */ + if (mode == OOmode || mode == XOmode || mode == TDOmode) { nregs = hard_regno_nregs (reg, mode); int reg_mode_nregs = hard_regno_nregs (reg, reg_mode); @@ -27477,9 +27680,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) unsigned offset = 0; unsigned size = GET_MODE_SIZE (reg_mode); - /* If we are reading an accumulator register, we have to - deprime it before we can access it. */ - if (TARGET_MMA + /* If we are reading an accumulator register, we have to deprime it + before we can access it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27511,9 +27714,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) emit_insn (gen_rtx_SET (dst2, src2)); } - /* If we are writing an accumulator register, we have to - prime it after we've written it. */ - if (TARGET_MMA + /* If we are writing an accumulator register, we have to prime it + after we've written it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst)); @@ -27527,7 +27730,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) || XINT (src, 1) == UNSPECV_MMA_ASSEMBLE); gcc_assert (REG_P (dst)); if (GET_MODE (src) == XOmode) - gcc_assert (FP_REGNO_P (REGNO (dst))); + gcc_assert ((TARGET_DENSE_MATH + ? VSX_REGNO_P (REGNO (dst)) + : FP_REGNO_P (REGNO (dst)))); if (GET_MODE (src) == OOmode) gcc_assert (VSX_REGNO_P (REGNO (dst))); @@ -27580,9 +27785,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) emit_insn (gen_rtx_SET (dst_i, op)); } - /* We are writing an accumulator register, so we have to - prime it after we've written it. */ - if (GET_MODE (src) == XOmode) + /* We are writing an accumulator register, so we have to prime it + after we've written it unless we have dense math registers. */ + if (GET_MODE (src) == XOmode && !TARGET_DENSE_MATH) emit_insn (gen_mma_xxmtacc (dst, dst)); return; @@ -27593,9 +27798,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) if (REG_P (src) && REG_P (dst) && (REGNO (src) < REGNO (dst))) { - /* If we are reading an accumulator register, we have to - deprime it before we can access it. */ - if (TARGET_MMA + /* If we are reading an accumulator register, we have to deprime it + before we can access it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27603,7 +27808,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) overlap. */ int i; /* XO/OO are opaque so cannot use subregs. */ - if (mode == OOmode || mode == XOmode ) + if (mode == OOmode || mode == XOmode || mode == TDOmode) { for (i = nregs - 1; i >= 0; i--) { @@ -27621,9 +27826,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) i * reg_mode_size))); } - /* If we are writing an accumulator register, we have to - prime it after we've written it. */ - if (TARGET_MMA + /* If we are writing an accumulator register, we have to prime it after + we've written it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst)); } @@ -27758,9 +27963,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) gcc_assert (rs6000_offsettable_memref_p (dst, reg_mode, true)); } - /* If we are reading an accumulator register, we have to - deprime it before we can access it. */ - if (TARGET_MMA && REG_P (src) + /* If we are reading an accumulator register, we have to deprime it + before we can access it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && REG_P (src) && GET_MODE (src) == XOmode && FP_REGNO_P (REGNO (src))) emit_insn (gen_mma_xxmfacc (src, src)); @@ -27777,7 +27982,7 @@ rs6000_split_multireg_move (rtx dst, rtx src) continue; /* XO/OO are opaque so cannot use subregs. */ - if (mode == OOmode || mode == XOmode ) + if (mode == OOmode || mode == XOmode || mode == TDOmode) { rtx dst_i = gen_rtx_REG (reg_mode, REGNO (dst) + j); rtx src_i = gen_rtx_REG (reg_mode, REGNO (src) + j); @@ -27790,9 +27995,9 @@ rs6000_split_multireg_move (rtx dst, rtx src) j * reg_mode_size))); } - /* If we are writing an accumulator register, we have to - prime it after we've written it. */ - if (TARGET_MMA && REG_P (dst) + /* If we are writing an accumulator register, we have to prime it after + we've written it unless we have dense math registers. */ + if (TARGET_MMA_NO_DENSE_MATH && REG_P (dst) && GET_MODE (dst) == XOmode && FP_REGNO_P (REGNO (dst))) emit_insn (gen_mma_xxmtacc (dst, dst)); @@ -28805,7 +29010,8 @@ rs6000_invalid_conversion (const_tree fromtype, const_tree totype) if (frommode != tomode) { - /* Do not allow conversions to/from XOmode and OOmode types. */ + /* Do not allow conversions to/from XOmode, OOmode, and TDOmode + types. */ if (frommode == XOmode) return N_("invalid conversion from type %<__vector_quad%>"); if (tomode == XOmode) @@ -28814,6 +29020,10 @@ rs6000_invalid_conversion (const_tree fromtype, const_tree totype) return N_("invalid conversion from type %<__vector_pair%>"); if (tomode == OOmode) return N_("invalid conversion to type %<__vector_pair%>"); + if (frommode == TDOmode) + return N_("invalid conversion from type %<__dm1024%>"); + if (tomode == TDOmode) + return N_("invalid conversion to type %<__dm1024%>"); } /* Conversion allowed. */ diff --git a/gcc/config/rs6000/rs6000.h b/gcc/config/rs6000/rs6000.h index 401f50ead4f4..eef25768b5af 100644 --- a/gcc/config/rs6000/rs6000.h +++ b/gcc/config/rs6000/rs6000.h @@ -500,6 +500,12 @@ extern int rs6000_vector_align[]; #define TARGET_MINMAX (TARGET_HARD_FLOAT && TARGET_PPC_GFXOPT \ && (TARGET_P9_MINMAX || !flag_trapping_math)) +/* Define if the MMA subsystem uses ISA 3.1 where the accumulators are overlaid + over VSX registers 0..31 or whether MMA uses separate dense math + accumulators. */ +#define TARGET_MMA_DENSE_MATH (TARGET_MMA && TARGET_DENSE_MATH) +#define TARGET_MMA_NO_DENSE_MATH (TARGET_MMA && !TARGET_DENSE_MATH) + /* In switching from using target_flags to using rs6000_isa_flags, the options machinery creates OPTION_MASK_<xxx> instead of MASK_<xxx>. The MASK_<xxxx> options that have not yet been replaced by their OPTION_MASK_<xxx> @@ -657,6 +663,7 @@ extern unsigned char rs6000_recip_bits[]; #define UNITS_PER_FP_WORD 8 #define UNITS_PER_ALTIVEC_WORD 16 #define UNITS_PER_VSX_WORD 16 +#define UNITS_PER_DMF_WORD 128 /* Type used for ptrdiff_t, as a string used in a declaration. */ #define PTRDIFF_TYPE "int" @@ -770,7 +777,7 @@ enum data_align { align_abi, align_opt, align_both }; Another pseudo (not included in DWARF_FRAME_REGISTERS) is soft frame pointer, which is eventually eliminated in favor of SP or FP. */ -#define FIRST_PSEUDO_REGISTER 111 +#define FIRST_PSEUDO_REGISTER 119 /* Use standard DWARF numbering for DWARF debugging information. */ #define DEBUGGER_REGNO(REGNO) rs6000_debugger_regno ((REGNO), 0) @@ -807,7 +814,9 @@ enum data_align { align_abi, align_opt, align_both }; /* cr0..cr7 */ \ 0, 0, 0, 0, 0, 0, 0, 0, \ /* vrsave vscr sfp */ \ - 1, 1, 1 \ + 1, 1, 1, \ + /* Dense math registers. */ \ + 0, 0, 0, 0, 0, 0, 0, 0 \ } /* Like `CALL_USED_REGISTERS' except this macro doesn't require that @@ -831,7 +840,9 @@ enum data_align { align_abi, align_opt, align_both }; /* cr0..cr7 */ \ 1, 1, 0, 0, 0, 1, 1, 1, \ /* vrsave vscr sfp */ \ - 0, 0, 0 \ + 0, 0, 0, \ + /* Dense math registers. */ \ + 0, 0, 0, 0, 0, 0, 0, 0 \ } #define TOTAL_ALTIVEC_REGS (LAST_ALTIVEC_REGNO - FIRST_ALTIVEC_REGNO + 1) @@ -868,6 +879,7 @@ enum data_align { align_abi, align_opt, align_both }; v2 (not saved; incoming vector arg reg; return value) v19 - v14 (not saved or used for anything) v31 - v20 (saved; order given to save least number) + dmr0 - dmr7 (not saved) vrsave, vscr (fixed) sfp (fixed) */ @@ -910,6 +922,9 @@ enum data_align { align_abi, align_opt, align_both }; 66, \ 83, 82, 81, 80, 79, 78, \ 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, \ + /* Dense math registers. */ \ + 111, 112, 113, 114, 115, 116, 117, 118, \ + /* Vrsave, vscr, sfp. */ \ 108, 109, \ 110 \ } @@ -936,6 +951,9 @@ enum data_align { align_abi, align_opt, align_both }; /* True if register is a VSX register. */ #define VSX_REGNO_P(N) (FP_REGNO_P (N) || ALTIVEC_REGNO_P (N)) +/* True if register is a Dense math register. */ +#define DMF_REGNO_P(N) ((N) >= FIRST_DMF_REGNO && (N) <= LAST_DMF_REGNO) + /* Alternate name for any vector register supporting floating point, no matter which instruction set(s) are available. */ #define VFLOAT_REGNO_P(N) \ @@ -975,7 +993,7 @@ enum data_align { align_abi, align_opt, align_both }; /* Modes that are not vectors, but require vector alignment. Treat these like vectors in terms of loads and stores. */ #define VECTOR_ALIGNMENT_P(MODE) \ - (FLOAT128_VECTOR_P (MODE) || (MODE) == OOmode || (MODE) == XOmode) + (FLOAT128_VECTOR_P (MODE) || OPAQUE_MODE_P (MODE)) #define ALTIVEC_VECTOR_MODE(MODE) \ ((MODE) == V16QImode \ @@ -1073,6 +1091,7 @@ enum reg_class FLOAT_REGS, ALTIVEC_REGS, VSX_REGS, + DMF_REGS, VRSAVE_REGS, VSCR_REGS, GEN_OR_FLOAT_REGS, @@ -1102,6 +1121,7 @@ enum reg_class "FLOAT_REGS", \ "ALTIVEC_REGS", \ "VSX_REGS", \ + "DMF_REGS", \ "VRSAVE_REGS", \ "VSCR_REGS", \ "GEN_OR_FLOAT_REGS", \ @@ -1136,6 +1156,8 @@ enum reg_class { 0x00000000, 0x00000000, 0xffffffff, 0x00000000 }, \ /* VSX_REGS. */ \ { 0x00000000, 0xffffffff, 0xffffffff, 0x00000000 }, \ + /* DMF_REGS. */ \ + { 0x00000000, 0x00000000, 0x00000000, 0x007f8000 }, \ /* VRSAVE_REGS. */ \ { 0x00000000, 0x00000000, 0x00000000, 0x00001000 }, \ /* VSCR_REGS. */ \ @@ -1163,7 +1185,7 @@ enum reg_class /* CA_REGS. */ \ { 0x00000000, 0x00000000, 0x00000000, 0x00000004 }, \ /* ALL_REGS. */ \ - { 0xffffffff, 0xffffffff, 0xffffffff, 0x00007fff } \ + { 0xffffffff, 0xffffffff, 0xffffffff, 0x007fffff } \ } /* The same information, inverted: @@ -1187,6 +1209,7 @@ enum r6000_reg_class_enum { RS6000_CONSTRAINT_wr, /* GPR register if 64-bit */ RS6000_CONSTRAINT_wx, /* FPR register for STFIWX */ RS6000_CONSTRAINT_wA, /* BASE_REGS if 64-bit. */ + RS6000_CONSTRAINT_wD, /* Accumulator regs if MMA/Dense Math. */ RS6000_CONSTRAINT_MAX }; @@ -2063,7 +2086,16 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */ &rs6000_reg_names[108][0], /* vrsave */ \ &rs6000_reg_names[109][0], /* vscr */ \ \ - &rs6000_reg_names[110][0] /* sfp */ \ + &rs6000_reg_names[110][0], /* sfp */ \ + \ + &rs6000_reg_names[111][0], /* dmr0 */ \ + &rs6000_reg_names[112][0], /* dmr1 */ \ + &rs6000_reg_names[113][0], /* dmr2 */ \ + &rs6000_reg_names[114][0], /* dmr3 */ \ + &rs6000_reg_names[115][0], /* dmr4 */ \ + &rs6000_reg_names[116][0], /* dmr5 */ \ + &rs6000_reg_names[117][0], /* dmr6 */ \ + &rs6000_reg_names[118][0], /* dmr7 */ \ } /* Table of additional register names to use in user input. */ @@ -2117,6 +2149,8 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */ {"vs52", 84}, {"vs53", 85}, {"vs54", 86}, {"vs55", 87}, \ {"vs56", 88}, {"vs57", 89}, {"vs58", 90}, {"vs59", 91}, \ {"vs60", 92}, {"vs61", 93}, {"vs62", 94}, {"vs63", 95}, \ + {"dmr0", 111}, {"dmr1", 112}, {"dmr2", 113}, {"dmr3", 114}, \ + {"dmr4", 115}, {"dmr5", 116}, {"dmr6", 117}, {"dmr7", 118}, \ } /* This is how to output an element of a case-vector that is relative. */ @@ -2174,7 +2208,7 @@ extern char rs6000_reg_names[][8]; /* register names (0 vs. %r0). */ /* Define which CODE values are valid. */ -#define PRINT_OPERAND_PUNCT_VALID_P(CODE) ((CODE) == '&') +#define PRINT_OPERAND_PUNCT_VALID_P(CODE) ((CODE) == '&' || (CODE) == '!') /* Print a memory address as an operand to reference that memory location. */ @@ -2250,6 +2284,7 @@ enum rs6000_builtin_type_index RS6000_BTI_const_str, /* pointer to const char * */ RS6000_BTI_vector_pair, /* unsigned 256-bit types (vector pair). */ RS6000_BTI_vector_quad, /* unsigned 512-bit types (vector quad). */ + RS6000_BTI_dm1024, /* unsigned 1,024-bit types (dmf). */ RS6000_BTI_const_ptr_void, /* const pointer to void */ RS6000_BTI_ptr_V16QI, RS6000_BTI_ptr_V1TI, @@ -2288,6 +2323,7 @@ enum rs6000_builtin_type_index RS6000_BTI_ptr_dfloat128, RS6000_BTI_ptr_vector_pair, RS6000_BTI_ptr_vector_quad, + RS6000_BTI_ptr_dm1024, RS6000_BTI_ptr_long_long, RS6000_BTI_ptr_long_long_unsigned, RS6000_BTI_INTPTI, @@ -2349,6 +2385,7 @@ enum rs6000_builtin_type_index #define const_str_type_node (rs6000_builtin_types[RS6000_BTI_const_str]) #define vector_pair_type_node (rs6000_builtin_types[RS6000_BTI_vector_pair]) #define vector_quad_type_node (rs6000_builtin_types[RS6000_BTI_vector_quad]) +#define dm1024_type_node (rs6000_builtin_types[RS6000_BTI_dm1024]) #define pcvoid_type_node (rs6000_builtin_types[RS6000_BTI_const_ptr_void]) #define ptr_V16QI_type_node (rs6000_builtin_types[RS6000_BTI_ptr_V16QI]) #define ptr_V1TI_type_node (rs6000_builtin_types[RS6000_BTI_ptr_V1TI]) @@ -2387,6 +2424,7 @@ enum rs6000_builtin_type_index #define ptr_dfloat128_type_node (rs6000_builtin_types[RS6000_BTI_ptr_dfloat128]) #define ptr_vector_pair_type_node (rs6000_builtin_types[RS6000_BTI_ptr_vector_pair]) #define ptr_vector_quad_type_node (rs6000_builtin_types[RS6000_BTI_ptr_vector_quad]) +#define ptr_dm1024_type_node (rs6000_builtin_types[RS6000_BTI_ptr_dm1024]) #define ptr_long_long_integer_type_node (rs6000_builtin_types[RS6000_BTI_ptr_long_long]) #define ptr_long_long_unsigned_type_node (rs6000_builtin_types[RS6000_BTI_ptr_long_long_unsigned]) diff --git a/gcc/config/rs6000/rs6000.md b/gcc/config/rs6000/rs6000.md index 4ec3d3da8a99..5b590bc9b0d9 100644 --- a/gcc/config/rs6000/rs6000.md +++ b/gcc/config/rs6000/rs6000.md @@ -51,6 +51,8 @@ (VRSAVE_REGNO 108) (VSCR_REGNO 109) (FRAME_POINTER_REGNUM 110) + (FIRST_DMF_REGNO 111) + (LAST_DMF_REGNO 118) ]) ;; diff --git a/gcc/config/rs6000/rs6000.opt b/gcc/config/rs6000/rs6000.opt index 2b6ec5222fc1..5bf1b98e4e77 100644 --- a/gcc/config/rs6000/rs6000.opt +++ b/gcc/config/rs6000/rs6000.opt @@ -638,6 +638,10 @@ mieee128-constant Target Var(TARGET_IEEE128_CONSTANT) Init(1) Save Generate (do not generate) code that uses the LXVKQ instruction. +mdense-math +Target Mask(DENSE_MATH) Var(rs6000_isa_flags) +Generate (do not generate) instructions that use dense math registers. + ; Documented parameters -param=rs6000-vect-unroll-limit= diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi index 9546523fd657..78c19c3fdd84 100644 --- a/gcc/doc/invoke.texi +++ b/gcc/doc/invoke.texi @@ -32908,6 +32908,13 @@ This option is enabled by default. Enable or disable warnings about deprecated @samp{vector long ...} Altivec type usage. This option is enabled by default. +@opindex mdense-math +@opindex mno-dense-math +@item -mdense-math +@itemx -mno-dense-math +Generate (do not generate) code that uses the dense math registers. +This option is enabled by default. + @end table @node RX Options diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi index 5ad4a201273d..1f0a79c178ce 100644 --- a/gcc/doc/md.texi +++ b/gcc/doc/md.texi @@ -3294,6 +3294,11 @@ Like @code{d}, if @option{-mpowerpc-gfxopt} is used; otherwise, @code{NO_REGS}. @item wA Like @code{b}, if @option{-mpowerpc64} is used; otherwise, @code{NO_REGS}. +@item wD +Accumulator register if @option{-mma} is used; otherwise, +@code{NO_REGS}. For @option{-mcpu=power10} the accumulator registers +overlap with VSX vector registers 0..31. + @item wB Signed 5-bit constant integer that can be loaded into an Altivec register. diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-1.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-1.c new file mode 100644 index 000000000000..b2aab020ca46 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-1.c @@ -0,0 +1,313 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo0 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvi4ger8 (&acc, vec0, vec1); + __builtin_mma_xvi4ger8pp (&acc, vec0, vec1); + dst[0] = acc; +} + +void +foo1 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvi8ger4 (&acc, vec0, vec1); + __builtin_mma_xvi8ger4pp (&acc, vec0, vec1); + __builtin_mma_xvi8ger4spp(&acc, vec0, vec1); + dst[1] = acc; +} + +void +foo2 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvi16ger2 (&acc, vec0, vec1); + __builtin_mma_xvi16ger2pp (&acc, vec0, vec1); + dst[2] = acc; +} + +void +foo3 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvi16ger2s (&acc, vec0, vec1); + __builtin_mma_xvi16ger2spp (&acc, vec0, vec1); + dst[3] = acc; +} + +void +foo4 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvf16ger2 (&acc, vec0, vec1); + __builtin_mma_xvf16ger2pp (&acc, vec0, vec1); + __builtin_mma_xvf16ger2pn (&acc, vec0, vec1); + dst[4] = acc; +} + +void +foo4b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_xvf16ger2np (&acc, vec0, vec1); + __builtin_mma_xvf16ger2nn (&acc, vec0, vec1); + dst[4] = acc; +} + +void +foo5 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvbf16ger2 (&acc, vec0, vec1); + __builtin_mma_xvbf16ger2pp (&acc, vec0, vec1); + __builtin_mma_xvbf16ger2pn (&acc, vec0, vec1); + dst[5] = acc; +} + +void +foo5b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_xvbf16ger2np (&acc, vec0, vec1); + __builtin_mma_xvbf16ger2nn (&acc, vec0, vec1); + dst[5] = acc; +} + +void +foo6 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_xvf32ger (&acc, vec0, vec1); + __builtin_mma_xvf32gerpp (&acc, vec0, vec1); + __builtin_mma_xvf32gerpn (&acc, vec0, vec1); + dst[6] = acc; +} + +void +foo6b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_xvf32gernp (&acc, vec0, vec1); + __builtin_mma_xvf32gernn (&acc, vec0, vec1); + dst[6] = acc; +} + +void +foo7 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvi4ger8 (&acc, vec0, vec1, 15, 15, 255); + __builtin_mma_pmxvi4ger8pp (&acc, vec0, vec1, 15, 15, 255); + dst[7] = acc; +} + +void +foo8 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvi8ger4 (&acc, vec0, vec1, 15, 15, 15); + __builtin_mma_pmxvi8ger4pp (&acc, vec0, vec1, 15, 15, 15); + __builtin_mma_pmxvi8ger4spp(&acc, vec0, vec1, 15, 15, 15); + dst[8] = acc; +} + +void +foo9 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvi16ger2 (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvi16ger2pp (&acc, vec0, vec1, 15, 15, 3); + dst[9] = acc; +} + +void +foo10 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvi16ger2s (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvi16ger2spp (&acc, vec0, vec1, 15, 15, 3); + dst[10] = acc; +} + +void +foo11 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvf16ger2 (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvf16ger2pp (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvf16ger2pn (&acc, vec0, vec1, 15, 15, 3); + dst[11] = acc; +} + +void +foo11b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_pmxvf16ger2np (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvf16ger2nn (&acc, vec0, vec1, 15, 15, 3); + dst[11] = acc; +} + +void +foo12 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvbf16ger2 (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvbf16ger2pp (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvbf16ger2pn (&acc, vec0, vec1, 15, 15, 3); + dst[12] = acc; +} + +void +foo12b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_pmxvbf16ger2np (&acc, vec0, vec1, 15, 15, 3); + __builtin_mma_pmxvbf16ger2nn (&acc, vec0, vec1, 15, 15, 3); + dst[12] = acc; +} + +void +foo13 (__vector_quad *dst, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + __builtin_mma_pmxvf32ger (&acc, vec0, vec1, 15, 15); + __builtin_mma_pmxvf32gerpp (&acc, vec0, vec1, 15, 15); + __builtin_mma_pmxvf32gerpn (&acc, vec0, vec1, 15, 15); + dst[13] = acc; +} + +void +foo13b (__vector_quad *dst, __vector_quad *src, vec_t *vec) +{ + __vector_quad acc; + vec_t vec0 = vec[0]; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_pmxvf32gernp (&acc, vec0, vec1, 15, 15); + __builtin_mma_pmxvf32gernn (&acc, vec0, vec1, 15, 15); + dst[13] = acc; +} + +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-times {\mlxv\M} 40 } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 12 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 40 } } */ +/* { dg-final { scan-assembler-times {\mdmxvbf16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvbf16ger2nn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvbf16ger2np\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvbf16ger2pn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvbf16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf16ger2nn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf16ger2np\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf16ger2pn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf32ger\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf32gernn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf32gernp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf32gerpn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf32gerpp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi16ger2s\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi16ger2spp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi4ger8\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi4ger8pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi8ger4\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi8ger4pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvi8ger4spp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvbf16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvbf16ger2nn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvbf16ger2np\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvbf16ger2pn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvbf16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf16ger2nn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf16ger2np\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf16ger2pn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf32ger\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf32gernn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf32gernp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf32gerpn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf32gerpp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi16ger2\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi16ger2pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi16ger2s\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi16ger2spp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi4ger8\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi4ger8pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi8ger4\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi8ger4pp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvi8ger4spp\M} 1 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-pair.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-pair.c new file mode 100644 index 000000000000..e6cb49fb667a --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-pair.c @@ -0,0 +1,21 @@ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo (__vector_pair *dst, vec_t *src) +{ + __vector_pair pair0, pair1; + /* Adjacent loads should be combined into one lxvp instruction + and identical build pairs should be combined. */ + __builtin_vsx_build_pair (&pair0, src[0], src[1]); + __builtin_vsx_build_pair (&pair1, src[0], src[1]); + dst[0] = pair0; + dst[2] = pair1; +} + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mstxv\M} } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 2 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-quad.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-quad.c new file mode 100644 index 000000000000..589b3dca842c --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-10-quad.c @@ -0,0 +1,23 @@ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo (__vector_quad *dst, vec_t *src) +{ + __vector_quad quad0, quad1; + /* Adjacent loads should be combined into two lxvp instructions. + and identical build accs should not be combined. */ + __builtin_mma_build_acc (&quad0, src[0], src[1], src[2], src[3]); + __builtin_mma_build_acc (&quad1, src[0], src[1], src[2], src[3]); + dst[0] = quad0; + dst[2] = quad1; +} + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mstxv\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 4 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 4 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-2.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-2.c new file mode 100644 index 000000000000..880e7a871aa0 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-2.c @@ -0,0 +1,72 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo0 (__vector_quad *dst, vec_t *vec, __vector_pair *pvecp) +{ + __vector_quad acc; + __vector_pair vecp0 = *pvecp; + vec_t vec1 = vec[1]; + + __builtin_mma_xvf64ger (&acc, vecp0, vec1); + __builtin_mma_xvf64gerpp (&acc, vecp0, vec1); + __builtin_mma_xvf64gerpn (&acc, vecp0, vec1); + dst[0] = acc; +} + +void +foo1 (__vector_quad *dst, __vector_quad *src, vec_t *vec, __vector_pair *pvecp) +{ + __vector_quad acc; + __vector_pair vecp0 = *pvecp; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_xvf64gernp (&acc, vecp0, vec1); + __builtin_mma_xvf64gernn (&acc, vecp0, vec1); + dst[0] = acc; +} + +void +foo2 (__vector_quad *dst, vec_t *vec, __vector_pair *pvecp) +{ + __vector_quad acc; + __vector_pair vecp0 = *pvecp; + vec_t vec1 = vec[1]; + __builtin_mma_pmxvf64ger (&acc, vecp0, vec1, 15, 3); + __builtin_mma_pmxvf64gerpp (&acc, vecp0, vec1, 15, 3); + __builtin_mma_pmxvf64gerpn (&acc, vecp0, vec1, 15, 3); + dst[1] = acc; +} + +void +foo3 (__vector_quad *dst, __vector_quad *src, vec_t *vec, __vector_pair *pvecp) +{ + __vector_quad acc; + __vector_pair vecp0 = *pvecp; + vec_t vec1 = vec[1]; + + acc = src[0]; + __builtin_mma_pmxvf64gernp (&acc, vecp0, vec1, 15, 3); + __builtin_mma_pmxvf64gernn (&acc, vecp0, vec1, 15, 3); + dst[1] = acc; +} + +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-times {\mlxv\M} 4 } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 8 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 8 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf64ger\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf64gerpp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf64gerpn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf64gernp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxvf64gernn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf64ger\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf64gerpp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf64gerpn\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf64gernp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpmdmxvf64gernn\M} 1 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-3.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-3.c new file mode 100644 index 000000000000..798a9dc69f9e --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-3.c @@ -0,0 +1,31 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +void +foo0 (void) +{ + __vector_quad acc; + asm ("#..." : "=d" (acc)); + __builtin_mma_xxmtacc (&acc); + __builtin_mma_xxmfacc (&acc); + asm ("#..." :: "d" (acc)); +} + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo1 (vec_t *vec) +{ + vec[1] = __builtin_vsx_xvcvspbf16 (vec[0]); + vec[3] = __builtin_vsx_xvcvbf16spn (vec[2]); +} + +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-not {\mlxvp\M} } } */ +/* { dg-final { scan-assembler-not {\mstxvp\M} } } */ +/* { dg-final { scan-assembler-times {\mlxv\M} 2 } } */ +/* { dg-final { scan-assembler-times {\mstxv\M} 2 } } */ +/* { dg-final { scan-assembler-times {\mxvcvspbf16\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mxvcvbf16spn\M} 1 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-4.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-4.c new file mode 100644 index 000000000000..cd62fc8579a5 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-4.c @@ -0,0 +1,73 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo (__vector_pair *dst, vec_t *src) +{ + __vector_pair pair; + __builtin_mma_assemble_pair (&pair, src[0], src[4]); + *dst = pair; +} + +void +foo2 (__vector_pair *dst, vec_t *src) +{ + __vector_pair pair; + __builtin_vsx_assemble_pair (&pair, src[0], src[4]); + *dst = pair; +} + +void +foo3 (__vector_pair *dst, vec_t *src) +{ + __vector_pair pair; + __builtin_vsx_build_pair (&pair, src[4], src[0]); + *dst = pair; +} + +void +bar (vec_t *dst, __vector_pair *src) +{ + vec_t res[2]; + __builtin_mma_disassemble_pair (res, src); + dst[0] = res[0]; + dst[4] = res[1]; +} + +void +bar2 (vec_t *dst, __vector_pair *src) +{ + vec_t res[2]; + __builtin_vsx_disassemble_pair (res, src); + dst[0] = res[0]; + dst[4] = res[1]; +} + +#if !__has_builtin (__builtin_vsx_assemble_pair) +# error "__has_builtin (__builtin_vsx_assemble_pair) failed" +#endif + +#if !__has_builtin (__builtin_vsx_disassemble_pair) +# error "__has_builtin (__builtin_vsx_disassemble_pair) failed" +#endif + +#if !__has_builtin (__builtin_mma_assemble_pair) +# error "__has_builtin (__builtin_mma_assemble_pair) failed" +#endif + +#if !__has_builtin (__builtin_mma_disassemble_pair) +# error "__has_builtin (__builtin_mma_disassemble_pair) failed" +#endif + +#if !__has_builtin (__builtin_vsx_build_pair) +# error "__has_builtin (__builtin_vsx_build_pair) failed" +#endif + +/* { dg-final { scan-assembler-times {\mlxv\M} 6 } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 2 } } */ +/* { dg-final { scan-assembler-times {\mstxv\M} 4 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 3 } } */ + diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-5.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-5.c new file mode 100644 index 000000000000..6890d87bfb96 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-5.c @@ -0,0 +1,47 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo (__vector_quad *dst, vec_t *src) +{ + __vector_quad acc; + __builtin_mma_assemble_acc (&acc, src[0], src[4], src[8], src[12]); + *dst = acc; +} + +void +foo2 (__vector_quad *dst, vec_t *src) +{ + __vector_quad acc; + __builtin_mma_build_acc (&acc, src[12], src[8], src[4], src[0]); + *dst = acc; +} + +void +bar (vec_t *dst, __vector_quad *src) +{ + vec_t res[4]; + __builtin_mma_disassemble_acc (res, src); + dst[0] = res[0]; + dst[4] = res[1]; + dst[8] = res[2]; + dst[12] = res[3]; +} + +#if !__has_builtin (__builtin_mma_assemble_acc) +# error "__has_builtin (__builtin_mma_assemble_acc) failed" +#endif + +#if !__has_builtin (__builtin_mma_build_acc) +# error "__has_builtin (__builtin_mma_build_acc) failed" +#endif + +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-times {\mlxv\M} 8 } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 2 } } */ +/* { dg-final { scan-assembler-times {\mstxv\M} 4 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 4 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-6.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-6.c new file mode 100644 index 000000000000..37706f79ce53 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-6.c @@ -0,0 +1,21 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-Wno-psabi -mdejagnu-cpu=future -O2" } */ + +void +foo (__vector_quad *dst) +{ + __vector_quad acc0, acc1; + __builtin_mma_xxsetaccz (&acc0); + __builtin_mma_xxsetaccz (&acc1); + dst[0] = acc0; + dst[1] = acc1; +} + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mlxvp\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmtacc\M} } } */ +/* { dg-final { scan-assembler-not {\mxxmfacc\M} } } */ +/* { dg-final { scan-assembler-times {\mdmsetdmrz\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mdmxxextfdmr512\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 4 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-7.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-7.c new file mode 100644 index 000000000000..6e1f62b26c96 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-7.c @@ -0,0 +1,26 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-mdejagnu-cpu=future -O2" } */ + +void +foo (__vector_pair *dst, __vector_pair *src, long idx) +{ + dst[0] = __builtin_vsx_lxvp (0, src); + dst[2] = __builtin_vsx_lxvp (32, src); + dst[4] = __builtin_vsx_lxvp (64, src); + /* Non-constant offset should generate a lxvpx. */ + dst[6] = __builtin_vsx_lxvp (idx, src); + /* Non-aligned offset should generate a plxvp. */ + dst[8] = __builtin_vsx_lxvp (257, src); +} + +#if !__has_builtin (__builtin_vsx_lxvp) +# error "__has_builtin (__builtin_vsx_lxvp) failed" +#endif + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mstxv\M} } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 3 } } */ +/* { dg-final { scan-assembler-times {\mlxvpx\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mplxvp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 5 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-8.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-8.c new file mode 100644 index 000000000000..45d4e9b52903 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-8.c @@ -0,0 +1,27 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-mdejagnu-cpu=future -O2" } */ + +void +foo (__vector_pair *dst, __vector_pair *src, long idx) +{ + __vector_pair pair = *src; + __builtin_vsx_stxvp (pair, 0, dst); + __builtin_vsx_stxvp (pair, 32, dst); + __builtin_vsx_stxvp (pair, 64, dst); + /* Non-constant offset should generate a stxvpx. */ + __builtin_vsx_stxvp (pair, idx, dst); + /* Non-aligned offset should generate a pstxvp. */ + __builtin_vsx_stxvp (pair, 257, dst); +} + +#if !__has_builtin (__builtin_vsx_stxvp) +# error "__has_builtin (__builtin_vsx_stxvp) failed" +#endif + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mstxv\M} } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 3 } } */ +/* { dg-final { scan-assembler-times {\mstxvpx\M} 1 } } */ +/* { dg-final { scan-assembler-times {\mpstxvp\M} 1 } } */ diff --git a/gcc/testsuite/gcc.target/powerpc/dm-builtin-9.c b/gcc/testsuite/gcc.target/powerpc/dm-builtin-9.c new file mode 100644 index 000000000000..c62040c0fcb4 --- /dev/null +++ b/gcc/testsuite/gcc.target/powerpc/dm-builtin-9.c @@ -0,0 +1,28 @@ +/* { dg-do compile } */ +/* { dg-require-effective-target powerpc_future_ok } */ +/* { dg-options "-mdejagnu-cpu=future -O2" } */ + +typedef unsigned char vec_t __attribute__((vector_size(16))); + +void +foo (__vector_pair *dst, vec_t *src) +{ + __vector_pair pair; + /* Adjacent loads should be combined into one lxvp instruction. */ + __builtin_vsx_build_pair (&pair, src[0], src[1]); + *dst = pair; +} + +void +bar (__vector_quad *dst, vec_t *src) +{ + __vector_quad quad; + /* Adjacent loads should be combined into two lxvp instructions. */ + __builtin_mma_build_acc (&quad, src[0], src[1], src[2], src[3]); + *dst = quad; +} + +/* { dg-final { scan-assembler-not {\mlxv\M} } } */ +/* { dg-final { scan-assembler-not {\mstxv\M} } } */ +/* { dg-final { scan-assembler-times {\mlxvp\M} 3 } } */ +/* { dg-final { scan-assembler-times {\mstxvp\M} 3 } } */
