On 19/08/2026 10:20, Claudio Bantaloukas wrote:
This patch adds support for the intrinsics implementing FEAT_SME_TMOP listed
below. All of these require the +sme-tmop arch option.
A new intrinsic shape and a new register constraint is required. This patch
adds these, along with tests. Also added a pair of predicates to tighten the
immediates selecting ZA tiles based on tile width.
Specifically:
- svtmopa_lane_za16[_mf8_mf8] (requires +sme-f8f16)
- svtmopa_lane_za16[_bf16_bf16] (requires +sme-b16b16)
- svtmopa_lane_za16[_f16_f16] (requires +sme-f16f16)
- svtmopa_lane_za32[_mf8_mf8] (requires +sme-f8f32)
- svtmopa_lane_za32[_bf16_bf16] (requires +sme-b16b16)
- svtmopa_lane_za32[_f16_f16]
- svtmopa_lane_za32[_f32_f32]
- svtmopa_lane_za32[_s16_s16]
- svtmopa_lane_za32[_u16_u16]
- svtmopa_lane_za32[_s8_s8]
- svtmopa_lane_za32[_u8_u8]
- svtmopa_lane_za32[_s8_u8]
- svtmopa_lane_za32[_u8_s8]
gcc/
* config/aarch64/aarch64-acle-builtins.cc:
(function_resolver::resolve_to): Fix documentation comment.
* config/aarch64/aarch64-acle-builtins.h: (TYPES_tmop_base,
TYPES_tmop_h_bfloat, TYPES_tmop_h_float, TYPES_tmop_h_mf8,
TYPES_tmop_s_mf8): Add new DEF_SVE_TYPES_ARRAYs.
* config/aarch64/aarch64-c.cc: (__ARM_FEATURE_SME_TMOP): Add define.
* config/aarch64/aarch64-sme.md:
(@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_BHI:mode>): Add
new insn.
(@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><VNx16QI_ONLY:mode>):
Likewise.
(@aarch64_sme_lane_<optab><mode><mode>):
Likewise.
(@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_HF:mode>):
Likewise.
(@aarch64_sme_lane_<optab><SME_TMOPA_FP8:mode><VNx16QI_ONLY:mode>):
Likewise.
* config/aarch64/aarch64-sve-builtins-shapes.cc:
(ternary_za_uint_dual_single): Add new shape.
* config/aarch64/aarch64-sve-builtins-shapes.h: Likewise.
* config/aarch64/aarch64-sve-builtins-sme.cc: (svtmopa_lane_za_impl):
Add new function impl class.
(svtmopa_lane_za): Add new FUNCTION.
* config/aarch64/aarch64-sve-builtins-sme.def: (svtmopa_lane): Add new
DEF_SME_ZA_FUNCTION_GS entries.
* config/aarch64/aarch64-sve-builtins-sme.h: (svtmopa_lane_za): Add new.
* config/aarch64/aarch64.h: (TARGET_STREAMING_SME_TMOP): Add new define.
* config/aarch64/constraints.md (Uwo): Add new constraint for Control
Vector Register in TMOP operations.
* config/aarch64/iterators.md: (SME_TMOPA_BHSF, SME_TMOPA_FP8): Add new
mode
iterators.
(UNSPEC_SME_FTMOPA, UNSPEC_SME_FTMOPA_FP8, UNSPEC_SME_STMOPA,
UNSPEC_SME_SUTMOPA, UNSPEC_SME_USTMOPA, UNSPEC_SME_UTMOPA): Add new
unspecs.
(SME_TMOP_INT, SME_TMOP_INT_CROSS, SME_TMOP_FP, SME_TMOP_FP8): Add new
iterators.
(za_imm_bits): Add new mode_attr.
(optab): Update with new unspecs.
* config/aarch64/predicates.md: (aarch64_imm1, aarch64_imm4): Add new
predicates.
gcc/testsuite/
* gcc.target/aarch64/pragma_cpp_predefs_6.c: Add tests checking that the
sme-tmop prefef is off by default, and checks for feature dependencies.
* gcc.target/aarch64/sme2/acle-asm/test_sme2_acle.h: (TEST_ZA_TMOP) Add
testing macro.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c:
New test.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_f16_f16.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f16_f16.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f32_f32.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_s8.c: Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_u8.c: Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s16_s16.c:
Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_s8.c: Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_u8.c: Likewise.
* gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u16_u16.c:
Likewise.
* gcc.target/aarch64/sve/acle/general-c/ternary_za_uint_dual_single_1.c:
Likewise.
* lib/target-supports.exp: Add check_effective_target support for
sme-tmop.
---
gcc/config/aarch64/aarch64-acle-builtins.cc | 2 +-
gcc/config/aarch64/aarch64-acle-builtins.h | 29 +++++
gcc/config/aarch64/aarch64-c.cc | 2 +
gcc/config/aarch64/aarch64-sme.md | 112 ++++++++++++++++++
.../aarch64/aarch64-sve-builtins-shapes.cc | 39 ++++++
.../aarch64/aarch64-sve-builtins-shapes.h | 1 +
.../aarch64/aarch64-sve-builtins-sme.cc | 36 ++++++
.../aarch64/aarch64-sve-builtins-sme.def | 24 ++++
gcc/config/aarch64/aarch64-sve-builtins-sme.h | 1 +
gcc/config/aarch64/aarch64.h | 3 +
gcc/config/aarch64/constraints.md | 4 +
gcc/config/aarch64/iterators.md | 30 +++++
gcc/config/aarch64/predicates.md | 11 ++
.../gcc.target/aarch64/pragma_cpp_predefs_6.c | 17 +++
.../aarch64/sme2/acle-asm/test_sme2_acle.h | 18 +++
.../sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za16_f16_f16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c | 83 +++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_f16_f16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_f32_f32.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c | 83 +++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_s16_s16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_s8_s8.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_s8_u8.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_u16_u16.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_u8_s8.c | 76 ++++++++++++
.../sme2/acle-asm/tmopa_lane_za32_u8_u8.c | 76 ++++++++++++
.../general-c/ternary_za_uint_dual_single_1.c | 87 ++++++++++++++
gcc/testsuite/lib/target-supports.exp | 1 +
30 files changed, 1418 insertions(+), 1 deletion(-)
create mode 100644 gcc/testsuite/gcc.target/aarch64/pragma_cpp_predefs_6.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_f16_f16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f16_f16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f32_f32.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s16_s16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_s8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_u8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u16_u16.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_s8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_u8.c
create mode 100644
gcc/testsuite/gcc.target/aarch64/sve/acle/general-c/ternary_za_uint_dual_single_1.c
<snip>
diff --git a/gcc/config/aarch64/aarch64-sme.md
b/gcc/config/aarch64/aarch64-sme.md
index 7091f566ba8..fa4c3016e1d 100644
--- a/gcc/config/aarch64/aarch64-sme.md
+++ b/gcc/config/aarch64/aarch64-sme.md
@@ -46,11 +46,13 @@
;; ---- [INT] Dot product
;; ---- [INT] Ternary widening arithmetic on ZA slice
;; ---- [INT] Sum of outer products
+;; ---- [INT] Sparse outer product
;; ---- [FP] Dot product
;; ---- [FP8] Dot product
;; ---- [FP] Ternary arithmetic on ZA slice
;; ---- [FP] Ternary widening arithmetic on ZA slice
;; ---- [FP] Sum of outer products
+;; ---- [FP] Sparse outer product
;;
;; == Table lookup
;; ---- Table lookup
@@ -1814,6 +1816,53 @@ (define_insn
"@aarch64_sme_<optab><VNx4SI_ONLY:mode><VNx4SI_ONLY:mode>"
"<optab>\tza%0.s, %1/m, %2/m, %3.s, %4.s"
)
+;; -------------------------------------------------------------------------
+;; ---- [INT] Sparse outer product
+;; -------------------------------------------------------------------------
+;; Includes:
+;; - STMOPA
+;; - UTMOPA
+;; - SUTMOPA
+;; - USTMOPA
+;; -------------------------------------------------------------------------
+;; svtmopa_lane_za32[_s16_s16]
+;; svtmopa_lane_za32[_u16_u16]
+;; svtmopa_lane_za32[_s8_s8]
+;; svtmopa_lane_za32[_u8_u8]
+(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_BHI:mode>"
+ [(set (reg:VNx4SI_ONLY ZA_REGNUM)
+ (unspec:VNx4SI_ONLY
+ [(reg:VNx4SI_ONLY ZA_REGNUM)
+ (reg:DI SME_STATE_REGNUM)
+ (match_operand:DI 0 "aarch64_imm2")
+ (match_operand:<SVE_FULL_BHI:VDOUBLE> 1 "aligned_register_operand"
"Uw2")
+ (match_operand:SVE_FULL_BHI 2 "register_operand" "w")
+ (match_operand:VNx16QI 3 "register_operand" "Uwo")
+ (match_operand:DI 4 "aarch64_imm2")
+ ]
+ SME_TMOP_INT))]
+ "TARGET_STREAMING_SME_TMOP"
+ "<optab>\tza%0.s, %1, %2.<SVE_FULL_BHI:Vetype>, %3[%4]"
+)
+
+;; svtmopa_lane_za32[_s8_u8]
+;; svtmopa_lane_za32[_u8_s8]
+(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><VNx16QI_ONLY:mode>"
+ [(set (reg:VNx4SI_ONLY ZA_REGNUM)
+ (unspec:VNx4SI_ONLY
+ [(reg:VNx4SI_ONLY ZA_REGNUM)
+ (reg:DI SME_STATE_REGNUM)
+ (match_operand:DI 0 "aarch64_imm2")
+ (match_operand:<VNx16QI_ONLY:VDOUBLE> 1 "aligned_register_operand"
"Uw2")
+ (match_operand:VNx16QI_ONLY 2 "register_operand" "w")
+ (match_operand:VNx16QI 3 "register_operand" "Uwo")
+ (match_operand:DI 4 "aarch64_imm2")
+ ]
+ SME_TMOP_INT_CROSS))]
+ "TARGET_STREAMING_SME_TMOP"
+ "<optab>\tza%0.s, %1, %2.<VNx16QI_ONLY:Vetype>, %3[%4]"
+)
+
;; -------------------------------------------------------------------------
;; ---- [FP] Dot product
;; -------------------------------------------------------------------------
@@ -2737,6 +2786,69 @@ (define_insn
"@aarch64_sme_<optab><SME_ZA_F8F16_32:mode><VNx16QI_ONLY:mode>"
"<optab>\tza%0.<SME_ZA_F8F16_32:Vetype>, %1/m, %2/m, %3.b, %4.b"
)
+;; -------------------------------------------------------------------------
+;; ---- [FP] Sparse outer product
+;; -------------------------------------------------------------------------
+;; Includes:
+;; - BFTMOPA (SME_TMOP)
+;; - FTMOPA (SME_TMOP)
+;; -------------------------------------------------------------------------
+;; svtmopa_lane_za16[_bf16_bf16]
+;; svtmopa_lane_za16[_f16_f16]
+;; svtmopa_lane_za32[_f32_f32]
+(define_insn "@aarch64_sme_lane_<optab><mode><mode>"
+ [(set (reg:SME_TMOPA_BHSF ZA_REGNUM)
+ (unspec:SME_TMOPA_BHSF
+ [(reg:SME_TMOPA_BHSF ZA_REGNUM)
+ (reg:DI SME_STATE_REGNUM)
+ (match_operand:DI 0 "aarch64_imm<SME_TMOPA_BHSF:za_imm_bits>")
Does this work without qualifying the mode iterator (I would have thought so,
given there is only a single mode iterator). If it works, I'd prefer to drop
it.
+ (match_operand:<VDOUBLE> 1 "aligned_register_operand" "Uw2")
+ (match_operand:SME_TMOPA_BHSF 2 "register_operand" "w")
+ (match_operand:VNx16QI 3 "register_operand" "Uwo")
+ (match_operand:DI 4 "aarch64_imm2")
+ ]
+ SME_TMOP_FP))]
+ "TARGET_STREAMING_SME_TMOP"
+ "<b><optab>\tza%0.<Vetype>, %1, %2.<Vetype>, %3[%4]"
+)
+
+;; svtmopa_lane_za32[_bf16_bf16]
+;; svtmopa_lane_za32[_f16_f16]
+(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_HF:mode>"
+ [(set (reg:SVE_FULL_HF ZA_REGNUM)
+ (unspec:SVE_FULL_HF
+ [(reg:VNx4SI_ONLY ZA_REGNUM)
+ (reg:DI SME_STATE_REGNUM)
+ (match_operand:DI 0 "aarch64_imm2")
+ (match_operand:<SVE_FULL_HF:VDOUBLE> 1 "aligned_register_operand"
"Uw2")
+ (match_operand:SVE_FULL_HF 2 "register_operand" "w")
+ (match_operand:VNx16QI 3 "register_operand" "Uwo")
+ (match_operand:DI 4 "aarch64_imm2")
+ ]
+ SME_TMOP_FP))]
+ "TARGET_STREAMING_SME_TMOP"
+ "<SVE_FULL_HF:b><optab>\tza%0.s, %1, %2.<SVE_FULL_HF:Vetype>, %3[%4]"
+)
+
+;; svtmopa_lane_za16[_mf8_mf8]_fpm
+;; svtmopa_lane_za32[_mf8_mf8]_fpm
+(define_insn "@aarch64_sme_lane_<optab><SME_TMOPA_FP8:mode><VNx16QI_ONLY:mode>"
+ [(set (reg:SME_TMOPA_FP8 ZA_REGNUM)
+ (unspec:SME_TMOPA_FP8
+ [(reg:SME_TMOPA_FP8 ZA_REGNUM)
+ (reg:DI SME_STATE_REGNUM)
+ (match_operand:DI 0 "aarch64_imm<SME_TMOPA_FP8:za_imm_bits>")
+ (match_operand:<VNx16QI_ONLY:VDOUBLE> 1 "aligned_register_operand"
"Uw2")
+ (match_operand:VNx16QI_ONLY 2 "register_operand" "w")
+ (match_operand:VNx16QI 3 "register_operand" "Uwo")
+ (match_operand:DI 4 "aarch64_imm2")
+ (reg:DI FPM_REGNUM)
+ ]
+ SME_TMOP_FP8))]
+ "TARGET_STREAMING_SME_TMOP"
+ "<optab>\tza%0.<SME_TMOPA_FP8:Vetype>, %1, %2.b, %3[%4]"
+)
+
;; =========================================================================
;; == Table lookup
;; =========================================================================
<snip>
diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md
index 8bfa6429026..9ca59218a36 100644
--- a/gcc/config/aarch64/iterators.md
+++ b/gcc/config/aarch64/iterators.md
@@ -859,6 +859,13 @@ (define_mode_iterator SME_MOP_HSDF [VNx4SF
(VNx8HF "TARGET_STREAMING_SME_F16F16")
(VNx8BF "TARGET_STREAMING_SME_B16B16")])
+(define_mode_iterator SME_TMOPA_BHSF [(VNx8HF "TARGET_STREAMING_SME_F16F16")
+ (VNx8BF "TARGET_STREAMING_SME_B16B16")
+ VNx4SF])
+
+(define_mode_iterator SME_TMOPA_FP8 [(VNx8HI "TARGET_STREAMING_SME_F8F16")
+ (VNx4SI "TARGET_STREAMING_SME_F8F32")])
+
;; ------------------------------------------------------------------
;; Unspec enumerations for Advance SIMD. These could well go into
;; aarch64.md but for their use in int_iterators here.
@@ -1367,6 +1374,8 @@ (define_c_enum "unspec"
UNSPEC_SME_FMOPA
UNSPEC_SME_FMOPS
UNSPEC_SME_FSUB
+ UNSPEC_SME_FTMOPA
+ UNSPEC_SME_FTMOPA_FP8
UNSPEC_SME_LD1_HOR
UNSPEC_SME_LD1_VER
UNSPEC_SME_READ
@@ -1383,12 +1392,14 @@ (define_c_enum "unspec"
UNSPEC_SME_SMOPS
UNSPEC_SME_ST1_HOR
UNSPEC_SME_ST1_VER
+ UNSPEC_SME_STMOPA
UNSPEC_SME_SUB
UNSPEC_SME_SUB_WRITE
UNSPEC_SME_SUDOT
UNSPEC_SME_SUVDOT
UNSPEC_SME_SUMOPA
UNSPEC_SME_SUMOPS
+ UNSPEC_SME_SUTMOPA
UNSPEC_SME_UDOT
UNSPEC_SME_UVDOT
UNSPEC_SME_UMLA
@@ -1399,6 +1410,8 @@ (define_c_enum "unspec"
UNSPEC_SME_USVDOT
UNSPEC_SME_USMOPA
UNSPEC_SME_USMOPS
+ UNSPEC_SME_USTMOPA
+ UNSPEC_SME_UTMOPA
UNSPEC_SME_WRITE
UNSPEC_SME_WRITE_HOR
UNSPEC_SME_WRITE_VER
@@ -2970,6 +2983,12 @@ (define_mode_attr za16_32_last_offset [(VNx4SI "3")(VNx8HI
"1")])
(define_mode_attr za32_last_offset [(VNx16QI "3") (VNx32QI "3") (VNx64QI "3")
(VNx8HI "1") (VNx16HI "1") (VNx32HI "1")])
+;; The number of bits required to specify a ZA tile of a certain element size.
+(define_mode_attr za_imm_bits [(VNx8HI "1") (VNx4SI "2") (VNx2DI "3")
+ (VNx1TI "4")
+ (VNx8BF "1") (VNx4BF "2") (VNx2BF "3")
Of the BF modes, I think only VNx8BF should be needed here: we only use
fully-packed SVE modes to reference ZA.
OK with those changes, thanks!
