On 19/08/2026 10:20, Claudio Bantaloukas wrote: > This patch adds support for the intrinsics implementing FEAT_SME_TMOP listed > below. All of these require the +sme-tmop arch option. > A new intrinsic shape and a new register constraint is required. This patch > adds these, along with tests. Also added a pair of predicates to tighten the > immediates selecting ZA tiles based on tile width. > > Specifically: > - svtmopa_lane_za16[_mf8_mf8] (requires +sme-f8f16) > - svtmopa_lane_za16[_bf16_bf16] (requires +sme-b16b16) > - svtmopa_lane_za16[_f16_f16] (requires +sme-f16f16) > - svtmopa_lane_za32[_mf8_mf8] (requires +sme-f8f32) > - svtmopa_lane_za32[_bf16_bf16] (requires +sme-b16b16) > - svtmopa_lane_za32[_f16_f16] > - svtmopa_lane_za32[_f32_f32] > - svtmopa_lane_za32[_s16_s16] > - svtmopa_lane_za32[_u16_u16] > - svtmopa_lane_za32[_s8_s8] > - svtmopa_lane_za32[_u8_u8] > - svtmopa_lane_za32[_s8_u8] > - svtmopa_lane_za32[_u8_s8] > > gcc/ > * config/aarch64/aarch64-acle-builtins.cc: > (function_resolver::resolve_to): Fix documentation comment. > * config/aarch64/aarch64-acle-builtins.h: (TYPES_tmop_base, > TYPES_tmop_h_bfloat, TYPES_tmop_h_float, TYPES_tmop_h_mf8, > TYPES_tmop_s_mf8): Add new DEF_SVE_TYPES_ARRAYs. > * config/aarch64/aarch64-c.cc: (__ARM_FEATURE_SME_TMOP): Add define. > * config/aarch64/aarch64-sme.md: > (@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_BHI:mode>): Add > new insn. > (@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><VNx16QI_ONLY:mode>): > Likewise. > (@aarch64_sme_lane_<optab><mode><mode>): > Likewise. > (@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_HF:mode>): > Likewise. > (@aarch64_sme_lane_<optab><SME_TMOPA_FP8:mode><VNx16QI_ONLY:mode>): > Likewise. > * config/aarch64/aarch64-sve-builtins-shapes.cc: > (ternary_za_uint_dual_single): Add new shape. > * config/aarch64/aarch64-sve-builtins-shapes.h: Likewise. > * config/aarch64/aarch64-sve-builtins-sme.cc: (svtmopa_lane_za_impl): > Add new function impl class. > (svtmopa_lane_za): Add new FUNCTION. > * config/aarch64/aarch64-sve-builtins-sme.def: (svtmopa_lane): Add new > DEF_SME_ZA_FUNCTION_GS entries. > * config/aarch64/aarch64-sve-builtins-sme.h: (svtmopa_lane_za): Add new. > * config/aarch64/aarch64.h: (TARGET_STREAMING_SME_TMOP): Add new define. > * config/aarch64/constraints.md (Uwo): Add new constraint for Control > Vector Register in TMOP operations. > * config/aarch64/iterators.md: (SME_TMOPA_BHSF, SME_TMOPA_FP8): Add new > mode > iterators. > (UNSPEC_SME_FTMOPA, UNSPEC_SME_FTMOPA_FP8, UNSPEC_SME_STMOPA, > UNSPEC_SME_SUTMOPA, UNSPEC_SME_USTMOPA, UNSPEC_SME_UTMOPA): Add new > unspecs. > (SME_TMOP_INT, SME_TMOP_INT_CROSS, SME_TMOP_FP, SME_TMOP_FP8): Add new > iterators. > (za_imm_bits): Add new mode_attr. > (optab): Update with new unspecs. > * config/aarch64/predicates.md: (aarch64_imm1, aarch64_imm4): Add new > predicates. > > gcc/testsuite/ > * gcc.target/aarch64/pragma_cpp_predefs_6.c: Add tests checking that the > sme-tmop prefef is off by default, and checks for feature dependencies. > * gcc.target/aarch64/sme2/acle-asm/test_sme2_acle.h: (TEST_ZA_TMOP) Add > testing macro. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c: > New test. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_f16_f16.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f16_f16.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f32_f32.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_s8.c: Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_u8.c: Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s16_s16.c: > Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_s8.c: Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_u8.c: Likewise. > * gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u16_u16.c: > Likewise. > * gcc.target/aarch64/sve/acle/general-c/ternary_za_uint_dual_single_1.c: > Likewise. > * lib/target-supports.exp: Add check_effective_target support for > sme-tmop. > --- > gcc/config/aarch64/aarch64-acle-builtins.cc | 2 +- > gcc/config/aarch64/aarch64-acle-builtins.h | 29 +++++ > gcc/config/aarch64/aarch64-c.cc | 2 + > gcc/config/aarch64/aarch64-sme.md | 112 ++++++++++++++++++ > .../aarch64/aarch64-sve-builtins-shapes.cc | 39 ++++++ > .../aarch64/aarch64-sve-builtins-shapes.h | 1 + > .../aarch64/aarch64-sve-builtins-sme.cc | 36 ++++++ > .../aarch64/aarch64-sve-builtins-sme.def | 24 ++++ > gcc/config/aarch64/aarch64-sve-builtins-sme.h | 1 + > gcc/config/aarch64/aarch64.h | 3 + > gcc/config/aarch64/constraints.md | 4 + > gcc/config/aarch64/iterators.md | 30 +++++ > gcc/config/aarch64/predicates.md | 11 ++ > .../gcc.target/aarch64/pragma_cpp_predefs_6.c | 17 +++ > .../aarch64/sme2/acle-asm/test_sme2_acle.h | 18 +++ > .../sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za16_f16_f16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c | 83 +++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_f16_f16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_f32_f32.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c | 83 +++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_s16_s16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_s8_s8.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_s8_u8.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_u16_u16.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_u8_s8.c | 76 ++++++++++++ > .../sme2/acle-asm/tmopa_lane_za32_u8_u8.c | 76 ++++++++++++ > .../general-c/ternary_za_uint_dual_single_1.c | 87 ++++++++++++++ > gcc/testsuite/lib/target-supports.exp | 1 + > 30 files changed, 1418 insertions(+), 1 deletion(-) > create mode 100644 gcc/testsuite/gcc.target/aarch64/pragma_cpp_predefs_6.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_bf16_bf16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_f16_f16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za16_mf8_mf8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_bf16_bf16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f16_f16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_f32_f32.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_mf8_mf8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s16_s16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_s8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_s8_u8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u16_u16.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_s8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sme2/acle-asm/tmopa_lane_za32_u8_u8.c > create mode 100644 > gcc/testsuite/gcc.target/aarch64/sve/acle/general-c/ternary_za_uint_dual_single_1.c > <snip> > diff --git a/gcc/config/aarch64/aarch64-sme.md > b/gcc/config/aarch64/aarch64-sme.md > index 7091f566ba8..fa4c3016e1d 100644 > --- a/gcc/config/aarch64/aarch64-sme.md > +++ b/gcc/config/aarch64/aarch64-sme.md > @@ -46,11 +46,13 @@ > ;; ---- [INT] Dot product > ;; ---- [INT] Ternary widening arithmetic on ZA slice > ;; ---- [INT] Sum of outer products > +;; ---- [INT] Sparse outer product > ;; ---- [FP] Dot product > ;; ---- [FP8] Dot product > ;; ---- [FP] Ternary arithmetic on ZA slice > ;; ---- [FP] Ternary widening arithmetic on ZA slice > ;; ---- [FP] Sum of outer products > +;; ---- [FP] Sparse outer product > ;; > ;; == Table lookup > ;; ---- Table lookup > @@ -1814,6 +1816,53 @@ (define_insn > "@aarch64_sme_<optab><VNx4SI_ONLY:mode><VNx4SI_ONLY:mode>" > "<optab>\tza%0.s, %1/m, %2/m, %3.s, %4.s" > ) > > +;; ------------------------------------------------------------------------- > +;; ---- [INT] Sparse outer product > +;; ------------------------------------------------------------------------- > +;; Includes: > +;; - STMOPA > +;; - UTMOPA > +;; - SUTMOPA > +;; - USTMOPA > +;; ------------------------------------------------------------------------- > +;; svtmopa_lane_za32[_s16_s16] > +;; svtmopa_lane_za32[_u16_u16] > +;; svtmopa_lane_za32[_s8_s8] > +;; svtmopa_lane_za32[_u8_u8] > +(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_BHI:mode>" > + [(set (reg:VNx4SI_ONLY ZA_REGNUM) > + (unspec:VNx4SI_ONLY > + [(reg:VNx4SI_ONLY ZA_REGNUM) > + (reg:DI SME_STATE_REGNUM) > + (match_operand:DI 0 "aarch64_imm2") > + (match_operand:<SVE_FULL_BHI:VDOUBLE> 1 "aligned_register_operand" > "Uw2") > + (match_operand:SVE_FULL_BHI 2 "register_operand" "w") > + (match_operand:VNx16QI 3 "register_operand" "Uwo") > + (match_operand:DI 4 "aarch64_imm2") > + ] > + SME_TMOP_INT))] > + "TARGET_STREAMING_SME_TMOP" > + "<optab>\tza%0.s, %1, %2.<SVE_FULL_BHI:Vetype>, %3[%4]" > +) > + > +;; svtmopa_lane_za32[_s8_u8] > +;; svtmopa_lane_za32[_u8_s8] > +(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><VNx16QI_ONLY:mode>" > + [(set (reg:VNx4SI_ONLY ZA_REGNUM) > + (unspec:VNx4SI_ONLY > + [(reg:VNx4SI_ONLY ZA_REGNUM) > + (reg:DI SME_STATE_REGNUM) > + (match_operand:DI 0 "aarch64_imm2") > + (match_operand:<VNx16QI_ONLY:VDOUBLE> 1 "aligned_register_operand" > "Uw2") > + (match_operand:VNx16QI_ONLY 2 "register_operand" "w") > + (match_operand:VNx16QI 3 "register_operand" "Uwo") > + (match_operand:DI 4 "aarch64_imm2") > + ] > + SME_TMOP_INT_CROSS))] > + "TARGET_STREAMING_SME_TMOP" > + "<optab>\tza%0.s, %1, %2.<VNx16QI_ONLY:Vetype>, %3[%4]" > +) > + > ;; ------------------------------------------------------------------------- > ;; ---- [FP] Dot product > ;; ------------------------------------------------------------------------- > @@ -2737,6 +2786,69 @@ (define_insn > "@aarch64_sme_<optab><SME_ZA_F8F16_32:mode><VNx16QI_ONLY:mode>" > "<optab>\tza%0.<SME_ZA_F8F16_32:Vetype>, %1/m, %2/m, %3.b, %4.b" > ) > > +;; ------------------------------------------------------------------------- > +;; ---- [FP] Sparse outer product > +;; ------------------------------------------------------------------------- > +;; Includes: > +;; - BFTMOPA (SME_TMOP) > +;; - FTMOPA (SME_TMOP) > +;; ------------------------------------------------------------------------- > +;; svtmopa_lane_za16[_bf16_bf16] > +;; svtmopa_lane_za16[_f16_f16] > +;; svtmopa_lane_za32[_f32_f32] > +(define_insn "@aarch64_sme_lane_<optab><mode><mode>" > + [(set (reg:SME_TMOPA_BHSF ZA_REGNUM) > + (unspec:SME_TMOPA_BHSF > + [(reg:SME_TMOPA_BHSF ZA_REGNUM) > + (reg:DI SME_STATE_REGNUM) > + (match_operand:DI 0 "aarch64_imm<SME_TMOPA_BHSF:za_imm_bits>")
Does this work without qualifying the mode iterator (I would have thought so, given there is only a single mode iterator). If it works, I'd prefer to drop it. > + (match_operand:<VDOUBLE> 1 "aligned_register_operand" "Uw2") > + (match_operand:SME_TMOPA_BHSF 2 "register_operand" "w") > + (match_operand:VNx16QI 3 "register_operand" "Uwo") > + (match_operand:DI 4 "aarch64_imm2") > + ] > + SME_TMOP_FP))] > + "TARGET_STREAMING_SME_TMOP" > + "<b><optab>\tza%0.<Vetype>, %1, %2.<Vetype>, %3[%4]" > +) > + > +;; svtmopa_lane_za32[_bf16_bf16] > +;; svtmopa_lane_za32[_f16_f16] > +(define_insn "@aarch64_sme_lane_<optab><VNx4SI_ONLY:mode><SVE_FULL_HF:mode>" > + [(set (reg:SVE_FULL_HF ZA_REGNUM) > + (unspec:SVE_FULL_HF > + [(reg:VNx4SI_ONLY ZA_REGNUM) > + (reg:DI SME_STATE_REGNUM) > + (match_operand:DI 0 "aarch64_imm2") > + (match_operand:<SVE_FULL_HF:VDOUBLE> 1 "aligned_register_operand" > "Uw2") > + (match_operand:SVE_FULL_HF 2 "register_operand" "w") > + (match_operand:VNx16QI 3 "register_operand" "Uwo") > + (match_operand:DI 4 "aarch64_imm2") > + ] > + SME_TMOP_FP))] > + "TARGET_STREAMING_SME_TMOP" > + "<SVE_FULL_HF:b><optab>\tza%0.s, %1, %2.<SVE_FULL_HF:Vetype>, %3[%4]" > +) > + > +;; svtmopa_lane_za16[_mf8_mf8]_fpm > +;; svtmopa_lane_za32[_mf8_mf8]_fpm > +(define_insn > "@aarch64_sme_lane_<optab><SME_TMOPA_FP8:mode><VNx16QI_ONLY:mode>" > + [(set (reg:SME_TMOPA_FP8 ZA_REGNUM) > + (unspec:SME_TMOPA_FP8 > + [(reg:SME_TMOPA_FP8 ZA_REGNUM) > + (reg:DI SME_STATE_REGNUM) > + (match_operand:DI 0 "aarch64_imm<SME_TMOPA_FP8:za_imm_bits>") > + (match_operand:<VNx16QI_ONLY:VDOUBLE> 1 "aligned_register_operand" > "Uw2") > + (match_operand:VNx16QI_ONLY 2 "register_operand" "w") > + (match_operand:VNx16QI 3 "register_operand" "Uwo") > + (match_operand:DI 4 "aarch64_imm2") > + (reg:DI FPM_REGNUM) > + ] > + SME_TMOP_FP8))] > + "TARGET_STREAMING_SME_TMOP" > + "<optab>\tza%0.<SME_TMOPA_FP8:Vetype>, %1, %2.b, %3[%4]" > +) > + > ;; ========================================================================= > ;; == Table lookup > ;; ========================================================================= <snip> > diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md > index 8bfa6429026..9ca59218a36 100644 > --- a/gcc/config/aarch64/iterators.md > +++ b/gcc/config/aarch64/iterators.md > @@ -859,6 +859,13 @@ (define_mode_iterator SME_MOP_HSDF [VNx4SF > (VNx8HF "TARGET_STREAMING_SME_F16F16") > (VNx8BF "TARGET_STREAMING_SME_B16B16")]) > > +(define_mode_iterator SME_TMOPA_BHSF [(VNx8HF "TARGET_STREAMING_SME_F16F16") > + (VNx8BF "TARGET_STREAMING_SME_B16B16") > + VNx4SF]) > + > +(define_mode_iterator SME_TMOPA_FP8 [(VNx8HI "TARGET_STREAMING_SME_F8F16") > + (VNx4SI "TARGET_STREAMING_SME_F8F32")]) > + > ;; ------------------------------------------------------------------ > ;; Unspec enumerations for Advance SIMD. These could well go into > ;; aarch64.md but for their use in int_iterators here. > @@ -1367,6 +1374,8 @@ (define_c_enum "unspec" > UNSPEC_SME_FMOPA > UNSPEC_SME_FMOPS > UNSPEC_SME_FSUB > + UNSPEC_SME_FTMOPA > + UNSPEC_SME_FTMOPA_FP8 > UNSPEC_SME_LD1_HOR > UNSPEC_SME_LD1_VER > UNSPEC_SME_READ > @@ -1383,12 +1392,14 @@ (define_c_enum "unspec" > UNSPEC_SME_SMOPS > UNSPEC_SME_ST1_HOR > UNSPEC_SME_ST1_VER > + UNSPEC_SME_STMOPA > UNSPEC_SME_SUB > UNSPEC_SME_SUB_WRITE > UNSPEC_SME_SUDOT > UNSPEC_SME_SUVDOT > UNSPEC_SME_SUMOPA > UNSPEC_SME_SUMOPS > + UNSPEC_SME_SUTMOPA > UNSPEC_SME_UDOT > UNSPEC_SME_UVDOT > UNSPEC_SME_UMLA > @@ -1399,6 +1410,8 @@ (define_c_enum "unspec" > UNSPEC_SME_USVDOT > UNSPEC_SME_USMOPA > UNSPEC_SME_USMOPS > + UNSPEC_SME_USTMOPA > + UNSPEC_SME_UTMOPA > UNSPEC_SME_WRITE > UNSPEC_SME_WRITE_HOR > UNSPEC_SME_WRITE_VER > @@ -2970,6 +2983,12 @@ (define_mode_attr za16_32_last_offset [(VNx4SI > "3")(VNx8HI "1")]) > (define_mode_attr za32_last_offset [(VNx16QI "3") (VNx32QI "3") (VNx64QI "3") > (VNx8HI "1") (VNx16HI "1") (VNx32HI "1")]) > > +;; The number of bits required to specify a ZA tile of a certain element > size. > +(define_mode_attr za_imm_bits [(VNx8HI "1") (VNx4SI "2") (VNx2DI "3") > + (VNx1TI "4") > + (VNx8BF "1") (VNx4BF "2") (VNx2BF "3") Of the BF modes, I think only VNx8BF should be needed here: we only use fully-packed SVE modes to reference ZA. OK with those changes, thanks! Alex > + (VNx8HF "1") (VNx4SF "2") (VNx2DF "3")]) > + > (define_mode_attr vg_modifier [(VNx16QI "") > (VNx32QI ", vgx2") > (VNx64QI ", vgx4") > @@ -4344,6 +4363,11 @@ (define_int_iterator SME_FP8_FVDOT_HALF [ > UNSPEC_SME_FVDOTT_FP8 > ]) > > +(define_int_iterator SME_TMOP_INT [UNSPEC_SME_STMOPA UNSPEC_SME_UTMOPA]) > +(define_int_iterator SME_TMOP_INT_CROSS [UNSPEC_SME_SUTMOPA > UNSPEC_SME_USTMOPA]) > +(define_int_iterator SME_TMOP_FP [UNSPEC_SME_FTMOPA]) > +(define_int_iterator SME_TMOP_FP8 [UNSPEC_SME_FTMOPA_FP8]) > + <snip>
