On Tue, 15 Sep 2026, Tamar Christina wrote: > > -----Original Message----- > > From: Richard Biener <[email protected]> > > Sent: 15 September 2026 12:50 > > To: Tamar Christina <[email protected]> > > Cc: Ezra Sitorus <[email protected]>; [email protected]; > > [email protected] > > Subject: RE: [RFC PATCH 3/5] gcc: Add new COMPLEX_DOT_PROD_ROT0 > > IFN/optabs > > > > On Tue, 15 Sep 2026, Tamar Christina wrote: > > > > > > -----Original Message----- > > > > From: Richard Biener <[email protected]> > > > > Sent: 15 September 2026 10:16 > > > > To: Tamar Christina <[email protected]> > > > > Cc: Ezra Sitorus <[email protected]>; [email protected]; > > > > [email protected] > > > > Subject: RE: [RFC PATCH 3/5] gcc: Add new COMPLEX_DOT_PROD_ROT0 > > > > IFN/optabs > > > > > > > > On Tue, 15 Sep 2026, Tamar Christina wrote: > > > > > > > > > > -----Original Message----- > > > > > > From: Richard Biener <[email protected]> > > > > > > Sent: 15 September 2026 09:23 > > > > > > To: Tamar Christina <[email protected]> > > > > > > Cc: Ezra Sitorus <[email protected]>; [email protected]; > > > > > > [email protected] > > > > > > Subject: RE: [RFC PATCH 3/5] gcc: Add new > > COMPLEX_DOT_PROD_ROT0 > > > > > > IFN/optabs > > > > > > > > > > > > On Tue, 15 Sep 2026, Tamar Christina wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: [email protected] <[email protected]> > > > > > > > > Sent: 09 September 2026 19:12 > > > > > > > > To: [email protected] > > > > > > > > Cc: Tamar Christina <[email protected]>; > > [email protected]; > > > > > > > > [email protected]; Ezra Sitorus > > > > <[email protected]> > > > > > > > > Subject: [RFC PATCH 3/5] gcc: Add new > > COMPLEX_DOT_PROD_ROT0 > > > > > > > > IFN/optabs > > > > > > > > > > > > > > > > From: Ezra Sitorus <[email protected]> > > > > > > > > > > > > > > > > Add new internal function COMPLEX_DOT_PROD_ROT0 and optab > > > > > > > > cdot_prod_rot0_optab > > > > > > > > to represent complex integer dot product. > > > > > > > > > > > > > > > > COMPLEX_DOT_PROD_ROT0 (A, B, ACC) is a lane-reducing signed > > > > integer > > > > > > > > complex > > > > > > > > dot product. The 2 input vectors contain interleaved W-bit real > > > > > > > > and > > > > > > imaginary > > > > > > > > components in even/odd numbered lanes respectively. Each 4W-bit > > > > result > > > > > > > > element > > > > > > > > accumulates the 0-rotation products of the 2 complex input > > > > > > > > pairs in > > the > > > > > > > > corresponding group of 4 input lanes: > > > > > > > > > > > > > > > > result[j] = acc[j] > > > > > > > > + a[k].re * b[k].re > > > > > > > > - a[k].im * b[k].im > > > > > > > > + a[k + 1].re * b[k + 1].re > > > > > > > > - a[k + 1].im * b[k + 1].im > > > > > > > > > > > > > > > > where 2*j = k. > > > > > > > > > > > > > > > > 0-rotation means that the imaginary components in the second > > source > > > > > > vector > > > > > > > > are > > > > > > > > negated, and so the overall operation is accumulating the real > > > > > > > > part > > of > > > > the > > > > > > > > complex dot product. Other rotations can be added in future > > patches. > > > > > > > > > > > > I other places we use _conj, not rot0: > > > > > > > > > > > > OPTAB_D (cmul_conj_optab, "cmul_conj$a3") > > > > > > > > > > > > so why not use that for consistency? > > > > > > > > > > That's a different operation. complex dotprod is numerically > > > > > > > > Sure, I meant why cdot_prod_rot0 rather than cdot_prod_conj, > > > > complex conjugate is exactly negating the imaginary part? > > > > > > Because the existing cmul/cadd/cdot *_conj optabs are intended to > > > represent the full mathematical complex operation which create both > > > the real and imaginary results. > > > > > > Yes, cdot #0 conjugates the second operand in the sense that it uses > > > ar * br - ai * bi. But it only computes the real component of the > > > complex dot product. The imaginary component is not computed. > > > > Oh, but then the name cdot_* is even more confusing? Why even bother > > with naming it "complex" then? It's not even a dot product? > > Agreed! Hence my ask for Ezra to document the operations so we could see > what they all mean. > > > > > res += a[i].real * b[i].real > > - a[i].imag * b[i].imag; > > > > that suggests cdot0 computes > > > > res + x * y - z * w > > > > or is it, for vectors, even more complicates as > > > > res + x[2*n] * y[2*n] - x[2*n+1] * y[2*n+1] > > > > ? Aka it's an alternating plus/minus non-"complex" dot-product > > > > (res + x[2*n] * y[2*n]) - x[2*n+1] * y[2*n+1] > > Yeah. > > > > > as if negating every other lane in x[] and then using a regular > > (non-complex) dot product? And it's also a reduction, so > > reduc_dot_<?>_scal? with <?> standing for whatever we invent > > for the alternating negation (we have vec_addsub and vec_fmaddsub > > and vec_fmsubadd for example, but that's not reductions). > > > > Yeah, reduc_dot_addsub_scal sounds quite funky hehe. Is the "dot" in > that case relevant? Why not just reduc_dot_addsub_scal? But that said > my issue is still I'm not sure this scales to the other operations. > > Like rot90 is > > res + x[2*n] * y[2*n+1] + x[2*n+1] * y[2*n] > > so do we do reduc_dot_addadd_scal or reduc_addadd_scal? That's why > we just kept the rot0 in the name before on BZ.
So there's still the question - is the "accumulator" really a scalar or is it a vector? From reading the SVE2 'CDOT' docs it appears that it's a vector but with half of the lanes as the input vectors? So _scal would be inappropriate. Instead it would be a lane-reducing (but not widening? CDOT is an integer vector operation?) operation? In which case reduc_dot_addsub. Or if we want to stick with 'cdot' and 'rot' maybe cdot_rot0_real and while the docs say rot90 is _imag it's rotated to real position, so cdot_rot90_real as well (only _real would remain). Or cdot_real and cdot_imag. But then the other rotates would have to be _conj in addition to _real and _imag? Bah. But it's still lane-reducing(?), so reduc_cdot_rot0_real? :/ > > I see you are following the aarch64 SVE2 'CDOT' instruction > > notation with its #rotate specification. I'm a bit unsure > > how to translate that appropriately also given the unfortunate(?) > > choice to compute a "packed" real part from two-component inputs. > > > > Yeah there's also the unfortunate issue that extracting just the real > results still requires a vec_select due to the pack representation. > > Thanks, > Tamar > > > > > > For example, cmul_conj today lowers to both: > > > > > > fcmla #180 > > > fcmla #270 > > > > > > because both operations are needed to compute the full complex multiply > > > by a conjugated operand. > > > > > > Similarly, computing both components of a complex dot product would need > > > Two instructions and rotations: > > > > > > cdot #0 > > > cdot #90 > > > > > > So naming the single cdot #0 operation cdot_prod_conj would be > > > misleading: it suggests the optab computes the full conjugated complex > > > dot product, whereas it only computes one rotated component. > > > > > > This is the same distinction as for complex multiply rotations, discussed > > > in PR121925, where we settled on the "rot" suffix for the lane-rotation > > forms. > > > > > > This isn't dissimilar to complex multiply and rotate which we discussed > > > here > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=121925 and settled on the > > "rot" suffix for them. > > > > > > Thanks, > > > Tamar > > > > > > > > > #include <stdint.h> > > > > > > > > > > typedef struct { > > > > > int8_t real; > > > > > int8_t imag; > > > > > } cint16_t; > > > > > > > > > > typedef struct { > > > > > int32_t real; > > > > > int32_t imag; > > > > > } cint32_t; > > > > > > > > > > cint32_t cdot(cint16_t *a, cint16_t *b, uint32_t N) > > > > > { > > > > > cint32_t res = {0, 0}; > > > > > > > > > > for (uint32_t i = 0; i < N; i++) { > > > > > res.real += a[i].real * b[i].real > > > > > - a[i].imag * b[i].imag; > > > > > > > > > > res.imag += a[i].real * b[i].imag > > > > > + a[i].imag * b[i].real; > > > > > } > > > > > > > > > > return res; > > > > > } > > > > > > > > > > Which would be what we'd call cdot. > > > > > > > > > > cdot_conj would be > > > > > > > > > > #include <stdint.h> > > > > > > > > > > typedef struct { > > > > > int8_t real; > > > > > int8_t imag; > > > > > } cint16_t; > > > > > > > > > > typedef struct { > > > > > int32_t real; > > > > > int32_t imag; > > > > > } cint32_t; > > > > > > > > > > cint32_t cdot_conj(const cint16_t *a, > > > > > const cint16_t *b, > > > > > uint32_t N) > > > > > { > > > > > cint32_t res = {0, 0}; > > > > > > > > > > for (uint32_t i = 0; i < N; i++) { > > > > > res.real += a[i].real * b[i].real > > > > > + a[i].imag * b[i].imag; > > > > > > > > > > res.imag += a[i].real * b[i].imag > > > > > - a[i].imag * b[i].real; > > > > > } > > > > > > > > > > return res; > > > > > } > > > > > > > > > > i.e. normal dual lane operations. > > > > > > > > > > however this patch recognizes the single lane reduction, > > > > > > > > > > #include <stdint.h> > > > > > > > > > > typedef struct { > > > > > int8_t real; > > > > > int8_t imag; > > > > > } cint16_t; > > > > > > > > > > int32_t cdot0(const cint16_t *a, const cint16_t *b, uint32_t N) > > > > > { > > > > > int32_t res = 0; > > > > > > > > > > for (uint32_t i = 0; i < N; i++) > > > > > res += a[i].real * b[i].real > > > > > - a[i].imag * b[i].imag; > > > > > > > > > > return res; > > > > > } > > > > > > > > > > Which is not numerically a complex dot product. Note that the end > > > > > goal > > > > > is to recognize all of these. > > > > > > > > > > Thanks, > > > > > Tamar > > > > > > > > > > > > > > In internal-fn.cc: complex_dot_prod_direct is defined to { -1, > > > > > > > > 1, false > > } > > > > > > > > because it uses the return type as the accumulator/result type > > > > > > > > and > > the > > > > > > input > > > > > > > > type is either of the narrower source type. > > > > > > > > > > > > > > > > In AArch64, this corresponds to the CDOT instruction. This patch > > also > > > > > > teaches > > > > > > > > GCC to convert the optab into the correct define_insn pattern. > > > > > > > > > > > > > > > > gcc/ChangeLog: > > > > > > > > * config/aarch64/aarch64-sve2.md > > > > > > > > (cdot_prod_rot0<mode><vsi2qi>): New > > > > > > > > define_expand. > > > > > > > > * internal-fn.cc (complex_dot_prod_direct): New static > > initializers. > > > > > > > > * internal-fn.def (COMPLEX_DOT_PROD_ROT0) New internal > > > > > > > > function. > > > > > > > > * optabs.def (cdot_prod_rot0_optab): New optab. > > > > > > > > --- > > > > > > > > gcc/config/aarch64/aarch64-sve2.md | 10 ++++++++++ > > > > > > > > gcc/internal-fn.cc | 5 +++++ > > > > > > > > gcc/internal-fn.def | 1 + > > > > > > > > gcc/optabs.def | 1 + > > > > > > > > > > > > > > The new optabs and IFN need to be documented in ifn.texi and > > md.texi > > > > > > > and split the aarch64 implementation to it's own patch. > > > > > > > > > > > > > > But why not add the other rotations at the same time? > > > > > > > > > > > > > > We'd likely also later (in a different patch) need cdot_prod and > > > > > > cdot_prod_conj, > > > > > > > because then both lanes are visible the SLP tree will look much > > different. > > > > i.e. > > > > > > > > > > > > > > #include <stdint.h> > > > > > > > > > > > > > > typedef _Complex int8_t cint16_t; > > > > > > > typedef _Complex int32_t cint32_t; > > > > > > > > > > > > > > cint32_t cdot(cint16_t *a, cint16_t *b, uint32_t N) > > > > > > > { > > > > > > > cint32_t res = 0; > > > > > > > > > > > > > > for (uint32_t i = 0; i < N; i++) > > > > > > > res += a[i] * b[i]; > > > > > > > > > > > > > > return res; > > > > > > > } > > > > > > > > > > > > > > > 4 files changed, 17 insertions(+) > > > > > > > > > > > > > > > > diff --git a/gcc/config/aarch64/aarch64-sve2.md > > > > > > > > b/gcc/config/aarch64/aarch64-sve2.md > > > > > > > > index c33df283cc2a..ad4febb60bf8 100644 > > > > > > > > --- a/gcc/config/aarch64/aarch64-sve2.md > > > > > > > > +++ b/gcc/config/aarch64/aarch64-sve2.md > > > > > > > > @@ -3578,6 +3578,16 @@ > > > > > > > > ;; - CDOT > > > > > > > > ;; > > > > > > > > ------------------------------------------------------------------------- > > > > > > > > > > > > > > > > +(define_expand "cdot_prod_rot0<mode><vsi2qi>" > > > > > > > > + [(set (match_operand:SVE_FULL_SDI 0 "register_operand") > > > > > > > > + (unspec:SVE_FULL_SDI > > > > > > > > + [(match_operand:SVE_FULL_SDI 3 "register_operand") > > > > > > > > + (match_operand:<VSI2QI> 1 "register_operand") > > > > > > > > + (match_operand:<VSI2QI> 2 "register_operand")] > > > > > > > > + UNSPEC_CDOT))] > > > > > > > > + "TARGET_SVE2" > > > > > > > > +) > > > > > > > > + > > > > > > > > (define_insn "@aarch64_sve_<optab><mode>" > > > > > > > > [(set (match_operand:SVE_FULL_SDI 0 "register_operand") > > > > > > > > (unspec:SVE_FULL_SDI > > > > > > > > diff --git a/gcc/internal-fn.cc b/gcc/internal-fn.cc > > > > > > > > index c57b594ac591..cc968109261b 100644 > > > > > > > > --- a/gcc/internal-fn.cc > > > > > > > > +++ b/gcc/internal-fn.cc > > > > > > > > @@ -180,6 +180,7 @@ init_internal_fns () > > > > > > > > #define unary_convert_direct { -1, 0, true } > > > > > > > > #define binary_direct { 0, 0, true } > > > > > > > > #define ternary_direct { 0, 0, true } > > > > > > > > +#define complex_dot_prod_direct { -1, 1, false } > > > > > > > > #define cond_unary_direct { 1, 1, true } > > > > > > > > #define cond_binary_direct { 1, 1, true } > > > > > > > > #define cond_ternary_direct { 1, 1, true } > > > > > > > > @@ -4315,6 +4316,9 @@ expand_reduc_sbool_optab_fn > > (internal_fn > > > > fn, > > > > > > > > gcall *stmt, direct_optab optab) > > > > > > > > #define expand_select_vl_optab_fn(FN, STMT, OPTAB) \ > > > > > > > > expand_convert_optab_fn (FN, STMT, OPTAB, 3) > > > > > > > > > > > > > > > > +#define expand_complex_dot_prod_optab_fn(FN, STMT, OPTAB) \ > > > > > > > > + expand_convert_optab_fn (FN, STMT, OPTAB, 3) > > > > > > > > + > > > > > > > > /* Expanders for optabs that can use expand_convert_optab_fn. > > > > > > > > */ > > > > > > > > > > > > > > > > #define expand_unary_convert_optab_fn(FN, STMT, OPTAB) \ > > > > > > > > @@ -4398,6 +4402,7 @@ multi_vector_optab_supported_p > > > > > > (convert_optab > > > > > > > > optab, tree_pair types, > > > > > > > > #define direct_unary_convert_optab_supported_p > > > > > > > > convert_optab_supported_p > > > > > > > > #define direct_binary_optab_supported_p > > direct_optab_supported_p > > > > > > > > #define direct_ternary_optab_supported_p > > direct_optab_supported_p > > > > > > > > +#define direct_complex_dot_prod_optab_supported_p > > > > > > > > convert_optab_supported_p > > > > > > > > #define direct_cond_unary_optab_supported_p > > > > > > direct_optab_supported_p > > > > > > > > #define direct_cond_binary_optab_supported_p > > > > > > direct_optab_supported_p > > > > > > > > #define direct_cond_ternary_optab_supported_p > > > > > > direct_optab_supported_p > > > > > > > > diff --git a/gcc/internal-fn.def b/gcc/internal-fn.def > > > > > > > > index af9f92950c73..08bdedc30045 100644 > > > > > > > > --- a/gcc/internal-fn.def > > > > > > > > +++ b/gcc/internal-fn.def > > > > > > > > @@ -453,6 +453,7 @@ DEF_INTERNAL_OPTAB_FN > > > > > > (COMPLEX_ADD_ROT90, > > > > > > > > ECF_CONST, cadd90, binary) > > > > > > > > DEF_INTERNAL_OPTAB_FN (COMPLEX_ADD_ROT270, ECF_CONST, > > > > > > cadd270, > > > > > > > > binary) > > > > > > > > DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL, ECF_CONST, cmul, > > > > binary) > > > > > > > > DEF_INTERNAL_OPTAB_FN (COMPLEX_MUL_CONJ, ECF_CONST, > > > > > > cmul_conj, > > > > > > > > binary) > > > > > > > > +DEF_INTERNAL_OPTAB_FN (COMPLEX_DOT_PROD_ROT0, > > > > ECF_CONST, > > > > > > > > cdot_prod_rot0, complex_dot_prod) > > > > > > > > DEF_INTERNAL_OPTAB_FN (VEC_ADDSUB, ECF_CONST, > > vec_addsub, > > > > > > binary) > > > > > > > > DEF_INTERNAL_OPTAB_FN (VEC_TRUNC_ADD_HIGH, ECF_CONST | > > > > > > > > ECF_NOTHROW, > > > > > > > > vec_trunc_add_high, binary) > > > > > > > > diff --git a/gcc/optabs.def b/gcc/optabs.def > > > > > > > > index 9d95f047cb50..b30e2bec6b2b 100644 > > > > > > > > --- a/gcc/optabs.def > > > > > > > > +++ b/gcc/optabs.def > > > > > > > > @@ -114,6 +114,7 @@ OPTAB_CD(vec_init_optab, "vec_init$a$b") > > > > > > > > OPTAB_CD (sdot_prod_optab, "sdot_prod$I$a$b") > > > > > > > > OPTAB_CD (udot_prod_optab, "udot_prod$I$a$b") > > > > > > > > OPTAB_CD (usdot_prod_optab, "usdot_prod$I$a$b") > > > > > > > > +OPTAB_CD (cdot_prod_rot0_optab, "cdot_prod_rot0$I$a$b") > > > > > > > > > > > > > > > > OPTAB_CD (while_ult_optab, "while_ult$a$b") > > > > > > > > OPTAB_CD (select_vl_optab, "select_vl$a$b") > > > > > > > > -- > > > > > > > > 2.53.0 > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Richard Biener <[email protected]> > > > > > > SUSE Software Solutions Germany GmbH, > > > > > > Frankenstrasse 146, 90461 Nuernberg, Germany; > > > > > > GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG > > Nuernberg) > > > > > > > > > > > > > -- > > > > Richard Biener <[email protected]> > > > > SUSE Software Solutions Germany GmbH, > > > > Frankenstrasse 146, 90461 Nuernberg, Germany; > > > > GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG Nuernberg) > > > > > > > -- > > Richard Biener <[email protected]> > > SUSE Software Solutions Germany GmbH, > > Frankenstrasse 146, 90461 Nuernberg, Germany; > > GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG Nuernberg) > -- Richard Biener <[email protected]> SUSE Software Solutions Germany GmbH, Frankenstrasse 146, 90461 Nuernberg, Germany; GF: Stefan Gaiser, Jochen Jaser, Abhinav Puri; (HRB 36809, AG Nuernberg)
