Hi Tamar, > On 19 Oct 2026, at 08:39, Tamar Christina <[email protected]> wrote: > > Hi Kyrill, > >> -----Original Message----- >> From: [email protected] <[email protected]> >> Sent: 06 October 2026 15:02 >> To: [email protected] >> Cc: Kyrylo Tkachov <[email protected]>; Tamar Christina >> <[email protected]>; Wilco Dijkstra <[email protected]> >> Subject: [PATCH] aarch64: Use standard RTL for FADDP >> >> From: Kyrylo Tkachov <[email protected]> >> >> The FADDP patterns use UNSPEC_FADDV, so RTL optimizations cannot see that >> FADDP adds pairs of elements. ADDP already uses standard RTL. Describe >> FADDP in the same way and let it share the ADDP patterns, which get >> parameterized names. >> >> For two-lane vectors, FADDP and the scalar FADDP reduction add the two >> elements of each input. RTL simplification describes the low element of >> a vector as a lowpart subreg, so use that form for these sums: >> >> (plus:SF (vec_select:SF (reg:V2SF x) (parallel [(const_int 1)])) >> (subreg:SF (reg:V2SF x) 0)) >> >> The insn condition checks that the subreg is the lowpart of the vector >> operand. The instruction only reads the vector operand. >> >> Combine can now use FADDP for the sum of the two elements of a vector, and >> pack two such sums into one FADDP. For example, with -O2: >> >> float32x2_t >> f (float32x2_t a, float32x2_t b) >> { >> return (float32x2_t) { vaddv_f32 (a), vaddv_f32 (b) }; >> } >> >> Before: >> faddp s1, v1.2s >> faddp s0, v0.2s >> zip1 v0.2s, v0.2s, v1.2s >> >> After: >> faddp v0.2s, v0.2s, v1.2s >> >> For x[0] + x[1], where x is a float32x2_t: >> >> Before: >> dup s31, v0.s[1] >> fadd s0, s31, s0 >> >> After: >> faddp s0, v0.2s >> >> Tests check lane extraction from FADDP results for both byte orders. The >> runtime test also passes under big-endian QEMU. >> >> Bootstrapped and tested on aarch64-none-linux-gnu. >> Ok for trunk? >> Thanks, >> Kyrill >> >> Cc: Tamar Christina <[email protected]> >> Cc: Wilco Dijkstra <[email protected]> >> >> gcc/ChangeLog: >> >> * config/aarch64/aarch64-protos.h (aarch64_simd_lowpart_of_p): >> Declare. >> * config/aarch64/aarch64.cc (aarch64_simd_lowpart_of_p): New >> function. >> * config/aarch64/iterators.md (VD_ADDP, VQ_ADDP, VDQ_ADDP): >> New mode >> iterators. >> (addp_type): New mode attribute. >> * config/aarch64/aarch64-simd.md >> (aarch64_faddp<mode><vczle><vczbe>): >> Remove. >> (reduc_plus_scal_<mode>): Use standard RTL for V2F. >> (aarch64_faddp_scal<mode>): New pattern. >> (aarch64_addp<mode>_insn): Rename to... >> (@aarch64_addp<mode>_concat): ...this. Handle FADDP. >> (aarch64_addp<mode><vczle><vczbe>_insn): Rename to... >> (@aarch64_addp<mode>_select<vczle><vczbe>): ...this. Handle >> FADDP. >> (@aarch64_faddp_scal_pair<mode><vczle><vczbe>): New pattern. >> (*aarch64_addp_same_reg<mode>): Handle FADDP. >> (aarch64_addp<mode>): Rename to... >> (aarch64_<f>addp<mode>): ...this. Handle FADDP. >> >> gcc/testsuite/ChangeLog: >> >> * gcc.target/aarch64/faddp-concat-1.c: New test. >> * gcc.target/aarch64/faddp-concat-3.c: Likewise. >> * gcc.target/aarch64/faddp-concat-be.c: Likewise. >> * gcc.target/aarch64/simd/faddp_lane_1.c: Likewise. >> * gcc.target/aarch64/simd/faddp_lane_2.c: Likewise. >> * gcc.target/aarch64/simd/faddp_reinterpret_1.c: Likewise. >> >> Signed-off-by: Kyrylo Tkachov <[email protected]> >> --- >> gcc/config/aarch64/aarch64-protos.h | 1 + >> gcc/config/aarch64/aarch64-simd.md | 183 ++++++++++++------ >> gcc/config/aarch64/aarch64.cc | 9 + >> gcc/config/aarch64/iterators.md | 20 ++ >> .../gcc.target/aarch64/faddp-concat-1.c | 64 ++++++ >> .../gcc.target/aarch64/faddp-concat-3.c | 25 +++ >> .../gcc.target/aarch64/faddp-concat-be.c | 51 +++++ >> .../gcc.target/aarch64/simd/faddp_lane_1.c | 76 ++++++++ >> .../gcc.target/aarch64/simd/faddp_lane_2.c | 53 +++++ >> .../aarch64/simd/faddp_reinterpret_1.c | 51 +++++ >> 10 files changed, 471 insertions(+), 62 deletions(-) >> create mode 100644 gcc/testsuite/gcc.target/aarch64/faddp-concat-1.c >> create mode 100644 gcc/testsuite/gcc.target/aarch64/faddp-concat-3.c >> create mode 100644 gcc/testsuite/gcc.target/aarch64/faddp-concat-be.c >> create mode 100644 gcc/testsuite/gcc.target/aarch64/simd/faddp_lane_1.c >> create mode 100644 gcc/testsuite/gcc.target/aarch64/simd/faddp_lane_2.c >> create mode 100644 >> gcc/testsuite/gcc.target/aarch64/simd/faddp_reinterpret_1.c >> >> diff --git a/gcc/config/aarch64/aarch64-protos.h >> b/gcc/config/aarch64/aarch64-protos.h >> index e8ae3d42794..b825ff5b584 100644 >> --- a/gcc/config/aarch64/aarch64-protos.h >> +++ b/gcc/config/aarch64/aarch64-protos.h >> @@ -963,6 +963,7 @@ bool aarch64_regno_ok_for_index_p (int, bool); >> bool aarch64_reinterpret_float_as_int (rtx value, unsigned HOST_WIDE_INT >> *fail); >> bool aarch64_simd_check_vect_par_cnst_half (rtx op, machine_mode mode, >> bool high); >> +bool aarch64_simd_lowpart_of_p (rtx, rtx); >> bool aarch64_parallel_select_half_p (machine_mode, rtx); >> bool aarch64_pars_overlap_p (rtx, rtx); >> bool aarch64_simd_scalar_immediate_valid_for_move (rtx, scalar_int_mode); >> diff --git a/gcc/config/aarch64/aarch64-simd.md >> b/gcc/config/aarch64/aarch64-simd.md >> index f928c5958cd..e1ce90bfafb 100644 >> --- a/gcc/config/aarch64/aarch64-simd.md >> +++ b/gcc/config/aarch64/aarch64-simd.md >> @@ -3760,16 +3760,6 @@ >> >> ;; 'across lanes' add. >> >> -(define_insn "aarch64_faddp<mode><vczle><vczbe>" >> - [(set (match_operand:VHSDF 0 "register_operand" "=w") >> - (unspec:VHSDF [(match_operand:VHSDF 1 "register_operand" "w") >> - (match_operand:VHSDF 2 "register_operand" "w")] >> - UNSPEC_FADDV))] >> - "TARGET_SIMD" >> - "faddp\t%0.<Vtype>, %1.<Vtype>, %2.<Vtype>" >> - [(set_attr "type" "neon_fp_reduc_add_<stype><q>")] >> -) >> - >> (define_insn "reduc_plus_scal_<mode>" >> [(set (match_operand:<VEL> 0 "register_operand" "=w") >> (unspec:<VEL> [(match_operand:VDQV 1 "register_operand" "w")] >> @@ -3799,12 +3789,31 @@ >> [(set_attr "type" "neon_reduc_add<VDQV_E:q>")] >> ) >> >> -(define_insn "reduc_plus_scal_<mode>" >> - [(set (match_operand:<VEL> 0 "register_operand" "=w") >> - (unspec:<VEL> [(match_operand:V2F 1 "register_operand" "w")] >> - UNSPEC_FADDV))] >> - "TARGET_SIMD" >> - "faddp\\t%<Vetype>0, %1.<Vtype>" >> +(define_expand "reduc_plus_scal_<mode>" >> + [(match_operand:<VEL> 0 "register_operand") >> + (match_operand:V2F 1 "register_operand")] >> + "TARGET_SIMD" >> + { >> + rtx hi = aarch64_gen_stepped_int_parallel (1, ENDIAN_LANE_N (2, 1), 1); >> + rtx x = force_reg (<MODE>mode, operands[1]); >> + emit_insn (gen_aarch64_faddp_scal<mode> (operands[0], x, hi, >> + gen_lowpart (<VEL>mode, x))); >> + DONE; >> + } >> +) >> + >> +;; The sum of the two elements of a vector. RTL simplification describes >> +;; the low element as a lowpart of the vector, so operand 3 is that lowpart >> +;; of operand 1. >> +(define_insn "aarch64_faddp_scal<mode>" >> + [(set (match_operand:<VEL> 0 "register_operand" "=w") >> + (plus:<VEL> >> + (vec_select:<VEL> >> + (match_operand:V2F 1 "register_operand" "w") >> + (match_operand:V2F 2 "vect_par_cnst_hi_half")) >> + (match_operand:<VEL> 3 "register_operand" "w")))] >> + "TARGET_SIMD && aarch64_simd_lowpart_of_p (operands[3], >> operands[1])" >> + "faddp\t%<Vetype>0, %1.<Vtype>" >> [(set_attr "type" "neon_fp_reduc_add_<Vetype><q>")] >> ) >> >> @@ -8304,27 +8313,27 @@ >> [(set_attr "type" "neon_fp_compare_<stype><q>")] >> ) >> >> -;; addp >> +;; addp and faddp >> >> -;; ADDP with two registers semantically concatenates them and performs >> -;; a pairwise addition on the result. For 128-bit input modes represent >> this >> -;; as a concatenation of the pairwise addition results of the two input >> -;; registers. This allow us to avoid using intermediate 256-bit modes. >> -;; The RTL uses GCC's lane numbering, so on big-endian targets operand 1 >> -;; provides the architecturally high half of the result. >> -(define_insn "aarch64_addp<mode>_insn" >> - [(set (match_operand:VQ_I 0 "register_operand" "=w") >> - (vec_concat:VQ_I >> +;; ADDP and FADDP with two registers semantically concatenate them and >> +;; perform a pairwise addition on the result. For 128-bit input modes >> +;; represent this as a concatenation of the pairwise addition results of the >> +;; two input registers. This allows us to avoid using intermediate 256-bit >> +;; modes. The RTL uses GCC's lane numbering, so on big-endian targets >> +;; operand 1 provides the architecturally high half of the result. >> +(define_insn "@aarch64_addp<mode>_concat" >> + [(set (match_operand:VQ_ADDP 0 "register_operand" "=w") >> + (vec_concat:VQ_ADDP >> (plus:<VHALF> >> (vec_select:<VHALF> >> - (match_operand:VQ_I 1 "register_operand" "w") >> - (match_operand:VQ_I 3 "vect_par_cnst_even_or_odd_half")) >> + (match_operand:VQ_ADDP 1 "register_operand" "w") >> + (match_operand:VQ_ADDP 3 "vect_par_cnst_even_or_odd_half")) >> (vec_select:<VHALF> >> (match_dup 1) >> - (match_operand:VQ_I 4 "vect_par_cnst_even_or_odd_half"))) >> + (match_operand:VQ_ADDP 4 >> "vect_par_cnst_even_or_odd_half"))) >> (plus:<VHALF> >> (vec_select:<VHALF> >> - (match_operand:VQ_I 2 "register_operand" "w") >> + (match_operand:VQ_ADDP 2 "register_operand" "w") >> (match_dup 3)) >> (vec_select:<VHALF> >> (match_dup 2) >> @@ -8332,26 +8341,27 @@ >> "TARGET_SIMD && !rtx_equal_p (operands[3], operands[4])" >> { >> if (BYTES_BIG_ENDIAN) >> - return "addp\t%<v>0<Vmtype>, %<v>2<Vmtype>, %<v>1<Vmtype>"; >> - return "addp\t%<v>0<Vmtype>, %<v>1<Vmtype>, %<v>2<Vmtype>"; >> + return "<f>addp\t%<v>0<Vmtype>, %<v>2<Vmtype>, %<v>1<Vmtype>"; >> + return "<f>addp\t%<v>0<Vmtype>, %<v>1<Vmtype>, %<v>2<Vmtype>"; >> } >> - [(set_attr "type" "neon_reduc_add<q>")] >> + [(set_attr "type" "<addp_type>")] >> ) >> >> -;; For 64-bit input modes an ADDP is represented as a concatenation >> +;; For 64-bit input modes an ADDP or FADDP is represented as a >> concatenation >> ;; of the input registers into an 128-bit register which is then fed >> ;; into a pairwise add. That way we avoid having to create intermediate >> -;; 32-bit vector modes. As above, operand 1 provides the architecturally >> -;; high half of the result on big-endian targets. >> -(define_insn "aarch64_addp<mode><vczle><vczbe>_insn" >> - [(set (match_operand:VD_BHSI 0 "register_operand" "=w") >> - (plus:VD_BHSI >> - (vec_select:VD_BHSI >> +;; 32-bit vector modes. vpadd_f32 uses @aarch64_faddp_scal_pair<mode> >> +;; instead. As above, operand 1 provides the architecturally high half of >> +;; the result on big-endian targets. >> +(define_insn "@aarch64_addp<mode>_select<vczle><vczbe>" >> + [(set (match_operand:VD_ADDP 0 "register_operand" "=w") >> + (plus:VD_ADDP >> + (vec_select:VD_ADDP >> (vec_concat:<VDBL> >> - (match_operand:VD_BHSI 1 "register_operand" "w") >> - (match_operand:VD_BHSI 2 "register_operand" "w")) >> + (match_operand:VD_ADDP 1 "register_operand" "w") >> + (match_operand:VD_ADDP 2 "register_operand" "w")) >> (match_operand:<VDBL> 3 "vect_par_cnst_even_or_odd_half")) >> - (vec_select:VD_BHSI >> + (vec_select:VD_ADDP >> (vec_concat:<VDBL> >> (match_dup 1) >> (match_dup 2)) >> @@ -8359,10 +8369,39 @@ >> "TARGET_SIMD && !rtx_equal_p (operands[3], operands[4])" >> { >> if (BYTES_BIG_ENDIAN) >> - return "addp\t%<v>0<Vmtype>, %<v>2<Vmtype>, %<v>1<Vmtype>"; >> - return "addp\t%<v>0<Vmtype>, %<v>1<Vmtype>, %<v>2<Vmtype>"; >> + return "<f>addp\t%<v>0<Vmtype>, %<v>2<Vmtype>, %<v>1<Vmtype>"; >> + return "<f>addp\t%<v>0<Vmtype>, %<v>1<Vmtype>, %<v>2<Vmtype>"; >> + } >> + [(set_attr "type" "<addp_type>")] >> +) >> + >> +;; FADDP of two-lane vectors, represented as the concatenation of the sums >> of >> +;; the two elements of each input. RTL simplification describes the low >> +;; element of a vector as a lowpart of the vector, so operands 4 and 5 are >> +;; those lowparts of operands 1 and 2. As above, operand 1 provides the >> +;; architecturally high half of the result on big-endian targets. >> +(define_insn "@aarch64_faddp_scal_pair<mode><vczle><vczbe>" >> + [(set (match_operand:V2F 0 "register_operand" "=w") >> + (vec_concat:V2F >> + (plus:<VEL> >> + (vec_select:<VEL> >> + (match_operand:V2F 1 "register_operand" "w") >> + (match_operand:V2F 3 "vect_par_cnst_hi_half")) >> + (match_operand:<VEL> 4 "register_operand" "w")) >> + (plus:<VEL> >> + (vec_select:<VEL> >> + (match_operand:V2F 2 "register_operand" "w") >> + (match_dup 3)) >> + (match_operand:<VEL> 5 "register_operand" "w"))))] >> + "TARGET_SIMD >> + && aarch64_simd_lowpart_of_p (operands[4], operands[1]) >> + && aarch64_simd_lowpart_of_p (operands[5], operands[2])" >> + { >> + if (BYTES_BIG_ENDIAN) >> + return "faddp\t%0.<Vtype>, %2.<Vtype>, %1.<Vtype>"; >> + return "faddp\t%0.<Vtype>, %1.<Vtype>, %2.<Vtype>"; >> } >> - [(set_attr "type" "neon_reduc_add<q>")] >> + [(set_attr "type" "<addp_type>")] >> ) >> >> ;; A common usecase of 64-bit ADDP is to have both operands come from the >> same >> @@ -8372,11 +8411,11 @@ >> [(set (match_operand:<VHALF> 0 "register_operand" "=w") >> (plus:<VHALF> >> (vec_select:<VHALF> >> - (match_operand:VQ_I 1 "register_operand" "w") >> - (match_operand:VQ_I 2 "vect_par_cnst_even_or_odd_half")) >> + (match_operand:VQ_ADDP 1 "register_operand" "w") >> + (match_operand:VQ_ADDP 2 "vect_par_cnst_even_or_odd_half")) >> (vec_select:<VHALF> >> (match_dup 1) >> - (match_operand:VQ_I 3 "vect_par_cnst_even_or_odd_half"))))] >> + (match_operand:VQ_ADDP 3 >> "vect_par_cnst_even_or_odd_half"))))] >> "TARGET_SIMD && !rtx_equal_p (operands[2], operands[3])" >> "#" >> "&& 1" >> @@ -8388,29 +8427,49 @@ >> else >> scratch = lowpart_subreg (<MODE>mode, operands[0], <VHALF>mode); >> >> - emit_insn (gen_aarch64_addp<mode>_insn (scratch, operands[1], >> operands[1], >> - operands[2], operands[3])); >> + emit_insn (gen_aarch64_addp_concat (<MODE>mode, scratch, >> operands[1], >> + operands[1], operands[2], >> + operands[3])); >> emit_move_insn (operands[0], gen_lowpart (<VHALF>mode, scratch)); >> DONE; >> } >> ) >> >> -(define_expand "aarch64_addp<mode>" >> - [(match_operand:VDQ_I 0 "register_operand") >> - (match_operand:VDQ_I 1 "register_operand") >> - (match_operand:VDQ_I 2 "register_operand")] >> +(define_expand "aarch64_<f>addp<mode>" >> + [(match_operand:VDQ_ADDP 0 "register_operand") >> + (match_operand:VDQ_ADDP 1 "register_operand") >> + (match_operand:VDQ_ADDP 2 "register_operand")] >> "TARGET_SIMD" >> { >> - int nunits = GET_MODE_NUNITS (<MODE>mode).to_constant (); >> - if (known_eq (GET_MODE_BITSIZE (<MODE>mode), 128)) >> - nunits /= 2; >> - rtx par_even = aarch64_gen_stepped_int_parallel (nunits, 0, 2); >> - rtx par_odd = aarch64_gen_stepped_int_parallel (nunits, 1, 2); >> /* The intrinsic uses architectural lane numbering. */ >> if (BYTES_BIG_ENDIAN) >> std::swap (operands[1], operands[2]); >> - emit_insn (gen_aarch64_addp<mode>_insn (operands[0], operands[1], >> - operands[2], par_even, par_odd)); >> + int nunits = GET_MODE_NUNITS (<MODE>mode).to_constant (); >> + if (nunits == 2 && FLOAT_MODE_P (<MODE>mode)) >> + { >> + rtx hi = aarch64_gen_stepped_int_parallel (1, ENDIAN_LANE_N (2, 1), >> 1); >> + rtx x = force_reg (<MODE>mode, operands[1]); >> + rtx y = force_reg (<MODE>mode, operands[2]); >> + emit_insn (gen_aarch64_faddp_scal_pair (<MODE>mode, >> operands[0], x, y, >> + hi, gen_lowpart (<VEL>mode, >> x), >> + gen_lowpart (<VEL>mode, >> y))); >> + } >> + else if (known_eq (GET_MODE_BITSIZE (<MODE>mode), 128)) >> + { >> + rtx even = aarch64_gen_stepped_int_parallel (nunits / 2, 0, 2); >> + rtx odd = aarch64_gen_stepped_int_parallel (nunits / 2, 1, 2); >> + emit_insn (gen_aarch64_addp_concat (<MODE>mode, operands[0], >> + operands[1], operands[2], even, >> + odd)); >> + } >> + else >> + { >> + rtx even = aarch64_gen_stepped_int_parallel (nunits, 0, 2); >> + rtx odd = aarch64_gen_stepped_int_parallel (nunits, 1, 2); >> + emit_insn (gen_aarch64_addp_select (<MODE>mode, operands[0], >> + operands[1], operands[2], even, >> + odd)); >> + } >> DONE; >> } > > Minor nit, I think I preferred the old approach of nunit /2 for the 128 bits > case > better. > > Patch is OK with that change. >
Thanks, here’s what I’ll commit. Kyrill > Thanks, > Tamar
0001-aarch64-Use-standard-RTL-for-FADDP.patch
Description: 0001-aarch64-Use-standard-RTL-for-FADDP.patch
