> On 28 Sep 2026, at 09:20, Tamar Christina <[email protected]> wrote: > > This patch teaches AArch64 extends the SVE CC flag producing instructions to > annotate which instructions support eliminating the PTEST instruction > > For example: > > bool > f (svbool_t pg, svuint32_t x, svuint32_t y) > { > svbool_t res = svcmpge_u32 (pg, x, y); > return svptest_any (pg, res); > } > > Before this series we generated a compare followed by a separate PTEST: > > cmphs p0.s, p0/z, z0.s, z1.s > ptest p0, p0.b > cset w0, ne > > With the generic compare-elim change and the new AArch64 z-only patterns, the > compare itself provides the zero/nonzero flag result: > > cmphs p0.s, p0/z, z0.s, z1.s > cset w0, ne > > > As a reminder, in SVE, PTEST derives NZCV from predicate bits selected by a > governing predicate. The important distinction is that a full PTEST condition > code carries more information than a zero/nonzero test. If later code only > branches using EQ or NE on a comparison with 0 then only the Z flag is > meaningful. In that case the earlier predicate-producing instruction can be > used instead of the PTEST flags and the PTEST can be eliminated. In such > cases > it only needs to provide an equivalent zero/nonzero answer. > > However the SVE architecture definition for values of predicates (i.e. BImode) > indicates that only the bottom bit of the lane is checked. That means that > e.g. > on a VNx16BImode vector boolean for an HI emode data vector, both 0x00 and > 0x10 > indicate that the lane is inactive and e.g. that no match is found so the Z > flag > is set. > > But the PTEST can re-shape the CC flag. and so if the PTEST's GP is .b then > `0x10` is re-interpreted as `0x1, 0x0` and so the Z flag is cleared as there > is > now and active lane. > > What makes the elimination possible is that even though the architecture > allows > this flexibility, the individual instructions explicitly say which bits they > set. > > For instance an SVE Integer comparison explicitly says the lanes are 0x1 and > 0x0 > zero extended. > > This is why the backend needs to provide enough context to compare-elim. To do > so for most instructions it's simple, we just have to provide a CC_Z variant > of the instruction next to the full CC_NZC. This is required because CC elim > will rewrite the instruction changing CC_NZC to CC_Z to indicate that we have > validated that only the Z flag is now relavant. The new overloads of > aarch64_sve_same_pred_for_ptest_p are now used to validated cases where ptest > can't be eliminated safely. > > The aarch64_select_cc_mode allows us to say that if we're comparing predicate > results with 0 that only CC_Z is needed. compare-elim uses this to start: > > Consider the example above, the RTL would look like > > ;; svcmpge_u32 > (insn 8 > (parallel [ > (set (reg:VNx16BI res) > (and:VNx16BI > (subreg:VNx16BI > (unspec:VNx4BI [pg 0 (geu x y)] UNSPEC_PRED_Z) 0) > ptrue.s-mask)) > (clobber (reg:CC_NZC cc))])) > > ;; svptest_any > (insn 9 > (set (reg:CC_NZC cc) > (unspec:CC_NZC [pg pg 0 res] UNSPEC_PTEST))) > > ;; return boolean > (insn 16 > (set (reg:QI x0) > (ne:QI (reg:CC_NZC cc) (const_int 0)))) > > 1. compare-elim will identify insn 9 as a non-compare flag setting operation. > 2. it walks backwards from the result "res" from the PTEST and finds insn 8. > 3. It identifies the use of insn'9s flag result and finds insn 16. > 4. It substitutes the source of res into the use. so we get > > (set (reg:CC_NZC cc) > (unspec:CC_NZC > [pg pg 0 > (and:VNx16BI > (subreg:VNx16BI > (unspec:VNx4BI [pg 0 (geu x y)] UNSPEC_PRED_Z) 0) > ptrue.s-mask)] > UNSPEC_PTEST)) > > 5. through insn 16 and the target hook select_cc_mode we tell it that for a > comparision against 0 of only eq/ne only CC_Z is relevant. > 6. It rewrites the merged instruction into a CC_Z only form > > (set (reg:CC_Z cc) > (unspec:CC_Z > [pg pg 0 > (and:VNx16BI > (subreg:VNx16BI > (unspec:VNx4BI [pg 0 (geu x y)] UNSPEC_PRED_Z) 0) > ptrue.s-mask)] > UNSPEC_PTEST)) > > 7. and it calls recog to validate the new form. If the target says the new > form > is valid with the given operands then insn 8 is rewritten to the new form > and > insn 9 is eliminated committing the change. > > The new patterns are to deal with akwardness around the ACLE definition of the > compares since they all return VNx16BI. > > Bootstrapped Regtested on aarch64-none-linux-gnu, > arm-none-linux-gnueabihf, x86_64-pc-linux-gnu > -m32, -m64 and no issues. > > OK for master? > Ok. Thanks, Kyrill > Thanks, > Tamar > > gcc/ChangeLog > > * config/aarch64/aarch64-protos.h > (aarch64_sve_same_pred_for_ptest_p): Declare new overloads. > * config/aarch64/aarch64-simd.md (reduc_sbool_and_scal_<mode>): > Pass CC_NZCmode to gen_aarch64_pred_cmp_ptest. > (reduc_sbool_ior_scal_<mode>): Likewise. > (<optab><mode>): Pass CC_NZCmode to gen_aarch64_ptest and > gen_aarch64_pred_cmp_ptest. > * config/aarch64/aarch64-sve.md: Use PTEST_CC iterator for selected > PTEST-like patterns and use mode-aware PTEST predicate checks. > (*aarch64_sve_ptruevnx16bi_cc): Rename to > *aarch64_sve_ptruevnx16bi_cc<ptest_cc>. > (*aarch64_sve_ptrue<mode>_cc): Rename to > *aarch64_sve_ptrue<PRED_HSD:mode>_cc<ptest_cc>. > (*aarch64_sve_ptruevnx16bi_ptest): Rename to > *aarch64_sve_ptruevnx16bi_ptest<ptest_cc>. > (*aarch64_sve_ptrue<mode>_ptest): Rename to > *aarch64_sve_ptrue<PRED_HSD:mode>_ptest<ptest_cc>. > (*aarch64_rdffr_z_ptest): Rename to > *aarch64_rdffr_z_ptest<ptest_cc>. > (*aarch64_rdffr_ptest): Rename to > *aarch64_rdffr_ptest<ptest_cc>. > (*aarch64_rdffr_z_cc): Rename to *aarch64_rdffr_z_cc<ptest_cc>. > (*aarch64_rdffr_cc): Rename to *aarch64_rdffr_cc<ptest_cc>. > (*one_cmpl<mode>3_cc): Rename to > *one_cmpl<PRED_ALL:mode>3_cc<ptest_cc>. > (*one_cmpl<mode>3_ptest): Rename to > *one_cmpl<PRED_ALL:mode>3_ptest<ptest_cc>. > (*<optab><mode>3_cc): Rename to > *<optab><PRED_ALL:mode>3_cc<ptest_cc>. > (*<optab><mode>3_ptest): Rename to > *<optab><PRED_ALL:mode>3_ptest<ptest_cc>. > (*<nlogical><mode>3_cc): Rename to > *<nlogical><PRED_ALL:mode>3_cc<ptest_cc>. > (*<nlogical><mode>3_ptest): Rename to > *<nlogical><PRED_ALL:mode>3_ptest<ptest_cc>. > (*<logical_nn><mode>3_cc): Rename to > *<logical_nn><PRED_ALL:mode>3_cc<ptest_cc>. > (*<logical_nn><mode>3_ptest): Rename to > *<logical_nn><PRED_ALL:mode>3_ptest<ptest_cc>. > (*cmp<cmp_op><mode>_cc): Rename to > *cmp<cmp_op><SVE_I:mode>_cc<ptest_cc>. > (*cmp<cmp_op><mode>_acle_cc): Rename to > *cmp<cmp_op><SVE_FULL_HSDI:mode>_acle_ccnzc. > (*cmp<cmp_op><mode>_acle_zonly_ptest): New pattern. > (*cmp<cmp_op><mode>_acle_ccz): Likewise. > (@aarch64_pred_cmp<cmp_op><mode>_ptest): Rename to > @aarch64_pred_cmp<cmp_op><SVE_I:mode>_ptest<ptest_cc_suffix>. > (*aarch64_pred_cmp<cmp_op><mode>_wide_cc): Rename B/H/S/I form to > *aarch64_pred_cmp<cmp_op><SVE_FULL_BHSI:mode>_wide_cc<ptest_cc>. > (*aarch64_pred_cmp<cmp_op><mode>_wide_cc): Rename H/S/I form to > *aarch64_pred_cmp<cmp_op><SVE_FULL_HSI:mode>_wide_cc<ptest_cc>. > (*aarch64_pred_cmp<cmp_op><mode>_wide_ptest): Rename to > *aarch64_pred_cmp<cmp_op><SVE_FULL_BHSI:mode>_wide_ptest<ptest_cc>. > (*while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_cc): Rename to > *while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_cc<ptest_cc>. > (*while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle_cc): > Rename to > *while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle_cc<ptest_cc>. > (@while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_ptest): Rename to > @while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_ptest<ptest_cc_suffix>. > (@aarch64_ptest<mode>): Rename to > @aarch64_<ptest_cc_suffix>ptest<PRED_ALL:mode>. > (*aarch64_brk<brk_op>_cc): Rename both forms to > *aarch64_brk<brk_op>_cc<ptest_cc>. > (*aarch64_brk<brk_op>_ptest): Rename both forms to > *aarch64_brk<brk_op>_ptest<ptest_cc>. > (*aarch64_brkn_cc): Rename to *aarch64_brkn_cc<ptest_cc>. > (*aarch64_brkn_ptest): Rename to *aarch64_brkn_ptest<ptest_cc>. > (*aarch64_sve_<sve_pred_op><mode>_cc): Rename PRED_ALL form to > *aarch64_sve_<sve_pred_op><PRED_ALL:mode>_cc<ptest_cc>. > (*aarch64_sve_<sve_pred_op><mode>_cc): Rename PRED_HSD form to > *aarch64_sve_<sve_pred_op><PRED_HSD:mode>_cc<ptest_cc>. > (*aarch64_sve_<sve_pred_op><mode>_ptest): Rename to > *aarch64_sve_<sve_pred_op><PRED_ALL:mode>_ptest<ptest_cc>. > * config/aarch64/aarch64-sve2.md > (check_<raw_war>_ptrs<mode>): Pass CC_NZCmode to gen_while_ptest. > (*aarch64_pred_<sve_int_op><mode>_cc): Rename to > *aarch64_pred_<sve_int_op><SVE_FULL_BHI:mode>_cc<ptest_cc>. > (*aarch64_pred_<sve_int_op><mode>_ptest): Rename to > *aarch64_pred_<sve_int_op><SVE_FULL_BHI:mode>_ptest<ptest_cc>. > * config/aarch64/aarch64.cc (aarch64_sve_same_pred_for_ptest_p): Add > mode-aware overloads for Z-only PTEST checks. > (aarch64_select_cc_mode): Use CC_Zmode for equality comparisons of > vector predicates against zero. > * config/aarch64/iterators.md (PTEST_CC, ptest_cc, ptest_cc_suffix): > New iterators and attributes. > > gcc/testsuite/ChangeLog > > * gcc.target/aarch64/sve/acle/general/brka_1.c: Update ptest count. > * gcc.target/aarch64/sve/acle/general/brkb_1.c: Likewise. > * gcc.target/aarch64/sve/acle/general/brkb_zonly_1.c: New test. > * gcc.target/aarch64/sve/acle/general/brkb_zonly_1_run.c: Likewise. > * gcc.target/aarch64/sve/acle/general/cmpge_zonly_1.c: Likewise. > * gcc.target/aarch64/sve/acle/general/cmpge_zonly_1_run.c: Likewise. > * gcc.target/aarch64/sve/acle/general/pnext_ptest_1_run.c: Likewise. > * gcc.target/aarch64/sve/acle/general/ptest_zonly_1.c: Likewise. > * gcc.target/aarch64/sve/cmpelim-sve-ptest.c: Update ptest count. > * gcc.target/aarch64/sve/cmpelim-sve-ptest-2.c: Likewise. > * gcc.target/aarch64/sve/noeffect.c: Remove redundant ptest. > * gcc.target/aarch64/sve/noeffect10.c: Likewise. > * gcc.target/aarch64/sve/noeffect7.c: Allow equivalent branch form. > * gcc.target/aarch64/sve2/acle/general/match_zonly_1.c: New test. > * gcc.target/aarch64/sve2/acle/general/match_zonly_1_run.c: Likewise. > * gcc.target/aarch64/sve2/acle/general/ptest_zonly_1.c: Likewise. > > --- > diff --git a/gcc/config/aarch64/aarch64-protos.h > b/gcc/config/aarch64/aarch64-protos.h > index > e8ae3d427949896f5a34d21bd55ce770cdcdcd76..97ff1668b3762c1eb9386e9fa466e69d33f294a1 > 100644 > --- a/gcc/config/aarch64/aarch64-protos.h > +++ b/gcc/config/aarch64/aarch64-protos.h > @@ -1052,6 +1052,10 @@ rtx aarch64_ptrue_reg (machine_mode, unsigned int); > rtx aarch64_ptrue_reg (machine_mode, machine_mode); > rtx aarch64_pfalse_reg (machine_mode); > bool aarch64_sve_same_pred_for_ptest_p (rtx *, rtx *); > +bool aarch64_sve_same_pred_for_ptest_p (machine_mode, rtx *, rtx *); > +bool aarch64_sve_same_pred_for_ptest_p (machine_mode, rtx, rtx *, rtx *, rtx, > + machine_mode = VOIDmode); > +bool aarch64_sve_same_pred_for_ptest_p (machine_mode, rtx, rtx *, rtx, rtx); > rtx aarch64_sve_packed_pred (machine_mode); > rtx aarch64_sve_fp_pred (machine_mode, rtx *); > rtx aarch64_sve_emit_masked_fp_pred (machine_mode, rtx); > diff --git a/gcc/config/aarch64/aarch64-simd.md > b/gcc/config/aarch64/aarch64-simd.md > index > 98c2da034c55af8d56e63e3e9b8644d870b2d097..52a99c724d1d572534a65bde1e7129cd98b8d483 > 100644 > --- a/gcc/config/aarch64/aarch64-simd.md > +++ b/gcc/config/aarch64/aarch64-simd.md > @@ -3854,8 +3854,8 @@ (define_expand "reduc_sbool_and_scal_<mode>" > rtx cast_gp = lowpart_subreg (pred_mode, gp, VNx16BImode); > rtx gp_flag = gen_int_mode (SVE_MAYBE_NOT_PTRUE, SImode); > emit_insn ( > - gen_aarch64_pred_cmp_ptest (EQ, full_mode, pred_res, gp, in, > - CONST0_RTX (full_mode), cast_gp, > + gen_aarch64_pred_cmp_ptest (EQ, full_mode, CC_NZCmode, pred_res, gp, > + in, CONST0_RTX (full_mode), cast_gp, > gp_flag, cast_gp, gp_flag)); > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp = gen_rtx_fmt_ee (EQ, SImode, cc_reg, const0_rtx); > @@ -3917,8 +3917,8 @@ (define_expand "reduc_sbool_ior_scal_<mode>" > rtx cast_gp = lowpart_subreg (pred_mode, gp, VNx16BImode); > rtx gp_flag = gen_int_mode (SVE_MAYBE_NOT_PTRUE, SImode); > emit_insn ( > - gen_aarch64_pred_cmp_ptest (NE, full_mode, pred_res, gp, in, > - CONST0_RTX (full_mode), cast_gp, > + gen_aarch64_pred_cmp_ptest (NE, full_mode, CC_NZCmode, pred_res, gp, > + in, CONST0_RTX (full_mode), cast_gp, > gp_flag, cast_gp, gp_flag)); > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp = gen_rtx_fmt_ee (NE, SImode, cc_reg, const0_rtx); > @@ -4567,13 +4567,14 @@ (define_expand "<optab><mode>" > { > aarch64_expand_sve_vec_cmp<sve_cmp_suff> (tmp, code, in1, in2); > emit_insn (gen_and3 (pred_mode, tmp, tmp, cast_ptrue)); > - emit_insn (gen_aarch64_ptest (pred_mode, ptrue, cast_ptrue, hint, > - tmp)); > + emit_insn (gen_aarch64_ptest (CC_NZCmode, pred_mode, ptrue, > + cast_ptrue, hint, tmp)); > } > else > - emit_insn (gen_aarch64_pred_cmp_ptest (code, full_mode, tmp, ptrue, in1, > - in2, cast_ptrue, hint, > - cast_ptrue, hint)); > + emit_insn (gen_aarch64_pred_cmp_ptest (code, full_mode, CC_NZCmode, > + tmp, ptrue, in1, in2, > + cast_ptrue, hint, cast_ptrue, > + hint)); > > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp_reg = gen_rtx_<cbranch_op> (VOIDmode, cc_reg, const0_rtx); > diff --git a/gcc/config/aarch64/aarch64-sve.md > b/gcc/config/aarch64/aarch64-sve.md > index > dc92cef4d888c36fd6dba02fa6628878dcc96322..526c73549b38eb2fae7d6d02747593f88f749ea1 > 100644 > --- a/gcc/config/aarch64/aarch64-sve.md > +++ b/gcc/config/aarch64/aarch64-sve.md > @@ -245,7 +245,9 @@ > ;; > ;; where the conditions marked (*) are also true when there are no active > ;; lanes (i.e. when the governing predicate is a PFALSE). The flags results > -;; of a PTEST use the condition code mode CC_NZC. > +;; of a full PTEST use the condition code mode CC_NZC. Some optimized forms > +;; only preserve the any/none information; those use CC_Z to indicate that > only > +;; the CC_Z flag is relevant for these patterns. > ;; > ;; PTEST is always a .B operation (i.e. it always operates on VNx16BI). > ;; This means that for other predicate modes, we need a governing predicate > @@ -276,6 +278,10 @@ > ;; > ;; (unspec:CC_NZC [gp cast_gp ptrue_flag op] UNSPEC_PTEST) > ;; > +;; or, when only any/none is relevant: > +;; > +;; (unspec:CC_Z [gp cast_gp ptrue_flag op] UNSPEC_PTEST) > +;; > ;; where: > ;; > ;; - GP is the real VNx16BI governing predicate > @@ -991,9 +997,9 @@ (define_insn "*aarch64_sve_mov<mode>" > ) > > ;; Match PTRUES Pn.B when both the predicate and flags are useful. > -(define_insn_and_rewrite "*aarch64_sve_ptruevnx16bi_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_sve_ptruevnx16bi_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 2) > (match_operand 3) > (const_int SVE_KNOWN_PTRUE) > @@ -1017,9 +1023,9 @@ (define_insn_and_rewrite "*aarch64_sve_ptruevnx16bi_cc" > ) > > ;; Match PTRUES Pn.[HSD] when both the predicate and flags are useful. > -(define_insn_and_rewrite "*aarch64_sve_ptrue<mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_sve_ptrue<PRED_HSD:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 2) > (match_operand 3) > (const_int SVE_KNOWN_PTRUE) > @@ -1039,16 +1045,16 @@ (define_insn_and_rewrite "*aarch64_sve_ptrue<mode>_cc" > "&& (!CONSTANT_P (operands[2]) || !CONSTANT_P (operands[3]))" > { > operands[2] = CONSTM1_RTX (VNx16BImode); > - operands[3] = CONSTM1_RTX (<MODE>mode); > + operands[3] = CONSTM1_RTX (<PRED_HSD:MODE>mode); > } > [(set_attr "sve_type" "sve_pred_misc")] > ) > > ;; Match PTRUES Pn.B when only the flags result is useful (which is > ;; a way of testing VL). > -(define_insn_and_rewrite "*aarch64_sve_ptruevnx16bi_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_sve_ptruevnx16bi_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 2) > (match_operand 3) > (const_int SVE_KNOWN_PTRUE) > @@ -1072,9 +1078,9 @@ (define_insn_and_rewrite > "*aarch64_sve_ptruevnx16bi_ptest" > > ;; Match PTRUES Pn.[HWD] when only the flags result is useful (which is > ;; a way of testing VL). > -(define_insn_and_rewrite "*aarch64_sve_ptrue<mode>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_sve_ptrue<PRED_HSD:mode>_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 2) > (match_operand 3) > (const_int SVE_KNOWN_PTRUE) > @@ -1093,7 +1099,7 @@ (define_insn_and_rewrite > "*aarch64_sve_ptrue<mode>_ptest" > "&& (!CONSTANT_P (operands[2]) || !CONSTANT_P (operands[3]))" > { > operands[2] = CONSTM1_RTX (VNx16BImode); > - operands[3] = CONSTM1_RTX (<MODE>mode); > + operands[3] = CONSTM1_RTX (<PRED_HSD:MODE>mode); > } > [(set_attr "sve_type" "sve_pred_misc")] > ) > @@ -1203,9 +1209,9 @@ (define_insn "aarch64_rdffr_z" > ) > > ;; Read the FFR to test for a fault, without using the predicate result. > -(define_insn "*aarch64_rdffr_z_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*aarch64_rdffr_z_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_dup 1) > (match_operand:SI 2 "aarch64_sve_ptrue_flag") > @@ -1224,9 +1230,9 @@ (define_insn "*aarch64_rdffr_z_ptest" > ) > > ;; Same for unpredicated RDFFR when tested with a known PTRUE. > -(define_insn "*aarch64_rdffr_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*aarch64_rdffr_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_dup 1) > (const_int SVE_KNOWN_PTRUE) > @@ -1243,9 +1249,9 @@ (define_insn "*aarch64_rdffr_ptest" > ) > > ;; Read the FFR with zero predication and test the result. > -(define_insn "*aarch64_rdffr_z_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*aarch64_rdffr_z_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_dup 1) > (match_operand:SI 2 "aarch64_sve_ptrue_flag") > @@ -1267,9 +1273,9 @@ (define_insn "*aarch64_rdffr_z_cc" > ) > > ;; Same for unpredicated RDFFR when tested with a known PTRUE. > -(define_insn "*aarch64_rdffr_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*aarch64_rdffr_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_dup 1) > (const_int SVE_KNOWN_PTRUE) > @@ -4229,9 +4235,9 @@ (define_insn "@aarch64_pred_one_cmpl<mode>_z" > > ;; Predicated predicate inverse in which the flags are set in the same > ;; way as a PTEST. > -(define_insn "*one_cmpl<mode>3_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*one_cmpl<PRED_ALL:mode>3_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upa") > (match_operand 3) > (match_operand:SI 4 "aarch64_sve_ptrue_flag") > @@ -4248,9 +4254,9 @@ (define_insn "*one_cmpl<mode>3_cc" > ) > > ;; Same, where only the flags result is interesting. > -(define_insn "*one_cmpl<mode>3_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*one_cmpl<PRED_ALL:mode>3_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upa") > (match_operand 3) > (match_operand:SI 4 "aarch64_sve_ptrue_flag") > @@ -7301,9 +7307,9 @@ (define_insn "@aarch64_pred_<optab><mode>_z" > ;; Perform a logical operation on operands 2 and 3, using operand 1 as > ;; the GP. Store the result in operand 0 and set the flags in the same > ;; way as for PTEST. > -(define_insn "*<optab><mode>3_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<optab><PRED_ALL:mode>3_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -7326,9 +7332,9 @@ (define_insn "*<optab><mode>3_cc" > ) > > ;; Same with just the flags result. > -(define_insn "*<optab><mode>3_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<optab><PRED_ALL:mode>3_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -7374,9 +7380,9 @@ (define_insn "aarch64_pred_<nlogical><mode>_z" > ) > > ;; Same, but set the flags as a side-effect. > -(define_insn "*<nlogical><mode>3_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<nlogical><PRED_ALL:mode>3_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -7402,9 +7408,9 @@ (define_insn "*<nlogical><mode>3_cc" > ) > > ;; Same with just the flags result. > -(define_insn "*<nlogical><mode>3_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<nlogical><PRED_ALL:mode>3_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -7451,9 +7457,9 @@ (define_insn "aarch64_pred_<logical_nn><mode>_z" > ) > > ;; Same, but set the flags as a side-effect. > -(define_insn "*<logical_nn><mode>3_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<logical_nn><PRED_ALL:mode>3_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -7480,9 +7486,9 @@ (define_insn "*<logical_nn><mode>3_cc" > ) > > ;; Same with just the flags result. > -(define_insn "*<logical_nn><mode>3_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn "*<logical_nn><PRED_ALL:mode>3_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -8870,10 +8876,11 @@ (define_insn "*aarch64_pred_cmp<cmp_op><mode>_acle" > ) > > ;; Predicated integer comparisons in which both the flag and predicate > -;; results are interesting. > -(define_insn_and_rewrite "*cmp<cmp_op><mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +;; results are interesting. The CC_Z form is useful when a later PTEST > +;; against the comparison result feeds only EQ/NE. > +(define_insn_and_rewrite "*cmp<cmp_op><SVE_I:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -8894,16 +8901,19 @@ (define_insn_and_rewrite "*cmp<cmp_op><mode>_cc" > (match_dup 3))] > UNSPEC_PRED_Z))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > - {@ [ cons: =0 , 1 , 2 , 3 ; attrs: pred_clobber ] > - [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, #%3 > - [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > - [ Upa , Upl, w , <sve_imm_con>; no ] ^ > - [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.<Vetype> > - [ ?Upl , 0 , w , w ; yes ] ^ > - [ Upa , Upl, w , w ; no ] ^ > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[4], &operands[6], > + operands[0], <SVE_I:MODE>mode)" > + {@ [ cons: =0 , 6 , 2 , 3 ; attrs: pred_clobber ] > + [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, #%3 > + [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > + [ Upa , Upl, w , <sve_imm_con>; no ] ^ > + [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, %3.<Vetype> > + [ ?Upl , 0 , w , w ; yes ] ^ > + [ Upa , Upl, w , w ; no ] ^ > } > - "&& !rtx_equal_p (operands[4], operands[6])" > + "&& !rtx_equal_p (operands[4], operands[6]) > + && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > { > operands[6] = copy_rtx (operands[4]); > operands[7] = operands[5]; > @@ -8911,7 +8921,7 @@ (define_insn_and_rewrite "*cmp<cmp_op><mode>_cc" > [(set_attr "sve_type" "sve_int_cmp_set")] > ) > > -(define_insn_and_rewrite "*cmp<cmp_op><mode>_acle_cc" > +(define_insn_and_rewrite "*cmp<cmp_op><SVE_FULL_HSDI:mode>_acle_ccnzc" > [(set (reg:CC_NZC CC_REGNUM) > (unspec:CC_NZC > [(match_operand:VNx16BI 1 "register_operand") > @@ -8938,7 +8948,8 @@ (define_insn_and_rewrite "*cmp<cmp_op><mode>_acle_cc" > 0) > (match_operand:<VPRED> 8 "aarch64_ptrue_all_operand")))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > + && aarch64_sve_same_pred_for_ptest_p (CC_NZCmode, &operands[4], > + &operands[6])" > {@ [ cons: =0 , 1 , 2 , 3 ; attrs: pred_clobber ] > [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, #%3 > [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > @@ -8954,16 +8965,108 @@ (define_insn_and_rewrite "*cmp<cmp_op><mode>_acle_cc" > } > ) > > +;; Predicated integer comparisons, formed through the ACLE svbool_t type, > +;; in which only the Z flag is interesting. ACLE exposes predicate results > +;; as VNx16BI plus an AND with the all-true mask for the element size, so it > +;; does not have the same RTL shape as the native predicate result above. > +(define_insn "*cmp<cmp_op><mode>_acle_zonly_ptest" > + [(set (reg:CC_Z CC_REGNUM) > + (unspec:CC_Z > + [(match_operand:VNx16BI 1 "register_operand") > + (match_operand 4) > + (match_operand:SI 5 "aarch64_sve_ptrue_flag") > + (and:VNx16BI > + (subreg:VNx16BI > + (unspec:<VPRED> > + [(match_operand 6 "register_operand") > + (match_operand:SI 7 "aarch64_sve_ptrue_flag") > + (SVE_INT_CMP:<VPRED> > + (match_operand:SVE_FULL_HSDI 2 "register_operand") > + (match_operand:SVE_FULL_HSDI 3 > + "aarch64_sve_cmp_<sve_imm_con>_operand"))] > + UNSPEC_PRED_Z) > + 0) > + (match_operand:<VPRED> 8 "aarch64_ptrue_all_operand"))] > + UNSPEC_PTEST)) > + (clobber (match_scratch:<VPRED> 0))] > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p (CC_Zmode, operands[1], > + &operands[4], &operands[6], > + operands[0], <MODE>mode)" > + {@ [ cons: =0 , 6 , 2 , 3 ; attrs: pred_clobber ] > + [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, #%3 > + [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > + [ Upa , Upl, w , <sve_imm_con>; no ] ^ > + [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, %3.<Vetype> > + [ ?Upl , 0 , w , w ; yes ] ^ > + [ Upa , Upl, w , w ; no ] ^ > + } > + [(set_attr "sve_type" "sve_int_cmp_set")] > +) > + > +;; Z-only form of the ACLE compare-and-CC pattern above. Here the PTEST > +;; operand is the full VNx16BI predicate value, since this is the form that > +;; compare-elim creates when folding a separate PTEST into the original > +;; predicate-producing comparison. > +(define_insn "*cmp<cmp_op><mode>_acle_ccz" > + [(set (reg:CC_Z CC_REGNUM) > + (unspec:CC_Z > + [(match_operand:VNx16BI 1 "register_operand") > + (match_operand 4) > + (match_operand:SI 5 "aarch64_sve_ptrue_flag") > + (and:VNx16BI > + (subreg:VNx16BI > + (unspec:<VPRED> > + [(match_operand 6 "register_operand") > + (match_operand:SI 7 "aarch64_sve_ptrue_flag") > + (SVE_INT_CMP:<VPRED> > + (match_operand:SVE_FULL_HSDI 2 "register_operand") > + (match_operand:SVE_FULL_HSDI 3 > + "aarch64_sve_cmp_<sve_imm_con>_operand"))] > + UNSPEC_PRED_Z) > + 0) > + (match_operand:<VPRED> 8 "aarch64_ptrue_all_operand"))] > + UNSPEC_PTEST)) > + (set (match_operand:VNx16BI 0 "register_operand") > + (and:VNx16BI > + (subreg:VNx16BI > + (unspec:<VPRED> > + [(match_dup 6) > + (match_dup 7) > + (SVE_INT_CMP:<VPRED> > + (match_dup 2) > + (match_dup 3))] > + UNSPEC_PRED_Z) > + 0) > + (match_dup 8)))] > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p (CC_Zmode, operands[1], > + &operands[4], &operands[6], > + operands[0], <MODE>mode)" > + {@ [ cons: =0 , 6 , 2 , 3 ; attrs: pred_clobber ] > + [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, #%3 > + [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > + [ Upa , Upl, w , <sve_imm_con>; no ] ^ > + [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, %3.<Vetype> > + [ ?Upl , 0 , w , w ; yes ] ^ > + [ Upa , Upl, w , w ; no ] ^ > + } > + [(set_attr "sve_type" "sve_int_cmp_set")] > +) > + > ;; Predicated integer comparisons in which only the flags result is > -;; interesting. > -(define_insn_and_rewrite "@aarch64_pred_cmp<cmp_op><mode>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +;; interesting. The CC_NZC form keeps enough information for first/last > +;; tests, so its PTEST predicate must match the compare predicate. The CC_Z > +;; form is only for any/none tests, where a wider PTEST predicate is OK. > +(define_insn_and_rewrite > + "@aarch64_pred_cmp<cmp_op><SVE_I:mode>_ptest<ptest_cc_suffix>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > (unspec:<VPRED> > - [(match_operand 6) > + [(match_operand 6 "register_operand") > (match_operand:SI 7 "aarch64_sve_ptrue_flag") > (SVE_INT_CMP:<VPRED> > (match_operand:SVE_I 2 "register_operand") > @@ -8972,16 +9075,19 @@ (define_insn_and_rewrite > "@aarch64_pred_cmp<cmp_op><mode>_ptest" > UNSPEC_PTEST)) > (clobber (match_scratch:<VPRED> 0))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > - {@ [ cons: =0, 1 , 2 , 3 ; attrs: pred_clobber ] > - [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, #%3 > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[4], &operands[6], > + operands[0], <SVE_I:MODE>mode)" > + {@ [ cons: =0, 6 , 2 , 3 ; attrs: pred_clobber ] > + [ &Upa , Upl, w , <sve_imm_con>; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, #%3 > [ ?Upl , 0 , w , <sve_imm_con>; yes ] ^ > [ Upa , Upl, w , <sve_imm_con>; no ] ^ > - [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.<Vetype> > + [ &Upa , Upl, w , w ; yes ] > cmp<cmp_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, %3.<Vetype> > [ ?Upl , 0 , w , w ; yes ] ^ > [ Upa , Upl, w , w ; no ] ^ > } > - "&& !rtx_equal_p (operands[4], operands[6])" > + "&& !rtx_equal_p (operands[4], operands[6]) > + && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > { > operands[6] = copy_rtx (operands[4]); > operands[7] = operands[5]; > @@ -9128,9 +9234,9 @@ (define_insn "*aarch64_pred_cmp<cmp_op><mode>_wide" > > ;; Predicated integer wide comparisons in which both the flag and > ;; predicate results are interesting. > -(define_insn_and_rewrite "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><SVE_FULL_BHSI:mode>_wide_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -9153,7 +9259,10 @@ (define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > SVE_COND_INT_CMP_WIDE)] > UNSPEC_PRED_Z))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[4], &operands[6], > + operands[0], > + <SVE_FULL_BHSI:MODE>mode)" > {@ [ cons: =0, 1 , 2, 3, 6 ; attrs: pred_clobber ] > [ &Upa , Upl, w, w, Upl; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.d > [ ?Upl , 0 , w, w, Upl; yes ] ^ > @@ -9167,9 +9276,9 @@ (define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > [(set_attr "sve_type" "sve_int_cmp_set")] > ) > > -(define_insn_and_rewrite "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><SVE_FULL_HSI:mode>_wide_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -9196,7 +9305,10 @@ (define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > 0) > (match_operand:<VPRED> 8 "aarch64_ptrue_all_operand")))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[4], &operands[6], > + operands[0], > + <SVE_FULL_HSI:MODE>mode)" > {@ [ cons: =0, 1 , 2, 3, 6 ; attrs: pred_clobber ] > [ &Upa , Upl, w, w, Upl; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.d > [ ?Upl , 0 , w, w, Upl; yes ] ^ > @@ -9212,9 +9324,9 @@ (define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><mode>_wide_cc" > > ;; Predicated integer wide comparisons in which only the flags result > ;; is interesting. > -(define_insn_and_rewrite "*aarch64_pred_cmp<cmp_op><mode>_wide_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><SVE_FULL_BHSI:mode>_wide_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -9229,7 +9341,10 @@ (define_insn_and_rewrite > "*aarch64_pred_cmp<cmp_op><mode>_wide_ptest" > UNSPEC_PTEST)) > (clobber (match_scratch:<VPRED> 0))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[4], &operands[6], > + operands[0], > + <SVE_FULL_BHSI:MODE>mode)" > {@ [ cons: =0, 1 , 2, 3, 6 ; attrs: pred_clobber ] > [ &Upa , Upl, w, w, Upl; yes ] > cmp<cmp_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.d > [ ?Upl , 0 , w, w, Upl; yes ] ^ > @@ -9334,9 +9449,9 @@ (define_insn > "*aarch64_sve_while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle > ;; The WHILE instructions set the flags in the same way as a PTEST with > ;; a PTRUE GP. Handle the case in which both results are useful. The GP > ;; operands to the PTEST aren't needed, so we allow them to be anything. > -(define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 3) > (match_operand 4) > (const_int SVE_KNOWN_PTRUE) > @@ -9363,9 +9478,9 @@ (define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_cc" > [(set_attr "sve_type" "sve_pred_cnt_ctrl")] > ) > > -(define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 3) > (match_operand 4) > (const_int SVE_KNOWN_PTRUE) > @@ -9396,9 +9511,13 @@ (define_insn_and_rewrite > "*while_<while_optab_cmp><GPI:mode><PRED_HSD:mode>_acle > ) > > ;; Same, but handle the case in which only the flags result is useful. > -(define_insn_and_rewrite > "@while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +;; Keep the unspec mode flexible: normal callers use CC_NZC so first/last > +;; are available, while compare-elim can use CC_Z after proving that only > +;; any/none is observed. > +(define_insn_and_rewrite > + "@while_<while_optab_cmp><GPI:mode><PRED_ALL:mode>_ptest<ptest_cc_suffix>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand 3) > (match_operand 4) > (const_int SVE_KNOWN_PTRUE) > @@ -9928,7 +10047,8 @@ (define_expand "cbranch<mode>4" > emit_insn (gen_aarch64_pred_xor<mode>_z (pred, cast_ptrue, operands[1], > operands[2])); > } > - emit_insn (gen_aarch64_ptest<mode> (ptrue, cast_ptrue, ptrue_flag, > pred)); > + emit_insn (gen_aarch64_ptest (CC_NZCmode, <MODE>mode, ptrue, > + cast_ptrue, ptrue_flag, pred)); > operands[1] = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > operands[2] = const0_rtx; > } > @@ -9960,7 +10080,8 @@ (define_expand "<optab><mode>" > emit_insn (gen_aarch64_pred_xor<mode>_z (pred, cast_ptrue, operands[1], > operands[2])); > } > - emit_insn (gen_aarch64_ptest<mode> (ptrue, cast_ptrue, ptrue_flag, > pred)); > + emit_insn (gen_aarch64_ptest (CC_NZCmode, <MODE>mode, ptrue, > + cast_ptrue, ptrue_flag, pred)); > > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp_reg = gen_rtx_<cbranch_op> (VOIDmode, cc_reg, const0_rtx); > @@ -9995,8 +10116,9 @@ (define_expand "<optab><mode>" > rtx cast_gp = operands[1]; > rtx flag = gen_int_mode (SVE_MAYBE_NOT_PTRUE, SImode); > > - emit_insn (gen_aarch64_pred_cmp_ptest (code, <MODE>mode, res, gp, in1, in2, > - cast_gp, flag, cast_gp, flag)); > + emit_insn (gen_aarch64_pred_cmp_ptest (code, <MODE>mode, CC_NZCmode, res, > + gp, in1, in2, cast_gp, flag, > + cast_gp, flag)); > > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp_reg = gen_rtx_<cbranch_op> (VOIDmode, cc_reg, const0_rtx); > @@ -10004,14 +10126,17 @@ (define_expand "<optab><mode>" > DONE; > }) > > -;; See "Description of UNSPEC_PTEST" above for details. > -(define_insn "@aarch64_ptest<mode>" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC [(match_operand:VNx16BI 0 "register_operand" "Upa") > - (match_operand 1) > - (match_operand:SI 2 "aarch64_sve_ptrue_flag") > - (match_operand:PRED_ALL 3 "register_operand" "Upa")] > - UNSPEC_PTEST))] > +;; See "Description of UNSPEC_PTEST" above for details. The CC_Z alternative > +;; is for svptest_any-style tests and for cases where only any/none can be > +;; observed. > +(define_insn "@aarch64_<ptest_cc_suffix>ptest<PRED_ALL:mode>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > + [(match_operand:VNx16BI 0 "register_operand" "Upa") > + (match_operand 1) > + (match_operand:SI 2 "aarch64_sve_ptrue_flag") > + (match_operand:PRED_ALL 3 "register_operand" "Upa")] > + UNSPEC_PTEST))] > "TARGET_SVE" > "ptest\t%0, %3.b" > [(set_attr "sve_type" "sve_pred_logical")] > @@ -10199,9 +10324,8 @@ (define_expand "reduc_sbool_and_scal_<mode>" > emit_insn (gen_aarch64_pred_one_cmpl_z (<MODE>mode, tmp, cast_ptrue, > operands[1])); > emit_insn ( > - gen_aarch64_ptest<mode> (ptrue, cast_ptrue, > - gen_int_mode (SVE_KNOWN_PTRUE, SImode), > - tmp)); > + gen_aarch64_ptest (CC_NZCmode, <MODE>mode, ptrue, cast_ptrue, > + gen_int_mode (SVE_KNOWN_PTRUE, SImode), tmp)); > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp = gen_rtx_fmt_ee (EQ, SImode, cc_reg, const0_rtx); > rtx tmp2 = gen_reg_rtx (SImode); > @@ -10226,9 +10350,9 @@ (define_expand "reduc_sbool_ior_scal_<mode>" > { > rtx ptrue = lowpart_subreg (VNx16BImode, operands[1], <MODE>mode); > emit_insn ( > - gen_aarch64_ptest<mode> (ptrue, operands[1], > - gen_int_mode (SVE_MAYBE_NOT_PTRUE, SImode), > - operands[1])); > + gen_aarch64_ptest (CC_NZCmode, <MODE>mode, ptrue, operands[1], > + gen_int_mode (SVE_MAYBE_NOT_PTRUE, SImode), > + operands[1])); > rtx cc_reg = gen_rtx_REG (CC_NZCmode, CC_REGNUM); > rtx cmp = gen_rtx_fmt_ee (NE, SImode, cc_reg, const0_rtx); > rtx tmp = gen_reg_rtx (SImode); > @@ -11714,14 +11838,14 @@ (define_insn "@aarch64_brk<brk_op>" > ) > > ;; Same, but also producing a flags result. > -(define_insn "*aarch64_brk<brk_op>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > - [(match_operand:VNx16BI 1 "register_operand") > - (match_dup 1) > - (match_operand:SI 4 "aarch64_sve_ptrue_flag") > +(define_insn "*aarch64_brk<brk_op>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > + [(match_operand:VNx16BI 4 "register_operand") > + (match_operand 5) > + (match_operand:SI 6 "aarch64_sve_ptrue_flag") > (unspec:VNx16BI > - [(match_dup 1) > + [(match_operand:VNx16BI 1 "register_operand") > (match_operand:VNx16BI 2 "register_operand") > (match_operand:VNx16BI 3 "aarch64_simd_imm_zero")] > SVE_BRK_UNARY)] > @@ -11732,7 +11856,10 @@ (define_insn "*aarch64_brk<brk_op>_cc" > (match_dup 2) > (match_dup 3)] > SVE_BRK_UNARY))] > - "TARGET_SVE" > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[4], > + &operands[5], operands[1], > + operands[0])" > {@ [ cons: =0, 1 , 2 ; attrs: pred_clobber ] > [ &Upa , Upa , Upa ; yes ] brk<brk_op>s\t%0.b, %1/z, > %2.b > [ ?Upa , 0Upa, 0Upa; yes ] ^ > @@ -11742,20 +11869,24 @@ (define_insn "*aarch64_brk<brk_op>_cc" > ) > > ;; Same, but with only the flags result being interesting. > -(define_insn "*aarch64_brk<brk_op>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > - [(match_operand:VNx16BI 1 "register_operand") > - (match_dup 1) > - (match_operand:SI 4 "aarch64_sve_ptrue_flag") > +(define_insn "*aarch64_brk<brk_op>_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > + [(match_operand:VNx16BI 4 "register_operand") > + (match_operand 5) > + (match_operand:SI 6 "aarch64_sve_ptrue_flag") > (unspec:VNx16BI > - [(match_dup 1) > + [(match_operand:VNx16BI 1 "register_operand") > (match_operand:VNx16BI 2 "register_operand") > - (match_operand:VNx16BI 3 "aarch64_simd_imm_zero")] > + (match_operand:VNx16BI 3 "aarch64_simd_reg_or_zero")] > SVE_BRK_UNARY)] > UNSPEC_PTEST)) > (clobber (match_scratch:VNx16BI 0))] > - "TARGET_SVE" > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p > + (rtx_equal_p (operands[3], CONST0_RTX (VNx16BImode)) > + ? <PTEST_CC:MODE>mode : CC_NZCmode, > + operands[4], &operands[5], operands[1], operands[0])" > {@ [ cons: =0, 1 , 2 ; attrs: pred_clobber ] > [ &Upa , Upa , Upa ; yes ] brk<brk_op>s\t%0.b, %1/z, > %2.b > [ ?Upa , 0Upa, 0Upa; yes ] ^ > @@ -11795,9 +11926,9 @@ (define_insn "@aarch64_brk<brk_op>" > > ;; BRKN, producing both a predicate and a flags result. Unlike other > ;; flag-setting instructions, these flags are always set wrt a ptrue. > -(define_insn_and_rewrite "*aarch64_brkn_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_brkn_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 4) > (match_operand:VNx16BI 5) > (const_int SVE_KNOWN_PTRUE) > @@ -11825,9 +11956,9 @@ (define_insn_and_rewrite "*aarch64_brkn_cc" > ) > > ;; Same, but with only the flags result being interesting. > -(define_insn_and_rewrite "*aarch64_brkn_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite "*aarch64_brkn_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 4) > (match_operand:VNx16BI 5) > (const_int SVE_KNOWN_PTRUE) > @@ -11850,14 +11981,14 @@ (define_insn_and_rewrite "*aarch64_brkn_ptest" > ) > > ;; BRKPA and BRKPB, producing both a predicate and a flags result. > -(define_insn "*aarch64_brk<brk_op>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > - [(match_operand:VNx16BI 1 "register_operand") > - (match_dup 1) > - (match_operand:SI 4 "aarch64_sve_ptrue_flag") > +(define_insn "*aarch64_brk<brk_op>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > + [(match_operand:VNx16BI 4 "register_operand") > + (match_operand 5) > + (match_operand:SI 6 "aarch64_sve_ptrue_flag") > (unspec:VNx16BI > - [(match_dup 1) > + [(match_operand:VNx16BI 1 "register_operand") > (match_operand:VNx16BI 2 "register_operand") > (match_operand:VNx16BI 3 "register_operand")] > SVE_BRKP)] > @@ -11868,30 +11999,36 @@ (define_insn "*aarch64_brk<brk_op>_cc" > (match_dup 2) > (match_dup 3)] > SVE_BRKP))] > - "TARGET_SVE" > - {@ [ cons: =0, 1 , 2 , 3 , 4; attrs: pred_clobber ] > - [ &Upa , Upa , Upa , Upa , ; yes ] > brk<brk_op>s\t%0.b, %1/z, %2.b, %3.b > - [ ?Upa , 0Upa, 0Upa, 0Upa, ; yes ] ^ > - [ Upa , Upa , Upa , Upa , ; no ] ^ > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[4], > + &operands[5], operands[1], > + operands[0])" > + {@ [ cons: =0, 1 , 2 , 3 ; attrs: pred_clobber ] > + [ &Upa , Upa , Upa , Upa ; yes ] brk<brk_op>s\t%0.b, > %1/z, %2.b, %3.b > + [ ?Upa , 0Upa, 0Upa, 0Upa; yes ] ^ > + [ Upa , Upa , Upa , Upa ; no ] ^ > } > [(set_attr "sve_type" "sve_pred_cnt_ctrl")] > ) > > ;; Same, but with only the flags result being interesting. > -(define_insn "*aarch64_brk<brk_op>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > - [(match_operand:VNx16BI 1 "register_operand") > - (match_dup 1) > - (match_operand:SI 4 "aarch64_sve_ptrue_flag") > +(define_insn "*aarch64_brk<brk_op>_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > + [(match_operand:VNx16BI 4 "register_operand") > + (match_operand 5) > + (match_operand:SI 6 "aarch64_sve_ptrue_flag") > (unspec:VNx16BI > - [(match_dup 1) > + [(match_operand:VNx16BI 1 "register_operand") > (match_operand:VNx16BI 2 "register_operand") > (match_operand:VNx16BI 3 "register_operand")] > SVE_BRKP)] > UNSPEC_PTEST)) > (clobber (match_scratch:VNx16BI 0))] > - "TARGET_SVE" > + "TARGET_SVE > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[4], > + &operands[5], operands[1], > + operands[0])" > {@ [ cons: =0, 1 , 2 , 3 ; attrs: pred_clobber ] > [ &Upa , Upa , Upa , Upa ; yes ] brk<brk_op>s\t%0.b, > %1/z, %2.b, %3.b > [ ?Upa , 0Upa, 0Upa, 0Upa; yes ] ^ > @@ -11957,9 +12094,9 @@ (define_insn "*aarch64_sve_<sve_pred_op><mode>" > ) > > ;; Same, but also producing a flags result. > -(define_insn_and_rewrite "*aarch64_sve_<sve_pred_op><mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><PRED_ALL:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upa") > (match_operand 2) > (match_operand:SI 3 "aarch64_sve_ptrue_flag") > @@ -11977,9 +12114,12 @@ (define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><mode>_cc" > SVE_PITER))] > "TARGET_SVE > && <max_elem_bits> >= <elem_bits> > - && aarch64_sve_same_pred_for_ptest_p (&operands[2], &operands[4])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[2], &operands[4], > + operands[0])" > "<sve_pred_op>\t%0.<Vetype>, %1, %0.<Vetype>" > - "&& !rtx_equal_p (operands[2], operands[4])" > + "&& <PTEST_CC:MODE>mode == CC_NZCmode > + && !rtx_equal_p (operands[2], operands[4])" > { > operands[4] = operands[2]; > operands[5] = operands[3]; > @@ -11987,9 +12127,9 @@ (define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><mode>_cc" > [(set_attr "sve_type" "sve_pred_cnt_ctrl")] > ) > > -(define_insn_and_rewrite "*aarch64_sve_<sve_pred_op><mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><PRED_HSD:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upa") > (match_operand 2) > (match_operand:SI 3 "aarch64_sve_ptrue_flag") > @@ -12010,9 +12150,12 @@ (define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><mode>_cc" > 0) > (match_operand:PRED_HSD 7 "aarch64_ptrue_all_operand")))] > "TARGET_SVE > - && aarch64_sve_same_pred_for_ptest_p (&operands[2], &operands[4])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[2], &operands[4], > + operands[0])" > "<sve_pred_op>\t%0.<Vetype>, %1, %0.<Vetype>" > - "&& !rtx_equal_p (operands[2], operands[4])" > + "&& <PTEST_CC:MODE>mode == CC_NZCmode > + && !rtx_equal_p (operands[2], operands[4])" > { > operands[4] = operands[2]; > operands[5] = operands[3]; > @@ -12020,9 +12163,9 @@ (define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><mode>_cc" > ) > > ;; Same, but with only the flags result being interesting. > -(define_insn_and_rewrite "*aarch64_sve_<sve_pred_op><mode>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><PRED_ALL:mode>_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upa") > (match_operand 2) > (match_operand:SI 3 "aarch64_sve_ptrue_flag") > @@ -12035,9 +12178,12 @@ (define_insn_and_rewrite > "*aarch64_sve_<sve_pred_op><mode>_ptest" > (clobber (match_scratch:PRED_ALL 0 "=Upa"))] > "TARGET_SVE > && <max_elem_bits> >= <elem_bits> > - && aarch64_sve_same_pred_for_ptest_p (&operands[2], &operands[4])" > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, operands[1], > + &operands[2], &operands[4], > + operands[0])" > "<sve_pred_op>\t%0.<Vetype>, %1, %0.<Vetype>" > - "&& !rtx_equal_p (operands[2], operands[4])" > + "&& <PTEST_CC:MODE>mode == CC_NZCmode > + && !rtx_equal_p (operands[2], operands[4])" > { > operands[4] = operands[2]; > operands[5] = operands[3]; > diff --git a/gcc/config/aarch64/aarch64-sve2.md > b/gcc/config/aarch64/aarch64-sve2.md > index > c33df283cc2abbc64a7c9e161086b11a00dc7467..23f940a2ed5156848101f7e278977b4eb721a60d > 100644 > --- a/gcc/config/aarch64/aarch64-sve2.md > +++ b/gcc/config/aarch64/aarch64-sve2.md > @@ -4633,7 +4633,7 @@ (define_expand "check_<raw_war>_ptrs<mode>" > /* Emit a WHILERW or WHILEWR, setting the condition codes based on > the result. */ > emit_insn (gen_while_ptest > - (<SVE2_WHILE_PTR:unspec>, <MODE>mode, pred_mode, > + (<SVE2_WHILE_PTR:unspec>, <MODE>mode, pred_mode, CC_NZCmode, > gen_rtx_SCRATCH (pred_mode), operands[1], operands[2], > CONSTM1_RTX (VNx16BImode), CONSTM1_RTX (pred_mode))); > > @@ -4752,9 +4752,9 @@ (define_insn "*aarch64_pred_<sve_int_op><mode>" > > ;; Predicated string matching in which both the flag and predicate results > ;; are interesting. > -(define_insn_and_rewrite "*aarch64_pred_<sve_int_op><mode>_cc" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +(define_insn_and_rewrite > "*aarch64_pred_<sve_int_op><SVE_FULL_BHI:mode>_cc<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upl") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > @@ -4828,14 +4828,18 @@ (define_insn_and_rewrite > "*aarch64_pred_<sve_int_op><mode>_cc" > ) > > ;; Predicated string matching in which only the flags result is interesting. > -(define_insn_and_rewrite "*aarch64_pred_<sve_int_op><mode>_ptest" > - [(set (reg:CC_NZC CC_REGNUM) > - (unspec:CC_NZC > +;; MATCH/NMATCH set predicate bits in the same zero-extended way as integer > +;; compares, so a wider PTEST predicate is OK when the only live information > +;; is any/none. Keep the stricter same-predicate rule for the full CC_NZC > +;; form, since first/last still depend on the exact governing predicate. > +(define_insn_and_rewrite > "*aarch64_pred_<sve_int_op><SVE_FULL_BHI:mode>_ptest<ptest_cc>" > + [(set (reg:PTEST_CC CC_REGNUM) > + (unspec:PTEST_CC > [(match_operand:VNx16BI 1 "register_operand" "Upl") > (match_operand 4) > (match_operand:SI 5 "aarch64_sve_ptrue_flag") > (unspec:<VPRED> > - [(match_operand 6) > + [(match_operand 6 "register_operand" "Upl") > (match_operand:SI 7 "aarch64_sve_ptrue_flag") > (unspec:<VPRED> > [(match_operand:SVE_FULL_BHI 2 "register_operand" "w") > @@ -4846,9 +4850,11 @@ (define_insn_and_rewrite > "*aarch64_pred_<sve_int_op><mode>_ptest" > (clobber (match_scratch:<VPRED> 0 "=Upa"))] > "TARGET_SVE2 > && TARGET_NON_STREAMING > + && aarch64_sve_same_pred_for_ptest_p (<PTEST_CC:MODE>mode, > + &operands[4], &operands[6])" > + "<sve_int_op>\t%0.<Vetype>, %6/z, %2.<Vetype>, %3.<Vetype>" > + "&& !rtx_equal_p (operands[4], operands[6]) > && aarch64_sve_same_pred_for_ptest_p (&operands[4], &operands[6])" > - "<sve_int_op>\t%0.<Vetype>, %1/z, %2.<Vetype>, %3.<Vetype>" > - "&& !rtx_equal_p (operands[4], operands[6])" > { > operands[6] = copy_rtx (operands[4]); > operands[7] = operands[5]; > diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc > index > 107271b040454605fc40abba3fe5d678a8e27104..04b6a092fa865f50cf2b2cc8fb777b38416485dd > 100644 > --- a/gcc/config/aarch64/aarch64.cc > +++ b/gcc/config/aarch64/aarch64.cc > @@ -4239,21 +4239,25 @@ aarch64_pfalse_reg (machine_mode mode) > return gen_lowpart (mode, reg); > } > > -/* PRED1[0] is a PTEST predicate and PRED1[1] is an aarch64_sve_ptrue_flag > - for it. PRED2[0] is the predicate for the instruction whose result > - is tested by the PTEST and PRED2[1] is again an aarch64_sve_ptrue_flag > - for it. Return true if we can prove that the two predicates are > - equivalent for PTEST purposes; that is, if we can replace PRED2[0] > - with PRED1[0] without changing behavior. */ > +/* PRED1[0] is the predicate used by the PTEST-like flags result and > + PRED1[1] is the associated aarch64_sve_ptrue_flag. PRED2[0] and > + PRED2[1] describe the predicate used by the instruction that produces > + the tested value. > + > + This strict form is for patterns that need full PTEST flags. It therefore > + requires the two predicates to be the same, modulo the usual "known ptrue" > + representation. */ > > bool > aarch64_sve_same_pred_for_ptest_p (rtx *pred1, rtx *pred2) > { > machine_mode mode = GET_MODE (pred1[0]); > - gcc_assert (aarch64_sve_pred_mode_p (mode) > - && mode == GET_MODE (pred2[0]) > - && aarch64_sve_ptrue_flag (pred1[1], SImode) > - && aarch64_sve_ptrue_flag (pred2[1], SImode)); > + if (!aarch64_sve_pred_mode_p (mode) > + || mode != GET_MODE (pred2[0])) > + return false; > + > + gcc_assert (aarch64_sve_ptrue_flag (pred1[1], SImode)); > + gcc_assert (aarch64_sve_ptrue_flag (pred2[1], SImode)); > > bool ptrue1_p = (pred1[0] == CONSTM1_RTX (mode) > || INTVAL (pred1[1]) == SVE_KNOWN_PTRUE); > @@ -4262,6 +4266,175 @@ aarch64_sve_same_pred_for_ptest_p (rtx *pred1, rtx > *pred2) > return (ptrue1_p && ptrue2_p) || rtx_equal_p (pred1[0], pred2[0]); > } > > +/* Return true if a PTEST-like flags result governed by PRED1 can reuse the > + flags set by an instruction governed by PRED2, when the live flags are > + CC_MODE. > + > + Full PTEST semantics need the strict predicate equivalence checked by the > + overload above. A CC_Z result only answers any/none, so it is enough for > + the PTEST predicate to test every bit that the producer could set. > + > + A predicate with smaller elements has more element bits set. In one > + 16-bit chunk of predicate storage, the element bits are: > + > + .B: 1111111111111111 > + .H: 1x1x1x1x1x1x1x1x > + .S: 1xxx1xxx1xxx1xxx > + .D: 1xxxxxxx1xxxxxxx > + > + So, for a .S-producing instruction: > + > + possible result bits: RxxxRxxxRxxxRxxx > + ptrue .B tests: 1111111111111111 OK, superset > + ptrue .S tests: 1xxx1xxx1xxx1xxx OK, same bits > + ptrue .D tests: 1xxxxxxx1xxxxxxx no, misses bits > + > + But for a .B-producing instruction: > + > + possible result bits: RRRRRRRRRRRRRRRR > + ptrue .S tests: 1xxx1xxx1xxx1xxx no, misses bits > + > + This is why the PTEST predicate may be wider in storage terms, but not > + narrower. */ > + > +bool > +aarch64_sve_same_pred_for_ptest_p (machine_mode cc_mode, rtx *pred1, > + rtx *pred2) > +{ > + machine_mode pred1_mode = GET_MODE (pred1[0]); > + machine_mode pred2_mode = GET_MODE (pred2[0]); > + if (!aarch64_sve_pred_mode_p (pred1_mode) > + || !aarch64_sve_pred_mode_p (pred2_mode)) > + return false; > + > + gcc_assert (aarch64_sve_ptrue_flag (pred1[1], SImode)); > + gcc_assert (aarch64_sve_ptrue_flag (pred2[1], SImode)); > + > + if (pred1_mode == pred2_mode > + && aarch64_sve_same_pred_for_ptest_p (pred1, pred2)) > + return true; > + > + if (cc_mode != CC_Zmode) > + return false; > + > + /* A Z-only test can use a wider PTEST predicate than the producer's > + governing predicate, but not a narrower one. For example, a .B PTRUE > + covers all bits that a .S comparison can set, but a .S PTRUE does not > + cover all bits that a .B comparison can set. */ > + if (!known_ge (GET_MODE_NUNITS (pred1_mode), > + GET_MODE_NUNITS (pred2_mode))) > + return false; > + > + if (pred1[0] == CONSTM1_RTX (pred1_mode) > + || INTVAL (pred1[1]) == SVE_KNOWN_PTRUE) > + return true; > + > + return rtx_equal_p (pred1[0], gen_lowpart (pred1_mode, pred2[0])); > +} > + > +/* Like the overload above, but for RTL forms that keep both the real > + byte-level PTEST predicate and a version cast to the result mode. > + > + REAL_GP is the actual predicate operand passed to PTEST. PTEST contains > + the cast GP and PTRUE flag from the PTEST unspec; PRODUCER contains the > + producer's predicate and PTRUE flag. > + > + Full CC modes use the cast GP to require strict predicate equivalence. > + > + For CC_Z there are two extra cases: > + > + - If REAL_GP is the result itself, as in: > + > + cmpge p1.s, p0/z, z0.s, z1.s > + ptest p1, p1.b > + > + result bits: 0100100000000000 > + real GP bits: 0100100000000000 > + > + then the zero/nonzero answer is unchanged. A set bit in RESULT is also > + tested by REAL_GP, and an all-zero RESULT makes REAL_GP all zero too. > + > + - If the producer writes a full predicate mode, a wider known-ptrue PTEST > + predicate can test a superset of the bits that the producer can set: > + > + cmpge p1.s, p0/z, z0.s, z1.s > + ptrue p2.b > + ptest p2, p1.b > + > + The .s compare can only set the S element bits of p1, and p2.b tests all > + of those bits. This does not work in the other direction: a .s PTEST > + predicate would miss some bits that a .b producer could set. > + > + Partial data modes are different. Their inactive bits can be don't-care > + bits in the predicate value, but PTEST is a byte-level operation. In > those > + cases we only accept forms that preserve the original PTEST element > + interpretation. Predicate-only producers pass VOIDmode here; they already > + produce byte predicates, so there is no separate data mode to check. */ > + > +bool > +aarch64_sve_same_pred_for_ptest_p (machine_mode cc_mode, rtx real_gp, > + rtx *ptest, rtx *producer, rtx result, > + machine_mode producer_data_mode) > +{ > + rtx cast_gp = ptest[0]; > + rtx ptrue_flag = ptest[1]; > + machine_mode result_mode = GET_MODE (result); > + > + gcc_assert (aarch64_sve_pred_mode_p (GET_MODE (real_gp))); > + gcc_assert (aarch64_sve_ptrue_flag (ptrue_flag, SImode)); > + > + if (!aarch64_sve_pred_mode_p (GET_MODE (cast_gp))) > + return false; > + > + if (cc_mode != CC_Zmode) > + return aarch64_sve_same_pred_for_ptest_p (ptest, producer); > + > + /* A Z-only PTEST is redundant when it tests the result itself. */ > + if (aarch64_sve_pred_mode_p (result_mode)) > + { > + rtx byte_result = result; > + if (result_mode != VNx16BImode) > + byte_result = gen_lowpart (VNx16BImode, result); > + > + if (rtx_equal_p (real_gp, byte_result)) > + return true; > + } > + > + /* Known-ptrue is enough here only when PTEST sees the predicate bytewise. > */ > + if (real_gp == CONSTM1_RTX (VNx16BImode) > + || (GET_MODE (cast_gp) == VNx16BImode > + && INTVAL (ptrue_flag) == SVE_KNOWN_PTRUE)) > + return true; > + > + /* A full-mode producer can be tested by a wider known-ptrue PTEST. */ > + if (INTVAL (ptrue_flag) == SVE_KNOWN_PTRUE > + && (producer_data_mode == VOIDmode > + || aarch64_classify_vector_mode (producer_data_mode) == VEC_SVE_DATA) > + && result_mode == GET_MODE (producer[0]) > + && known_ge (GET_MODE_NUNITS (GET_MODE (cast_gp)), > + GET_MODE_NUNITS (GET_MODE (producer[0])))) > + return true; > + > + rtx producer_gp = producer[0]; > + if (GET_MODE (producer_gp) != VNx16BImode) > + producer_gp = gen_lowpart (VNx16BImode, producer_gp); > + > + return rtx_equal_p (real_gp, producer_gp); > +} > + > +/* Like the overload above, but for byte-predicate producers that have no > + separate ptrue flag in their RTL representation. Treat the producer as > + being governed by a known ptrue. */ > + > +bool > +aarch64_sve_same_pred_for_ptest_p (machine_mode cc_mode, rtx real_gp, > + rtx *ptest, rtx producer_gp, > + rtx result) > +{ > + rtx producer[] = { producer_gp, GEN_INT (SVE_KNOWN_PTRUE) }; > + return aarch64_sve_same_pred_for_ptest_p (cc_mode, real_gp, ptest, > + producer, result); > +} > > /* Generate a predicate to control partial SVE mode DATA_MODE as if it > were fully packed, enabling the defined elements only. */ > @@ -12718,6 +12891,13 @@ aarch64_select_cc_mode (RTX_CODE code, rtx x, rtx y) > if (GET_MODE_CLASS (mode_x) == MODE_FLOAT) > return aarch64_fp_cc_mode (code, mode_x); > > + /* Equality comparisons of vector predicates against zero only need the > + Z flag. */ > + if (GET_MODE_CLASS (mode_x) == MODE_VECTOR_BOOL > + && rtx_equal_p (y, CONST0_RTX (mode_x)) > + && (code == EQ || code == NE)) > + return CC_Zmode; > + > /* Equality comparisons of short modes against zero can be performed > using the TST instruction with the appropriate bitmask. */ > if (y == const0_rtx && (REG_P (x) || SUBREG_P (x)) > diff --git a/gcc/config/aarch64/iterators.md b/gcc/config/aarch64/iterators.md > index > 568786e37149a1292a7a0ed391f88fde51e23a62..0363a0398bcbe4bd2cf62c96f2dc18853d10995c > 100644 > --- a/gcc/config/aarch64/iterators.md > +++ b/gcc/config/aarch64/iterators.md > @@ -26,6 +26,13 @@ > (define_mode_iterator CC_ONLY [CC]) > (define_mode_iterator CCFP_CCFPE [CCFP CCFPE]) > > +;; PTEST-like SVE instructions normally set CC_NZC, since callers can ask > +;; about first/last active elements as well as any/none. When the only > +;; observable question is any/none, compare-elim can narrow the use to CC_Z. > +;; This iterator lets the flag-setting instruction still be recognized after > +;; that narrowing. > +(define_mode_iterator PTEST_CC [CC_NZC CC_Z]) > + > ;; Iterator for General Purpose Integer registers (32- and 64-bit modes) > (define_mode_iterator GPI [SI DI]) > > @@ -1501,6 +1508,8 @@ (define_c_enum "unspecv" > > ;; "e" for signaling operations, "" for quiet operations. > (define_mode_attr e [(CCFP "") (CCFPE "e")]) > +(define_mode_attr ptest_cc [(CC_NZC "nzc") (CC_Z "z")]) > +(define_mode_attr ptest_cc_suffix [(CC_NZC "") (CC_Z "z")]) > > ;; In GPI templates, a string like "%<w>0" will expand to "%w0" in the > ;; 32-bit version and "%x0" in the 64-bit version. > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brka_1.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brka_1.c > index > 6146ef828490e8482aff547ad6f4726723a134c2..087323939063965a18d4432c9b319ad5c4676c22 > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brka_1.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brka_1.c > @@ -18,6 +18,6 @@ test2 (svbool_t pg, svbool_t x, svbool_t y, int *any) > return svptest_any (pg, res); > } > > -/* { dg-final { scan-assembler-times {\tbrka\t} 2 } } */ > -/* { dg-final { scan-assembler-times {\tptest\t} 2 } } */ > -/* { dg-final { scan-assembler-not {\tbrkas\t} } } */ > +/* { dg-final { scan-assembler-times {\tbrka\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkas\t} 2 } } */ > +/* { dg-final { scan-assembler-not {\tptest\t} } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_1.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_1.c > index > 7f1c75837de913d6439e5766975c1d3dddeb11f2..bee876308dd31cc3b783cf2c694bfac56cad85f2 > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_1.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_1.c > @@ -18,6 +18,6 @@ test2 (svbool_t pg, svbool_t x, svbool_t y, int *any) > return svptest_any (pg, res); > } > > -/* { dg-final { scan-assembler-times {\tbrkb\t} 2 } } */ > -/* { dg-final { scan-assembler-times {\tptest\t} 2 } } */ > -/* { dg-final { scan-assembler-not {\tbrkbs\t} } } */ > +/* { dg-final { scan-assembler-times {\tbrkb\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkbs\t} 2 } } */ > +/* { dg-final { scan-assembler-not {\tptest\t} } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..f80bebbd40583bea69a6e91ecdd09b284e835d27 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1.c > @@ -0,0 +1,41 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +BOOL > +any (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_any (res, res); > +} > + > +BOOL > +not_any (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return !svptest_any (res, res); > +} > + > +BOOL > +first (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_first (res, res); > +} > + > +BOOL > +last (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_last (res, res); > +} > + > +/* { dg-final { scan-assembler-times {\tbrkbs\t} 2 } } */ > +/* { dg-final { scan-assembler-times {\tptest\t} 2 } } */ > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1_run.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1_run.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..637073b2c65bee636ac5e61f2f5437ae5116ff86 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/brkb_zonly_1_run.c > @@ -0,0 +1,57 @@ > +/* { dg-do run { target aarch64_sve_hw } } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +#define ASSERT(COND) do { if (!(COND)) __builtin_abort (); } while (0) > + > +__attribute__((noipa)) BOOL > +any (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_any (res, res); > +} > + > +__attribute__((noipa)) BOOL > +not_any (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return !svptest_any (res, res); > +} > + > +__attribute__((noipa)) BOOL > +first (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_first (res, res); > +} > + > +__attribute__((noipa)) BOOL > +last (svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkb_z (x, y); > + return svptest_last (res, res); > +} > + > +int > +main (void) > +{ > + svbool_t none = svpfalse (); > + svbool_t all = svptrue_b8 (); > + svbool_t first_lane = svwhilelt_b8 (0, 1); > + > + ASSERT (!any (none, all)); > + ASSERT (not_any (none, all)); > + ASSERT (!any (all, all)); > + ASSERT (not_any (all, all)); > + ASSERT (first (first_lane, none)); > + ASSERT (last (first_lane, none)); > + > + return 0; > +} > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..e1f28837ba3ddc4cd04850a932e251bc78a2467c > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1.c > @@ -0,0 +1,90 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +BOOL > +any_u8 (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svcmpge_u8 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +any_u16 (svbool_t pg, svuint16_t x, svuint16_t y) > +{ > + svbool_t res = svcmpge_u16 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +any_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +any_u64 (svbool_t pg, svuint64_t x, svuint64_t y) > +{ > + svbool_t res = svcmpge_u64 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +any_self_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_any (res, res); > +} > + > +BOOL > +not_any_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return !svptest_any (pg, res); > +} > + > +BOOL > +wide_ptest_u16 (svuint16_t x, svuint16_t y) > +{ > + svbool_t res = svcmpge_u16 (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +wide_ptest_u32 (svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (svptrue_b32 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +wide_ptest_u64 (svuint64_t x, svuint64_t y) > +{ > + svbool_t res = svcmpge_u64 (svptrue_b64 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +first_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_first (pg, res); > +} > + > +BOOL > +last_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_last (pg, res); > +} > + > +/* { dg-final { scan-assembler-times {\tcmphs\t} 11 } } */ > +/* { dg-final { scan-assembler-times {\tptest\t} 2 } } */ > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1_run.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1_run.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..475d9b663167582ce08f6ada38b4e493e01a30fd > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/cmpge_zonly_1_run.c > @@ -0,0 +1,131 @@ > +/* { dg-do run { target aarch64_sve_hw } } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve" } */ > + > +#include <arm_sve.h> > + > +typedef __UINT8_TYPE__ uint8_t; > +typedef __UINT16_TYPE__ uint16_t; > +typedef __UINT32_TYPE__ uint32_t; > +typedef __UINT64_TYPE__ uint64_t; > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +#define ASSERT(COND) do { if (!(COND)) __builtin_abort (); } while (0) > + > +__attribute__((noipa)) BOOL > +any_u8 (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svcmpge_u8 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +any_u16 (svbool_t pg, svuint16_t x, svuint16_t y) > +{ > + svbool_t res = svcmpge_u16 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +any_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +any_u64 (svbool_t pg, svuint64_t x, svuint64_t y) > +{ > + svbool_t res = svcmpge_u64 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +any_self_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_any (res, res); > +} > + > +__attribute__((noipa)) BOOL > +not_any_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return !svptest_any (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +wide_ptest_u16 (svuint16_t x, svuint16_t y) > +{ > + svbool_t res = svcmpge_u16 (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) BOOL > +wide_ptest_u32 (svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (svptrue_b32 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) BOOL > +wide_ptest_u64 (svuint64_t x, svuint64_t y) > +{ > + svbool_t res = svcmpge_u64 (svptrue_b64 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) BOOL > +first_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_first (pg, res); > +} > + > +__attribute__((noipa)) BOOL > +last_u32 (svbool_t pg, svuint32_t x, svuint32_t y) > +{ > + svbool_t res = svcmpge_u32 (pg, x, y); > + return svptest_last (pg, res); > +} > + > +int > +main (void) > +{ > + svbool_t pg8 = svptrue_b8 (); > + svbool_t pg16 = svptrue_b16 (); > + svbool_t pg32 = svptrue_b32 (); > + svbool_t pg64 = svptrue_b64 (); > + > + ASSERT (!any_u8 (pg8, svdup_u8 (0), svdup_u8 (1))); > + ASSERT (!any_u16 (pg16, svdup_u16 (0), svdup_u16 (1))); > + ASSERT (!any_u32 (pg32, svdup_u32 (0), svdup_u32 (1))); > + ASSERT (!any_u64 (pg64, svdup_u64 (0), svdup_u64 (1))); > + > + ASSERT (any_u8 (pg8, svdup_u8 (2), svdup_u8 (1))); > + ASSERT (any_u16 (pg16, svdup_u16 (2), svdup_u16 (1))); > + ASSERT (any_u32 (pg32, svdup_u32 (2), svdup_u32 (1))); > + ASSERT (any_u64 (pg64, svdup_u64 (2), svdup_u64 (1))); > + ASSERT (any_self_u32 (pg32, svdup_u32 (2), svdup_u32 (1))); > + ASSERT (!any_self_u32 (pg32, svdup_u32 (0), svdup_u32 (1))); > + ASSERT (not_any_u32 (pg32, svdup_u32 (0), svdup_u32 (1))); > + ASSERT (!not_any_u32 (pg32, svdup_u32 (2), svdup_u32 (1))); > + > + ASSERT (!wide_ptest_u16 (svdup_u16 (0), svdup_u16 (1))); > + ASSERT (!wide_ptest_u32 (svdup_u32 (0), svdup_u32 (1))); > + ASSERT (!wide_ptest_u64 (svdup_u64 (0), svdup_u64 (1))); > + ASSERT (wide_ptest_u16 (svdup_u16 (2), svdup_u16 (1))); > + ASSERT (wide_ptest_u32 (svdup_u32 (2), svdup_u32 (1))); > + ASSERT (wide_ptest_u64 (svdup_u64 (2), svdup_u64 (1))); > + > + ASSERT (first_u32 (pg32, svdup_u32 (2), svdup_u32 (1))); > + ASSERT (last_u32 (pg32, svdup_u32 (2), svdup_u32 (1))); > + ASSERT (!first_u32 (pg32, svdup_u32 (0), svdup_u32 (1))); > + ASSERT (!last_u32 (pg32, svdup_u32 (0), svdup_u32 (1))); > + > + return 0; > +} > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/pnext_ptest_1_run.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/pnext_ptest_1_run.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..535d9be8c905055d5eefe393d72eb8ef823ac1ed > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/pnext_ptest_1_run.c > @@ -0,0 +1,20 @@ > +/* { dg-do run { target aarch64_sve_hw } } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve" } */ > + > +#include <arm_sve.h> > + > +int __attribute__((noipa)) > +f (svbool_t pg) > +{ > + svbool_t res = svpnext_b16 (svptrue_b8 (), svpfalse ()); > + return svptest_any (pg, res); > +} > + > +int > +main (void) > +{ > + if (f (svpfalse ()) != 0) > + __builtin_abort (); > + > + return 0; > +} > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve/acle/general/ptest_zonly_1.c > b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/ptest_zonly_1.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..58d37f2fb275cd941d1831319981525e3c604135 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve/acle/general/ptest_zonly_1.c > @@ -0,0 +1,163 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +BOOL > +not_any (svbool_t pg, svbool_t x) > +{ > + svbool_t res = svnot_b_z (pg, x); > + return svptest_any (pg, res); > +} > + > +BOOL > +and_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svand_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +eor_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = sveor_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +orr_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svorr_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +bic_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svbic_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +orn_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svorn_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +nand_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svnand_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +nor_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svnor_b_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +rdffr_z_any (svbool_t pg) > +{ > + svbool_t res = svrdffr_z (pg); > + return svptest_any (pg, res); > +} > + > +BOOL > +rdffr_any (void) > +{ > + svbool_t res = svrdffr (); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +whilelt_any (int x, int y) > +{ > + svbool_t res = svwhilelt_b8 (x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +whilele_any (int x, int y) > +{ > + svbool_t res = svwhilele_b16 (x, y); > + return svptest_any (svptrue_b16 (), res); > +} > + > +BOOL > +brka_any (svbool_t pg, svbool_t x) > +{ > + svbool_t res = svbrka_z (pg, x); > + return svptest_any (pg, res); > +} > + > +BOOL > +brkb_any (svbool_t pg, svbool_t x) > +{ > + svbool_t res = svbrkb_z (pg, x); > + return svptest_any (pg, res); > +} > + > +BOOL > +brkn_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkn_z (pg, x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +brkpa_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkpa_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +brkpb_any (svbool_t pg, svbool_t x, svbool_t y) > +{ > + svbool_t res = svbrkpb_z (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +pfirst_any (svbool_t pg, svbool_t x) > +{ > + svbool_t res = svpfirst (pg, x); > + return svptest_any (pg, res); > +} > + > +BOOL > +pnext_any (svbool_t pg, svbool_t x) > +{ > + svbool_t res = svpnext_b8 (pg, x); > + return svptest_any (pg, res); > +} > + > +/* { dg-final { scan-assembler-times {\tnots\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tands\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\teors\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\torrs\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbics\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\torns\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tnands\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tnors\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\trdffrs\t} 2 } } */ > +/* { dg-final { scan-assembler-times {\twhilelt\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\twhilele\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkas\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkbs\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkns\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkpas\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tbrkpbs\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tpfirst\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tpnext\t} 1 } } */ > +/* { dg-final { scan-assembler-not {\tptest\t} } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest-2.c > b/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest-2.c > index > 8e590d6171a3be4d4d32deb7b657d64f293983bb..f54b5e49ddf3ce2ff259f463a7ececdc156b4335 > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest-2.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest-2.c > @@ -11,4 +11,4 @@ any_gt1 (signed char *restrict a, signed char *restrict b, > int n) > } > > /* { dg-prune-output "disable pass rtl-cmpelim" } */ > -/* { dg-final { scan-assembler-times {\tcmpgt\tp[0-9]+\.b, p[0-9]+/z, > z[0-9]+\.b, z[0-9]+\.b} 2 } } */ > +/* { dg-final { scan-assembler-times {\tcmpgt\tp[0-9]+\.b, p[0-9]+/z, > z[0-9]+\.b, z[0-9]+\.b} 1 } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest.c > b/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest.c > index > 79263044fb7bf255eea4765aa5315e597641734a..1bc7a46b69595e254e70a3d06a94c6b580df3d35 > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/cmpelim-sve-ptest.c > @@ -11,4 +11,4 @@ any_gt1 (signed char *restrict a, signed char *restrict b, > int n) > } > > /* { dg-final { scan-assembler-times {\tcmpgt\tp[0-9]+\.b, p[0-9]+/z, > z[0-9]+\.b, z[0-9]+\.b} 1 } } */ > -/* { dg-final { scan-assembler-not {\tptest\t} } } */ > +/* { dg-final { scan-assembler-times {\tptest\t} 1 } } */ > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/noeffect.c > b/gcc/testsuite/gcc.target/aarch64/sve/noeffect.c > index > f7109b1483cb63f48123577b49ce4ee09cbda0bf..aac752277d3aff9fed53360d4469a1e344f9b14d > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/noeffect.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/noeffect.c > @@ -11,8 +11,7 @@ int b[N] = {0, 0, 0, 1}; > ** ... > ** ldp q[0-9]+, q[0-9]+, \[x[0-9]+\] > ** cmplt p[0-9]+\.s, p[0-9]+/z, z[0-9]+\.s, z[0-9]+\.s > -** ptest p[0-9]+, p[0-9]+\.b > -** cset w0, any > +** cset w0, ne > ** ret > */ > __attribute__ ((noipa, noinline)) > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/noeffect10.c > b/gcc/testsuite/gcc.target/aarch64/sve/noeffect10.c > index > 39ab9489b8e035a337d536fe2a9be10b865f717c..1e9326b7b7d58a78c69350afb02d249bbd365833 > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/noeffect10.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/noeffect10.c > @@ -11,8 +11,7 @@ short b[N] = {0}; > ** ... > ** ldp q[0-9]+, q[0-9]+, \[x[0-9]+\] > ** cmplt p[0-9]+\.h, p[0-9]+/z, z[0-9]+\.h, z[0-9]+\.h > -** ptest p[0-9]+, p[0-9]+\.b > -** cset w0, any > +** cset w0, ne > ** ret > */ > __attribute__ ((noipa, noinline)) > diff --git a/gcc/testsuite/gcc.target/aarch64/sve/noeffect7.c > b/gcc/testsuite/gcc.target/aarch64/sve/noeffect7.c > index > 9dab90e72b25b552daad81fdc27dc69dc329cbb0..a2122bb2ccdc5a62caca367d6d9e52e13ff482cc > 100644 > --- a/gcc/testsuite/gcc.target/aarch64/sve/noeffect7.c > +++ b/gcc/testsuite/gcc.target/aarch64/sve/noeffect7.c > @@ -12,7 +12,7 @@ int b[N] = {0}; > ** ld2w \{z[0-9]+\.s - z[0-9]+\.s\}, p[0-9]+/z, \[x[0-9]+, x[0-9]+, lsl 2\] > ** ... > ** cmpgt p[0-9]+\.s, p[0-9]+/z, z[0-9]+\.s, z[0-9]+\.s > -** ptest p[0-9]+, p[0-9]+\.b > +** b(?:eq|ne) \.L[0-9]+ > ** ... > ** ldr w[0-9]+, \[x[0-9]+, x[0-9]+, lsl 2\] > ** ldr w[0-9]+, \[x[0-9]+, x[0-9]+, lsl 2\] > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1.c > b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..7974de8aeb0be97ea90f840d055eb846c553c3cc > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1.c > @@ -0,0 +1,56 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +BOOL > +match_any (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +nmatch_any (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svnmatch (pg, x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +match_wide_ptest (svint16_t x, svint16_t y) > +{ > + svbool_t res = svmatch (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +nmatch_wide_ptest (svint16_t x, svint16_t y) > +{ > + svbool_t res = svnmatch (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +match_first (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_first (svptrue_b8 (), res); > +} > + > +BOOL > +match_last (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_last (svptrue_b8 (), res); > +} > + > +/* { dg-final { scan-assembler-times {\tmatch\t} 4 } } */ > +/* { dg-final { scan-assembler-times {\tnmatch\t} 2 } } */ > +/* { dg-final { scan-assembler-times {\tptest\t} 6 } } */ > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1_run.c > b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1_run.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..ab250559ff17b4ffd175d9f10abead0f354eff68 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/match_zonly_1_run.c > @@ -0,0 +1,71 @@ > +/* { dg-do run { target aarch64_sve2_hw } } */ > +/* { dg-options "-O2 -march=armv8.2-a+sve2" } */ > + > +#include <arm_sve.h> > + > +#define ASSERT(COND) do { if (!(COND)) __builtin_abort (); } while (0) > + > +__attribute__((noipa)) _Bool > +match_any (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) _Bool > +nmatch_any (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svnmatch (pg, x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) _Bool > +match_wide_ptest (svint16_t x, svint16_t y) > +{ > + svbool_t res = svmatch (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) _Bool > +nmatch_wide_ptest (svint16_t x, svint16_t y) > +{ > + svbool_t res = svnmatch (svptrue_b16 (), x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) _Bool > +match_first (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_first (svptrue_b8 (), res); > +} > + > +__attribute__((noipa)) _Bool > +match_last (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch (pg, x, y); > + return svptest_last (svptrue_b8 (), res); > +} > + > +int > +main (void) > +{ > + svbool_t all8 = svptrue_b8 (); > + > + ASSERT (match_any (all8, svdup_u8 (7), svdup_u8 (7))); > + ASSERT (!match_any (all8, svdup_u8 (7), svdup_u8 (9))); > + ASSERT (nmatch_any (all8, svdup_u8 (7), svdup_u8 (9))); > + ASSERT (!nmatch_any (all8, svdup_u8 (7), svdup_u8 (7))); > + > + ASSERT (match_wide_ptest (svdup_s16 (7), svdup_s16 (7))); > + ASSERT (!match_wide_ptest (svdup_s16 (7), svdup_s16 (9))); > + ASSERT (nmatch_wide_ptest (svdup_s16 (7), svdup_s16 (9))); > + ASSERT (!nmatch_wide_ptest (svdup_s16 (7), svdup_s16 (7))); > + > + ASSERT (match_first (all8, svdup_u8 (7), svdup_u8 (7))); > + ASSERT (match_last (all8, svdup_u8 (7), svdup_u8 (7))); > + ASSERT (!match_first (all8, svdup_u8 (7), svdup_u8 (9))); > + ASSERT (!match_last (all8, svdup_u8 (7), svdup_u8 (9))); > + > + return 0; > +} > diff --git > a/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/ptest_zonly_1.c > b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/ptest_zonly_1.c > new file mode 100644 > index > 0000000000000000000000000000000000000000..03f7635a401d50aa6bb8c9e8c84557af24121d26 > --- /dev/null > +++ b/gcc/testsuite/gcc.target/aarch64/sve2/acle/general/ptest_zonly_1.c > @@ -0,0 +1,61 @@ > +/* { dg-do compile } */ > +/* { dg-options "-O2 -march=armv8.3-a+sve2" } */ > + > +#include <arm_sve.h> > + > +#ifdef __cplusplus > +#define BOOL bool > +#else > +#define BOOL _Bool > +#endif > + > +BOOL > +whilege_any (int x, int y) > +{ > + svbool_t res = svwhilege_b8 (x, y); > + return svptest_any (svptrue_b8 (), res); > +} > + > +BOOL > +whilegt_any (int x, int y) > +{ > + svbool_t res = svwhilegt_b16 (x, y); > + return svptest_any (svptrue_b16 (), res); > +} > + > +BOOL > +whilehs_any (unsigned int x, unsigned int y) > +{ > + svbool_t res = svwhilege_b32 (x, y); > + return svptest_any (svptrue_b32 (), res); > +} > + > +BOOL > +whilehi_any (unsigned long x, unsigned long y) > +{ > + svbool_t res = svwhilegt_b64 (x, y); > + return svptest_any (svptrue_b64 (), res); > +} > + > +BOOL > +match_any (svbool_t pg, svuint8_t x, svuint8_t y) > +{ > + svbool_t res = svmatch_u8 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +BOOL > +nmatch_any (svuint16_t x, svuint16_t y) > +{ > + svbool_t pg = svptrue_b16 (); > + svbool_t res = svnmatch_u16 (pg, x, y); > + return svptest_any (pg, res); > +} > + > +/* { dg-final { scan-assembler-times {\twhilege\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\twhilegt\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\twhilehs\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\twhilehi\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tmatch\t} 1 } } */ > +/* { dg-final { scan-assembler-times {\tnmatch\t} 1 } } */ > +/* { dg-final { scan-assembler-not {\tptest\t} } } */ > > > -- > <rb20884.patch>
