> -----Original Message----- > From: Robin Dapp <[email protected]> > Sent: 24 September 2026 15:55 > To: Tamar Christina <[email protected]>; Robin Dapp > <[email protected]>; [email protected] > Cc: nd <[email protected]>; [email protected] > Subject: Re: [patch 1/14]middle-end: add new vector control flow IFNs and > optabs > > > Yeah, the idea atm is that the compare itself reduces to a known state > > so that > > COUNT_ACTIVE has a more defined input and output. I'll get back to this > > later responding to another point. > > > >> gets emitted. For RVV that approach should always better, for SVE I'm > >> not so sure. I could imagine that the predicate handling for ncopies > > >> 1 gets more expensive with that suggestion. > >> > >> Would masking COUNT_ACTIVE work for SVE, at least for the ncopies = 1 > >> case or am I missing something fundamental? Or maybe it just doesn't > >> scale beyond ncopies = 1? But wouldn't some of the extra work we need > >> to do (like the TER and duplicate use) in this series vanish if that > >> were a viable approach? > >> > > > > COUNT_ACTIVE doesn't work today when ncopies > 1 because the exits are > > reduced using exclusive OR. The vectorizer has a limitation that we can't > > introduce new cfg on the early exits. So with ncopies > 1 we know one > element > > is active but we don't know which one. > > > > The thing Artemiy is working on is not using an ORR reduction in such cases, > > and instead we reduce each copy individually so we know which exit was > taken. > > We'd still not need COUNT_ACTIVE to take an additional value however since > > The IV computation code will deal with the offset as part of the PHI node > with > > the vector "scalar" IV. > > > > So at the moment I'm having a hard time with why you'd want this, and > what the > > benefit would be vs normal combine patterns. For instance even with SVE > we > > transform x + COUNT_ACTIVE further into e.g. "INC_ACTIVE" in RTL.. > > IMHO the current RVV or length-based approach for early break has it > backwards. VCOND_MASK_LEN is a vehicle to convert a length into a mask > with built-in zeroing of tail lanes. In my opinion that's not necessary > and we should be able to work with the length directly. Right now, not > doing this limits our early-break usage unnecessarily. Conceptually, > VCOND_MASK_LEN cannot be fully fixed because of RVV ISA restrictions. > > Today, we already combine (popcount (VCOND_MASK_LEN)) into > "len_popcount" (under the restrictions above) so I'm suggesting to emit > the latter directly in order to avoid the restrictions.
Ok, but why would you want to use COUNT_ACTIVE which is there for non-masked and non-len targets instead of COND_LEN_COUNT_ACTIVE which does the exact thing you mentioned above? > > > The patch explicitly doesn't deal with ncopies > 1, along with reductions > > and stores since this grew things considerably. To support them I think we > > need to do more during materialization. > > I think what I'm proposing for RVV would work for ncopies > 1 in the > same way, reducing every copy individually. But I haven't implemented > it yet. > Yes, that's the only way it would work. Similarly this restriction in current codegen and ifcvt is also why I don't support more than one exit with this patch. It would grow it significantly to support so that's for GCC 18. > >> So to my understanding, right now with trapping math excluded, we don't > >> need a masked vec_cmp for RVV because we could just compare without > >> length, ignoring tail elements, then popcount the result and branch > >> dependent on it. With trapping math the cond_len variant is needed but > >> won't be optimized anyway. > > > > Not entirely, it's also needed for partial loops where the mask is required > > so > > we don't access or store elements beyond what e loaded. This is > > particularly > > important for targets where the else value of a load is not zero. > > Here I'd disagree to some degree and instead argue that it's actually > not needed. In partial vectorization we mask load and store and, > normally, let the rest of the loop operate at full length. Exceptions > to this rule are tail-observing operations like reductions, trapping > ops, etc. Those we always mask. > > Now, for early break I believe we have the choice where or what to mask: > > (1) The compare itself. > (2) The "reduction" (COUNT_ACTIVE) after the compare. > (3) The "branch". > > For SVE (1) and (2) are the same instruction (and the same operation in > gimple because flags aren't exposed) and we can't even choose what to > mask. For RVV all three are separate insns and we have more freedom. > > Now if we mask the reduction (2), there is no need for zeroing because > it will only look at the active values and the tail stays unobserved. > That way, RVV early-break handling wouldn't need to jump through as many > hoops as the SVE approach. That's not quite true. SVE compares are side-effectful in more than just fp exceptions. They also set CC flags. And it's the reason the example code in the cover letter has cmphi p6.b, p7/z, z31.b, #5 b.none .L8 splice z31.b, p6, z31.b, z31.b fmov w0, s31 ret notice the branch after the compare, because the compares themselves set the zero flag. Without masking, this would never be correct, this is why SVE does not have "unpredicated" compares. Because GIMPLE does not carry this information, the "cbranch" today re-sets the flags using a PTEST. But this means we're setting the flags twice, and this because expanding to an "unpredicated compare", i.e. a compare with an all true mask, and then later moving the & mask into the compare by combine means we're quite far down the optimization pipeline and so by that time we have no other passes that can optimize the compare futher. This is part of the motivation for the new compare IFNs. The propagate the mask at expand time directly into the initial RTL. Also in SVE (1) and (2) are not the same instruction as you say. COUNT_ACTIVE is an mask reduction, And SVE would use COND_COUNT_ACTIVE because the SVE instruction CNTP requires both the loop mask and the result mask [1] https://support.arm.com/documentation/ddi0487/mb/-Part-C-The-AArch64-Instruction-Set/-Chapter-C8-SVE-Instruction-Descriptions/-C8-2-Alphabetical-list-of-SVE-instructions/-C8-2-116-CNTP--predicate- The documentations for these are @cindex @code{IFN_COUNT_ACTIVE} internal function @item @samp{IFN_COUNT_ACTIVE} The @code{IFN_COUNT_ACTIVE} internal function directly maps to the @code{count_active} instruction pattern. It returns the number of active elements in its argument. @cindex @code{IFN_COND_COUNT_ACTIVE} internal function @item @samp{IFN_COND_COUNT_ACTIVE} The @code{IFN_COND_COUNT_ACTIVE} internal function directly maps to the @code{cond_count_active} instruction pattern. It returns the number of active elements in its second argument under the mask in its first argument. This is used in the cases where If (<condition>) return i; Or another computation involving the loop invariant. So (1) and (2) are unrelated. The codegen here is cmp_mask = cmp & loop_mask if (cmp_mask != 0) { partial_mask = . COND_TAKE_EXCL_ACTIVE (loop_mask, cmp_mask); index = .COND_COUNT_ACTIVE (loop_mask, partial_mask); return x + index; } where x is the scalar IV the vectorizer is maintaining. COUNT_ACTIVE* and friends are about manipulating the loop IV. Both here and in Alfie's patches. Now masking (3) is not possible, because Richi has stated repeatedly in the past not to want an IFN within a cbranch in GIMPLE. i.e. VECTOR_BOOLEAN semantics on comparisons are the same for every target in GIMPLE. This was the point of cond_[len]_vec_cbranch_{any/all}, at transition out of GIMPLE it would explicitly mask and len your branch. So I never understood why you said it wasn't useful for RVV, you're arguing for the exact semantics here :) FWIW this is also how x86_64 has implemented this optabs. This is again the idea behind the new compare IFNs, to make it so that by the time You leave GIMPLE all early break control statements have an explicit len. > > The current approach of masking (1) (or maybe 1 and 2) makes sense for > SVE because the operations cannot be separated anyway. But they are separate. I think there's a big misunderstanding here of what these operations do, which also becomes clear from your example below. > Or maybe they could, synthetically, but then we'd have a costly re-test. > Also, > zeroing is free. > > > Another reason I added the masked vec_cmp is because at least for SVE > > compares can set flags. And because of that we introduced in GCC 15 > > vec_cbranch_{any/all} which allow a target to distinguish between when the > > compare is setting a flag or when the non-flags results of the compare is > needed. > > > > By doing cmp & mask, we rely on combine to the compare into a masked > compare, > > which is great but then CSE cannot eliminate duplicate compares since > there's > > no CSE after combine. > > > > So cond_vec_cmp is intended to not rely on combine to push in the masks, > and as > > a side-effect also allows the expand patterns to be much simpler than > > before. > > > > I have chosen to do this quite late instead of through the vectorizer > > because > > passes such as cunroll can deal with & on VLA compares, but wouldn't be > able > > to deal with the new IFN. i.e. we'd lose many GIMPLE optimizations by > transforming > > it too early. > > > > Do you have an example of why you'd want to do the emission so early? > > Ideal RVV code for ncopies = 1 would look like: > > # ... > # loop body > vmslt v0, v2, v3 # v0 = v2 < v3, unmasked > vfirst a0, v0 # COND_LEN_INDEX_FIRST_ACTIVE, masked > bltz a0 # branch if < 0 > # scalar "tail" for the mismatched element > # ... > That's not what the optabs are supposed to be or do, COND_LEN_INDEX_FIRST_ACTIVE is an IV extraction optab. For instance If (<condition>) return i; gets transformed into for SVE cmp_mask = cmp & loop_mask if (cmp_mask != 0) return x + .COND_INDEX_FIRST_ACTIVE (loop_mask, cmp_mask); In SVE2 this maps to FIRSTP, https://support.arm.com/documentation/ddi0487/mb/-Part-C-The-AArch64-Instruction-Set/-Chapter-C8-SVE-Instruction-Descriptions/-C8-2-Alphabetical-list-of-SVE-instructions/-C8-2-192-FIRSTP and COND_INDEX_FIRST_ACTIVE is an optimization over using partial_mask = . COND_TAKE_EXCL_ACTIVE (loop_mask, cmp_mask); index = .COND_COUNT_ACTIVE (loop_mask, partial_mask); And this is the exact reason that COND_INDEX_FIRST_ACTIVE is undefined when the cmp_mask is empty, vs the combination of TAKE_EXCL + COUNT_ACTIVE which is defined to be num_lanes. . COND_INDEX_FIRST_ACTIVE and friends are an optimization that can only be applied by the vectorizer which generates the branch. The vectorizer knows that because the result is used within an early exit edge, that if that branch is taken the mask would never but empty. You cannot do this in RTL (well not easily, without a new RTL pass) because you'd have to walk the loop and identify which exit is an early exit and which is the main exit. Something you cannot tell anymore after the vectorizer, since the vectorizer can pick a main exit *other* than the natural loop latch. That's the EARLY_BREAK_PEELED cases the code deals with. And because operations that map to COND_INDEX_FIRST_ACTIVE typically define the empty case to be -1, this also means in RTL you can't blindly combine the mask partition + count operation. The vectorizer gives targets a choice here, Either implement *_INDEX_FIRST_ACTIVE or _TAKE_EXCL_* + COUND_ACTIVE. Note that for instance _TAKE_EXCL_ will be used for e.g. reductions or creating masks for side-effectful operations like stores. So I don't see why the undefinedness of this IFN for an empty mask would hinder RVV, it would never be called with it. As an IV update operation x + -1, would make no sense. For RVV the sequence described above would be > # ... > # loop body > vmslt v0, v2, v3 # v0 = v2 < v3, unmasked > vfirst a0, v0 # cond_len_vec_cbranch_any , masked > bltz a0 # branch if < 0 > # no scalar tail > # ... > cond_len_vec_cbranch_any was explicitly designed to be a reduction for the cases where you know your result it a vector boolean. Vs normal cbranch which requires a scalar boolean comparison. vec_cbranch_any explicitly because of that did not define a *result* value just that you must branch. This is why normally cbranch for SVE uses "PTEST". So In your case your cond_len_vec_cbranch_any can expand to vmslt + vfirst + bltz This is exactly what Adv. SIMD does. https://godbolt.org/z/E66hG1czr Notice how out of vec_cbranch_any Adv. SIMD expands this sequence: cmeq v31.4s, v31.4s, v30.4s umaxp v31.4s, v31.4s, v31.4s fmov x4, d31 cbz x4, .L2 which is precisely what you want no? > For ncopies > 1 we'd have/want ncopies * vfirst, so one per copy. > Probably the same direction Artemiy is heading? > > > Out of interest and as a future question, not for this series: All the > mask duplication, TER, optabs, isel changes are a churn as well and feel > very tailored to just a few early-break operations. What would need > changing to handle this more gracefully? You mentioned RTL > representation issues with flags, that should generally be solvable? > I also think we can do more in the gimple pipeline by stripping cond/len > and reassembling after simplification. > > >> Could we define the result as -1? RVV's vfirst returns -1 if no bits > >> are set and that would help to rely on it. > >> > > > > No, since the undefined is to deal with targes which don't have a native > instruction > > for this optimally. For instance adv. Simd does not have a single > > instruction to > > do so, and instead relies on CTZ, which is undefined for 0. So having to > > force > it to > > -1 requires such targets to artificially have additional instructions they > > could > never > > optimize away. > > And presumably for AdvSIMD the -1 sequence would be too expensive? But > it's in the exit block, right? Anyway, then I suppose I need to define > a separate IFN for RVV. Being able to rely on -1 makes things much > easier. We can discuss this once I have the rest in place. The `-1` would never ever be hit. Adv. SIMD does today cmhi v31.16b, v30.16b, v29.16b addhn v31.8b, v31.8h, v31.8h umov x0, v31.d[0] cbz x0, .L8 rbit x0, x0 <-- clz x0, x0 <-- where the two <-- are the result of COUNT_ACTIVE. Your suggestion is to force it to do cmhi v31.16b, v30.16b, v29.16b addhn v31.8b, v31.8h, v31.8h umov x0, v31.d[0] cbz x0, .L8 rbit x1, x0 clz x1, x1 cmp x0, 0 mov x0, -1 csel x0, x1, x0, ne The last 3 operations are dead. The csel condition would never be true. But it cannot be eliminated by any RTL optimizer. For that to happen it needs to know that the BB the instruction is in is guaranteed that there is always at least one bit set. This is not the same as nonzero_bits because this is a mask, and we simply don't know which bit may be non-zero. We just know there is one. So I don't understand why -1 makes it easier for RVV, it would quite literally never be used. Thanks, Tamar > > >> Usually we add len ifns (as well as mask ifns) to > >> internal_fn_len_index > >> and internal_fn_mask_index but as we can't re-use > >> add_mask_else_and_len_args due to optab operand order, it won't help > >> anyway. And I really hope we can untangle all of this eventually when > >> we have an RTL vector predicate. > >> > > > > One Issue I've faced is that genuinely many of the CFG control IFN don't > > work with an else other than zero. That leads to an unfortunate number > > of churn to use them since as you say I can't use the existing helpers. > > > > This is becoming a bigger issue as we keep adding CFG support and needs > > more looking at. I thought to bring it up at Cauldron. > > I think that might also be a consequence of the whole approach being > "mask focused". Under my RVV interpretation they would be "reductions" > (i.e. observers of tail elements) and must be explicitly masked > themselves. See my response to 7/14. > > -- > Regards > Robin
