[Bug target/127177] New: __builtin_isinf vectorization incorrectly signals FE_INVALID

[mkretz at gcc dot gnu.org via Gcc-bugs]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=127177

            Bug ID: 127177
           Summary: __builtin_isinf vectorization incorrectly signals
                    FE_INVALID
           Product: gcc
           Version: 17.0
            Status: UNCONFIRMED
          Keywords: missed-optimization
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: mkretz at gcc dot gnu.org
  Target Milestone: ---
            Target: x86_64-*-*

A loop over __builtin_isinff is vectorized to

        vbroadcastss    xmm1, DWORD PTR .LC1[rip]
        vandps  xmm1, xmm0, xmm1
        vbroadcastss    xmm0, DWORD PTR .LC3[rip]
        vcmpnltps       xmm0, xmm0, xmm1
        vpcmpeqd        xmm1, xmm1, xmm1
        vpandn  xmm0, xmm0, xmm1

The vcmpnltps instruction signals invalid on quiet NaNs.

Test case (https://compiler-explorer.com/z/Gx66KWEvs):

#include <cfenv>
#include <utility>

using V [[gnu::vector_size(16)]] = float;

[[gnu::noipa]] auto visinf(V x) {
  using K [[gnu::vector_size(16)]] = int;
  return K{-__builtin_isinff(x[0]), -__builtin_isinff(x[1]),
           -__builtin_isinff(x[2]), -__builtin_isinff(x[3])};
}

int main() {
  V v = {0, 0, __builtin_inff(), __builtin_nanf("")};
  std::feclearexcept(FE_ALL_EXCEPT);
  visinf(v);
  return std::fetestexcept(FE_INVALID);
}


A correct and more efficient vectorization would be:

        vpcmpeqd        ymm1, ymm1, ymm1
        vpslld  ymm2, ymm1, 31
        vorps   ymm0, ymm0, ymm2
        vpslld  ymm1, ymm1, 24
        vpcmpeqd        ymm0, ymm0, ymm1

I.e., set the sign bit -> integer-compare against -inf