Issue 241290
Summary [ConstantFolding] nextafter/nexttoward overflow folded as errno-free; ERANGE lost (miscompile at -O3)
Labels new issue
Assignees
Reporter jeffchen006
    `nextafter(DBL_MAX, INFINITY)` overflows and glibc sets `errno = ERANGE`. With the default `-fmath-errno`, EarlyCSE folds the call to +inf and then deletes it, because `isMathLibCallNoop` says it writes no errno.

**Reproducer** (`repro.c`)
```c
/* nextafter(DBL_MAX, INFINITY) overflows to inf, so glibc sets errno = ERANGE.
   Correct output: 34 (ERANGE on Linux) */
#include <errno.h>
#include <float.h>
#include <math.h>
#include <stdio.h>
int main(void) {
  errno = 0;
 nextafter(DBL_MAX, INFINITY);
  printf("%d\n", errno);
  return 0;
}
```

**Commands and output** (clang version 24.0.0git (https://github.com/llvm/llvm-project.git 8207e9143575), Target: x86_64-unknown-linux-gnu):
```console
$ clang -O0 repro.c -o o0 -lm
$ ./o0
34
$ clang -O3 repro.c -o o3 -lm
$ ./o3
0
```
Expected (as `-O0`): `34` (ERANGE). `-O3` prints `0`.

**Why the expected output is right**
One libm call with constant arguments; `errno` is set to 0 before and read after. C11 7.12.11.3 allows a range error on overflow and glibc sets ERANGE. The library is the same at -O0 and -O3; clang on Linux defaults to `-fmath-errno`, and the IR call has `memory(errnomem: write)`. UBSan/ASan/MSan at -O0 are clean.

**Potential Root cause**
`llvm/lib/Analysis/ConstantFolding.cpp`, `llvm::isMathLibCallNoop`, case `LibFunc_nextafter*` / `nexttoward*`: `return ConstantFoldNextToward(...) != nullptr;`. `ConstantFoldNextToward` returns null only for zero / denormal / signaling results; for DBL_MAX -> +inf it returns +inf. EarlyCSE replaces the call by the constant, and `wouldInstructionBeTriviallyDead` asks `isMathLibCallNoop`, which says true. (From ff125ae1bae7, #168794.)

**Possible Proposed fix**
```diff
 case LibFunc_nexttowardl: {
-        return ConstantFoldNextToward(Op0, Op1, F->getReturnType()) != nullptr;
+        auto *C = dyn_cast_or_null<ConstantFP>(
+            ConstantFoldNextToward(Op0, Op1, F->getReturnType()));
+        return C && !(C->getValueAPF().isInfinity() && !Op0.isInfinity());
       }
```
The value fold can stay; only "the call can be deleted" changes.

Related: PR #168794 (adds this fold; the review asked to also fold the overflow case, with FP exceptions in mind). errno under `-fmath-errno` was not discussed there, and no issue reports the lost ERANGE.

_______________________________________________
llvm-bugs mailing list
[email protected]
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-bugs

Reply via email to