csanchezdll opened a new pull request, #17901:
URL: https://github.com/apache/nuttx/pull/17901
Without this fix, values of x where x * x + 1 are rounded down could make
asin() argument to be out of range.
## Summary
Some arguments to `atanf()` can make it (incorrectly) return NaN, because
the approximate formula
uses `asin()` with produces NaN for values ouf of [-1, 1] range, which
rounding operations combined with
`sqrtf()` can cause. In particular, NuttX libm implementation of `sqrtf()`
rounding triggers the problem easily,
though its rounding is correct. The problem lies in `atanf()` not being
totally safe.
## Impact
`atanf()` is defined in the range (-INFINITY, INFINITY), so it should never
return NaN for any finite input. For high numbers, it should return PI/2.
## Testing
This can be replicated with a simple program:
```
#include <math.h>
volatile float f;
int
main (int argc, char *argv[])
{
f = atanf(6418.04346);
return 0;
}
```
I tested on an ARMv7, but probably all platforms are affected. Run it after
configuring NuttX with `CONFIG_LIBM`, and you you will see `f` is NaN. The
problem is in the `atanf()` implementation:
```
(gdb) next
42 return asinf(x / sqrtf(x * x + 1.0F));
(gdb) p x
$2 = 6418.04346
(gdb) p x*x
$3 = 41191280
(gdb) p x*x + 1
$4 = 41191280
(gdb) p sqrtf(x*x + 1)
$5 = 6418.04297
(gdb)
```
An accurate result for `sqrtf(41191280)` would be 6418.043315528, but
6418.04297 is a **valid** rounding. IEEE 754 section 4.3.3 "Rounding attribute
requirements" states round-to-nearest is not required for binary
implementations, just recommended; round-toward-zero is valid as well, and
6418.04297 is the next SP float lower than 6418.043315528 which can be
represented.
The problem is 6418.04297 < x, making the division > 1 and `asinf()` to
return NaN.
This patch ensures the input to `asinf()` in `atanf()` implementation is
always in-bounds.
(Note enabling CONFIG_ARMV7_LIBM to enable HW-assisted `sqrtf()` square root
implementation and makes it round-to-nearest, which does not reproduce the
issue; this can happen with other HW implementations; ensure you are using SW
`sqrtf()` when trying to reproduce).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
