malloc() does not check its size arithmetic for overflow, so malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX) return a single page instead of NULL, and slightly smaller sizes fail with EINVAL instead of ENOMEM.
Patch 1 fixes it. Patch 2 adds nolibc-test cases for huge sizes, which would have caught it. Tested on top of nolibc/for-next 0e1c44b472e1, on x86_64 (GCC and clang) and i386, and on arm, arm64 and sparc64 under qemu-user: - nolibc-test, all tests: no failures with the series. The only difference from before is the three new tests, which pass. They also pass against glibc (make libc-test), which builds without warnings. - Without patch 1, all three new tests fail on every build: malloc(SIZE_MAX) and calloc(SIZE_MAX, 1) return a pointer, and malloc(SIZE_MAX - 4096) fails with EINVAL. - On x86_64, a separate program shows realloc(ptr, SIZE_MAX) returning a pointer before patch 1 and NULL with ENOMEM after it. Danish Khateeb (2): tools/nolibc: check for overflow in malloc() selftests/nolibc: test malloc() and calloc() with huge sizes tools/include/nolibc/stdlib.h | 12 +++++++++--- tools/testing/selftests/nolibc/nolibc-test.c | 11 +++++++++++ 2 files changed, 20 insertions(+), 3 deletions(-) base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5 -- 2.55.0
