malloc() does not check the addition of its header size for overflow, so malloc(SIZE_MAX), calloc(SIZE_MAX, 1) and realloc(ptr, SIZE_MAX) return a single page instead of NULL.
As suggested by Thomas, patch 1 drops the rounding to 4096 bytes, and patch 2 adds the overflow check on top. Tested on x86_64 (GCC and clang), i386, and on arm, arm64 and sparc64 under qemu-user: - nolibc-test: no failures, with patch 1 alone and with the series. - A separate program: malloc(), calloc() and realloc() with sizes close to SIZE_MAX fail with ENOMEM, and free() still unmaps every page. Changes in v2: - Drop the rounding in a separate patch, instead of checking it for overflow (Thomas) - Drop the test patch (Thomas) - Link to v1: https://lore.kernel.org/r/[email protected] Danish Khateeb (2): tools/nolibc: stop rounding malloc() sizes up to 4096 bytes tools/nolibc: check for overflow in malloc() tools/include/nolibc/stdlib.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) base-commit: 0e1c44b472e1ec21efdad1df21b10e5c568b65b5 -- 2.55.0
