I ran into some problems with GNU sed on NetBSD and endeavored to fix some of them by altering the mbrtoc32 replacement, so that in the C locale it always converts bytes with the top bit set to 0xDFxx, instead of doing it only sometimes. This patch also updates commentary to match current POSIX standards and some newer OS releases. * lib/mbrtoc32.c (mbrtoc32): Also convert (e.g.) 0xED to 0xDFED when !GNULIB_defined_mbstate_t && HAVE_WORKING_MBRTOC32 && HAVE_WORKING_C32RTOMB && !MBRTOC32_MULTIBYTE_LOCALE_BUG && MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ. * tests/test-mbrtoc16.c (main): * tests/test-mbrtoc32.c (main): Simplify, now that we always do that conversion. --- ChangeLog | 17 +++++++++++++++++ doc/posix-functions/c32rtomb.texi | 7 +++++++ doc/posix-functions/mbrtoc32.texi | 4 ++-- lib/mbrtoc32.c | 2 +- m4/mbrtoc16.m4 | 9 ++++----- m4/mbrtoc32.m4 | 9 ++++----- tests/test-mbrtoc16.c | 11 ++++------- tests/test-mbrtoc32.c | 11 ++++------- 8 files changed, 43 insertions(+), 27 deletions(-)
diff --git a/ChangeLog b/ChangeLog index 3fc2cfad5e..01f9e74297 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,20 @@ +2026-09-13 Paul Eggert <[email protected]> + + mbrtoc32: also protect high-bit bytes when EILSEQ + I ran into some problems with GNU sed on NetBSD and + endeavored to fix some of them by altering the mbrtoc32 replacement, + so that in the C locale it always converts bytes with the top + bit set to 0xDFxx, instead of doing it only sometimes. + This patch also updates commentary to match current POSIX standards + and some newer OS releases. + * lib/mbrtoc32.c (mbrtoc32): Also convert (e.g.) 0xED to 0xDFED + when !GNULIB_defined_mbstate_t && HAVE_WORKING_MBRTOC32 && + HAVE_WORKING_C32RTOMB && !MBRTOC32_MULTIBYTE_LOCALE_BUG && + MBRTOC32_IN_C_LOCALE_MAYBE_EILSEQ. + * tests/test-mbrtoc16.c (main): + * tests/test-mbrtoc32.c (main): + Simplify, now that we always do that conversion. + 2026-09-13 Bruno Haible <[email protected]> setlocale: Use bool. diff --git a/doc/posix-functions/c32rtomb.texi b/doc/posix-functions/c32rtomb.texi index 782df8ebeb..a03e3c2239 100644 --- a/doc/posix-functions/c32rtomb.texi +++ b/doc/posix-functions/c32rtomb.texi @@ -22,6 +22,13 @@ c32rtomb Portability problems not fixed by Gnulib: @itemize @item +On some platforms, when Gnulib's @code{mbrtoc32} module is used, +in the C locale @code{mbrtoc32} translates bytes in the range 0x80--0xFF +to @code{char32_t} values in the range 0xDF80--0xDFFF, +but when those @code{char32_t} values are given to this function +it fails with @code{EILSEQ} instead of translating them back: +glibc 2.44, FreeBSD 15, NetBSD 11, OpenBSD 7.9. +@item On some platforms, when using a multibyte encoding other than UTF-8, this function may incorrectly convert an invalid character to an invalid byte sequence instead of returning @code{(size_t) -1}: diff --git a/doc/posix-functions/mbrtoc32.texi b/doc/posix-functions/mbrtoc32.texi index cbb12e27a5..b64fb4b9ce 100644 --- a/doc/posix-functions/mbrtoc32.texi +++ b/doc/posix-functions/mbrtoc32.texi @@ -20,7 +20,7 @@ mbrtoc32 and set @code{errno} to @code{EILSEQ}: @c https://sourceware.org/PR19932 @c https://sourceware.org/PR29511 -glibc 2.35. +glibc 2.44, NetBSD 11.1. @item In the C or POSIX locales, this function returns values like in ISO-8859-1 locales, @@ -40,7 +40,7 @@ mbrtoc32 This function does not recognize multibyte sequences that @code{mbrtowc} recognizes on some platforms: @c https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=272293 -FreeBSD 13.2, +FreeBSD 15.1, Solaris 11.4, mingw, MSVC 14. @c For MSVC this is because it assumes that the input is always UTF-8 encoded. @c See https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/mbrtoc16-mbrtoc323 diff --git a/lib/mbrtoc32.c b/lib/mbrtoc32.c index 34ac7b8348..9cbac3504c 100644 --- a/lib/mbrtoc32.c +++ b/lib/mbrtoc32.c @@ -305,7 +305,7 @@ mbrtoc32 (char32_t *pwc, const char *s, size_t n, mbstate_t *ps) if ((size_t) -2 <= ret && n != 0 && ! hard_locale (LC_CTYPE)) { if (pwc != NULL) - *pwc = (unsigned char) *s; + *pwc = (*s & 0x80 ? 0xDF00 : 0) + (unsigned char) {*s}; return 1; } # endif diff --git a/m4/mbrtoc16.m4 b/m4/mbrtoc16.m4 index f8a8be460a..6233678100 100644 --- a/m4/mbrtoc16.m4 +++ b/m4/mbrtoc16.m4 @@ -1,5 +1,5 @@ # mbrtoc16.m4 -# serial 5 +# serial 6 dnl Copyright (C) 2014-2026 Free Software Foundation, Inc. dnl This file is free software; the Free Software Foundation dnl gives unlimited permission to copy and/or distribute it, @@ -268,10 +268,9 @@ AC_DEFUN([gl_MBRTOC16_EMPTY_INPUT] ]) ]) -dnl <https://pubs.opengroup.org/onlinepubs/9699919799/functions/mbrtowc.html> -dnl POSIX:2018 says regarding mbrtowc: "In the POSIX locale an [EILSEQ] error -dnl cannot occur since all byte values are valid characters." It is reasonable -dnl to expect mbrtoc16 to behave in the same way. +dnl <https://pubs.opengroup.org/onlinepubs/9799919799/functions/mbrtoc16.html> +dnl POSIX:2024 says: "In the POSIX locale an [EILSEQ] error cannot occur +dnl since all byte values are valid characters." AC_DEFUN([gl_MBRTOC16_C_LOCALE], [ diff --git a/m4/mbrtoc32.m4 b/m4/mbrtoc32.m4 index 36e9d4632b..1ed4ac4af6 100644 --- a/m4/mbrtoc32.m4 +++ b/m4/mbrtoc32.m4 @@ -1,5 +1,5 @@ # mbrtoc32.m4 -# serial 28 +# serial 29 dnl Copyright (C) 2014-2026 Free Software Foundation, Inc. dnl This file is free software; the Free Software Foundation dnl gives unlimited permission to copy and/or distribute it, @@ -187,10 +187,9 @@ AC_DEFUN([gl_MBRTOC32_EMPTY_INPUT] ]) ]) -dnl <https://pubs.opengroup.org/onlinepubs/9699919799/functions/mbrtowc.html> -dnl POSIX:2018 says regarding mbrtowc: "In the POSIX locale an [EILSEQ] error -dnl cannot occur since all byte values are valid characters." It is reasonable -dnl to expect mbrtoc32 to behave in the same way. +dnl <https://pubs.opengroup.org/onlinepubs/9799919799/functions/mbrtoc32.html> +dnl POSIX:2024 says: "In the POSIX locale an [EILSEQ] error cannot occur +dnl since all byte values are valid characters." AC_DEFUN([gl_MBRTOC32_C_LOCALE], [ diff --git a/tests/test-mbrtoc16.c b/tests/test-mbrtoc16.c index abebfbe68c..bb1c98c27d 100644 --- a/tests/test-mbrtoc16.c +++ b/tests/test-mbrtoc16.c @@ -151,18 +151,15 @@ main (int argc, char *argv[]) wc = (char16_t) 0xBADF; ret = mbrtoc16 (&wc, buf, 1, &state); - /* POSIX:2018 says regarding mbrtowc: "In the POSIX locale an - [EILSEQ] error cannot occur since all byte values are valid - characters." It is reasonable to expect mbrtoc16 to behave - in the same way. */ + /* POSIX:2024 says: "In the POSIX locale an [EILSEQ] error + cannot occur since all byte values are valid + characters." */ ASSERT (ret == 1); if (c < 0x80) /* c is an ASCII character. */ ASSERT (wc == c); else - /* On most platforms, the bytes 0x80..0xFF map to U+0080..U+00FF. - But on musl libc, the bytes 0x80..0xFF map to U+DF80..U+DFFF. */ - ASSERT (wc == (btoc32 (c) == 0xDF00 + c ? btoc32 (c) : c)); + ASSERT (wc == (0xDF00 | c)); ASSERT (mbsinit (&state)); ret = mbrtoc16 (NULL, buf, 1, &state); diff --git a/tests/test-mbrtoc32.c b/tests/test-mbrtoc32.c index 01a9180dd8..de2d5aebe7 100644 --- a/tests/test-mbrtoc32.c +++ b/tests/test-mbrtoc32.c @@ -151,18 +151,15 @@ main (int argc, char *argv[]) wc = 0xBADFACE; ret = mbrtoc32 (&wc, buf, 1, &state); - /* POSIX:2018 says regarding mbrtowc: "In the POSIX locale an - [EILSEQ] error cannot occur since all byte values are valid - characters." It is reasonable to expect mbrtoc32 to behave - in the same way. */ + /* POSIX:2024 says: "In the POSIX locale an [EILSEQ] + error cannot occur since all byte values are valid + characters." */ ASSERT (ret == 1); if (c < 0x80) /* c is an ASCII character. */ ASSERT (wc == c); else - /* On most platforms, the bytes 0x80..0xFF map to U+0080..U+00FF. - But on musl libc, the bytes 0x80..0xFF map to U+DF80..U+DFFF. */ - ASSERT (wc == (btoc32 (c) == 0xDF00 + c ? btoc32 (c) : c)); + ASSERT (wc == (0xDF00 | c)); ASSERT (mbsinit (&state)); ret = mbrtoc32 (NULL, buf, 1, &state); -- 2.55.0
