Re: multibyte characters in the Info reader
> From: Gavin Smith > Date: Fri, 16 Jan 2026 19:18:23 + > Cc: Eli Zaretskii , [email protected], [email protected] > > On Thu, Jan 15, 2026 at 10:33:58PM +0100, Bruno Haible via Bug reports for > the GNU Texinfo documentation system wrote: > > Eli Zaretskii wrote: > > > Regarding the encoding of the Info file, it is not a serious problem, > > > because (a) most Info files use UTF-8 anyway, and (b) the Info reader > > > already includes support for re-encoding other codesets to UTF-8 > > > (provided that the Info reader is build with libiconv). So the only > > > case where the encoding of the Info file is relevant is if the Info > > > reader was built without libiconv. > > > > I see. So the problem is reduced to displaying > > - (U) UTF-8 text in memory (most frequent case), or > > - (L) locale-encoded text in memory (only if no iconv API available). > > As I understand, it is case (L) that is handled in info. > > info attempts to recode Info files to the locale encoding (using the > iconv function in libc). It then uses locale-aware functions to process > the contents of Info files. It does not make explicit use of UTF-8 in > many places. Yes. > The proposal to use "libiconv" appears to assume that the target encoding > is always "UTF-8", which would require a slight change in how info loads > Info files: it would not be recoding Info files to the locale encoding, > but to UTF-8 always. It wasn't clear to me from this discussion whether > people understood that info already uses the iconv function. The proposal is to convert the file's text to UTF-8, process it in UTF-8, then convert to the target encoding when outputting the text to the screen. > It would then requiring rewriting the whole program to use libunistring > instead of libc functions. Yes. > Eli: what is missing from my understanding of your use case is what > is going on in scan.c:copy_converting, when the Info file is first > read in. Does conversion of input files to UTF-8, based on the locale, > actually happen? In the cases I tried, that conversion was not needed: the Info file was already in UTF-8. (In fact, I'm yet to see an Info file encoded in some Windows codepage -- it just doesn't happen IME, definitely not with GNU projects. Either the files are pure ASCII or they are in UTF-8.) The current code of the Info reader, after the patches that I submitted recently, reports "UTF-8" from nl_langinfo when the terminal's encoding is UTF-8. And since the file is in UTF-8, the iconv_to_output conversion is a no-op. But the character classification and iteration in printed_representation are still done, and they use the locale's encoding, because they eventually call locale-aware functions from the C runtime. > Can I clarify that "shown as raw bytes" means that they look like > "\302\251", i.e. as backslash escape sequences? Actually, even worse: some look like control characters, some (e.g., \200) look like ASCII strings produced to represent non-printable characters, i.e, with actual ASCII backslash and 3 octal digits. That's because printed_representation uses the locale-aware functions from the C runtime, and the locale hasn't been changed to use UTF-8 (and with the older Windows runtime MSVCRT it cannot be changed in principle, because MSVCRT didn't support UTF-8). What I wanted to accomplish was simple: have Info interpret the text as UTF-8, and output it as UTF-8. But because the C runtime functions like mbrlen and iswprint, which are called by mb_len and mb_isprint, don't recognize UTF-8, they return results which get in the way. > If the iteration over codepoints in printed_representation does not > work, not recognising non-ASCII UTF-8 sequences even though the terminal > supports them, then it would be better to fall back to ASCII substitutes > when the file is first read in. This would not be the best but would > be better then getting the "\302\251" everywhere. This would mean using > the degrade_utf8 function in scan.c. Another possibility is using > the //TRANSLIT flag for an encoding passed to iconv (I didn't know about > this possibility when I wrote the ASCII degradation code, as it wasn't > documented in the libc manual or anywhere else I looked.) This already works, and worked in previous versions of Texinfo. If the terminal's encoding is anything other than UTF-8, the Info reader degrades the non-ASCII characters to their ASCII equivalents, whether via //TRANSLIT or degrade_utf8. What I wanted was to allow Info output the original UTF-8 encoded characters to the Windows terminal when the terminal's encoding is UTF-8, even if the locale's codeset is different. And that will not work unless Info will learn to use UTF-8 aware functions from libunistring to handle multibyte characters _instead_ of the C runtime functions. So, to summarize: . there's no regression in the Info reader wrt Texinfo 7.2 . I would like to improve Info to support UTF-8 when possible, but that req
Re: multibyte characters in the Info reader
> From: Bruno Haible > Cc: [email protected], [email protected] > Date: Fri, 16 Jan 2026 14:41:00 +0100 > > > as the > > Windows runtime doesn't support well (or not at all) characters beyond > > the BMP, replacing its standard C functions in Gnulib with versions > > that accept char32_t codepoints > > This is an effort that is already completed in Gnulib: Since we can't > redefine the 'wchar_t' type, we had to create another set of functions > that work on char32_t[] strings. [2][3] A couple of GNU packages already > make use of it, so as to support characters outside the BMP correctly > on Cygwin and native Windows. Then maybe Texinfo could use that as well. > > and paying attention to the console's > > output codepage rather than the system locale's codeset > > This is a request of the past. For a couple of years now, the output > functions in the Microsoft runtime library have automatic conversion > from the locale encoding to the console's output codepage (e.g. > from CP1252 to CP850). [4] There is no need any more to care about > this difference in Gnulib or in GNU packages, except for the workarounds > mentioned in [4]. I think we are miscommunicating. I didn't mean to allude to what the Windows runtime does, I meant to allude to what GNU packages do when they run on Windows. While on Posix platforms the terminal's encoding is (AFAIK) determined only by the locale's codeset, on Windows users can change the encoding of the terminal without changing the system-wide locale. However, many GNU packages, being of Posix origin, only look at nl_langinfo(CODEEST) when they decide whether they should use UTF-8. What I suggest is that console programs which output text to the terminal pay attention to the terminal's encoded (via calling GetConsoleOutputCP) in preference to GetACP, and if the former returns codepage 65001, use UTF-8 internally and for writing to the console, even though the locale's codeset might not be UTF-8. This requires use of functions that convert between multibyte and wide-character representation of text which don't depend on the Windows runtime, because the Windows runtime won't support UTF-8 if the system locale is not set to something.UTF-8. For example, there are programs which refrain from supporting output of emoji when GetACP returns something other than 65001. But the Windows terminal on modern versions of Windows is entirely capable of displaying emoji if the console's codepage is 65001. So I'm about to install changes in GDB that remove this unnecessary limitation when the console's encoding is UTF-8. As another example, the next release of GNU Awk will improve support for Unicode on MS-Windows by using UTF-8 and char32_t internally when the console's encoding is UTF-8. I'm saying that other text-mode programs can and should move in this direction. But to be able to do so, they need to bypass the Windows runtime for conversions between multibyte and char32_t representations and for stuff like collation and character classification, and use either the Gnulib functions or their own equivalents.
Re: multibyte characters in the Info reader
On Thu, Jan 15, 2026 at 10:33:58PM +0100, Bruno Haible via Bug reports for the GNU Texinfo documentation system wrote: > Eli Zaretskii wrote: > > Regarding the encoding of the Info file, it is not a serious problem, > > because (a) most Info files use UTF-8 anyway, and (b) the Info reader > > already includes support for re-encoding other codesets to UTF-8 > > (provided that the Info reader is build with libiconv). So the only > > case where the encoding of the Info file is relevant is if the Info > > reader was built without libiconv. > > I see. So the problem is reduced to displaying > - (U) UTF-8 text in memory (most frequent case), or > - (L) locale-encoded text in memory (only if no iconv API available). As I understand, it is case (L) that is handled in info. info attempts to recode Info files to the locale encoding (using the iconv function in libc). It then uses locale-aware functions to process the contents of Info files. It does not make explicit use of UTF-8 in many places. The proposal to use "libiconv" appears to assume that the target encoding is always "UTF-8", which would require a slight change in how info loads Info files: it would not be recoding Info files to the locale encoding, but to UTF-8 always. It wasn't clear to me from this discussion whether people understood that info already uses the iconv function. It would then requiring rewriting the whole program to use libunistring instead of libc functions. (Texinfo already uses a lot of libunistring via gnulib in texi2any, although that part of the code is completely separate from info and uses a separate gnulib checkout.) Eli: what is missing from my understanding of your use case is what is going on in scan.c:copy_converting, when the Info file is first read in. Does conversion of input files to UTF-8, based on the locale, actually happen? Can I clarify that "shown as raw bytes" means that they look like "\302\251", i.e. as backslash escape sequences? If the iteration over codepoints in printed_representation does not work, not recognising non-ASCII UTF-8 sequences even though the terminal supports them, then it would be better to fall back to ASCII substitutes when the file is first read in. This would not be the best but would be better then getting the "\302\251" everywhere. This would mean using the degrade_utf8 function in scan.c. Another possibility is using the //TRANSLIT flag for an encoding passed to iconv (I didn't know about this possibility when I wrote the ASCII degradation code, as it wasn't documented in the libc manual or anywhere else I looked.)
Re: multibyte characters in the Info reader
Eli Zaretskii wrote: > > * Alternatively, you could create an extended copy of gnulib/lib/mbchar.h, > > defining an abstract "multibyte character" that is UTF-8 encoded in case > > (U) and locale encoded in case (L), i.e. depending on a global variable. > > And then, an equally extended copy of gnulib/lib/mbiter.h, defining the > > iterator over such multibyte characters. > > Yes. > > It's up to you as a Gnulib developer, but I tend to think that ... maybe > in the long run Gnulib should add the above-mentioned extensions to > its mbchar.h and mbiter.h. If there was large demand for it, I would do that. But I think most programs use _either_ strings are represented in locale encoding _or_ Unicode strings, for the reason explained in [1]. The current 'info' reader is quite particular in working on strings in locale encoding sometimes and on Unicode strings sometimes, in the same places. > as the > Windows runtime doesn't support well (or not at all) characters beyond > the BMP, replacing its standard C functions in Gnulib with versions > that accept char32_t codepoints This is an effort that is already completed in Gnulib: Since we can't redefine the 'wchar_t' type, we had to create another set of functions that work on char32_t[] strings. [2][3] A couple of GNU packages already make use of it, so as to support characters outside the BMP correctly on Cygwin and native Windows. > and paying attention to the console's > output codepage rather than the system locale's codeset This is a request of the past. For a couple of years now, the output functions in the Microsoft runtime library have automatic conversion from the locale encoding to the console's output codepage (e.g. from CP1252 to CP850). [4] There is no need any more to care about this difference in Gnulib or in GNU packages, except for the workarounds mentioned in [4]. Bruno [1] https://www.gnu.org/software/libunistring/manual/html_node/In_002dmemory-representation.html [2] https://www.gnu.org/software/gnulib/manual/html_node/Comparison-of-string-APIs.html [3] https://www.gnu.org/software/gnulib/manual/html_node/Comparison-of-character-APIs.html [4] https://lists.gnu.org/archive/html/bug-gnulib/2025-09/msg00199.html
Re: multibyte characters in the Info reader
> From: Bruno Haible > Cc: [email protected], [email protected] > Date: Thu, 15 Jan 2026 22:33:58 +0100 > > > Is there a way to use (B) when the locale's codeset is UTF-8 and use > > (A) otherwise? > > It *is* possible with Gnulib if you use the different APIs: > - u8_mbtouc etc. for the case (U) above, > - mbrtowc (or better: mbrtoc32) etc. for the case (L) above. > > Upon first sight, this would mean that you would need to duplicate > the logic of the functions 'find_diff', 'display_process_line', > 'printed_representation', 'display_update_node_text' for the two cases. > > But that would be unmaintainable. > > To avoid such code duplication, I can think of two maintainable > approaches: > > * You could declare libiconv a mandatory dependency, e.g. like in > gettext/DEPENDENCIES: > [...] > This means that case (L) would not occur any more, and you could use > the libunistring API (u8_mbtouc etc.) throughout display.c. Gavin, is this acceptable for you? If yes, it would be best, I think. (You could decide to defer such changes to the next release, of course.) > * Alternatively, you could create an extended copy of gnulib/lib/mbchar.h, > defining an abstract "multibyte character" that is UTF-8 encoded in case > (U) and locale encoded in case (L), i.e. depending on a global variable. > And then, an equally extended copy of gnulib/lib/mbiter.h, defining the > iterator over such multibyte characters. Yes. It's up to you as a Gnulib developer, but I tend to think that, as the Windows runtime doesn't support well (or not at all) characters beyond the BMP, replacing its standard C functions in Gnulib with versions that accept char32_t codepoints, and paying attention to the console's output codepage rather than the system locale's codeset, will allow GNU projects to have decent support for Unicode on Windows. So maybe in the long run Gnulib should add the above-mentioned extensions to its mbchar.h and mbiter.h. Thanks.
Re: multibyte characters in the Info reader
Eli Zaretskii wrote: > Regarding the encoding of the Info file, it is not a serious problem, > because (a) most Info files use UTF-8 anyway, and (b) the Info reader > already includes support for re-encoding other codesets to UTF-8 > (provided that the Info reader is build with libiconv). So the only > case where the encoding of the Info file is relevant is if the Info > reader was built without libiconv. I see. So the problem is reduced to displaying - (U) UTF-8 text in memory (most frequent case), or - (L) locale-encoded text in memory (only if no iconv API available). > Your questions above are mostly about encoding of the Info file, but > the actual problem I was talking about is with how text is displayed. > The encoding of the Windows terminal is, of course, only relevant to > how text is displayed. > > So what I would like to have is a way of bypassing the Windows CRT > functions, and using Gnulib's code to do the likes of mbrlen and > mbrtowc, but only when the locale's codeset is UTF-8. ... > Is that possible, with the current Gnulib? This is *not* possible if you only use the Gnulib mbrlen, mbrtowc, etc. functions. Because, as explained earlier, the functions mbrtowc and MB_CUR_MAX are one API, and Gnulib can't implement MB_CUR_MAX == 4 when the system libraries only support MB_CUR_MAX ≤ 2. > Is there a way to use (B) when the locale's codeset is UTF-8 and use > (A) otherwise? It *is* possible with Gnulib if you use the different APIs: - u8_mbtouc etc. for the case (U) above, - mbrtowc (or better: mbrtoc32) etc. for the case (L) above. Upon first sight, this would mean that you would need to duplicate the logic of the functions 'find_diff', 'display_process_line', 'printed_representation', 'display_update_node_text' for the two cases. But that would be unmaintainable. To avoid such code duplication, I can think of two maintainable approaches: * You could declare libiconv a mandatory dependency, e.g. like in gettext/DEPENDENCIES: GNU libiconv + Not needed on systems with glibc and on NetBSD. But highly recommended on all other systems. Needed for character set conversion of PO files from/to Unicode and for the iconv_ostream class of libtextstyle. + Homepage: https://www.gnu.org/software/libiconv/ + Download: https://ftp.gnu.org/gnu/libiconv/ + Pre-built package name: - On Debian and Debian-based systems: --, - On Red Hat distributions: --. - Other: https://repology.org/project/libiconv/versions + If it is installed in a nonstandard directory, pass the option --with-libiconv-prefix=DIR to 'configure'. + On mingw, a slim alternative is the 'win-iconv' package version 0.0.8 from https://github.com/win-iconv/win-iconv . This means that case (L) would not occur any more, and you could use the libunistring API (u8_mbtouc etc.) throughout display.c. * Alternatively, you could create an extended copy of gnulib/lib/mbchar.h, defining an abstract "multibyte character" that is UTF-8 encoded in case (U) and locale encoded in case (L), i.e. depending on a global variable. And then, an equally extended copy of gnulib/lib/mbiter.h, defining the iterator over such multibyte characters. This way, the dispatch between the two APIs (U) and (L) is in the files mbchar.h and mbiter.h, with only minor code duplications in display.c. Bruno
Re: multibyte characters in the Info reader
> From: Bruno Haible > Cc: [email protected], [email protected] > Date: Thu, 15 Jan 2026 15:52:26 +0100 > > You are right regarding the limited support of UTF-8 locales on native > Windows. And the problem is not limited to Windows, it's a basic choice > of programming APIs. > > There are three ways to read files that contain multibyte characters: > > (A) Use locale-aware functions. > (B) Use a specific encoding always (e.g. UTF-8), independently of the > locale. > (C) Support many encodings, independently of the locale. > > Approach (A) consists of the function mbrtowc(), the macro MB_CUR_MAX, and > higher layers based on mbrtowc(): mbi_iterator_t etc. > > Since locales are defined by the system (and many systems don't have a POSIX > compliant 'localedef' utility), this limits the available encodings: > - On native Windows with MSVCRT, UTF-8 locales are not supported. > Not without Gnulib and not with Gnulib, because we can't support > MB_CUR_MAX == 4 if the system supports only MB_CUR_MAX ≤ 2. > - On macOS, musl libc, Android, and other platforms, unibyte locales are not > supported because all locales use UTF-8 or ASCII. > - On AIX, the system supports UTF-8 locales; but if your sysadmin has only > installed ISO-8859-1 locales, you are doomed as well. > > Approach (B) consists of using e.g. libunistring with the various u8_* > functions. > This is independent of the locale, but it's only one ASCII-compatible > encoding. > > Approach (C) generalizes (B) by supporting several encodings. Up to 10 types > of encodings can be supported (unibyte, UTF-8, EUC, EUC-JP, EUC-TW, BIG5, > BIG5-HKSCS, GBK, GB18030, Shift_JIS). > > The code in texinfo/info/display.c uses approach (A), with the limitations > mentioned above. > > If you want to overcome these limitations, the following questions need > to be answered first: > - Which text encodings can occur in Info files? > - Who decides about the text encoding in an Info file? > - Are there commands for converting an Info file from one encoding to > another (kind of info-iconv)? What I would like to do is to have full support when the terminal can support UTF-8 encoding, and delegate the other output encodings to the system libraries. Regarding the encoding of the Info file, it is not a serious problem, because (a) most Info files use UTF-8 anyway, and (b) the Info reader already includes support for re-encoding other codesets to UTF-8 (provided that the Info reader is build with libiconv). So the only case where the encoding of the Info file is relevant is if the Info reader was built without libiconv. Your questions above are mostly about encoding of the Info file, but the actual problem I was talking about is with how text is displayed. The encoding of the Windows terminal is, of course, only relevant to how text is displayed. So what I would like to have is a way of bypassing the Windows CRT functions, and using Gnulib's code to do the likes of mbrlen and mbrtowc, but only when the locale's codeset is UTF-8. (The Windows port of the Info reader already overrides the locale's codeset with the codepage returned by GetConsoleOutputCP, so if the user sets that to codepage 65001, the Info reader will consider the locale's codeset to be UTF-8.) Is that possible, with the current Gnulib? > If you want to use approach (B), it's a different API, documented in the > GNU libunistring manual [1] and available through Gnulib modules [2]. Is there a way to use (B) when the locale's codeset is UTF-8 and use (A) otherwise? Because the Info reader already converts all external encodings to UTF-8 internally (provided libiconv is available), see scan.c:copy_converting in the Info sources. Thanks.
Re: multibyte characters in the Info reader
[CCing bug-gnulib, because it's a question about Gnulib]
Eli Zaretskii wrote:
> I've tried to run the Info reader from Texinfo 7.2.90 after setting
> the console encoding to use UTF-8 (a.k.a. "codepage 65001"), showing
> the ELisp Reference manual (which uses UTF-8 to encode non-ASCII
> characters). It didn't work as I expected: the UTF-8 sequences were
> shown as raw bytes instead of Unicode characters.
>
> AFAICT, this happens because display.c:printed_representation doesn't
> regognize UTF-8 byte sequences as such, and instead decides that they
> are just raw bytes. And that led me to this fragment:
>
> const char *
> printed_representation (mbi_iterator_t *iter, int *delim, size_t pl_chars,
> int *pchars, int *pbytes)
> {
> struct text_buffer *rep = &printed_rep;
>
> char *cur_ptr = (char *) mbi_cur_ptr (*iter);
> int cur_len = mb_len (mbi_cur (*iter));
>
> text_buffer_reset (&printed_rep);
>
> if (mb_isprint (mbi_cur (*iter)))
>
> This uses multibyte iteration and functions/macros like mb_len and
> mb_isprint from Gnulib, and they evidently don't recognize UTF-8
> encoding in this case. I didn't have time to look closely enough at
> the implementation (which is quite complex, and seems to use 32-bit
> Unicode codepoints and various functions that replace(?) the likes of
> mbrlen and mbrtowc), but it seems to still use mbstate_t as declared
> on the system headers.
>
> So my question to Bruno is: do the above functions/macros rely on
> mbrlen and mbrtowc from the Windows C runtime, or do they replace them
> from the ground up?
>
> I'm asking because AFAIK these functions as implemented in the legacy
> MSVCRT run-time library don't support UTF-8 encoding, only the newer
> UCRT runtime does. And even UCRT only supports UTF-8 when the system
> locale was set to something.UTF-8; just setting the terminal's
> encoding to UTF-8 is not enough. By contrast, I would like the Info
> reader to be capable of UTF-8 output when it runs on a Windows
> terminal whose encoding is UTF-8, and I'd like to be able to support
> this both in the MSVCRT and UCRT builds of the Info reader. So if
> Gnulib replaces the above functions with its own (or can potentially
> replace them, given some build-time knobs), I'd like to try that
> and/or fix the code involved to pay attention to the terminal's
> encoding, not just to the current system locale, if that's possible.
You are right regarding the limited support of UTF-8 locales on native
Windows. And the problem is not limited to Windows, it's a basic choice
of programming APIs.
There are three ways to read files that contain multibyte characters:
(A) Use locale-aware functions.
(B) Use a specific encoding always (e.g. UTF-8), independently of the locale.
(C) Support many encodings, independently of the locale.
Approach (A) consists of the function mbrtowc(), the macro MB_CUR_MAX, and
higher layers based on mbrtowc(): mbi_iterator_t etc.
Since locales are defined by the system (and many systems don't have a POSIX
compliant 'localedef' utility), this limits the available encodings:
- On native Windows with MSVCRT, UTF-8 locales are not supported.
Not without Gnulib and not with Gnulib, because we can't support
MB_CUR_MAX == 4 if the system supports only MB_CUR_MAX ≤ 2.
- On macOS, musl libc, Android, and other platforms, unibyte locales are not
supported because all locales use UTF-8 or ASCII.
- On AIX, the system supports UTF-8 locales; but if your sysadmin has only
installed ISO-8859-1 locales, you are doomed as well.
Approach (B) consists of using e.g. libunistring with the various u8_*
functions.
This is independent of the locale, but it's only one ASCII-compatible encoding.
Approach (C) generalizes (B) by supporting several encodings. Up to 10 types
of encodings can be supported (unibyte, UTF-8, EUC, EUC-JP, EUC-TW, BIG5,
BIG5-HKSCS, GBK, GB18030, Shift_JIS).
The code in texinfo/info/display.c uses approach (A), with the limitations
mentioned above.
If you want to overcome these limitations, the following questions need
to be answered first:
- Which text encodings can occur in Info files?
- Who decides about the text encoding in an Info file?
- Are there commands for converting an Info file from one encoding to
another (kind of info-iconv)?
If you want to use approach (B), it's a different API, documented in the
GNU libunistring manual [1] and available through Gnulib modules [2].
If you want to use approach (C), such code exists in GNU gettext (files
po-charset.h, po-charset.c, read-po-internal.h, read-po-lex.c), but it would
need serious refactoring before it could be used outside GNU gettext.
Bruno
[1] https://www.gnu.org/software/libunistring/manual/html_node/index.html
[2] https://www.gnu.org/software/gnulib/manual/html_node/libunistring.html
