[CCing bug-gnulib, because it's a question about Gnulib]
Eli Zaretskii wrote:
> I've tried to run the Info reader from Texinfo 7.2.90 after setting
> the console encoding to use UTF-8 (a.k.a. "codepage 65001"), showing
> the ELisp Reference manual (which uses UTF-8 to encode non-ASCII
> characters). It didn't work as I expected: the UTF-8 sequences were
> shown as raw bytes instead of Unicode characters.
>
> AFAICT, this happens because display.c:printed_representation doesn't
> regognize UTF-8 byte sequences as such, and instead decides that they
> are just raw bytes. And that led me to this fragment:
>
> const char *
> printed_representation (mbi_iterator_t *iter, int *delim, size_t pl_chars,
> int *pchars, int *pbytes)
> {
> struct text_buffer *rep = &printed_rep;
>
> char *cur_ptr = (char *) mbi_cur_ptr (*iter);
> int cur_len = mb_len (mbi_cur (*iter));
>
> text_buffer_reset (&printed_rep);
>
> if (mb_isprint (mbi_cur (*iter)))
>
> This uses multibyte iteration and functions/macros like mb_len and
> mb_isprint from Gnulib, and they evidently don't recognize UTF-8
> encoding in this case. I didn't have time to look closely enough at
> the implementation (which is quite complex, and seems to use 32-bit
> Unicode codepoints and various functions that replace(?) the likes of
> mbrlen and mbrtowc), but it seems to still use mbstate_t as declared
> on the system headers.
>
> So my question to Bruno is: do the above functions/macros rely on
> mbrlen and mbrtowc from the Windows C runtime, or do they replace them
> from the ground up?
>
> I'm asking because AFAIK these functions as implemented in the legacy
> MSVCRT run-time library don't support UTF-8 encoding, only the newer
> UCRT runtime does. And even UCRT only supports UTF-8 when the system
> locale was set to something.UTF-8; just setting the terminal's
> encoding to UTF-8 is not enough. By contrast, I would like the Info
> reader to be capable of UTF-8 output when it runs on a Windows
> terminal whose encoding is UTF-8, and I'd like to be able to support
> this both in the MSVCRT and UCRT builds of the Info reader. So if
> Gnulib replaces the above functions with its own (or can potentially
> replace them, given some build-time knobs), I'd like to try that
> and/or fix the code involved to pay attention to the terminal's
> encoding, not just to the current system locale, if that's possible.
You are right regarding the limited support of UTF-8 locales on native
Windows. And the problem is not limited to Windows, it's a basic choice
of programming APIs.
There are three ways to read files that contain multibyte characters:
(A) Use locale-aware functions.
(B) Use a specific encoding always (e.g. UTF-8), independently of the locale.
(C) Support many encodings, independently of the locale.
Approach (A) consists of the function mbrtowc(), the macro MB_CUR_MAX, and
higher layers based on mbrtowc(): mbi_iterator_t etc.
Since locales are defined by the system (and many systems don't have a POSIX
compliant 'localedef' utility), this limits the available encodings:
- On native Windows with MSVCRT, UTF-8 locales are not supported.
Not without Gnulib and not with Gnulib, because we can't support
MB_CUR_MAX == 4 if the system supports only MB_CUR_MAX ≤ 2.
- On macOS, musl libc, Android, and other platforms, unibyte locales are not
supported because all locales use UTF-8 or ASCII.
- On AIX, the system supports UTF-8 locales; but if your sysadmin has only
installed ISO-8859-1 locales, you are doomed as well.
Approach (B) consists of using e.g. libunistring with the various u8_*
functions.
This is independent of the locale, but it's only one ASCII-compatible encoding.
Approach (C) generalizes (B) by supporting several encodings. Up to 10 types
of encodings can be supported (unibyte, UTF-8, EUC, EUC-JP, EUC-TW, BIG5,
BIG5-HKSCS, GBK, GB18030, Shift_JIS).
The code in texinfo/info/display.c uses approach (A), with the limitations
mentioned above.
If you want to overcome these limitations, the following questions need
to be answered first:
- Which text encodings can occur in Info files?
- Who decides about the text encoding in an Info file?
- Are there commands for converting an Info file from one encoding to
another (kind of info-iconv)?
If you want to use approach (B), it's a different API, documented in the
GNU libunistring manual [1] and available through Gnulib modules [2].
If you want to use approach (C), such code exists in GNU gettext (files
po-charset.h, po-charset.c, read-po-internal.h, read-po-lex.c), but it would
need serious refactoring before it could be used outside GNU gettext.
Bruno
[1] https://www.gnu.org/software/libunistring/manual/html_node/index.html
[2] https://www.gnu.org/software/gnulib/manual/html_node/libunistring.html
