On Fri, Aug 28, 2026 at 01:11:19PM +0200, Patrice Dumas wrote:
> Hello,
>
> In the past, I had proposed to always return Info string (from output(),
> with file name as an empty string, only output in tests) as a character
> string. See this message and following thread:
> https://lists.gnu.org/archive/html/bug-texinfo/2024-02/msg00043.html
>
> I have changed my mind, I now think that an Info string returned should
> always be considered as a binary string, and encoded. I think that
> a character string representation of Info is not meaningful, because the
> tags complement a string that is to be considered as a string of bytes,
> encoded such that the bytes count match. Also there are control
> characters which a re not classically in strings. (There is also the
> issue that the tag table is wrong, as noted by Gavin in the above
> thread, although this could be worked around it is much easier to
> output a correct string.)
>
> So, in commit
> https://cgit.git.savannah.gnu.org/cgit/texinfo.git/commit/?id=1d9d4c78663e18ab4ae2a613aa80f8176db3c9f0
> I changed that.
It makes sense for the reasons you explain.
> For the test output that goes to a .pl file, I decided to protect all
> the non-ascii characters in the string by \x** and generate a double
> quoted string. That way the string can be converted to UTF-8 like the
> other strings without problem. The drawback is that the files diff are
> less readable.
Yes, this kind of change in the output files is not an improvement:
-* error→ € ¡::
-* ª º ¿::
-* a b a sunny day å::
-* Å æ œ Æ Œ ø Ø ß ł Ł Ð ð þ::
-* ä ẽ î â à é ç ē e̊ e̋ ę::
-* ė ĕ e̲ ẹ ě ȷ e͡e::
-* ı Ḕ Ḉ::
-* “ ” ‘ ’ „ ‚::
-* « » « » ‹ ›::
+* error\xe2\x86\x92 \xe2\x82\xac \xc2\xa1::
+* \xc2\xaa \xc2\xba \xc2\xbf::
+* a b a sunny day \xc3\xa5::
+* \xc3\x85 \xc3\xa6 \xc5\x93 \xc3\x86 \xc5\x92 \xc3\xb8 \xc3\x98 \xc3\x9f
\xc5\x82 \xc5\x81 \xc3\x90 \xc3\xb0 \xc3\xbe::
+* \xc3\xa4 \xe1\xba\xbd \xc3\xae \xc3\xa2 \xc3\xa0 \xc3\xa9 \xc3\xa7 \xc4\x93
e\xcc\x8a e\xcc\x8b \xc4\x99::
+* \xc4\x97 \xc4\x95 e\xcc\xb2 \xe1\xba\xb9 \xc4\x9b \xc8\xb7 e\xcd\xa1e::
+* \xc4\xb1 \xe1\xb8\x94 \xe1\xb8\x88::
+* \xe2\x80\x9c \xe2\x80\x9d \xe2\x80\x98 \xe2\x80\x99 \xe2\x80\x9e
\xe2\x80\x9a::
+* \xc2\xab \xc2\xbb \xc2\xab \xc2\xbb \xe2\x80\xb9 \xe2\x80\xba::
It makes it harder to actually see what is in the output. Even though an
Info file is a "binary" output format, it does not mean that it doesn't matter
if the test output is unreadable.
Is there any way of getting back using UTF-8 in the reference test
files rather than the escape sequences with \x?
Could we wrap the strings with UTF-8 in them in a function call, like
instead of "error\xe2\x86\x92" we have something like
utf8::encode("error→") which should have the same value?
Another idea is to specify "no utf8;" instead of "use utf8;" for sections
of the test results files so that string literals are interpreted as
byte strings (i.e. with the UTF-8 flag off).
> A possible workaround would be to use output files
> systematically for info.
Generally I have found it harder in the past to investigate test results
when they are in separate output files rather than the *.pl files under
tta/perl/t/results.
> As a side note, this is also more or less needed to have a C
> implementation that give the same output as Perl without being
> cumbersome, but, although that is what made me rethink about the issue,
> I think that it is a good thing in itself to have Info as byte strings
> only independentrly of the C vs Perl output.
>
> --
> Pat
>
