Follow-up Comment #1, bug #59402 (group groff): Six years later, I think I understand the formatter enough to talk intelligently about these issues. We'll see.
[comment #0 original submission:]
> This bug report is the spawn of bug #57618; the relevant discussion begins at
> comment 8 in that bug report. This example illustrates the problem:
>
> $ printf "co\xF6perate\n" | uconv -f latin1 -x Any-NFD | groff -Kutf8 >
> /dev/null
> <standard input>:1: warning: can't find special character `u0308'
>
> Unpacking the above: the "printf" hard-codes a Latin-1 character in its
> otherwise plain ASCII output; the "uconv" converts its Latin-1 input to
> UTF-8, specifically requesting Normalization Form D; groff's -K option
> implicitly runs preconv first and tells it to expect UTF-8 input; and finally
> groff processes preconv's output, whereupon it doesn't understand the
> non-ASCII part.
I agree that GNU _troff_ should either accept the input _preconv_ generates,
or _preconv_ should not generate that input.
> (No error is generated if the above is changed to have uconv emit
> Normalization Form C (NFC) instead of NFD.)
Yes, because as a special case, GNU _troff_ accepts input characters in the
Unicode Latin-1 supplement, because they happen to be encodable as a single
byte in ISO 8859-1.
_groff_char_(7):
Unicode encodes far more characters than groff has names for;
special character escape forms based on numerical code points
enable access to any of them. Frequently used glyphs or glyph
combinations can be stored in strings, and new glyph names can be
created ad hoc with the char request; see groff(7).
\[unnnn[n[n]]]
is a Unicode numeric special character escape sequence. Any
Unicode code point can be accessed with four to six
hexadecimal digits, with hexadecimal letters accepted in
uppercase form only. Thus, \[u02DA] accesses the (spacing)
ring accent, producing “˚”.
Unicode code points can be composed as well; when they are, GNU
troff requires NFD (Normalization Form D), where all Unicode glyphs
are maximally decomposed. (Exception: precomposed characters in
the Latin‐1 supplement described above are also accepted. Do not
count on this exception remaining in a future GNU troff that
accepts UTF‐8 input directly.) Thus, GNU troff accepts
“caf\['e]”,
“caf\[e aa]”, and “caf\[u0065_0301]”, as ways to input
“café”.
(Due to its legacy 8‐bit encoding compatibility, at present it also
accepts “caf\[u00E9]” on ISO Latin‐1 systems.) Except for the
special character names listed in the “Input” columns of the glyph
tables below, GNU troff derives special character names
algorithmically, using a simplified version of the technique
employed by the Adobe Glyph List (AGL) algorithm ⟨https://github
.com/adobe-type-tools/agl-aglfn⟩. The (frozen) set of names that
can’t be derived algorithmically is called the groff glyph list
(GGL).
\[ubase‐character[_combining‐component]...]
constructs a composite glyph from Unicode numeric special
character escape sequences. The code points of the base
glyph and the combining components are each expressed in
hexadecimal, with an underscore (_) separating each
component. Thus, \[u006E_0303] produces “ñ”.
> The problem, then, is that preconv emits a Unicode sequence that groff
> doesn't understand.
At this point I must interrupt and beg for clear usage of terminology.
groff.info:
1.8 Conventions Used in This Manual
===================================
We apply the term "groff" to the language documented here, the GNU
implementation of the overall system, the project that develops that
system, and the command of that name. In the first sense, 'groff' is an
extended dialect of the 'roff' language, for which many similar
implementations exist. We say "the formatter" when speaking of behavior
that is generally true of 'troff' and 'nroff' programs.
So let's please say "troff" or "GNU troff" when referring to the formatter
program.
> Whether that's a problem with preconv, with groff, or with both, is unclear
> (to me), but if both, then this bug can fork into two bugs to track each
> problem separately.
I think this arguably a problem in both the _preconv_ and _troff_ programs,
but I have no hope of "fixing" the latter soon. I'll explain below.
> Branden's initial analysis was that this is "a bug in (1) preconv for not
> emitting composite Unicode glyph escapes and (2) troff for failing to accept
> them," though if the antecedent to "them" is "composite Unicode glyph
> escapes," there's a logic error there: troff can't be failing to accept
> something that it's not even being given. I read the "them" as referring to
> the sequence that preconv does in fact emit from the above input:
>
> * coo\[u0308]perate
>
> So this bug could be fixed by making groff as liberal as possible with the
> forms of Unicode escapes it recognizes.
Yes, you got the right antecedent. And GNU _troff_ could be fixed to be that
liberal, but it ain't gonna be easy. Nor, I propose, necessary.
> It already knows three equivalent forms of this string:
>
> * co\[:o]perate (historical roff style)
Historical (Clark-era) "groff" style, I think. I've never yet encountered any
other formatter that accepted syntax like `\(:o` for Latin-1 Supplement
characters. They do not appear in CSTR #54 (any edition).
> * co\[u00F6]perate (NFC)
Yes, because this specific character (and others in the range U+00A0 to
U+00FF) can be "cast" to a small integer and thrown into a bucket of type
`unsigned char`. That's going to prove to be a stifling limitation.
> * co\[u006F_0308]perate (NFD, but "spelled" differently)
Yes, this is pretty much NFD with a funny accent. It's a general and flexible
mechanism. Werner Lemberg brought us this feature (and the foregoing NFC
training wheel).
_groff_char_(7):
\[ubase‐character[_combining‐component]...]
constructs a composite glyph from Unicode numeric special
character escape sequences. The code points of the base
glyph and the combining components are each expressed in
hexadecimal, with an underscore (_) separating each
component. Thus, \[u006E_0303] produces “ñ”.
There's yet another possibility.
* co[o ad]perate
_groff_char_(7):
\[base‐character composite‐1 composite‐2 ... composite‐n]
is a composite glyph. Glyphs like a lowercase “e” with an
acute accent, as in the word “café”, can be expressed as \[e
aa].
(Hmm, I should probably quote that last bit of syntax since it can break, as
shown.)
> (A fix for bug #58796 would enable preconv to emit the first of the above.)
>
> The advantages of making groff grok the "o\[u0308]" sequence are:
>
> 0 it follows the “Be liberal in what you accept" principle
> 0 preconv would not have to change
> 0 groff could handle arbitrary combinations for which the other three forms
> do not exist, such as
> * Spin\[u0308]al Tap
>
> The drawback, as Branden elucidates it, is that it's significantly easier to
> change preconv to output something groff already recognizes.
Supporting this in GNU _troff_, as I understand it, would be dauntingly--I
almost said "stupefyingly"--hard.
The reason is that the formatter tries very hard, and largely succeeds at,
being an LR(1) parser, not LALR(1). This means that it generally does not
"look ahead" (LA) when scanning from left to right (LR), and it reads only one
input "token" (character, since that's what the Standard C Library stream
reading functions feed us) at a time.
So, when GNU _troff_ reads "coo\[u0308]perate", it works like this, pumping
the input stream character-by-character as it goes.
Upon each pump of the stream, it asks itself: "What did I just see?" Let's
work through the exercise. For brevity and our sanity, I'm ignoring the
ghastly complexity involved in the character-to-glyph resolution process, with
which I know you are at least passingly familiar.
* "A 'c'? Good, let me create a glyph node with the charinfo for 'c' and push
it onto the pending output line.
* An 'o'? Good, let me create a glyph node with the charinfo for 'o' and push
it onto the pending output line.
* An 'o'? Good, let me create a glyph node with the charinfo for 'o' and push
it onto the pending output line.
* A '\'. That's the escape character. Let me change state. Hi, I'm the
escape sequence interpreter. What do I see?
* A '['? Okay, we need to interpret a special character or class identifier
or composite character argument sequence. Hi, I'm the guy who does that.
What do I see?
* A 'u'? Hold on, this is ambiguous. Is the string of escape sequence
arguments longer than two characters?
* Yes. Try to resolve it as a `valid_unicode_code_sequence()`.
* It worked! What did I get?
* Uhhh, a combining character. What does this combine with? I don't know!
I've already forgotten everything that's come before in the input stream.
The intrepid reader may want to look at the
[https://cgit.git.savannah.gnu.org/cgit/groff.git/tree/src/roff/troff/input.cpp?h=1.24.2#n2740
code for the hairier part of the foregoing procedure].
Solving this problem would, I think, require that _every_
character-resolvable-to-a-glyph that GNU _troff_ reads not be processed
immediately, but held in some kind of staging area while input processing
continues, because when we read it we don't know if it will be followed by a
combining character/glyph. Only after we've read _another_ "base"
character/glyph, or some subset of the entire GNU _troff_ token universe that
I fear to enumerate that tells us we definitely won't have to combine anything
with the "staged" character/glyph that is "on deck", can we proceed.
The level of code complexity I foresee here is fearsome.
> In any case, the first step seems to be to define exactly what groff
> should--and, if relevant, should not--accept as valid forms of Unicode-escape
> input.
Humans tend to read entire words or even phrases at a time. Lexical analysis
is generally much more myopic.
I think the only thing to do here in the short-to-medium term is to fix the
_preconv_ bug. To truly support NFD with straight-up unescaped UTF-8, which I
is suppose is implicated by bug #40720, we're going to have to replace GNU
_troff_'s deepest character-reading function with something that does the "on
deck" thing. I nurse a feeble hope that GNU _libunistring_ or something like
that can be easily integrated, and that its API makes it easy to delegate
stream reading with a configurable set of "jam-on-the-brakes characters" that
effectively mark the end of combining character collection.
_______________________________________________________
Reply to this item at:
<https://savannah.gnu.org/bugs/?59402>
_______________________________________________
Message sent via Savannah
https://savannah.gnu.org/
signature.asc
Description: PGP signature
