[
https://issues.apache.org/jira/browse/FOP-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124651#comment-18124651
]
Jason Harrop commented on FOP-3341:
-----------------------------------
An FO, with Carlito as the distributions package it (OFL): attached
FOP-3341-language-fallback.fo and distro-fonts.xconf (adjust the font path).
{noformat}
fop -c distro-fonts.xconf -fo FOP-3341-language-fallback.fo -pdf out.pdf
pdftotext -bbox out.pdf -
{noformat}
Two blocks of {{office AVATAR}} in Carlito, 12pt, the second with
{{language="en"}}. Carlito has latn, grek and cyrl script tables and no DFLT,
and no legacy kern table.
On main at 5be8c69b6: the first block ligates (pdftotext gives {{office}},
U+FB03) and AVATAR is 37.02pt wide; the second block has no ligature
({{office}}, six letters) and AVATAR is 39.96pt, the bare advances:
language="en" matches no language system, the fallback looks for DFLT, and the
font has none, so nothing is shaped or kerned.
With the pull request's change both blocks read {{office}} and AVATAR is 37.02pt
in both.
> A font whose GSUB or GPOS has no DFLT script table silently gets no
> substitution and no kerning under a default script, with no warning
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FOP-3341
> URL: https://issues.apache.org/jira/browse/FOP-3341
> Project: FOP
> Issue Type: Bug
> Components: font/opentype
> Affects Versions: 2.11
> Reporter: Jason Harrop
> Priority: Minor
> Attachments: FOP-3341-language-fallback.fo, distro-fonts.xconf
>
>
> GlyphTable.matchLookups falls back to the default script when a lookup match
> comes up empty:
> {noformat}
> if (lm.isEmpty() && !OTFScript.isDefault(script) &&
> !OTFScript.isWildCard(script)) {
> return matchLookups(OTFScript.DEFAULT, OTFLanguage.DEFAULT, feature);
> }
> {noformat}
> The fallback targets DFLT. So for a font that has no DFLT entry in its script
> list, a script of DFLT finds nothing, and the fallback looks for DFLT again
> and finds nothing again. Nothing is applied and nothing is reported. The font
> is loaded, its tables are read, performsSubstitution and performsPositioning
> both return true, and every lookup silently misses.
> Fonts without a DFLT script table are not unusual. Carlito, the
> metric-compatible Calibri substitute packaged by most Linux distributions,
> declares cyrl, grek and latn and no DFLT. DejaVu declares DFLT alongside
> them, which is why the two behave differently on the same code path and why
> this is easy to mistake for a font-file or a loader problem.
> Measured, on the same font objects with only the script argument changing:
> {noformat}
> script Carlito 1.104 DejaVuLGCSerif
> latn 'fi' -> U+FB01 'fi' -> U+FB01
> * 'fi' -> U+FB01 'fi' -> U+FB01
> DFLT 'fi' -> 'f','i' 'fi' -> U+FB01
> zyyy 'fi' -> 'f','i' 'fi' -> U+FB01
> auto 'fi' -> 'f','i' 'fi' -> U+FB01
> {noformat}
> The language argument opens the same hole. A language with no langsys entry
> under the script makes the (script, language) match come up empty, and the
> fallback then goes to (DFLT, dflt), skipping the script's own default
> langsys. Measured 2026-09-27 through the command line, one fo:block, only the
> language attribute changing, read back with pdftotext:
> {noformat}
> attributes Carlito DejaVu Sans
> script=latn, no language ffi ffl fi fl ft ti ligated ffi ffl fi fl
> ligated
> script=latn language="en" nothing ligated nothing ligated
> {noformat}
> And the language never matches in the first place. The FO language property
> carries an ISO 639 code ("en", "tr"); the font's language systems are
> OpenType tags ("ENG", "TRK"); FOP passes the code through untranslated. So a
> language the font does distinguish is treated like one it does not:
> DejaVuLGCSerif's TRK system omits liga, and performSubstitution("fi", "latn",
> "tr") ligates while ("fi", "latn", "TRK") does not.
> Neither font has an ENG langsys under latn. DejaVu Sans does have DFLT, but
> its DFLT langsys carries case, ccmp, dlig and kern and no liga, so the
> fallback lands on a script table that lacks the feature. DejaVu Serif's DFLT
> langsys does carry liga, and we measured that it ligates under the same
> attributes. So whether a run ligates today is font-accidental: it depends on
> which features the font's author happened to put in DFLT, not on what the
> font offers for the script.
> This is not a corner case for a consumer. docx4j writes language="en"
> country="US" on every block and inline it produces, from w:lang and the
> document defaults, and it cannot drop the attribute because FOP selects
> hyphenation patterns by it. The first point of FOP-2092 (Thai shapes only
> under language="dflt") is this language half. The fix belongs in the fallback
> order, as OpenType layout engines do it: (script, language), then (script,
> dflt), then (DFLT, dflt). GlyphTable.matchLookups goes straight from the
> first to the third.
> It is not only substitution. The same script matching governs GPOS, so
> kerning disappears too, and that is the wider harm because it reaches every
> letter pair rather than a handful of ligature sequences:
> {noformat}
> Carlito legacy kern table: absent, 0 entries
> GPOS 'AV' latn kerned * kerned DFLT null
> DejaVu legacy kern table: present, 101 entries
> GPOS 'AV' latn kerned * kerned DFLT kerned
> {noformat}
> Carlito has no legacy kern table at all, so under a default script it gets no
> kerning from either mechanism. A consumer that sets kerning="true" on such a
> font gets nothing and is told nothing.
> Note that the wildcard script "*" works, because the wildcard matches the
> font's real script tables. GlyphMapping maps "auto" and "zyyy" to "*", so the
> common FO path is safe; it is a caller that passes DFLT, or "auto"/"zyyy"
> unnormalised, that loses everything.
> h3. Why it matters
> The failure is silent and invisible in rendering: the text is drawn, just
> unshaped and unkerned. Nobody notices until they compare advance widths
> against another renderer. It also produces exactly the misleading symptom
> this was first mistaken for: "FOP reads this font's tables but not that
> one's".
> h3. Proposed fix
> Done on the fork, branch CR-003-script-fallback (GlyphTable, OTFLanguage,
> OTFScript; tests GlyphTableFallbackTestCase on synthetic tables shaped like
> Carlito and DejaVu Sans, and LanguageSystemTestCase on the DejaVuLGCSerif in
> the test tree, whose TRK system omits liga):
> # Map the language to its OpenType tag first (a table of ISO 639 codes
> against the tags OTFLanguage already names: "tr" to TRK, "en-US" to ENG,
> "zh-TW" to ZHT; a tag is accepted as given; unknown codes match nothing and
> take the next step), then fall back in the order OpenType layout engines use:
> (script, language), then (script, dflt), then (DFLT, dflt). Each step is
> taken only when the one before found nothing.
> # OTFScript.isWildCard compares with the wildcard rather than DEFAULT
> (harmless today, corrected alongside).
> Measured after the change, same command line: Carlito under language="en"
> ligates (ffi fi fl ft ti, 26 glyphs to 21) and kerns (AVATAR 43.19 pt, was
> 46.62); DejaVu Sans under "en" ligates; DejaVuLGCSerif
> performSubstitution("fi", "latn", "tr") no longer ligates, which is what its
> TRK system says; Noto Sans Arabic under "ar" unchanged.
> This changes shaping and kerning for existing documents wherever a language
> was set and the font had no system for it, so it wants the release note. A
> safer first step, if preferred: an event when a font with GSUB or GPOS
> present yields no lookups for the requested script, turning the silent miss
> into a diagnosable one.
> Note that the kerning half is also gated by the font configuration's kerning
> flag, which did not reach GPOS at all (a separate report, FOP-3343), and by a
> reader defect that drops a script's default language system when a named
> language shares its table (FOP-3342): DejaVu Sans needs all three before it
> kerns under "en".
> h3. Provenance
> Found while implementing the docx4j FO renderer's gsub-features hook, tracing
> why one font shaped and another did not through the same pipeline. All the
> numbers above are from loading each font directly with FontLoader and calling
> performSubstitution and performPositioning, so no renderer or consumer
> pipeline is involved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
