Re: Character Sequences of Uncertain Rendering (was: Version linking?)

[Philippe Verdy via Unicode]

Actually the matras in questions in the first message were neither
left-to-right or right-to-left, they were two-part vowels, and repeatedly
encoded after a base letter.
Malayalam itself is left-to-right but this only makes sense for the order
of base letters. matras encoded after that are placed around it according
to the script rule, but two part vowels cause problem if multiple ones are
used. We know how to order the right parts that are postposed, but there's
no clear order for the left parts that are preposed (including when there
are also preposed one-part vowels).

This is kind of similar to the problem of defining the stacking order when
there are multiple diacritics above (or below) when they all compete for
the same position. If generally the option is to render them ordered from
the innermost to the outermost position (so successive diacritics noramlly
positioned above should stack vertically upward, but there are known
exception where they will be instead not stacking vertically but
horizontally either left-to-right or right-to-left, and some cases where
their order will also be reversed).

There are only common positions and stacking options which should be used
by default in absence of any kind of joiners between them. For all other
cases, we need additional joiner controls between them if this is not the
default. But here, what is the default for the uncomon case where there are
multliple occurences of the same two-part matras ? In my opinion, they
should still be ordering their respective left-part or right from from
innermost to outermost, so the left-parts will be rendered right to left,
and the right-parts will be rendered left-to-right.

Here the problem is that this is performed in Firefox only for a limited
number (2) of preposed one-part vowels or preposed diacritics, or preposed
left-parts (of two-part vowels). So after rendering the first two matras,
there's no space left for the third matra, which will then be rendered
entirely after the cluster, in a separate cluster (missing a base
consonnant so you see the dotted glyph in the middle). IE does seem to do
things correctly by supporting more left-side preposed matras or left-side
preposed "half-matras": it first decomposes the two-part matras into two
pseudo-matras for each part and then order the first pseudo-matra like
other preposed vowels, all by default right-to-left (i.e. from innermost to
outsermost when you place the center of view on the base letter).

But there's no special joiners encoded in Unicode to override the placement
(direction) or relative order of diacritics competing to the same position.
If one was used, it should be encoded just before that diacritic, but
twop-part diacritics are even more challenging as they could possibly need
one or two separate overrides (either for the left-part or the right-part,
or both !)

However for the case given above, it makes no sense to use what Google
Chrome currently renders for "കോോോ" (U+0D15, followed by 3 occurences of
U+0D4B).

To make it clear, I'll use ASCII-only notation :  for the base letter
(U+0D15) and  for the two-part diacritic U+0D4B, and  the dotted
circle.
- When we encode , the rendering should be "CMD". it is OK in all
browsers.
- When we encode  we also see "CCMDD" everywhere including in
Chrome or Firefox.
- Then comes the encoding  that IE correctly renders as
"CCCMDDD", but Chrome or Firefox cannot render this correctly, they first
render  as "CCMDD" then comes  left alone without base
consonnant, so a dotted circle is inserted and we see "CoD" as a glued (but
now separate) cluster, the final result is "CCMDDCoD" (which is still not
breakable whe ntrying to select it with keyboard/mouse/touch).

I think this is caused by the algorithm used in Chrome and Firefox
renderers that only offer at most two positions for preposed parts when
computing the reordered layout of glyphs. IE does this correctly by not
limiting the number of preposed glyphs or using a higher limit (I did not
test by using arbitrarily-long sequences of preposed vowels or two-part
vowels, or at least 4 of them then more).

I know that IE/Edge is capable now to stack very high stacks of diacritics
(and this was implemented probably for the Tibetan script, or for
supporting mathematical notations).

But still, overriding the default direction of stacking is unspecified in
Unicode, except for a few documented cases where some joiner controls are
used (for the "liquid" vowels that we consider as consonnants in Latin, and
that will be present in words borrowed to Indic languages in their script
using matras) to alter the restation of stacking (but without complex glyph
reordering)

consider also the case of Acute accent in Greek whose default position is
by default altered when they occur contextually with capital letters, from
above, to the left. so  is reordered as
, but most Greek fonts will render like their
precombined 

Re: Character Sequences of Uncertain Rendering (was: Version linking?)

[Richard Wordingham via Unicode]

On Sun, 27 Aug 2017 19:55:31 +0200
Philippe Verdy via Unicode  wrote:

> 2017-08-27 6:06 GMT+02:00 Richard Wordingham via Unicode <
> unicode@unicode.org>:  
 
> Canonical reordering is unambiguously refering to the canonical
> equivalences in TUS. These are automated and can occur at any time,
> and the only way to avoid them is to insert joiners. But they should
> never be needed for normal texts, except to split clusters or
> introduce semantic differences where they are relevant (and in that
> case the renderers will also try to distinguish them, otherwise they
> can freely reorder every sequence of diacritics with distinct
> non-zero combining classes and will represent all canonically
> equivlent sequences exactly the same way without distinguishing them).

This wasn't the sort of problem I was talking about.  The Indic
example with undefined rendering has two left matras with ccc=0.  The
questions was whether they should be displayed from left to right (as in
MS Edge) or right to left (as in Firefox).

The problem of diacritics below having different combining classes has
been raised for minority languages in Thai.  There seems a definite
prospect that the rendering order has to depend on the writing system -
and the other order would simply be wrong.  Standardisation occurs
outside the purview of the UTC.  The order may be forced by CGJ,
which is a joiner in name only when it occurs before combining marks.

Richard.

Re: Character Sequences of Uncertain Rendering (was: Version linking?)

[Philippe Verdy via Unicode]

2017-08-27 6:06 GMT+02:00 Richard Wordingham via Unicode <
unicode@unicode.org>:

> On Sat, 26 Aug 2017 21:52:19 +0200
> Philippe Verdy via Unicode  wrote:
>
> > 2017-08-26 21:28 GMT+02:00 Richard Wordingham via Unicode <
> > unicode@unicode.org>:
>
> > Of course SHY in this use is not suitable, but who knows if one will
> > not need this to split in tow parts what would be otherwise a single
> > cluster (possibly reordered by canonical reordering if one needs to
> > split between two Indic matras: this would suggest there's a need for
> > a new "empty base consonnant" for that Indic script, but SHY (U+00AD)
> > should probably not have the correct effect if it also inserts an
> > undesired line break opportunity, independantly of how the glyph
> > which would be rendered and the position (first or second line) where
> > it would be rendered if the linebreak is honored).
>
> I am confused as to what conceivable case you have in mind.  An example
> would help.  I wonder if I'm misunderstanding what you mean by
> 'canonical reordering'.


Canonical reordering is unambiguously refering to the canonical
equivalences in TUS. These are automated and can occur at any time, and the
only way to avoid them is to insert joiners. But they should never be
needed for normal texts, except to split clusters or introduce semantic
differences where they are relevant (and in that case the renderers will
also try to distinguish them, otherwise they can freely reorder every
sequence of diacritics with distinct non-zero combining classes and will
represent all canonically equivlent sequences exactly the same way without
distinguishing them).

Re: Character Sequences of Uncertain Rendering (was: Version linking?)

[Richard Wordingham via Unicode]

On Sat, 26 Aug 2017 21:52:19 +0200
Philippe Verdy via Unicode  wrote:

> 2017-08-26 21:28 GMT+02:00 Richard Wordingham via Unicode <
> unicode@unicode.org>:  

> Of course SHY in this use is not suitable, but who knows if one will
> not need this to split in tow parts what would be otherwise a single
> cluster (possibly reordered by canonical reordering if one needs to
> split between two Indic matras: this would suggest there's a need for
> a new "empty base consonnant" for that Indic script, but SHY (U+00AD)
> should probably not have the correct effect if it also inserts an
> undesired line break opportunity, independantly of how the glyph
> which would be rendered and the position (first or second line) where
> it would be rendered if the linebreak is honored).

I am confused as to what conceivable case you have in mind.  An example
would help.  I wonder if I'm misunderstanding what you mean by
'canonical reordering'.  Do you mean the order of codepoints, or the
arrangement of glyphs.  CGJ is available to preserve a specific
ordering of codepoints, though it is completely redundant in most Indic
scripts.

It is a fact that aksharas do get split between lines in manuscripts,
undesirable though it may be.  In a transcription intended to preserve
a division into lines, one would probably use NBSP at such a point,
and worry less about attempting to preserve the structure of the
line-broken akshara.  It seems that Unicode only supports word
boundaries and their absence where they provide or prohibit line
breaks.

Richard.

Re: Character Sequences of Uncertain Rendering (was: Version linking?)

[Philippe Verdy via Unicode]

2017-08-26 21:28 GMT+02:00 Richard Wordingham via Unicode <
unicode@unicode.org>:

>
> I'm wondering if there are any cases where a SHY _should_ go between a
> Latin letter and diacritic.  I can't think of any.
>

In standard Latin orthography you would not expect it, normally, but there
will be cases where this will still occur at random places between long
spans of letters.

However I did NOT suggest (like you are doing here) using SHY between a
Latin letter and any diacritic.

But may be you've been confused by the fact I took the example of free
insertion of SHY controls in alphabetic scripts in comparison to the free
insertion of joiner controls (not the same thing) between Indic letters
(including vowel matras or subjoined consonants that are encoded as
combining characters but are not really "diacritics").

Of course SHY in this use is not suitable, but who knows if one will not
need this to split in tow parts what would be otherwise a single cluster
(possibly reordered by canonical reordering if one needs to split between
two Indic matras: this would suggest there's a need for a new "empty base
consonnant" for that Indic script, but SHY (U+00AD) should probably not
have the correct effect if it also inserts an undesired line break
opportunity, independantly of how the glyph which would be rendered and the
position (first or second line) where it would be rendered if the linebreak
is honored).

If one wants an, empty base letter to combine with the diacritic after it,
I think it should be NBSP (U+00A0) to avoid the interpretation as a
"defective" cluster using a implied glyph such as the dotted circle (but
NBSP also has its own problems, notably for collation where it would
collate like a space instead of being ignorable at primary level: this can
be fixed however quite easily in collation tailorings, using collation
elements made with "NBSP+combining matra")