I am trying to understand the mysteries of unicode encodings; the following may 
(or may not) be useful (or confusing) to others.

 The docs say the full chunk expression for a unicode character is
      byte i of codeunit j of codepoint k of character c  of str
(with the warning that this is 'not of general utility’ … indeed!)

Taking a look at the Emoji ‘flag of Scotland’ character 🏴󠁧󠁢󠁳󠁣󠁴󠁿 which won’t 
display here but exists in the Apple Color Emoji font and in corresponding 
fonts for other platforms, I get

put 🏴󠁧󠁢󠁳󠁣󠁴󠁿 into str
number of chars of str: 1

char 1 of str : 🏴󠁧󠁢󠁳󠁣󠁴󠁿 number of codepoints of char 1 of str:  7
     codepoint:1        1F3F4   with 2 codeunits (D83C DFF4)
     codepoint:2        0               with 0 codeunits - seems to be a 
placeholder rather an actual codepoint
     codepoint:3        E0067   (DB40 DC67)
     codepoint:4        0       
     codepoint:5        E0062   (DB40 DC62)
     codepoint:6        0       
     codepoint:7        E0073   (DB40 DC73)

number of codepoints of str: 7
number of codeunits of str: 14
number of codeunits of char 1 of str: 14

So there are 6 codeunits which are not in any codepoints (or at least not as 
reported by LC). They can be enumerated by looping over “codeunit j of str” 
rather than 'codeunit j of codepoint k of ..' Or by textEnccode(str,”UTF-16”) 
and then by enumerating the bytes of the binary encoded str.

Bytes in binary encoding = all the codeunits (encoding is actually in 
littleendian byte order, but given here in bigendian order, which is the order 
reported by enumerating the codeunits)
       D83C DFF4 DB40 DC67 DB40 DC62 DB40 DC73 DB40 DC63 DB40 DC74 DB40 DC7F

Which should correspond to codepoints
       1F3F4 E0067 E0062 E0073 E0063 E0074 E007F
And indeed if I manually build a UTF-16 string with these code points it does 
display as the flag of Scotland. So the lesson is that the reported chunks are 
not to be naively trusted  --- tho not exactly a bug given the documentation 

1F3F4 by the way is a black flag; the remaining codepoints are in the 
Variations unicode block. Amusingly the Rainbow flag emoji  is made up of 3 
characters, char 1 is a white flag, char 2 is an invisible vertical join 
instruction, char 3 is a rainbow. BTW, backspacing over the displayed Rainbow 
flag actually has to be done in three steps to remove the displayed glyph, 
which I think is not correct behaviour for an editor since it appears to the 
user as one unicode character. Apple's TextEdit for example deletes the Rainbow 
flag with a single backspace. There are nasties lurking here for text 
manipulation LC code. Perhaps there should be a new string element 
‘unicodeChar’? BTW I have nothing but huge admiration for the LC unicode 
implementation team, it is a subject of extreme complexity.

Another question (which I think has been raised before but I don’t think there 
was an answer?). When a character (codepoint) in a string is displayed, if the 
requested font does not have that codepoint the OS substitutes a glyph from 
another font (or the missing character glyph if no font supports the 
codepoint). So for example if you change the font of the above flag of Scotland 
to Arial, it still displays as the flag of Scotland, even though this glyph is 
not in Arial. LC will still report that the font of this character is Arial: 
from what I can gather this is not the fault of LC, the OS is doing this 
substitution behind its back (TextEdit does the same). But is there any way to 
find out (programatically) the actual font being used? 

use-livecode mailing list
Please visit this url to subscribe, unsubscribe and manage your subscription 

Reply via email to