On 2017-03-09 19:06, Richard Gaskin via use-livecode wrote:
Thanks. I don't mind the verbosity, but I could use some clarity:

There's been talk of LC using UTF-16 internally, but when I do this:

on mouseUp
   put "Hello" into s
   put the number of bytes of s
end mouseUp

...I get "5".

When does LC use UTF-16, and when it's not UTF-16 is it still
ISO-8959-1 or UTF-8?

Internally strings are stored as either UTF-16 or in the native encoding (MacRoman, Latin-1) depending on the content of the string and such (the engine transparently switches internal encoding as necessary). However, this is an internal implementation detail - it might do something completely different in the future...

Before 7, the idea of 'byte' and 'char' were synonymous - if you used a string in the context of something expecting text it interpreted as being a string encoded in the native encoding, if you used a string in the context of something expecting binary data it interpreted as being just plain bytes.

With the advent of 7 it is necessary to treat text and binary separately - they aren't the same thing at all for the simple reason that text only becomes binary when you choose a specific text encoding and apply it to the unicode string.

In order to ensure that code written prior to 7 worked identically in 7 it was necessary to add an automatic conversion between text and binary which preserved the previous behavior which (essentially) viewed text and binary strings as being the same thing.

Indeed, in the above code what is actually happening is this:

on mouseUp
  put "Hello" into a
  put the number of bytes of <implicit-text-to-data>(s)
end mouseUp

One important property which existed before 7 was that:

   the number of bytes in s == the number of chars in s

However, the definition of 'char' changed in 7 to mean a Unicode grapheme - something which will often require many bytes to encode in any encoding (e.g. [e, combining-acute] is a perfectly valid way to express e-acute in Unicode - taking two codepoints, and not one). In order to keep the above equivalence (which would break many things if it were not kept) the implicit-text-to-data conversion is defined as follows:

  repeat for each char tChar in tString
    get textEncode(tChar, "native")
    if textDecode(it, "native") is tChar then
       put it after tData
    else
       put "?" after tData
    end if
  end repeat

(Note: The engine does work quite hard to keep things as equivalent as possible - it normalizes tString to NFC first so that it doesn't matter if the string has passed through a process which has happened to decompose it, or if it has come from a source which favours decomposed representations - most notably Mac HFS filenames).

This approach means that any multi-codepoint character in Unicode still maps to a single byte - and any non-updated code which manipulates strings as if they are data will still work (albeit with some data loss in regards the original Unicode string - which it wasn't written to understand anyway).

In the future, it is entirely possible that we will make it a runtime error to implicitly convert between data and string (don't worry, it wouldn't be the default behavior) because if you aren't clear about how you are doing the conversion (i.e. which conversion you are using) it is a potential source of hard to find errors in code.

Warmest Regards,

Mark.

--
Mark Waddingham ~ [email protected] ~ http://www.livecode.com/
LiveCode: Everyone can create apps

_______________________________________________
use-livecode mailing list
[email protected]
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Reply via email to