Daniël Mantione schrieb:
Op Wed, 22 Aug 2012, schreef Felipe Monteiro de Carvalho:
On Wed, Aug 22, 2012 at 9:36 PM, Martin Schreiber <mse00...@gmail.com>
wrote:
I am not talking about Unicode. I am talking about day by day
programming of
an average programmer where the live is easier with utf-16 than with
utf-8.
Unicode is not done by using pos() instead of character indexes.
I think everybody knows my opinion, I stop now.
Please be clear in the terminogy. Don't say "live is easier with
utf-16 than with utf-8" if you don't mean utf-16 as it is. Just say
"live is easier with ucs-2 than with utf-8", then everything is clear
that you are talking about ucs2 and not true utf-16.
That is nonsense.
* There are no whitespace characters beyond widechar range. This means you
can write a routine to split a string into words without bothing about
surrogate pairs and remain fully UTF-16 compliant.
How is this different for UTF-8?
* There are no characters with uppper/lowercase beyond widechar range.
That means if you write cade that deals with character case you don't
need to bother with surrogate pairs and still remain fully UTF-16
complaint.
How expensive is a Unicode Upper/LowerCase conversion per se?
* You can group Korean letters into Korean syllables, again without
bothering about surrogate pairs, as Korean is one of the many languages
that is entirely in widechar range.
The same applies to English and UTF-8 ;-)
Selected languages can be handled in special ways, but not all.
Many more examples exist. It's true there exist also many examples where
surrogates do need to be handled.
But... even if a certain piece of code doesn't handle e.g. Egyptian
hyroglyps correctly; there is no guarantee that a UTF-8 code would do,
since these scripts have many properties that are not compatible with
text processing codes designed for western languages, they need a lot of
custom code.
That's it!
In everydays coding I'm happy with AnsiStrings, covering English and
German. But when I want to deal with Unicode, except for display-only
purposes, I want to do it right and in the most simple way. This means
that I'd call the functions existing (in FPC?) for detecting
non-breakable character ranges, upper/lower case conversion etc., and
use (sub)strings all over to get rid of any byte/wordcount issues.
You mentioned Korean syllables splitting - is this a task occuring often
in Korean programs? I don't remember when I *ever* wanted to break
German or English words into syllables. At the begin of computer-based
publishing most German texts were hard to read, due to many wordbreak
errors. Finding syllables (as possible breakpoints), in detail in
foreign languages, still requires to use according library functions,
which do (hopefully) proper disambiguation. In my code I'd call the
GetSyllable function, and then split the string at the given points -
regardless of any encoding. Or, as I really did, break strings only at
word boundaries, again insensitive to any encoding.
Also breaking strings for display purposes, at a given pixel count, is
expensive. It's not sufficient to find possible breakpoints, it's also
required to narrow down the right breakpoint by repetitive tries. It's
not a good idea to simply add the width of individual characters,
instead the pixel width of every possible substring must be determined
individually. This means that the efficiency does not depend much on the
string encoding.
But another point becomes *really* important, when libraries with
beforementioned Unicode functions are used: The application and
libraries should use the *same* string encoding, to prevent frequent
conversions with every function call. This suggests to use the
library(=platform) specific string encoding, which can be different on
e.g. Windows and Linux.
Consequently a cross-platform program should be as insensitive as
possible to encodings, and the whole UTF-8/16 discussion turns out to be
purely academic. This leads again to an different issue: should we
declare an string type dedicated to Unicode text processing, which can
vary depending on the platform/library encoding? Then everybody can
decide whether to use one string type (RTL/FCL/LCL compatible) for
general tasks, and the library compatible type for text processing?
Or should we bite the bullet and support different flavors of the FPC
libraries, for best performance on any platform? This would also leave
it to the user to select his preferred encoding, stopping any UTF
discussion immediately :-]
DoDi
_______________________________________________
fpc-devel maillist - fpc-devel@lists.freepascal.org
http://lists.freepascal.org/mailman/listinfo/fpc-devel
