On 28.05.14 22:24, Andres Valloud wrote:
Hey Philippe,
Yes but #= is blissfully unaware of normalization in Squeak/Pharo. In
fact AFAIK Squeak/Pharo is unaware of normalization. Having a short look
at it doesn't even look as if case insensitivity worked in Squeak/Pharo
outside of Latin-1 (I could be wrong though).
Yes, that's what I am thinking about. To be more explicit, suppose
"Unicode" series of characters got into the image via the keyboard, a
file, a socket... once decoded, what could one do with them? Are all
types of decoded character series going to be represented as instances
of a single class, although they have inherently different behavior?
I don't understand the question.
In addition you probably don't want #= to do normalization "because
performance". And even if you did you probably still want a fast path
for ByteString receiver and ByteString argument in which case #size is
safe.
Assuming all fixed width representation strings (e.g. byte strings) will
always have the same encoding (e.g. same code page), then the size check
for those seems ok to me.
All Strings are fixed width in Pharo/Squeak. If you have a single
non-Latin1 character (code point) in a String all characters in the
String are promoted from a byte to an OOP. #size then answers the number
of OOPs instead of the number of bytes. So #size always answers the
number of characters (code points) non-normalized (because there is no
way to do normalization in Pharo/Squeak).
Just to make sure, I am not celebrating all this complexity in the
world... however, given that it's there, how are we going to deal with
it? I'm concerned about the long term consequences of making things
more complex than they are by reinterpreting them. The problem I see is
that ultimately programs just won't Work(TM).
This seems to me a more general discussion than the problem at hand.
Again at the moment Pharo/Squeak is largely unaware of Unicode. It
supports very large code points but without any semantics (similar to
let's say Erlang with doesn't even have a String type). What
Pharo/Squeak however does know about is encodings for mapping the
Strings to and from bytes.
And quite honestly doing normalization in #= may cause things to just
won't Work(TM). Consider for example an HTTP request with a query string
with two different query fields that are normalized equal. Would you
want the values to be stored under one dictionary key?
Cheers
Philippe
