On 7 Sep 2008, at 20:53, Owen Winkler wrote:
>> "[I]f present" frightens me. If we just use the native string >> functions we can end up with radically different results. The exact >> behaviour of strtoupper()/strtolower() is undefined, and is noted as >> being locale dependant. This means in some locales bytes above 0x7F >> are upper-/lower-cased. In a locale that uses Windows-1252, on a >> string b"\xC3\xA9" (UTF-8 encoded U+00E9 LATIN SMALL LETTER E WITH >> ACUTE (é)), lowercasing it would result in"\xE3\xA9" (obviously, as a >> lowercase character, it should remain changed), a invalid UTF-8 >> sequence. > > Just to satisfy my curiosity, and realizing it might not be ultimately > correct, what is the occurrence of the scenario you've described? > Does > having this happen 30% of the time? 80%? I mean to ask, with what > frequency do people supply é as data in Windows-1252 that results in > an > invalid UTF-8 character sequence? > > Sure, some effort should be made to make this work for as much as is > practical, but I'm getting the impression that there are edge cases > where the effort isn't worth the return. Oh, I think my description wasn't good enough. It's a UTF-8 string of the bytes b"\xE3\xA9", which is a single Unicode character, é. Lowercasing it using strtolower(), on a lot of PHP installs (those who default locale is ISO-8859-1/Windows-1252), results in an invalid UTF-8 sequence. This is more just a reason why we can't fallback on to the normal binary string functions. >> The basic options are to either require mbstring (whose exact >> behaviour is PHP version dependant (though unlike iconv not C-library >> dependant)), or to implement the exact behaviour in userland PHP. The >> latter is the solution I took for my Unicode class (in fact, it never >> uses mbstring to decode strings so I can make error handling user- >> configurable, as some things require fatal error handling, others >> rely >> on non-fatal error handling), and only uses native support on PHP 6 >> (as this _is_ user-configurable). > > Ok, your Unicode class is great. Can we use it in a String class? Or > did you have something else in mind? Actually, half of it is totally broken, and it is barely written. I was more than anything else just pointing out what route I'd gone down for this same issue (it is far from a small undertaking to implement Unicode in PHP userland, FWIW). -- Geoffrey Sneddon <http://gsnedders.com/> --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/habari-dev -~----------~----~----~----~------~----~------~--~---
