Here is another option: Only use UTF8. Process and display UTF8, store as UTF8 in the database, and require the database to support UTF8 (which all of them do...at least MySQL 4.1+ and PG and SQLite3).
In Drizzle, we have just decided to scrap all other character sets completely, going with 4-byte UTF8 for everything, and only handle collations, not character sets. For Habari, which doesn't even have to deal with various performance issues related to compression and other things with UTF8, I see no reason not to just do UTF8 across the board and just assume everything is UTF8, so all strings going into and out of Habari go through a String class which always does UTF8. Doing it this way will make porting to PHP6 a breeze, since only the String class would have to be modified. Sure, developers must learn to use the String class, but that's not a big deal (that's what code reviews are for, anyway) +1 on String class, +1 on UTF8 for everything, +1 on a requirement the DB must support UTF8, +1 on making all .schema files UTF8 as well... My 2 euro. ;) -jay On Sun, Sep 7, 2008 at 6:56 PM, Geoffrey Sneddon <[EMAIL PROTECTED]>wrote: > > > On 7 Sep 2008, at 20:53, Owen Winkler wrote: > > >> "[I]f present" frightens me. If we just use the native string > >> functions we can end up with radically different results. The exact > >> behaviour of strtoupper()/strtolower() is undefined, and is noted as > >> being locale dependant. This means in some locales bytes above 0x7F > >> are upper-/lower-cased. In a locale that uses Windows-1252, on a > >> string b"\xC3\xA9" (UTF-8 encoded U+00E9 LATIN SMALL LETTER E WITH > >> ACUTE (é)), lowercasing it would result in"\xE3\xA9" (obviously, as a > >> lowercase character, it should remain changed), a invalid UTF-8 > >> sequence. > > > > Just to satisfy my curiosity, and realizing it might not be ultimately > > correct, what is the occurrence of the scenario you've described? > > Does > > having this happen 30% of the time? 80%? I mean to ask, with what > > frequency do people supply é as data in Windows-1252 that results in > > an > > invalid UTF-8 character sequence? > > > > Sure, some effort should be made to make this work for as much as is > > practical, but I'm getting the impression that there are edge cases > > where the effort isn't worth the return. > > Oh, I think my description wasn't good enough. It's a UTF-8 string of > the bytes b"\xE3\xA9", which is a single Unicode character, é. > Lowercasing it using strtolower(), on a lot of PHP installs (those who > default locale is ISO-8859-1/Windows-1252), results in an invalid > UTF-8 sequence. This is more just a reason why we can't fallback on to > the normal binary string functions. > > >> The basic options are to either require mbstring (whose exact > >> behaviour is PHP version dependant (though unlike iconv not C-library > >> dependant)), or to implement the exact behaviour in userland PHP. The > >> latter is the solution I took for my Unicode class (in fact, it never > >> uses mbstring to decode strings so I can make error handling user- > >> configurable, as some things require fatal error handling, others > >> rely > >> on non-fatal error handling), and only uses native support on PHP 6 > >> (as this _is_ user-configurable). > > > > Ok, your Unicode class is great. Can we use it in a String class? Or > > did you have something else in mind? > > Actually, half of it is totally broken, and it is barely written. I > was more than anything else just pointing out what route I'd gone down > for this same issue (it is far from a small undertaking to implement > Unicode in PHP userland, FWIW). > > > -- > Geoffrey Sneddon > <http://gsnedders.com/> > > > > > --~--~---------~--~----~------------~-------~--~----~ To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/habari-dev -~----------~----~----~----~------~----~------~--~---
