Ahh, but that's not the case. You cannot just delete the check, since not all combinations of bytes are valid UTF8. UTF bytes FE & FF never appear in a byte sequence for instance. UTF8 is more that two bytes btw, up to 6 bytes are used to represent an UTF8 character. The 5 and 6 byte characters are currently not in use tho.
I didn't actually notice the difference in UTF8 width between my original patch and my last, so attached, updated patch. Regards, John Hansen -----Original Message----- From: Tom Lane [mailto:[EMAIL PROTECTED] Sent: Saturday, August 07, 2004 3:07 PM To: John Hansen Cc: Hackers; Patches Subject: Re: [HACKERS] UNICODE characters above 0x10000 "John Hansen" <[EMAIL PROTECTED]> writes: > My apologies for not reading the code properly. > Attached patch using pg_utf_mblen() instead of an indexed table. > It now also do bounds checks. I think you missed my point. If we don't need this limitation, the correct patch is simply to delete the whole check (ie, delete lines 827-836 of wchar.c, and for that matter we'd then not need the encoding local variable). What's really at stake here is whether anything else breaks if we do that. What else, if anything, assumes that UTF characters are not more than 2 bytes? Now it's entirely possible that the underlying support is a few bricks shy of a load --- for instance I see that pg_utf_mblen thinks there are no UTF8 codes longer than 3 bytes whereas your code goes to 4. I'm not an expert on this stuff, so I don't know what the UTF8 spec actually says. But I do think you are fixing the code at the wrong level. regards, tom lane
---------------------------(end of broadcast)--------------------------- TIP 6: Have you searched our list archives? http://archives.postgresql.org