Hi, In [1] Tom said the real fix for the psql problem was the WIN1252 validator. Here is a patch for master.
Thirteen single-byte encodings have bytes with no character. I took the lists from each *_to_utf8.map: ISO_8859_6 ISO_8859_7 ISO_8859_8 LATIN3 WIN874 WIN1250 WIN1251 WIN1252 WIN1253 WIN1254 WIN1255 WIN1257 WIN1258 For WIN1252 they are 0x81 0x8D 0x8F 0x90 0x9D. Today these bytes are accepted, then fail to convert to any other encoding. Bug #14197 [2] is one report of that. The patch rejects them in the verifystr functions. It also closes two other ways in. chr(129) skips the validator, so it now checks the result. And the LATIN2 to WIN1250 table mapped five bytes into WIN1250 holes, so those now fail to convert. A database that already has such bytes can still be dumped, but the restore fails on COPY for that table. pg_upgrade keeps the bytes as they are. Maybe pg_upgrade should check for them as well//// *The extra check costs something.* COPY of 1M rows of 60 bytes: WIN1252 ASCII only 0.221 s -> 0.226 s (noise) WIN1252 50% non-ASCII 0.218 s -> 0.261 s (+20%) ISO 8859-6 100% non-ASCII 0.219 s -> 0.271 s (+23%) UTF8 databases do not pay for it. Another idea is to map the holes to characters instead, for example 0x81 to U+0081. That costs nothing and keeps old data readable. But our maps would no longer match the Unicode.org files. ^^^ only works cleanly for holes in 0x80-0x9F, which covers 5/13. Thanks, Shihao
v1-0002-Add-tests-for-unassigned-bytes-in-single-byte-enc.patch
Description: Binary data
v1-0001-Reject-unassigned-bytes-in-single-byte-encodings.patch
Description: Binary data
