Eryk Sun added the comment: Thanks, Serihy. When I looked at this previously, I mistakenly assumed that any undefined codes would be decoded using the codepage's default Unicode character. But for single-byte codepages in the range above 0x9F, Windows instead maps undefined codes to the Private Use Area (PUA). For example, using decode() from above:
ERROR_NO_UNICODE_TRANSLATION = 0x0459 codepages = 857, 864, 874, 1253, 1255, 1257 for cp in codepages: undefined = [] for i in range(256): b = bytes([i]) try: decode(cp, b) except OSError as e: if e.winerror == ERROR_NO_UNICODE_TRANSLATION: c = decode(cp, b, False) undefined.append('{:02x}=>{:04x}'.format(ord(b), ord(c))) print(cp, *undefined, sep=', ') output: 857, d5=>f8bb, e7=>f8bc, f2=>f8bd 864, a6=>f8be, a7=>f8bf, ff=>f8c0 874, db=>f8c1, dc=>f8c2, dd=>f8c3, de=>f8c4, fc=>f8c5, fd=>f8c6, fe=>f8c7, ff=>f8c8 1253, aa=>f8f9, d2=>f8fa, ff=>f8fb 1255, d9=>f88d, da=>f88e, db=>f88f, dc=>f890, dd=>f891, de=>f892, df=>f893, fb=>f894, fc=>f895, ff=>f896 1257, a1=>f8fc, a5=>f8fd Do you think Python's 'replace' handler should prevent adding the MB_ERR_INVALID_CHARS flag for PyUnicode_DecodeCodePageStateful? One benefit is that the PUA code can be encoded back to the original byte value: >>> codecs.code_page_encode(1257, '\uf8fd') (b'\xa5', 1) > cp932: 0xA0, 0xFD, 0xFE, 0xFF are errors instead of mapping to U+F8F0-U+F8F3. Windows maps these byte values to PUA codes if the MB_ERR_INVALID_CHARS flag isn't used: >>> decode(932, b'\xa0\xfd\xfe\xff', False) '\uf8f0\uf8f1\uf8f2\uf8f3' ---------- _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue28712> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com