"Dave Angel" <da...@ieee.org> wrote in message
news:4ae43150.9010...@ieee.org...
Joe wrote:
For the reason BK explained, the important difference is that I ran in
the IDLE shell, which handles screen printing of unicode better ;-)
Something still does not seem right here to me.
In the example above the bytes were decoded to 'UTF-8' with the
*nope* you're decoding FROM utf-8 to unicode.
replace option so any characters that were not UTF-8 were replaced and
the resulting string is '\ufffdabc' as BK explained. I understand
that the replace worked.
Now consider this:
Python 3.1.1 (r311:74483, Aug 17 2009, 16:45:59) [MSC v.1500 64 bit
(AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
s = '\ufffdabc'
print(s)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "p:\SW64\Python.3.1.1\lib\encodings\cp437.py", line 19, in
encode
return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\ufffd' in
position
0: character maps to <undefined>
import sys
sys.getdefaultencoding()
'utf-8'
This too fails for the exact same reason (and doesn't invole decode).
In the original example I decoded to UTF-8 and in this example the
default encoding is UTF-8 so why is cp437 being used?
Thanks in advance for your assistance!
Benjamin had it right, but you still don't understand what he said.
The problem in your original example, and in the current one, is not in
decode(), but in encode(), which is implicitly called by print(), when
needed to convert from Unicode to some byte format of the console. Take
your original example:
b'\x80abc'.decode('utf-8', 'replace')
The decode() is explicit, and converts *FROM* utf8 string to a unicode
one. But since you're running in a debugger, there's an implicit print,
which is converting unicode into whatever your default console encoding
is. That calls encode() (or one of its variants, charmap_encode(), on
the unicode string. There is no relationship between the two steps.
In your current example, you're explicitly doing the print(), but still
have the same implicit encoding to cp437, which gets the equivalent error.
That's the encoding that your Python 3.x is choosing for the stdout
console, based on country-specific Windows settings. In the US, that
implicit encoding is ASCII. I don't know how to override it generically,
but I know it's possible to replace stdout with a wrapper that does your
preferred encoding. You probably want to keep cp437, but change the error
handling to ignore. Or if this is a one-time problem, I suspect you could
do the encoding manually, to a byte array, then print that.
You can also replace the Unicode replacement character U+FFFD with a valid
cp437 character before displaying it:
b'\x80abc'.decode('utf8','replace').replace('\ufffd','?')
'?abc'
-Mark
--
http://mail.python.org/mailman/listinfo/python-list
