Liam Clarke wrote:
> What's the difference between
>
> x = "Hi"
> y = x.encode("UTF-16")
>
> and
>
> y = unicode(x, "UTF-16")
They are more-or-less opposite.
encode() converts away from unicode. (Think of unicode as the 'normal' format,
anything else in 'encoded'.) Normally it is used on a unicode string, not a
byte string. It means, "interpret this string as unicode, then convert it to an
encoded byte string using the given encoding".
When you encode a non-unicode string (like "Hi"), the string is first converted
to unicode (decoded) using sys.getdefaultencoding(), then encoded using the
supplied encoding. So
'Hi'.encode('utf-16')
is the same as
'Hi'.decode(sys.getdefaultencoding()).encode('utf-16')
In either case, the result is a string in UTF-16 encoding:
>>> 'Hi'.encode('UTF-16')
'\xff\xfeH\x00i\x00'
>>> 'Hi'.decode(sys.getdefaultencoding()).encode('utf-16')
'\xff\xfeH\x00i\x00'
Note that the utf-16 codec puts a byte-order mark ('\xff\xfe') in the output;
then 'H' becomes 'H\x00' and 'i' becomes 'i\x00'.
Because sys.getdefaultencoding() is used to convert to unicode, you will get an
error if the original string cannot be decoded with this encoding:
>>> '\xe3'.encode('utf-16')
Traceback (most recent call last):
File "<stdin>", line 1, in ?
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe3 in position 0: ordinal
not in range(128)
What about unicode('Hi', 'utf-16')? This doesn't do anything useful:
>>> unicode('Hi', 'UTF-16')
u'\u6948'
unicode('Hi', 'utf-16') means the same as 'Hi'.decode('utf-16'). In this case
we are saying, "Interpret this string as an encoded byte string in the given
encoding, and convert it to a unicode string." Since 'Hi' is not, in fact, a
byte string encoded in UTF-16, the results are not very useful.
To summarize:
If you have an encoded byte string and you want a unicode string, use
str.decode() or unicode()
If you have a unicode string and you want an encoded byte string, use
unicode.encode().
If you are using str.encode() you probably haven't though through your problem
completely and you will likely get UnicodeDecodeErrors when you have non-ASCII
data.
If you are writing a unicode-aware application, a good strategy is to keep all
strings internally as unicode and to convert to and from the required encodings
at the boundaries.
Kent
_______________________________________________
Tutor maillist - [email protected]
http://mail.python.org/mailman/listinfo/tutor