On Wed, Mar 30, 2011 at 08:36:43AM +0200, Lennart Regebro wrote: > On Wed, Mar 30, 2011 at 07:54, Toshio Kuratomi <a.bad...@gmail.com> wrote: > > Lennart is missing that you just need to use the same encoding > > + surrogateescape (or stick with bytes) for decoding the byte strings that > > you are comparing. > > You lost me here. I need to do this for what? """ The lesson here seems to be "if you have to use blacklists, and you use unicode strings for those blacklists, also make sure the string you compare with doesn't have surrogates".> """
Really, surrogates are a red herring to this whole issue. The issue is that
the original code was trying to compare two different transformations of
byte sequences and expecting them to be equal. Let's say that you have the
following byte value::
b_test_value = b'\xa4\xaf'
This is something that's stored in a file or the filename of something on
a unix filesystem or stored in a database or any number of other things.
Now you want to compare that to another piece of data that you've read in
from somewhere outside of python. You'd expect any of the following to
work::
b_test_value == b_other_byte_value
b_test_value.encode('utf-8', 'surrogateescape') ==
b_other_byte_value('utf-8', 'surrogateescape')
b_test_value.encode('latin-1') == b_other_byte_value('latin-1')
b_test_value.encode('euc_jp') == b_other_byte_value('euc_jp')
You wouldn't expect this to work::
b_test_value.encode('latin-1') == b_other_byte_value('euc_jp')
Once you see that, you realize that the following is only a specific case of
the former, surrogateescape doesn't really matter::
b_test_value.encode('utf-8', 'surrogateescape') ==
b_other_byte_value('euc_jp')
-Toshio
pgpZiMIuYZION.pgp
Description: PGP signature
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
