Hi again,
I just added a section to the CEP about encoding issues. Any comments on
the solution (last paragraph) below?
Stefan
"""
=== Encoding ===
Letting non-identifier (i.e. non-ASCII) strings switch their type at
runtime introduces various issues with encodings. Following PEP 3120, the
default source code encoding in Cython is UTF-8, and users can override
this encoding as described in PEP 263.
Problems arise when users enter plain strings containing escaped characters
that are not representable in the current source code encoding. In this
case, there is no way to determine what bytes should be used in the Python
2 byte string. Example:
{{{
# encoding: ASCII
b = b'abc\xFF'
s = 'abc\xFF'
u = u'abc\xFF'
}}}
Both the byte string (b) and the unicode string (u) have well defined
content. If {{{\xFF}}} is interpreted as byte value, the str value (s) has
well defined byte content in Python 2, but it does not have a meaningful
content under Python 3, as there is no indication on how to interpret
{{{\xFF}}} as a unicode character. If, on the other hand, {{{\xFF}}} is
interpreted as unicode character value, the str value becomes well defined
in Python 3, but cannot be represented as a byte sequence in Python 2.
Depending on whether the string is read as unicode string or byte string by
the parser, further issues can arise. Imagine this case:
{{{
#encoding: UTF-8
s = 'abc\xFF'
}}}
If {{{\xFF}} is interpreted as byte sequence here, the string cannot be
decoded as UTF-8. If it is interpreted as a unicode character, i.e. the
string is read as a unicode string, the character can be encoded to UTF-8
to result in a valid two byte sequence. However, it is not clear if this is
what a user expects. Also, Python 2.6 reads the above string as a four byte
string, not a five byte UTF-8 string.
The only obvious way to prevent both confusion and runtime errors is to
disallow unprefixed strings that cannot be decoded under the current source
code encoding. Following Python 2 string semantics, Cython will therefore
read {{{str}}} literals as byte strings, and then try to decode them using
the source code encoding at compile time. All strings that fail to decode
will be rejected.
"""
_______________________________________________
Cython-dev mailing list
[email protected]
http://codespeak.net/mailman/listinfo/cython-dev
