> According to PEP 263, the Python tokenizer is supposed to work on a UTF-8 byte > sequence. However, since Cython runs under Python, I think it's easier to work > with unicode strings throughout the compiler. Any objections to that? I assume > that this will also make the transition of Cython itself to Py3 simpler. > Using unicode strings certainly makes a lot of sense. It seems natural to have the parser component handle everything that has to do with source encoding as such, and pass on unicode strings (as well as extra information that is necesarry in order to create the output).
How would this work with byte-strings, that is, with the contents of the nodes representing Python 2 strings/bytes literals? There seems to be two approaches: 1) They are explicitly passed around as str, and passed as bytes in Python 3 (this might allow one to completely "forget" the encoding of the input document after the parsing phase?) 2) Or, they are rather decoded to unicode using the document encoding, and the encoding used is stored so that the original bytes can be recreated on output. The latter one is the PEP 263 approach IIRC, though one isn't really bound by the "Implementation" section of that spec. The former seems fine as well...perhaps one could create a Bytes class (simply "Bytes = str" for now) that would make it clear (in Python 2) that the intent is passing binary data and not a string... Which one is best might also depend on the interface towards the current and perhaps also Fabrizio's parser? I.e, if it uses a tokenizer component that takes an encoding and only returns tokenized unicode strings, then 2) makes more sense. (I don't know that much about this aspect of Cython, take it for what it's worth.) Dag Sverre _______________________________________________ Cython-dev mailing list [email protected] http://codespeak.net/mailman/listinfo/cython-dev
