On Sat, Aug 16, 2008 at 8:34 PM, Brett Cannon <[EMAIL PROTECTED]> wrote: > If you import a module that uses '\r\n' line endings, Python does the > right thing. But if you read in the bytes for the same file and then > pass it to compile() you get an unhelpful syntax error pointing at a > blank line. > > Normally I would say one should just open the source file as 'r' > instead of 'rb', but with source code that does not work well as their > can be a source encoding set. Lib/test/test_pep263.py is the perfect > example of this; Windows newlines with a koi8-r encoding. > > What I would like to do is get compile() to work properly with a bytes > stream just as if Python itself was handling the compilation through > import and from a file directly. But before I try to dig into the > parser to figure out where the translation of newlines occurs (or > where the translation option is set), I thought I would ask to see if > anyone just happened to know (I have already spent a few hours > figuring out why Latin-1 encodings were not working with compile() so > I don't want to go diving into the maze of function calls in the > parser again).
Have a look at tok_nextc in Parser/tokenizer.c. > > -Brett > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1." _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
