On 2015-11-17 01:53, Serhiy Storchaka wrote:
I'm working on rewriting Python tokenizer (in particular the part that
reads and decodes Python source file). The code is complicated. For now
there are such cases:
* Reading from the string in memory.
* Interactive reading from the file.
* Reading from the file:
- Raw reading ignoring encoding in parser generator.
- Raw reading UTF-8 encoded file.
- Reading and recoding to UTF-8.
The file is read by the line. It makes hard to check correctness of the
first line if the encoding is specified in the second line. And it makes
very hard problems with null bytes and with desynchronizing buffered C
and Python files. All this problems can be easily solved if read all
Python source file in memory and then parse it as string. This would
allow to drop a large complex and buggy part of code.
Are there disadvantages in this solution? As for memory consumption, the
source text itself will consume only small part of the memory consumed
by AST tree and other structures. As for performance, reading and
decoding all file can be faster then by the line.
[1] http://bugs.python.org/issue25643
As I understand it, *nix expects the shebang to be b'#!', which means
that the
first line should be ASCII-compatible (it's possible that the UTF-8 BOM
might
be present). This kind of suggests that encodings like UTF-16 would cause a
problem on such systems.
The encoding line also needs to be ASCII-compatible.
I believe that the recent thread "Support of UTF-16 and UTF-32 source
encodings" also concluded that UTF-16 and UTF-32 shouldn't be supported.
This means that you could treat the first 2 lines as though they were some
kind of extended ASCII (Latin-1?), the line ending being '\n' or '\r' or
'\r\n'.
Once you'd identify the encoding, you could decode everything (including the
shebang line) using that encoding.
(What should happen if the encoding line then decoded differently, i.e.
encoding_line.decode(encoding) != encoding_line.decode('latin-1')?)
_______________________________________________
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
