Gareth Rees g...@garethrees.org added the comment:
Terry: agreed. Does anyone actually use this module? Does anyone know what the
design goals are for tokenize? If someone can tell me, I'll do my best to make
it meet them.
Meanwhile, here's another bug. Each character of trailing whitespace
Gareth Rees g...@garethrees.org added the comment:
Please find attached a patch containing four bug fixes for untokenize():
* untokenize() now always returns a bytes object, defaulting to UTF-8 if no
ENCODING token is found (previously it returned a string in this case).
* In compatibility
New submission from Gareth Rees g...@garethrees.org:
On Mac OS 10.7, test_faulthandler fails. See test output below.
It looks as though the tests may be at fault in expecting to see
(?:Segmentation fault|Bus error) instead of (?:Segmentation fault|Bus
error|Illegal instruction
Gareth Rees g...@garethrees.org added the comment:
Thanks Ezio for the review. I've made all the changes you requested, (except
for the re-ordering of paragraphs in the documentation, which I don't want to
do because that would lead to the round-trip property being mentioned before
it's
Gareth Rees g...@garethrees.org added the comment:
I'm having a look to see if I can make tokenize.py better match the real
tokenizer, but I need some feedback on a couple of design decisions.
First, how to handle tokenization errors? There are three possibilities:
1. Generate an ERRORTOKEN
Gareth Rees g...@garethrees.org added the comment:
Having looked at some of the consumers of the tokenize module, I don't think my
proposed solutions will work.
It seems to be the case that the resynchronization behaviour of tokenize.py is
important for consumers that are using
Gareth Rees g...@garethrees.org added the comment:
Ah ... TokenInfo is a *subclass* of namedtuple, so I can add extra properties
to it without breaking consumers that expect it to be a 5-tuple.
--
___
Python tracker rep...@bugs.python.org
http
New submission from Gareth Rees g...@garethrees.org:
tokenize.untokenize is completely broken.
Python 3.2.1 (default, Jul 19 2011, 00:09:43)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type help, copyright, credits or license for more information.
import tokenize, io
Gareth Rees g...@garethrees.org added the comment:
See my last paragraph: I propose to deliver a single patch that fixes both this
bug and issue12675. I hope this is OK. (If you prefer, I'll try to split the
patch in two.)
I just noticed another bug in untokenize(): in compatibility mode
New submission from Gareth Rees g...@garethrees.org:
The tokenize module is happy to tokenize Python source code that the real
tokenizer would reject. Pretty much any instance where tokenizer.c returns
ERRORTOKEN will illustrate this feature. Here are some examples:
Python 3.3.0a0
Gareth Rees g...@garethrees.org added the comment:
These errors are generated directly by the tokenizer. In tokenizer.c, the
tokenizer generates ERRORTOKEN when it encounters something it can't tokenize.
This causes parsetok() in parsetok.c to stop tokenizing and return an error
New submission from Gareth Rees g...@garethrees.org:
If you call timeit.timeit and the timed code raises an exception, then garbage
collection is disabled. I have verified this in Python 2.7 and 3.2. Here's an
interaction with Python 3.2:
Python 3.2 (r32:88445, Jul 7 2011, 15:52:49
Gareth Rees g...@garethrees.org added the comment:
Patch attached.
--
keywords: +patch
Added file: http://bugs.python.org/file22605/issue12514.patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue12514
101 - 113 of 113 matches
Mail list logo
