Guido van Rossum wrote:
it would be a good thing if it could, optionally, be made to report
horizontal whitespace as well.
It's remarkably easy to get this out of the existing API
sure, but it would be even easier if I didn't have to write that code
myself (last time I did that, I needed a
On 12/2/06, Fredrik Lundh [EMAIL PROTECTED] wrote:
Guido van Rossum wrote:
it would be a good thing if it could, optionally, be made to report
horizontal whitespace as well.
It's remarkably easy to get this out of the existing API
sure, but it would be even easier if I didn't have to
I've got a small tweak to tokenize.py that I'd like to run by folks here.
I'm working on a refactoring tool for Python 2.x-to-3.x conversion,
and my approach is to build a full parse tree with annotations that
show where the whitespace and comments go. I use the tokenize module
to scan the input.
At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote:
I've got a small tweak to tokenize.py that I'd like to run by folks here.
I'm working on a refactoring tool for Python 2.x-to-3.x conversion,
and my approach is to build a full parse tree with annotations that
show where the whitespace and
Are you opposed changing tokenize? If so, why (apart from
compatibility)? ISTM that it would be a good thing if it reported
everything except horizontal whitespace.
On 11/30/06, Phillip J. Eby [EMAIL PROTECTED] wrote:
At 09:49 AM 11/30/2006 -0800, Guido van Rossum wrote:
I've got a small tweak
At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
Are you opposed changing tokenize? If so, why (apart from
compatibility)?
Nothing apart from compatibility. I think you should have to explicitly
request the new behavior(s), since tools (like detokenize) written to work
around the old
It would be trivial to add another yield to tokenize.py when
the backslah is detected
+1
I think that it should probably yield a single NL pseudo-token
whose value is a backslash followed by a newline; or perhaps it
should yield the backslash as a comment token, or as a new token.
The
On 11/30/06, Phillip J. Eby [EMAIL PROTECTED] wrote:
At 10:28 AM 11/30/2006 -0800, Guido van Rossum wrote:
Are you opposed changing tokenize? If so, why (apart from
compatibility)?
Nothing apart from compatibility. I think you should have to explicitly
request the new behavior(s), since
On 11/30/06, Fredrik Lundh [EMAIL PROTECTED] wrote:
Guido van Rossum wrote:
Are you opposed changing tokenize? If so, why (apart from
compatibility)? ISTM that it would be a good thing if it reported
everything except horizontal whitespace.
it would be a good thing if it could,
