[issue5028] tokenize.generate_tokens doesn't always return logical line

[miss-islington]

miss-islington  added the comment:


New changeset 2a58b0636d1f620f8a85a2e4c030cc10551936a5 by Miss Islington (bot) 
(Anthony Sottile) in branch 'master':
bpo-5028: Fix up rest of documentation for tokenize documenting line (GH-13686)
https://github.com/python/cpython/commit/2a58b0636d1f620f8a85a2e4c030cc10551936a5


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Anthony Sottile]

Change by Anthony Sottile :


--
pull_requests: +13574
pull_request: https://github.com/python/cpython/pull/13686

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Cheryl Sabella]

Change by Cheryl Sabella :


--
nosy:  -miss-islington
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[miss-islington]

miss-islington  added the comment:


New changeset 1e36f75d634383eb243aa1798c0f2405c9ceb5d4 by Miss Islington (bot) 
(Andrew Carr) in branch 'master':
bpo-5028: fix doc bug for tokenize (GH-11683)
https://github.com/python/cpython/commit/1e36f75d634383eb243aa1798c0f2405c9ceb5d4


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch, patch, patch
pull_requests: +11518, 11519, 11520, 11521
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch
pull_requests: +11518, 11519
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch, patch, patch
pull_requests: +11518, 11519, 11520
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Roundup Robot]

Change by Roundup Robot :


--
keywords: +patch
pull_requests: +11518
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Cheryl Sabella]

Change by Cheryl Sabella :


--
components: +Documentation -Library (Lib)
keywords: +easy
stage:  -> needs patch
versions: +Python 3.8 -Python 2.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Georg Brandl]

Changes by Georg Brandl ge...@python.org:


--
assignee:  - d...@python
nosy: +d...@python

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5028
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Jack Diederich]

Jack Diederich jackd...@gmail.com added the comment:

+1 for a docbug.  The last item is always the physical line and not the
logical line.  Some other examples:

if True and \
   False: pass

if (True and
False): pass

--
nosy: +jackdied

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5028
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[Duncan Findlay]

New submission from Duncan Findlay du...@apache.org:

According to the documentation for tokenize.generate_tokens:

The generator produces 5-tuples with these members: the token type; the
token string; a 2-tuple (srow, scol) of ints specifying the row and
column where the token begins in the source; a 2-tuple (erow, ecol) of
ints specifying the row and column where the token ends in the source;
and the line on which the token was found. The line passed (the last
tuple item) is the logical line; continuation lines are included.

It seems though that the logical line -- the last element of the tuple
is the physical line unless the token being returned spans beyond the
end of the line. As an example, consider a test file test.py:

foo = 
%s  % 'bar'

 import pprint, tokenize
 pprint.pprint(list(tokenize.generate_tokens(open('test.py').readline)))
[(1, 'foo', (1, 0), (1, 3), 'foo = \n'),
 (51, '=', (1, 4), (1, 5), 'foo = \n'),
 (3, '\n%s ', (1, 6), (2, 6), 'foo = \n%s  % \'bar\'\n'),
 (51, '%', (2, 7), (2, 8), '%s  % \'bar\'\n'),
 (3, 'bar', (2, 9), (2, 14), '%s  % \'bar\'\n'),
 (4, '\n', (2, 14), (2, 15), '%s  % \'bar\'\n'),
 (0, '', (3, 0), (3, 0), '')]
 

Since there is only one logical line, I would expect the first 6 tokens
to have the same 5th element.

--
components: Library (Lib)
messages: 80353
nosy: duncf
severity: normal
status: open
title: tokenize.generate_tokens doesn't always return logical line
type: behavior
versions: Python 2.6

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5028
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue5028] tokenize.generate_tokens doesn't always return logical line

[David W. Lambert]

Changes by David W. Lambert lamber...@corning.com:


--
nosy: +LambertDW

___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue5028
___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com