Meador Inge <mead...@gmail.com> added the comment: The proposed documentation text seems too complicated and language expert speaky to me. We should try to link to standard definitions when possible to reduce the text here. For example, I believe the "Operators" and "Delimiters" tokens in the "Lexical Analysis" section of the docs (http://docs.python.org/dev/reference/lexical_analysis.html#operators) are exactly what we are trying to describe when referencing "literal tokens" and "affected tokens".
I like Nick's idea to introduce a new attribute for the exact type, while keeping the tuple structure itself backwards compatible. Attached is a patch for 3.3 that updates the docs, adds exact_type, adds new unit tests, and adds a new CLI option for displaying token names using the exact type. An example of the new CLI option is: $ echo '1+2**4' | ./python -m tokenize 1,0-1,1: NUMBER '1' 1,1-1,2: OP '+' 1,2-1,3: NUMBER '2' 1,3-1,5: OP '**' 1,5-1,6: NUMBER '4' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' $ echo '1+2**4' | ./python -m tokenize -e 1,0-1,1: NUMBER '1' 1,1-1,2: PLUS '+' 1,2-1,3: NUMBER '2' 1,3-1,5: DOUBLESTAR '**' 1,5-1,6: NUMBER '4' 1,6-1,7: NEWLINE '\n' 2,0-2,0: ENDMARKER '' ---------- Added file: http://bugs.python.org/file24045/tokenize-exact-type-v0.patch _______________________________________ Python tracker <rep...@bugs.python.org> <http://bugs.python.org/issue2134> _______________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com
