Duncan Booth wrote: > Why would you return a token rather than throwing an exception?
Tokenizers have lots of uses. Colorizing text in an editor, for example. We've got a MALFORMED_NUMBER when you type '0x'. We've got an INTEGER when we get your next keystroke (probably). -- http://mail.python.org/mailman/listinfo/python-list
