On Sun, Nov 3, 2019 at 7:23 PM Edward K. Ream <edream...@gmail.com> wrote:
Clearly, the new code would have to be repackaged if it were to be made > part of tokenize.py. That's all I would like to say now regarding your > comments. Perhaps other devs will pick up the ball. > After sleeping on your comments, I would like to distill them to their gist: 1. untokenize does not take a contents argument. 2. untokenize does not do exactly the same thing as the new code. These are legitimate complaints, and I'm not quite sure what, if anything, can be done about them. However, I am not quite ready dismiss the possibility of using the the new code in the tokenize module, because... *Motivations* For convenience, lets call the proposed new code the *gem *:-) Let's be clear, there is no reason for python libraries to include every little code gem. However, I have two motivations adding the gem to the tokenize module: 1. The gem would be useful for any token-based beautification code, such as the token-based versions of black or fstringify that Leo uses. Furthermore, the present untokenize method teaches away <https://www.google.com/search?client=firefox-b-1-d&q=teaches+away> from the gem. 2. issue 12691 <https://bugs.python.org/issue12691> is troubling, because the unit tests failed to detect a code blunder in add_whitespace. *Strengthening unit tests* It's not clear how the gem could strengthen unit tests. Indeed, the gem is always going to pass ;-) This is frustrating, because full sources *are *available to TestRoundtrip *.*check_roundtrip. Indeed, that method starts with: if isinstance(f, str): code = f.encode('utf-8') else: code = f.read() f.close() If f is a str, then f *is* "contents". Otherwise, "contents" can be recreated from the file's bytes. *Summary* Your objections are largely valid. Nevertheless, the gem may be worth further consideration. There is no need to add every little code gem to python's standard library. In this case, there are two reasons why adding the gem to tokenize.py might be a good idea: 1. The gem would be of use to anyone writing a token-based tool. Such tools *do* have access to full sources. 2. The present untokenize method teaches away from the gem. The gem corrects the mistaken impression that untokenizing in the presence of full sources is difficult. In fact, it's a snap. It is an open question whether it might be possible to strengthen the unit tests for the tokenize module using the gem. Full source*s are* available to TestRoundtrip*.*check_roundtrip. Edward
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-le...@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/S3SHIMJU4ICG7Q6GS2GT36UG4AB5DOWC/ Code of Conduct: http://python.org/psf/codeofconduct/
