Re: Leo's new beautifier: progress report

[Edward K. Ream]

On Thu, Feb 13, 2020 at 4:02 AM Edward K. Ream  wrote:

> ...it should be pretty safe to try out the new beautify-files command.

There is a problem with @doc and @ sections.

Leo uses a too-clever-by-half convention for determining whether blank
lines in such sections are "real". At present, beautifying a file destroys
that info. It will probably be possible to save some new state to deal with
this, but I haven't done so yet.

Another known bug: -1 sometimes gets formatted as - 1. This will likely be
fixed by consulting the ast node corresponding to one or more tokens.
Failing unit tests cover this bug.

A recent rev fixed a nasty bug: beautifying files would insert too many
blank lines at the end of the file. The result, within Leo, were unwanted
@last directives.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS2jZzJuxVxqXA3THu_sijbiF0nd4NvyqOBN5GsrRCR1Xw%40mail.gmail.com.

Re: Leo's new beautifier: progress report

[Edward K. Ream]

On Wednesday, February 12, 2020 at 6:23:45 AM UTC-6, Edward K. Ream wrote:

As of rev f8fc59 it should be pretty safe to try out the new beautify-files 
> command. Again, make sure you have backups of any to-be-beautified file. 
>

Still true.

I have beautified all of Leo's files in the fstrings branch. This revealed 
several minor bugs which will be fixed soon.

More importantly, I saw that splitting and joining lines involve subtle 
matters of policy that I had not considered at all. For now, my preference 
is *never* to join lines.

For example, when I have taken the trouble to call or def arguments on 
separate lines, I see no reason to undo that just because they actually 
could fit on a single line. It's not as if anyone is going to print the 
code, so "saving" lines makes no sense.

*Summary*

I'll check in the changes to devel when I have beautified and fstringified 
all of Leo's files to my satisfaction. This will likely take a day or two.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/4893a3cc-fb80-4298-85bf-e64e681f6100%40googlegroups.com.

Re: Leo's new beautifier: progress report

[Edward K. Ream]

On Tuesday, February 11, 2020 at 7:02:02 AM UTC-6, Edward K. Ream wrote:

[Monday] the beautify-files command passed a major milestone: all of Leo's 
> core files compile without syntax errors after being beautified :-)
>

As of rev f8fc59 it should be pretty safe to try out the new beautify-files 
command. Again, make sure you have backups of any to-be-beautified file. 

Yesterday's problems arose from the code that supports @nobeautify.  Rev 
f8fc59 fixes those problems, with just 6 lines of code. All files in 
LeoPyRef.leo beautify without (apparent) problems, and all unit tests pass 
on the beautified files.

I won't ever blindly check in beautified files. Unit tests do not suffice 
to give sufficient confidence. Instead, I'll carefully examine the diffs. 
I'll also ensure that beautifying each file a second time introduces no new 
changes.

*Summary*

The beautify-files and diff-beautify-files are ready for careful testing. 
Back up your files first.

These commands know about Leo sentinels, that is, Leo's outline structure 
and directives.

Support for @beautify/@nobeautify (#pragma: beautify / #pragma: nobeautify) 
is a feature that black lacks. It would likely be difficult for the black 
devs to add similar features.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/bdea0c11-c9c7-4d4b-ba25-5a7b0ddbdf44%40googlegroups.com.

Re: Leo's new beautifier: progress report

[Edward K. Ream]

On Tue, Feb 11, 2020 at 7:02 AM Edward K. Ream  wrote:

Yesterday the beautify-files command passed a major milestone: all of Leo's
> core files compile without syntax errors after being beautified :-)
>

That's no longer true, if it ever was. I'll let you know when I can
recommend using beautify-files.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS27BX%3DYVAdKYd%2BfPk6e-b_ZBj%3DFYt_cHErgzqX1Cr9A9w%40mail.gmail.com.

Leo's new beautifier: progress report

[Edward K. Ream]

Leo's beautify-files command now uses the Orange class in leoAst.py. This 
class is a proof of concept for #1440 
, unify the ast and 
token worlds. I've been working on this project for almost four months. It 
is nearing completion. Further testing remains.

Yesterday the beautify-files command passed a major milestone: all of Leo's 
core files compile without syntax errors after being beautified :-) No, I 
haven't committed any of the changes: that will require careful perusal of 
diffs.

*Commands*

The beautify-files commands beautifies all @ nodes in the selected 
outline.

The diff-beautify-files command (aka beautify-files-diff) reports the diffs 
(to the console) that would result from running the beautify-files command.

*Settings*

These commands take the following settings, with the indicated defaults:

@bool beautify-allow-joined-strings = False

By default, the beautifier will refrain from joining lines if they contain 
strings. This is my strong preference, because it suppresses rearranging of 
code such as:

message = (
f"\n"
f"  contents: {contents!r}\n"
f"  expected: {expected!r}\n"
f"   got: {results!r}")

I use this pattern all the time now. I would be most annoyed if some 
"helpful" beautifier collapsed it.

@bool beautify-keep-blank-lines = True

As I write this, I'm not sure this option works, or is even in effect. It's 
on the list of things to investigate.

@int beautify-max-join-line-length = 80
@int beautify-max-split-line-length = 90

Notice: these two settings have different defaults. The beautifier will 
join lines only if they won't soon be candidates for being split. Zero 
suppresses splitting or joining.The beautifier ensures that the join length 
is no larger than the split length.

In any case, the beautifier will join lines only if they have not just been 
split ;-)

*Enabling and disabling beautification*

Within Leo, @nobeautify suppresses all beautification until the end of the 
node. It's probably best to put this directive only a the start of a node. 
See below for an important warning.

The beautifier supports two special comments:

# pragma: no beautify
# pragma: beautify

These will allow a stand-alone version of the beautifier to suppress 
beautification in a range of lines.

*Warning**:* Beautification catastrophes could occur if 'indent' and 
'dedent' tokens are not matched within the "verbatim" range. It's probably 
best to use @nobeautify only at the start of a node. I might add support 
for @beautify, but that increases the danger. More noodling is required.

*To do*

1. 100% coverage of all non-debugging code.

Full coverage guarantees important properties. It will be an important 
advertising point when I announce this project.

2. Merge fstrings branch into devel.

This will be done as soon as full coverage is achieved.

3. Beautify all of Leo's core files.

This will be done in a separate branch. Imo, this work should be done 
before closing #1440.

4. Announce #1440 to the world.

The core python devs should definitely know about this work. What they do 
with it is up to them.

*Summary*

Leo's new beautify-files command is now a worthy competitor to black. It 
contains important settings and features not found in black.

Please feel free to play with the beautify-files command and report any 
problems. *Make sure you have backups of to-be-beautified files.* In 
particular, take care with @nobeautify.

Unit tests guarantee that Leo's beautify works exactly like black for all 
of the pet peeves  
listed in pep 8 . To do this, 
the beautifier must know which parse tree node corresponds to each ":" 
token. This single datum justifies using the entire TokenOrderGenerator 
class!

Completing a major project such as #1440 is *itself* a significant project. 
Surely #1440 is much nearer completion than it was a few days ago ;-)

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/a709ea6f-75e7-4b6f-896f-67343e6e673d%40googlegroups.com.