Re: SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

On Sun, Nov 29, 2020 at 9:10 PM tbp1...@gmail.com 
wrote:

I'm glad to see you taking this approach.  I think that different
> development tasks should be done in separate branches.  In the past I've
> not always been careful about this, and by now I think I may finally have
> learned the lesson.
>

Thanks for this comment.

My present plan regarding the ekr-undo and ekr-unit-test branches is as
follows:

1. (Done) Document both branches in Leo's branch info page
.

2. Merge ekr-unit-test into devel in stages.
After each merge, merge devel back into ekr-undo.

3. Merge ekr-undo into devel later, when Leo has better unit tests.

*Summary:* The lack of complete coverage tests is not an immediate crisis
:-)

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS2Aynb0A%3D8x4M2mZrGQKWRD4HTuQSAmsS%3DzpBWBQ3GM3g%40mail.gmail.com.

Re: SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

On Sun, Nov 29, 2020 at 8:49 PM Brian Theado  wrote:

> Having measurable code coverage of the automated tests will be useful.
>
> The coverage.py library is not at all tied to pytest.
>

I had forgotten that.

The big picture: Leo's old unit-testing framework, based on @test nodes in
unitTest.leo, is inflexible. To add new features, complex code must be
added to leoTest.py.

In contrast, leoTest2.py contains only a few simple classes, subclasses of
unittest.TestCase. I am about to simplify the code further, by adding
top-level helper functions in leoTest2.py, for use by various setUp
methods. This may reduce or eliminate the need for base classes.

To emphasize: leoTest2.py has *easy* access to all of the features of
python's unittest module, and by extension, all of the features of pytest,
nose, whatever. This is the real breakthrough.

About a year ago I tried running its standalone version against Leo's
> existing tests.
>
> However, some of the coverage results were not making sense. Possibly the
> way leo's unit test framework loads the code was causing confusion to
> coverage.py. I didn't do any investigation to see if that is a reasonable
> guess.
>

All such problems should go away, possibly to be replaced by new
complications :-)

For example, there is the question of how to init g.app only once for a
suite of tests. Googling this yields several useful leads involving
per-class and per-module *fixtures*. I'm still wrapping my head around this.

All this to say that your planned approach of extracting the tests into
> standard pytests is probably the better path to reliable code coverage
> measurements.
>

I agree. Yesterday's work completely proves the concept. I'll say more in a
new post.

Have you considered using pytest features such as fixtures and
> parameterized marks?
>

Not yet :-) My present plan is to create test *functions* with a single
test class, rather than creating separate test *classes.* This seems like
the simplest thing that could possibly work.

While I was debugging the new code I switched back and forth between
running the test class with unittest and with pytest. For debugging,
unittest seems a bit better. So I don't necessarily want to tie the code to
pytest features. Unless they add something essential.

I tend to favor composition over inheritance and functions over classes.
> The fixture feature helps with this preference.
>

In the present environment, the difference seems small. However, this
comment nudged me towards creating top-level helper functions in
leoTest2.py. That seems like a step forward.

I think your preference is the opposite, but I thought I'd ask.
>

I don't have a strong preference at present.

Many thanks for all your comments. They have been helpful.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAMF8tS17yh%2BeV8sywhotBbBH_7UhF1rP%3Do9p-H3_%3D-BhUfO8cA%40mail.gmail.com.

Re: SB: Please Read. Huge Ahas re unit testing

[tbp1...@gmail.com]

I'm glad to see you taking this approach.  I think that different 
development tasks should be done in separate branches.  In the past I've 
not always been careful about this, and by now I think I may finally have 
learned the lesson.

On Sunday, November 29, 2020 at 1:52:56 PM UTC-5 Edward K. Ream wrote:

> On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:
>
> > The ekr-undo9 branch contains preliminary work on the new testing 
> framework.
>
> Not sure that's true. Anyway, work on this topic will be done in the 
> ekr-unit-test branch. PR 1762 
>  records the work.
>
> After some thought, I've decided to base ekr-unit-test on devel, as usual. 
> The goals are to minimize diffs, and to make the unit test work distinct 
> from the undo work. When both branches are ready, I'll merge ekr-undo9 into 
> devel first, merge devel into ekr-unit-test, and run all the new unit tests.
>
> Edward
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/f44d4cc7-9e71-40ce-912a-28e89c497ec1n%40googlegroups.com.

Re: SB: Please Read. Huge Ahas re unit testing

[Brian Theado]

Having measurable code coverage of the automated tests will be useful.

The coverage.py library is not at all tied to pytest. About a year ago I
tried running its standalone version against Leo's existing tests.

However, some of the coverage results were not making sense. Possibly the
way leo's unit test framework loads the code was causing confusion to
coverage.py. I didn't do any investigation to see if that is a reasonable
guess.

All this to say that your planned approach of extracting the tests into
standard pytests is probably the better path to reliable code coverage
measurements.

Have you considered using pytest features such as fixtures and
parameterized marks?

I tend to favor composition over inheritance and functions over classes.
The fixture feature helps with this preference.

I think your preference is the opposite, but I thought I'd ask.

On Sun, Nov 29, 2020 at 1:52 PM Edward K. Ream  wrote:

> On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:
>
> > The ekr-undo9 branch contains preliminary work on the new testing
> framework.
>
> Not sure that's true. Anyway, work on this topic will be done in the
> ekr-unit-test branch. PR 1762
>  records the work.
>
> After some thought, I've decided to base ekr-unit-test on devel, as usual.
> The goals are to minimize diffs, and to make the unit test work distinct
> from the undo work. When both branches are ready, I'll merge ekr-undo9 into
> devel first, merge devel into ekr-unit-test, and run all the new unit tests.
>
> Edward
>
> --
> You received this message because you are subscribed to the Google Groups
> "leo-editor" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to leo-editor+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/leo-editor/7debef12-60e1-4463-9dda-8bd6a1ab7dc4n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/CAO5X8CywWr_fhJsAXh%3DGR-BfSjfdAbQ6ooYpnxmH_UZknNMn0A%40mail.gmail.com.

Re: SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:

> The ekr-undo9 branch contains preliminary work on the new testing 
framework.

Not sure that's true. Anyway, work on this topic will be done in the 
ekr-unit-test branch. PR 1762 
 records the work.

After some thought, I've decided to base ekr-unit-test on devel, as usual. 
The goals are to minimize diffs, and to make the unit test work distinct 
from the undo work. When both branches are ready, I'll merge ekr-undo9 into 
devel first, merge devel into ekr-unit-test, and run all the new unit tests.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/7debef12-60e1-4463-9dda-8bd6a1ab7dc4n%40googlegroups.com.

Re: SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

On Sunday, November 29, 2020 at 4:31:42 AM UTC-6 Edward K. Ream wrote:

*tl;dr: *It will be *straightforward *to provide complete coverage tests 
> for most of Leo!
>

The various Ahas have straightforward consequences:

Leo will continue to support @test and @suite nodes. Eventually, leoTest.py 
will contain only the minimum code to support *generic* @test and @suite 
nodes.

Considerable code will migrate from leoTest.py to leoTest2.py. leoTest2.py 
will also contain new code and classes. As before, most of Leo's unit tests 
must be aware of Leo's DOM and API. To make that happen, leoTest2.py will 
define test classes that define off-screen *test commanders*. leoTest.py 
already does this. The corresponding code in leoTest2.py should be 
substantially simpler because the code will not be running *inside *Leo.

Oh joy. The distinction between running unit tests internally or externally 
will disappear.

leoTest2.py must provide a way of running all unit tests. It will likely do 
that by looking for source files with a top-level "pytest_main" method.

20+ years of hindsight should improve leoTest2.py. For example, it's now 
clear that most unit tests can and should be run with a "null" 
(string-based) gui. leoTest2.py must also support desperately needed unit 
tests of the Qt gui. I'll be experimenting with running tests in an 
"off-screen" Qt gui.

Let me emphasize several points:

1. The conversion script 
 
is dead simple, both in concept and in code. The script extracts data from 
a tree of nodes copied from unitTest.py, and uses that data as kwargs to a 
"run_test" method of a test class. Neither the run_test method nor the test 
class exists at present, but that's a nit :-) I'll adapt that script to 
convert other @test nodes in unitTest.py. Other @test nodes can just be 
copied to their own unit tests.

Aside: The new unit tests will not have automatic access to c and g and p. 
However, that won't change much, because the various setup and teardown 
methods can compensate. Surely, it's no big deal.

2. The sabbatical will continue. The main focus of my sabbatical remains 
learning new web technologies. But now a secondary focus will be to 
revolutionize Leo's unit tests.

In the next few weeks, I *will *create leoTest2.py and create new unit 
tests covering most of Leo's commands. PR 1759 
 tells which files were 
changed. To that list I'll add files changed in the ekr-undo9 branch.

Other work can wait. I am not going to commit to covering all of Leo's 
source code immediately. Some unit tests can wait. Otoh, full coverage 
testing of Leo's code base will be a defining feature of Leo 6.4.

*Summary*

No new significant invention is needed to make full coverage testing of 
Leo's code base a reality.

The sabbatical will continue, now with twin objectives.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/leo-editor/0c87a970-e16e-45be-9244-50fc8e2a012en%40googlegroups.com.

SB: Please Read. Huge Ahas re unit testing

[Edward K. Ream]

Yesterday will surely rank as one of the most important days in Leo's 
history.

*tl;dr: *It will be *straightforward *to provide complete coverage tests 
for most of Leo!

Like all momentous Ahas, it has suddenly become difficult to remember what 
the world looked like before the Aha. I am writing this post in the middle 
of the night, attempting to recreate the Aha, explain the way forward, and 
get back to sleep :-)

*A slow-motion Aha*

Yesterday I thought I was finally done with work on Leo. At last, I could 
start studying web technologies. Indeed, yesterday I completed the "Hello 
World" example for VS Code plugins 
. But 
then fate intervened.

My subconscious nagged me about completing the elimination of 
c.onBodyChanged. This is #1733 
. Yesterday I 
reopened this item, removed the "Won'tDo" label, and reassigned it to Leo 
6.4. I did this because I saw that after simplifying c.onBodyChanged it 
would be easy to replace c.onBodyChanged with a *small* amount of code, 
given in #1733.

I eliminated several calls to c.onBodyChanged using this strategy.  I ran 
all unit tests, and they all passed. So far, seemingly so good.

But now my subconscious started screaming at me: "Why are you doing this 
work? You have no idea whether your changes have been covered by unit 
tests. Admit it. You are flying blind."

According to github, I created #1758 
 (Cover most of Leo 
with unit tests) *two days *ago*.* I am quite sure that when I created this 
item I did not have any idea how to do it! But presumably, this item primed 
my mental pump. That's probably why my subconscious was screaming at me 
yesterday.

Yesterday afternoon Rebecca and I took a long walk in some lovely woods 
overlooking the University of Wisconsin. After the walk, I saw the start of 
the way forward.

Strangely, I don't remember the exact moment. I remember discussing unit 
testing with Rebecca after the walk. Before dinner, I wrote a script to 
convert unit tests for edit commands in unitTest.leo to "traditional" unit 
tests.

It was only after discussing that script with Rebecca at dinner that I 
started to realize what I had just done! As I talked, I got more and more 
excited. The building excitement was similar to the "hand-waving Aha" about 
synchronizing tokens in leoAst.py.


*The first conversion script*

The second comment of #1758 
 
shows the first conversion script. The setup is:

- Copy @file activeUnitTests.txt-->Organized by file-->leoEditCommands
  from unitTest.leo to the top-level node of leoPy.leo.
- Create a new top-level node called 'new-tests'.
- Run the script.

The second comment also shows the output of the script, which is a 
"traditional" unit test.

*Important:* The script creates a list of Leo *nodes*. I'll copy these 
nodes to leoEditCommands.py. But like all other nodes in @file trees, the 
nodes create *flat text* in the external file. This means...

*Aha 1*: pytest does not have to know about Leo outlines!

*Support classes*

A new file, leo/core/leoTest2.py, will define the classes and helpers 
needed to make the new tests work. I'll adjust the conversion script to 
bring the generated nodes into "alignment" with the code in leoTest.py.

*Aha 2*: The kinds of support that leoTest2.py is much simpler than that 
contained in leoTest.py!

leoTest.py *creates* "flat" unit tests for python's unittest module. It's 
messy. In contrast, leoTest2.py just needs to define some base classes and 
simple support helpers.

*Aha 3*: here is *no way* to run coverage tests from unitTest.leo. 
unitTest.leo must go.

None of these three Ahas were clear to me until my dinner with my muse :-)

*Summary*

Running coverage tests is the crucial first step. Coverage failures will 
guide the testing process. 

Scripts will convert unit tests in unitTest.leo to corresponding nodes 
containing traditional unit tests. I'll copy the generated nodes to the 
corresponding source files. The actual unit tests will be flat text in the 
external source file.

A new file, leoTest2.py, will provide support for the new testing 
framework. leoTest2.py will be *much* simpler than leoTest.py because it 
doesn't need to convert outlines to tests.

unitTest.leo must go. It was a useful dead end.

The *ekr-undo9 branch* contains preliminary work on the new testing 
framework.

I had expected that my sabbatical would be refreshing. I did not expect a 
huge breakthrough in Leo's world.

Edward

-- 
You received this message because you are subscribed to the Google Groups 
"leo-editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to leo-editor+unsubscr...@googlegroups.com.
To view this discussion on the web visit