Vitalije, On Thu, May 7, 2020 at 2:42 PM vitalije <[email protected]> wrote:
> First of all I must say that testing with hypothesis is really great way > to discover hidden bugs. Several bugs were found in the previous > implementation that are very hard to imagine as a possible scenario. Of > course most of these bugs were related to clones. Hypothesis did find them > really quickly and I had to think hard how to solve them. After several > iterations of running hypothesis and solving found bugs, prototype is now > able to survive 5000 test sessions. At first I have started each test with > the complete LeoPyRef.leo outline, test would choose and select a random > position in this outline, and then it would perform a random sequence of > commands, checking after each command that both models (v-nodes and tree > widget items) are in sync. It used to take several minutes for a series of > 500 tests. > To speed up tests, I have changed test to use a smaller outline as > starting position. At start it inserts just five ordinary nodes > interspersed with five cloned nodes (total of 15 node). Now hypothesis runs > 5000 tests in about minute. After several executions no bug has been found. > That's awesome. A while back when I read about hypothesis property based testing, I was wondering how to apply it to leo outlines. I knew it would have to be done either by creating a strategy which could generate recursive structures or a strategy which could perform sequences of commands. I didn't see how to proceed. Looks like you nailed it with the second approach. Nice work! I looked at the code but I don't understand how to run the hypothesis tests. I would love to see it in action. Could you share the steps? > It is highly unlikely that a new bug will be discovered using this test. > That means we can be pretty sure that no matter what operations and in > whichever order user executes both models: v-nodes and tree widget items > will always remain in sync. The outline represented by each of them is > exactly the same. > Have you also considered using the property of random operation + undo = original tree widget state? And random operation + undo + redo = 2nd state? I'm not sure that would reveal anything from what you are already testing. Just a thought. Brian -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/leo-editor/CAO5X8CxszLQDNx%3DxWfUK37DiNXjHQH9r7jnKK-vSiMA2wskYqA%40mail.gmail.com.
