On 2017-04-28 06:35 PM, Lex Trotman wrote:
...
Geany is almost entirely an interactive application, so until
interactive tests are possible I don't think technical tests like
these will add a great deal to the committability of PRs.
If the tests just test functions, all it needs is to get Geany started up,
then the tests can call the new/changed functions testing with different
inputs and such. There are at least two PRs to do similar.
Sadly Geany isn't a pure functional program, most functions leave
messy side-effects on global data, the Scintilla buffers :(
So you need to be able to examine those.
You can, the tests are just regular extra functions called at runtime,
you have access to all state that normal code does, it just makes it
more trouble to setup/assert that state. When you have this in mind
while writing a test for the new/changed function, you're more likely to
make it more "pure" and single-task specific. The end result is better
code and more testable code, which would gradually spread through parts
of the codebase.
Clangalizers and sanitizers and formatters won't tell you that the PR
actually puts 'z' in the buffer instead of 'a'.
No, but they'll catch a number of runtime bugs that are often hard to
identify upon basic code inspection or manual testing.
Perhaps Columban knows more about using the accessibility framework
for testing now Scintilla supports it?
There are several UI testing frameworks that work with GTK+, though I've not
used any: autopilot, dogtail, and LDTP.
I don't think we really need fully automatic UI testing (seems like too much
work), but we could get a long way just testing at the function level,
ensuring functions uphold their contract and flexing them with unusual
inputs. Making a testable function usually means writing it better too,
avoiding global state and writing more "pure" functions, and making
functions do one thing and not writing huge functions or many small
functions.
We really NEED automatic UI testing and we NEED function unit testing,
but realistically we are not going to get either. If we don't have
enough resources to just run and test PRs we don't have the resources
to add these.
The contributors add the tests flexing the PR changes, giving the person
merging the change more confidence and less reason to test every little
corner case themselves, and are automated and repeatable to ensure the
assumptions those tests make are not broken by other changes in the
future. Instead of as the OP suggested, writing up a prose testing
report by hand, they just write a test function that tests the
assumptions they have checked, also showing missing assumptions.
Regards,
Matthew Brush
_______________________________________________
Devel mailing list
Devel@lists.geany.org
https://lists.geany.org/cgi-bin/mailman/listinfo/devel
