Below is a copy of what was written on the white board during the meeting.
Goals of functional tests/ tinderbox
1.
Knowing when the code is broken
2.
Knowing when chandler doesn't run
3.
Increase productivity
4.
We want constantly working code
5.
Points us to the error so we can fix it fast
Test users (Organization)
1.
QA
2.
Devs
3.
build/ release
Frustration:
1.
Too much time spent on running and fixing functional tests
2.
Functional tests are hard to fix
3.
Functional tests are hard to debug
4.
Reports are not clear enough
5.
Hard to run tests under debugger
6.
Too many different ways to run tests
7.
Functional tests are dependent on one another
8.
Tests are not 100% deterministic
9.
Test framework is not reliable
10.
Tests fail for the wrong reasons
11.
Too many assumptions in test framework
12.
Tests always trying to emulate a user makes the tests slower and
less reliable than necessary
13.
Some failures are not caught
14.
The scripting assert messages are logged and get in the way of
finding real errors
15.
Can't use debuggers with do_tests (can't do_tests.sh --gdb )
Suggestions:
1.
Re-write do_tests in python (there is a bug on this)
2.
Tests should be run both individually and together in suite
3.
Document how to run the tests better (all flags, options)
4.
Improve script library
5.
Make the framework less brittle
6.
Faster tinderbox hardware for faster test cycles
7.
Improve logs
8.
Change defaults for more log output
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
