On Sep 27, 2006, at 12:43 PM, Dan Steinicke wrote:
I would like to see this discussion continue. We probably need to have another meeting about this, I am not attached to that but I would like us to follow through on this and come up with some clear solutions to the problems that are causing so much frustration.Below are my ideas of what the issues are and possible ways we could start dealing with them. I encourage others to voice their opinions on what the issues are and what we should do about them. I especially encourage people to correct me if you feel I am miss-stating something.1) We need a new agreement on how to respond when the tinderboxes go red or orange for a long period of time. Bryan has made the following proposal:after one week of mostly red or orange tinderbox A) P1 bugs get filed for each failureB) The failing tests get disabled on platforms on which they fail (with bug numbers in the comments)C) People are assigned to fix those bugs and re-enable the tests
I agree with this.
2) A number of ideas were given about the functional tests and ChandlerTestLib (formerly known as QAUITestAppLib). We should decide what ideas we want to implement, give them priorities and assign people to make the changes. The ideas I heard were: (in no particular order)A) Not always try to emulate a user typing and clicking B) Have more asserts for self checking of the test library code C) Remove dependencies between the tests D) Regularly run the tests on their own as well as in the suiteE) Make the tests more robust. Remove assumptions and add abstraction code, example: TestLib should have functions that aid setting up tests moving from view to view and checking where you are etc F) Insure tests fail when there is an exception (there is already a bug on this)G) Default to stopping tests on first error (bug 6751) H) Document how to run the tests better
We could make item D happen in the tinderbox if there was some manifest of tests to run - or if some way of having the test runner be able to query and find out the test names. Otherwise we stand the chance of having the automated tools fall out-of-sync with tests that can be run.
3) A number of improvements to the way tests are logged. Again we should decide what we want to implement, prioritize and assign the tasks. A) Provide one log containing all log output (chandler, twisted, tests) in chronological order. B) Remove the false links at the top of the on line logsC) Failing tests should print a line showing how the test was runD) Make tests raise on failure so there would always be a traceback pointing to the failureE) Have the default produce more log output
+1 to the consolidation of chandler and twisted logs. Having the test output also go into that log works but we may have to scale back the amount of text that is created and save the more wordy version for the tinderbox console output.
Item C can be solved by having each test output to the log it's parameters so the run is self-documenting - fail or pass.
4) At least one suggestion was made about the scripting functions.A) Remove the 'magic' of the app_ns object and re-write so that the miss leading None type errors are reduced or avoided.5) HardwareA) Improve cycle times by using faster computers to run the tinderboxes
we are working with IT on this now.
6) do_tests A) Re-write do_tests in python
Also scheduled to be done - like you mentioned to me earlier we should start a wiki page to gather requirements so when we get to the point we are ready to do the coding we are prepared.
Great job summarizing the meeting! --- Bear Build and Release Engineer Open Source Applications Foundation (OSAF) [EMAIL PROTECTED] http://www.osafoundation.org [EMAIL PROTECTED] http://code-bear.com PGP Fingerprint = 9996 719F 973D B11B E111 D770 9331 E822 40B3 CD29
PGP.sig
Description: This is a digitally signed message part
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "chandler-dev" mailing list http://lists.osafoundation.org/mailman/listinfo/chandler-dev
