Mikeal Rogers wrote:
On Feb 10, 2006, at 11:20 AM, John Anderson wrote:
Mikeal Rogers wrote:
On Feb 10, 2006, at 10:04 AM, Philippe Bossut wrote:
John Anderson wrote:
Having written my first functional test yesterday I have some
thoughts. The biggest problem I encountered when trying to write
and debug tests is navigating all the layers:
my test <-> CATS <-> CPIA Script <-> Chandler
Fortunately I'm very familiar with Chandler, somewhat familiar
with CPIA Script and CATS is small enough to grock without much
effort. However, I suspect most developers would find all the
layers daunting and trying to debug things would only make it worse.
Agree with that.
One of the requirements is that the system be easy to use. Obviously
there is another layer of complexity over what we do with CATS but
it is still designed to be very easy for someone to pick up and
start writing scripts and to see legible output. Part of the
deliverables for the first version of this framework will be;
-Command line python wrapper (much like do_tests, a script is
imported and output is generated that is legible using a set of
default parameters for the framework)
Yes, we've all been using variation of this for unit tests and
functional tests and find it useful. Also, when you get a
particularly tricky functional test failing somewhere deep in
Chandler, were a traceback isn't enough to diagnose the problem, it's
often handy to track it down in a debugger. So you might set things
up so you can attach in wing, e.g. include wingdbstub.py.
This is probably a 2.0 feature. But, this framework has many
advantages for debugging test cases.
For example: Although the framework does all the reporting for you, it
uses methods in the TestCase class which you inherit from to define
the class for your own test. This means that in debugging you could
write custom code and push additional reporting into the output object
using same method the framework uses. The current CATS design requires
you write custom code in the Logger object to do this, and the logger
object has a bunch of logic to do fuzzy guessing as to which state the
test is in. The idea is to have the framework do most of the heavy
lifting for you but not keep you in such a tight box that you can't
extend it without hacking at the framework itself.
I think I'm confused here. I was considering the case where one of my
functional tests starts failing and I don't understand the problem from
looking at a stack trace. In that situation I need to poke around in a
debugger. Are you saying that I can't do that until 2.0? Or that I can
only add more debugging code using the framework and rerun the test. Or
something else?
-Sufficient Documentation ("Writting Chandler automation in 10
minutes" style doc, extended OAF documentation for developers who
wish to use non-default features in the system, and maybe most
importantly GOOD documentation for the chandler test library that
can facilitate both easy test script authoring and developer
improvements to the chandler testing library itself.
I think much of what makes writing functional tests difficult has
little to do with your proposed framework, and more to do with how
you access the pieces of Chandler, do menu commands, click on
buttons, etc., i.e. the stuff that is mostly in CPIA script and CPIA.
This is an issue that will be handled outside the framework. Again,
the _framework_ is not Chandler specific.
All of the QAUITestAppLib will need to be re-written to work with OAF,
during this rewrite we will be taking care of many of the current
issues developers have with it. I don't know if we can replace CPIA,
after the framework is signed off on my next task is to come up with a
plan to overhaul QAUITestAppLib.
I'm hoping that writing a new functional test for Chandler will be
almost the same as writing a CPIA script, and that a Chandler developer
needs to learn very little to write a functional test. For example, I
don't want to have to extend some QA library every time I need to drive
some new piece of the application.
At the very least we intend to make the new library more modular and
reuse code as much as possible. Making it easier for developers to
alter/extend the functionality inside of inherited objects in the test
scripts is a priority, as is making it easier for developers and QA to
extend the library to cover new functionality in Chandler.
The reason for all of the layers in the design is to make the
framework non-application specific and extendable. The implementation
will make these layers mostly transparent to the user. In the same way
we can layer the new Chandler QAUITestAppLib to be non-CPIA specific
in case you plan on implementing something better in the future.
"CPIA specific" means many things to many people, however, what I mean
by "CPIA specific" is simply that Chandler functional tests need to
access Chandler in a way that that is consistent with its internal
design, so I'm not sure we have the same idea of what "non-CPIA
specific" means.
The output can be very customized using this framework, but the
default output will be humanly legible and go directly to a file.
Also, a -debug flag can be set, which sets all output in the
framework to be processed as it comes in to the output object. This
is no good for performance tests but will make debugging issues
worlds easier than in CATS.
To finish up, many of the extra layers that developer might find
"daunting" will be transparent in the implementation, but the output
that developers depend on (such as a tracebacks in the log if a
failure occures) are made easy and reliable by this abstraction.
I hope this alleviates your concerns.
-Mikeal
I think it would be preferable to make the small changes necessary
to CPIA Script to make it appropriate for testing instead of
adding another layer, e.g. CATS.
Improving CPIA Script to make scripting easier is indeed a good
idea. I don't think it will replace entirely a test harness though
like CATS or, better, OAF (proposed by Mikeal). There's a lot of
test functions (batch, log, data gathering and stats) that have no
place in a Chandler level scripting language. John, I suggest you
read Mikeal proposal
(http://wiki.osafoundation.org/bin/view/Projects/OpenAutomationFramework)
first. Keep in mind also that Mikeal is trying to solve a problem
that includes Chandler and Cosmo.
Similarly, I think it's preferable to modify Chandler to eliminate
some of CPIA Script.
What alternative to CPIA scripting do you propose? No scripting at
all? Another script mechanism? Leverage an existing one?
Cheers,
- Philippe
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
