On Feb 10, 2006, at 12:04 PM, John Anderson wrote:
Mikeal Rogers wrote:
On Feb 10, 2006, at 11:20 AM, John Anderson wrote:
Mikeal Rogers wrote:
On Feb 10, 2006, at 10:04 AM, Philippe Bossut wrote:
John Anderson wrote:
Having written my first functional test yesterday I have some
thoughts. The biggest problem I encountered when trying to
write and debug tests is navigating all the layers:
my test <-> CATS <-> CPIA Script <-> Chandler
Fortunately I'm very familiar with Chandler, somewhat familiar
with CPIA Script and CATS is small enough to grock without
much effort. However, I suspect most developers would find all
the layers daunting and trying to debug things would only make
it worse.
Agree with that.
One of the requirements is that the system be easy to use.
Obviously there is another layer of complexity over what we do
with CATS but it is still designed to be very easy for someone
to pick up and start writing scripts and to see legible output.
Part of the deliverables for the first version of this framework
will be;
-Command line python wrapper (much like do_tests, a script is
imported and output is generated that is legible using a set of
default parameters for the framework)
Yes, we've all been using variation of this for unit tests and
functional tests and find it useful. Also, when you get a
particularly tricky functional test failing somewhere deep in
Chandler, were a traceback isn't enough to diagnose the problem,
it's often handy to track it down in a debugger. So you might set
things up so you can attach in wing, e.g. include wingdbstub.py.
This is probably a 2.0 feature. But, this framework has many
advantages for debugging test cases.
For example: Although the framework does all the reporting for
you, it uses methods in the TestCase class which you inherit from
to define the class for your own test. This means that in
debugging you could write custom code and push additional
reporting into the output object using same method the framework
uses. The current CATS design requires you write custom code in
the Logger object to do this, and the logger object has a bunch of
logic to do fuzzy guessing as to which state the test is in. The
idea is to have the framework do most of the heavy lifting for you
but not keep you in such a tight box that you can't extend it
without hacking at the framework itself.
I think I'm confused here. I was considering the case where one of
my functional tests starts failing and I don't understand the
problem from looking at a stack trace. In that situation I need to
poke around in a debugger. Are you saying that I can't do that
until 2.0? Or that I can only add more debugging code using the
framework and rerun the test. Or something else?
No, I'm saying that making sure that everything debugs in Wing in a
way that is usable for developers may be a 2.0 feature. I don't use
Wing so I don't know how much work that would be.
With the example, I was just showing some of the additional features
the new framework provides for debugging.
-Sufficient Documentation ("Writting Chandler automation in 10
minutes" style doc, extended OAF documentation for developers
who wish to use non-default features in the system, and maybe
most importantly GOOD documentation for the chandler test
library that can facilitate both easy test script authoring and
developer improvements to the chandler testing library itself.
I think much of what makes writing functional tests difficult has
little to do with your proposed framework, and more to do with
how you access the pieces of Chandler, do menu commands, click on
buttons, etc., i.e. the stuff that is mostly in CPIA script and
CPIA.
This is an issue that will be handled outside the framework.
Again, the _framework_ is not Chandler specific.
All of the QAUITestAppLib will need to be re-written to work with
OAF, during this rewrite we will be taking care of many of the
current issues developers have with it. I don't know if we can
replace CPIA, after the framework is signed off on my next task is
to come up with a plan to overhaul QAUITestAppLib.
I'm hoping that writing a new functional test for Chandler will be
almost the same as writing a CPIA script, and that a Chandler
developer needs to learn very little to write a functional test.
For example, I don't want to have to extend some QA library every
time I need to drive some new piece of the application.
Chandler developers are not the only users of this framework or the
new library for testing chandler with this framework (QAUITestAppLib
replacement).
Ideally we want people in the community to be writing automated test
cases. Contributing QA to an open source project is usually the first
stage of contribution for most people, so writing test scripts for
Chandler should be easy for someone who is NOT a Chandler developer.
The design of the framework and the upcoming changes to the Chandler
test library will be intended to make it as simple as possible for
anyone to write test scripts and should _not_ be targeted solely at
Chandler developers who are comfortable with the Chandler code base
and very comfortable with python. But the implementation will not
make it an uphill battle for Chandler developers to write automated
test cases either. We will be leveraging python in every way we can,
and all the methods we use for abstraction will be available via
inheritance for you to access directly. This way a developer, like
you, who is very familiar with CPIA and Chandler isn't limited by
what we've built abstraction for.
Also we are trying to move away from Chandler automation being a
collection of small scripts using simple libraries to automate
functionality and make it easier for anyone to come on and write a
script that is useful for our automation system. If we are depending
on the users to handle the reporting and keeping track of state
themselves we can expect them to make mistakes and contribute scripts
that provide differing output from scripts written by people at OSAF.
In addition, a big part of this framework is designed in order to get
accurate performance information. It's sole application is not
functional tests and debugging those issues in Chandler.
At the very least we intend to make the new library more modular
and reuse code as much as possible. Making it easier for
developers to alter/extend the functionality inside of inherited
objects in the test scripts is a priority, as is making it easier
for developers and QA to extend the library to cover new
functionality in Chandler.
The reason for all of the layers in the design is to make the
framework non-application specific and extendable. The
implementation will make these layers mostly transparent to the
user. In the same way we can layer the new Chandler QAUITestAppLib
to be non-CPIA specific in case you plan on implementing something
better in the future.
"CPIA specific" means many things to many people, however, what I
mean by "CPIA specific" is simply that Chandler functional tests
need to access Chandler in a way that that is consistent with its
internal design, so I'm not sure we have the same idea of what "non-
CPIA specific" means.
The output can be very customized using this framework, but the
default output will be humanly legible and go directly to a file.
Also, a -debug flag can be set, which sets all output in the
framework to be processed as it comes in to the output object.
This is no good for performance tests but will make debugging
issues worlds easier than in CATS.
To finish up, many of the extra layers that developer might find
"daunting" will be transparent in the implementation, but the
output that developers depend on (such as a tracebacks in the
log if a failure occures) are made easy and reliable by this
abstraction.
I hope this alleviates your concerns.
-Mikeal
I think it would be preferable to make the small changes
necessary to CPIA Script to make it appropriate for testing
instead of adding another layer, e.g. CATS.
Improving CPIA Script to make scripting easier is indeed a good
idea. I don't think it will replace entirely a test harness
though like CATS or, better, OAF (proposed by Mikeal). There's
a lot of test functions (batch, log, data gathering and stats)
that have no place in a Chandler level scripting language.
John, I suggest you read Mikeal proposal (http://
wiki.osafoundation.org/bin/view/Projects/
OpenAutomationFramework) first. Keep in mind also that Mikeal
is trying to solve a problem that includes Chandler and Cosmo.
Similarly, I think it's preferable to modify Chandler to
eliminate some of CPIA Script.
What alternative to CPIA scripting do you propose? No scripting
at all? Another script mechanism? Leverage an existing one?
Cheers,
- Philippe
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "Dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/dev
