On Sep 6, 2007, at 8:21 AM, Grant Baillie wrote:
Hi, Aparna et al
Overall, I can see why people like wx-level acceptance testing:
there is a certain warm fuzzy feeling from knowing that the app in
its entirety is behaving reasonably. (Even if you have to watch it
to be sure :). The concerns I have are:
+ How would these recorded scripts be maintained? If we changed the
app's layout, does someone have to go re-record them all?
The recorded tests are pretty independent of the apps layout. For
example if you move a widget they should still work. That said, if
you remove a widget or change what a widget does, the script will
report that as a failure since the point of the script is to verify
that the widget does what it did when recorded.
This will lead to situations were, when the application logic or
features change, scripts will need to be rerecorded. But hopefully
this can be done easier than writing some code and maintaining it by
hand like we have done in the past.
+ Each script currently requires a launch and quit of the app
(which is reasonable, given the nasty inter-test dependencies that
have arisen with the current functional tests). In the scenario
where we really cover all the current manual test specifications
with recorded ones, how long will this take? Will developers be
required to run them all before checkins?
Starting with a fresh Chandler before each test is a good place to
start since it minimized dependencies of one script on the others. If
it turns out that this is a performance bottleneck we could explore
alternatives, e.g. starting with a prebuilt repository or after each
script is run, roll back the repository to a known state, or require
each script remove side effect of the test (e.g. delete collections
it created, change the view back to a known state). We could also
have some scripts that run individually, and some that run back to back.
Today, few developers run all the tests on all the platform before
checkin. It might make sense to have developers run a subset of tests
before checkin but let the tinderboxes run a more complete set.
+ I think I remember John/Dan saying scripts currently have to be
recorded on Windows (something to do with keycodes). Is that likely
to still be the case?
Yes, today scripts need to be recorded on windows if you want them to
playback on the other platforms. This is because windows records
platform specific key codes which aren't recorded on the other
platforms. These kind of problems are solvable, but it's more a
question of it is worth the effort. I had hoped to start using the
scripts to gain some experience before I had fixed all the problems.
I still don't have enough experience with recorded scripts to
understand how much work it will be to script every possible user
action. However, I think there is a reasonable chance we could have
better than 90% coverage with a few weeks more effort.
Finally, I don't think recorded scripts need to be the only way to
test Chandler, but I'm hopeful that they can be an significant
improvement over our our old functional test framework.
John
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
