Today we had an edifying community meeting on the subject of our new testing infrastructure - now the 3rd in
a series of probably 4 meetings. This thread was suggested by an early post of Yura's - can be seen at
http://old.nabble.com/Testing-Framework-Community-Meeting-to34626170.html
The goals of the new framework include:
i) To facilitate the testing of demands blocks that may be issued by integrators against components deployed
in a particular (complex) context
ii) To automate and regularise the work of "setup" and "teardown" in complex integration scenarios, by
deferring this to our standard IoC infrastructure
iii) To simplify the often tortuous logic required when using the "nested callback style" to test a
particular sequence of asynchronous requests and responses (via events) issued against a component with
complex behaviour
iv) To facilitate the reuse of testing code by allowing test fixtures to be aggregated into what are
becoming the 2 standard forms for our delivery of implementation - a) pure JSON structures which can be
freely interchanged and transformed, b) free functions with minimum dependence on context and lifecycle
I presented the implementation I have so far, which is now good enough to demonstrate the approach we want
to take for iii), allowing testing of event sequences. The quadratically increasing complexity of doing this
by hand typically deters us from writing thorough tests of this kind - the following heroic code written by
Yura for testing a CollectionSpace component illustrates the route by which this complexity increases:
"recordEditorReady.test": {
path: "listeners",
listener: function (admin) {
var recordRenderer =
admin.adminRecordEditor.recordRenderer;
jqUnit.assertEquals("Selected username is", "Reader",
locateSelector(recordRenderer, "screenName").val());
jqUnit.notVisible("Confiration dialog is invisible initially",
admin.adminRecordEditor.confirmation.popup);
locateSelector(recordRenderer, "screenName").val("New
Name").change();
admin.adminRecordEditor.confirmation.popup.bind("dialogopen", function () {
jqUnit.isVisible("Confirmation dialog should now be visible",
admin.adminRecordEditor.confirmation.popup);
admin.events.onSelect.addListener(function () {
admin.events.recordEditorReady.addListener(function (admin) {
jqUnit.assertEquals("User Name should now be", "Administrator",
locateSelector(admin.adminRecordEditor.recordRenderer, "screenName").val());
start();
}, undefined, undefined, "last");
});
.....
(available at
https://github.com/collectionspace/ui/blob/master/src/test/js/AdminUsersTest.js#L586-594
)
The outer level shows a testing framework devised by Yura to start to attack this issue - although the outer
layer of event registration has been unwound, in the body of the listener we can see a set of callbacks
nested FOUR deep in order to issue an integration test making assertions about a sequence of 4 events.
A declarative structure would allow us to flatten this nesting into a simple array of sequential assertions.
Here is an example from the test cases I showed today:
fluid.defaults("fluid.tests.asyncTester", {
gradeNames: ["fluid.test.testCaseHolder", "autoInit"],
testCases: [ {
name: "Async test case",
tests: [{
name: "Rendering sequence",
expect: 2,
sequence: [ {
func: "fluid.tests.startRendering",
args: ["{asyncTest}", "{instantiator}"]
}, {
listener: "fluid.tests.checkEvent",
event: "{asyncTest}.events.buttonClicked"
}]
}
]
}]
});
(available in my FLUID-4850 branch at
https://github.com/amb26/infusion/blob/bc7a6a414d251a640e868aca38a9977f474cc9be/src/webapp/tests/test-core/testTests/js/TestingTests.js#L132-L149
)
The interesting aspects of this "test fixture holding grade" are as follows -
i) It is a pure configuration grade with no implementation code, offering good
potential for reusability
ii) The block of configuration "sequence" is a flat array, where each element of the array corresponds to a
sequential state of the component under test.
The "sequence" array may hold "fixture directives" of a small variety of types. We can currently imagine a
repertoire of 4 or 5 record types, which themselves can be assigned to one of two broader categories -
"executor" records which actively interact with the component under test, and "binder" records which
register listeners responding to events fired by the component under test. Any sequence of these records may
appear in any order. The types we imagine, indexed by a "duck typing field" system slightly reminiscent of
that used by the Fluid Renderer, are as follows:
func (executor): Execute a free function with arguments IoC-resolved against
the tree
listener (binder): Register a listener to a Fluid event good for ONE firing only at the appropriate point in
the sequence
jquery (binder): Register a listener to a jQuery event against a DOM element
resolvable in the tree
changeListener (binder): Register a listener to a ChangeApplier event fired by a model-bearing component in
the tree
jqueryTrigger (executor): Trigger a jQuery event from a DOM node resolvable in
the tree
We covered the overall idiom of the test framework in our meeting two weeks ago, but a few points have been
clarified since then. The overall scheme is that arbitrarily sized chunks of application code under test
will be embedded together with "fixture holders" of the type shown above together in the same overall
component tree, corresponding to a "testing environment". The setup and teardown for all of the test cases
held in these fixtures will proceed by the standard IoC semantics for construction and destruction of
component trees. The root of this tree will hold a component with the grade "fluid.test.testEnvironment"
with a specific name chosen by the fixture author which is suitable to issue demands blocks whose scope
consists of just this environment. The "fixture holders" scattered around the tree arbitrarily each have the
grade "fluid.test.testCaseHolder" - each of these contains configuration coding for a set of jqUnit (qunit)
test case "modules" and "test cases" which will be dispatched to the standard jqUnit framework once the
construction of the overall component tree is complete.
A few issues came up relating to the potential mismatch between this setup/teardown model, at the unit of
whole component trees and qunit "modules", and the native one operated by qunit which only operates at the
level of individual test cases. Our tentative decision is to sidestep the native teardown model completely
in favour of one operated by this new framework in a dedicated way. The native model comprises three parts -
i) setup/teardown at the level of JavaScript globals, checking for global namespace pollution - ii)
setup/teardown of markup nested inside a DOM node with the hard-coded ID "qunit-fixture" (formerly "main" in
earlier versions of qunit) - iii) user-supplied setup/teardown functions to a module which are operated for
each individual TestCase within the module.
Part i) is healthy enough, although there should be no instances of such global pollution caused by Fluid
components (unless their execution happens to cause further components to become defined - not expected with
our current code idioms). ii) will be replaced by a new scheme allowing ANY selector to be registered at the
root of the environment as the markup to participate in setup/teardown on the lifecycle of the entire
environment, rather than individual test cases, iii) will be replaced by the overall action of the
construction and destruction of the component tree.
Together with the presentation, there were a number of interesting questions from the group which I
reproduce here (not necessarily in order):
Testing onCreate:
================
Justin asked how the system could be used to test the action of the "onCreate" event of a component in the
tree under test. I replied that the creation of the component under test would need to be deferred by means
of the "createOnEvent" directive, and then an executor block inserted into the fixture description to
operate that event within the fixture sequence, as in the following sketch:
fluid.defaults("fluid.tests.myTestTree", {
gradeNames: ["fluid.test.testEnvironment", "autoInit"],
events: {
startCat: null
},
components: {
cat: {
createOnEvent: "startCat",
type: "fluid.tests.cat"
},
catTester: {
type: "fluid.tests.catTester"
}
},
and then in the tester:
fluid.defaults("fluid.tests.catTester", {
gradeNames: ["fluid.test.testCaseHolder", "autoInit"],
testCases: [ {
name: "Late cat tester",
tests: [{
name: "Rendering sequence",
expect: 2,
sequence: [ {
func: "{myTestTree}.events.startCat.fire",
}, {
event: "{cat}.events.onCreate",
listener: "fluid.tests.catCreationTester"
},
etc.
Spectrum between unit tests and integration tests:
=================================================
Michelle asked about how appropriate this system was for expressing unit tests as considered against
integration tests, how these two situations could be identified by those reading the code, and how we would
make recommendations about what tests to write. After some discussion, we resolved on a few things:
i) There is a spectrum between unit tests and integration tests, which can be roughly identified by the SIZE
of the component tree appearing in the "component under test" part of the overall environment test. If this
tree consists of a single component, the test situation would more clearly have the character of a "unit
test". A large tree or indeed an entire application present in the environment would represent a complex
integration test.
ii) One purpose of the framework would be to bridge between the worlds of unit and integration tests,
allowing the same code and idioms to be usable in both worlds. However, it's clear that in general the IoC
testing system is more appropriate and more economical the closer the situation represents an "integration
testing scenario"
iii) That said, even a single Fluid component which is event-driven could benefit significantly from having
its tests written in the new style - this enables us to write tests which are easier to read intent from, as
well as being more powerful than those we could write before - easily testing a PARTICULAR sequence of
events, rather than just placing ad hoc constraints on sequencing in the way we often would in our old-style
tests, usually involving some kind of scrawling into suitably scoped variables - the following style of code
will be familiar to us all:
var that = fluid.tests.eventParent3();
var received = {};
that.eventChild.events.relayEvent.addListener(function(arg) {
received.arg = arg;
});
that.events.parentEvent1.fire(that); // first event does nothing
iv) Non-event driven code which just implements ALGORITHMS (such as the model transformation system) can
continue to be profitably written in the plain old jqUnit style - however -
v) Given that "new" testing code is reduced to the status of free functions holding jqUnit assertions, there
is a lot of scope for sharing and reusing these fixture functions between code written in plain jqUnit style
implementing unit tests, and IoC style implementing integration tests - for example, the following fixture
function in the TestingTests we looked at:
fluid.tests.startRendering = function (asyncTest, instantiator) {
asyncTest.refreshView();
var decorators = fluid.renderer.getDecoratorComponents(asyncTest,
instantiator);
var decArray = fluid.values(decorators);
jqUnit.assertEquals("Constructed one component", 1, decArray.length);
asyncTest.locate("button").click();
};
could just as easily be invoked from within a manual jqUnit test as by the IoC
testing framework.
Appropriate unit of reusability:
===============================
Alex expressed the concern that the global namespace might endlessly clutter up with numerous specifically
named fragments of dedicated testing functions (in the "free" style we just looked at). We turned to an
analysis of Yura's very complex testing code (near the beginning of this post), and decided that the best
outcome would be if the ability to write such free functions ended up in a better factoring of testing
responsibilities rather than proliferating numerous similar functions. In fact, once all the complexity of
event binding were removed from this code, we would discover that the actual testing assertions issued
mostly consisted of single jqUnit assertions - which could be issued directly from the fixture sequence
rather than requiring a new dedicated free function to be written. Other opportunities for reuse could also
be more clearly identified - for example, the AdminUsersTest.js file shown above would reveal opportunities
centred around the existing "locateSelector" function as used often in the following pattern:
locateSelector(recordRenderer, "screenName").val("New
Name").change();
Testing particular instance of event firing:
===========================================
Justin asked a question relating to a situation encountered in Decapod where he needed to test (say) the
SECOND instance of firing an event (in this case, a rendering event) whilst ignoring the first, and asked
whether this kind of thing would be assisted by the framework. I replied that this was just the kind of use
case for which it was designed - a sequence record, say, holding just fluid.identity or jqUnit.assert could
be supplied for the first event firing, and a more complex one for the second event, verifying that
particular pieces of markup had indeed been rendered.
Dealing with "dropped sequence" failures:
========================================
JURA highlighted the frequently unhelpful behaviour of (j)qunit on encountering a failure in an asynchronous
test - this simply causes the UI to hang, without any clear indication of whether there really are more
tests or what the expected operation which failed to occur was. To be clear, this is a failure cause by some
path where an asyncTest fails to cause the "QUnit.start()" operation to be invoked, through a particular
event failing to be fired - rather than a direct failure held within code.
I replied that this problem scenario was to a certain extent a fundamental and irremovable feature of any
asynchronous testing framework - purely by virtue of being asynchronous, an event's occurence would be
unexpected and could never be finally deemed not to have occured. However, I suggested that as a result of
the clear declarative nature of the "sequence" structure, the new framework could make it considerably
clearer than formerly which the "missing event" was by providing information as to the most recently
correctly operated sequence point. This could (should) even be highlighted somehow in the existing QUnit UI
as a separate diagnostic to help the user pinpoint the expected and missing event.
Yura mentioned that another approach could be to assist the user in setting timeouts on events - I commented
that this was certainly valuable, but could itself be the cause of false positive test failures in the case
the browser was running slowly. However, it would be something that would be easy to add to the existing
declarative structure for "binding" sequence records by adding an extra "timeout" field - far easier than
writing the requisite timeout handling code by hand.
I feel there were some other interesting and useful questions but can't bring them to mind right now,
perhaps if someone recalls, they could add them in replies.
Implementation status:
I still have some work to do to bring the implementation to a usable status. On
the to-do list:
i) Implement the other 3 record types (jquery, changeListener, jqueryTrigger) and apply much more thorough
testing to ensure that all of the different patterns of sequential appearance are handled correctly in terms
of the binding and execution sequence chosen by the framework
ii) Fix the current very hacked system for chaining together the execution of several test environments in
sequence - given the markup setup/teardown system described above, this needs to be properly serialised to
avoid corrupting the document
iii) Implement the described markup setup/teardown system!
iv) Ensure that something sensible happens when the user selects just a single test, or a test case filter
to be operated via the HTML UI provided by qunit. This should be possible (perhaps by using our now freed-up
slot for qunit per-test setup/teardown functions) without needing to hack on the underlying qunit code!
This should be all ready in time for the final testing meeting next week!
Cheers,
Antranig
_______________________________________________________
fluid-work mailing list - [email protected]
To unsubscribe, change settings or access archives,
see http://lists.idrc.ocad.ca/mailman/listinfo/fluid-work
