Here is the "high-level" design of my test / samples restructuring effort. I began this undertaking with the following input as requirements:
- 1. Determine the needs and dependancies of each test and sample
2. Reduce the complexity of the xml that compiles, deploys, execs, and undeploys samples and tests
3. Enable a new test or sample to be "plugged in" to the suites, without disturbing currently working things
4. Enable the ability to instantiate any single sample or any single test
5. Identify tests and samples that need "network" or "internet" resources
6. Enable the ability to group tests and samples together in "bundles"
This was based on the following information which was relayed to me:
1. Developers don't like to run ALL of the tests, if they are changing only one part
2. Developers don't like adding new tests / samples because their integration into the big master build.xml recursive functions
threatens to break everything
3. Some development organizations test in an "off-line" mode, and therefore cannot exec ALL of the tests ALL of the time.
To these ends, I have done the following:
1. Broken out the buildTest and samples targets from build.xml into buildTest.xml and buildSamples.xml
2. Componentized those big recursive "build everything" calls into specific test compilation xml files
For example, the commands / code to build the sessions test is now located in a file, buildComponent.xml, in the directory
test/sessions under the target "compile"
3. Instead, in the buildTest and buildSamples files, there now exists a non-complex variant of the ant ForeachTask, which finds all files
named buildComponent.xml in test/ and samples/ respectively, and operates on them.
This allows a new test / sample to be "plugged in" to the test/ and samples/ trees, and then be exectued upon, withouth
having to muck with the master controlling files
4. If a single test or sample is needed to be instantiated directly (is a pre-req for another test or sample for example), then the
developer simply adds a block as below into either buildTest.xml or buildSample.xml to enable the single component use of that
specific test or sample
<target name="session">
<ant inheritAll="true" antfile="test/session/buildComponent.xml" target="compile"/>
</target>
5. I am still working on this part. The basic jist is that if there is no internet (network failure, firewalled development environment, etc)
the tests should not fail. I have created a property in the test.properties file which is test.runtest and is given a value of "LOCAL",
"REMOTE", or "BOTH" to control this function. This is one of the parts that I will need some help with.
6. As in item 4, in order to "group" a bunch of tests, this can be implemented in two methods:
A. Create a target task in the build<Test,Samples> file that instantiates them all
B. Create a second ForeachTask loop that looks for files called testBundle*.xml and exec each of these.
As for your issue with "stopping on the first failure", I beleive that there is already a flag set up to ignoreFailures in the xml files, which will then exec all tests, and give you a wrap up of what did, or did not, work. Also, to that end, I will need to add a "common output" method for these componentized tests to allow them (if the user wishes) to have their output placed in either separate log file (reports/test-session.xml), a "wrap-up" log file (reports/test.xml), or both. Suggestions and input here would be greatly appreciated.
Also, the buildComponent.xml are all a "template" type files, all with the same targets, the same dependancies, and the same ordering, all reading in common properties. In this way it should be much easier to trace through all of these tests, simply because they are a small bit easier to understand.
Please, everyone feel free to comment openly on this. Tell me what you hate, and what you like, and where you need more information.
Matt Seibert [EMAIL PROTECTED]
IBM External: (512) 838-3656 Internal: 678-3656
"Steve Loughran" <[EMAIL PROTECTED]>> Prompted by a note from Steve Loughran this morning, and being budged by
> some of my teammates, I am putting in that I am willing to chip in on the
> restructuring.
>
> Steve,
>
> I have been working for about two weeks on the re-structuring of the test/
> and samples/ compilation / execution process, being able to componentize
> and instantiate any given test singularly, and making the system easier to
> maintain and, hopefully, a little better organized.
>
> I would like to speak with you about how I foresee the build/execution of
> the test and samples structures changing in correlation to how you think
> the overall build process should be changing. I would very much like for
> you to review my proposed changes, as well as see what your proposals are,
> so as to produce the best possible output.
>
> Please feel free to ping me back. I am new to the axis community, and am
> not quite sure how we all go about sharing information......
I am new to axis too, but worked on other apache projects. we can discuss
stuff on the dev mail list as it stores the discussion and lets other people
get involved; it "eliminates surprises"
So, write up your proposed changes and see where you think we should go.
Me, I think that as well as cleaning up the build process, we need to work
on the test suite so it runs more like a big fat junit test; you run all the
tests and get back an XML file of what worked, what didnt...to date the way
we go right now the first failure seems to stops the run.
I dont know quite how to do that (yet), though I have some definate ideas,
which revolve around how ant's unit tests run build files inside junit, but
with a bit more template driven generation of build files, and maybe even
the junit test cases to call them.
-steve
