hey, this is a good start. I will look at it in more detail in a day or so,
as usual remind me if you hear nothing.
-steve
----- Original Message -----
From: "Matt Seibert" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Sent: Tuesday, May 28, 2002 7:05 AM
Subject: Re: AXIS Test Restructuring
> Steve,
>
> Here is the "high-level" design of my test / samples restructuring effort.
> I began this undertaking with the following input as requirements:
>
> 1. Determine the needs and dependancies of each test and sample
> 2. Reduce the complexity of the xml that compiles, deploys, execs, and
> undeploys samples and tests
> 3. Enable a new test or sample to be "plugged in" to the suites,
without
> disturbing currently working things
> 4. Enable the ability to instantiate any single sample or any single
> test
> 5. Identify tests and samples that need "network" or "internet"
> resources
> 6. Enable the ability to group tests and samples together in "bundles"
>
> This was based on the following information which was relayed to me:
> 1. Developers don't like to run ALL of the tests, if they are changing
> only one part
> 2. Developers don't like adding new tests / samples because their
> integration into the big master build.xml recursive functions
> threatens to break everything
> 3. Some development organizations test in an "off-line" mode, and
> therefore cannot exec ALL of the tests ALL of the time.
All good points
> To these ends, I have done the following:
> 1. Broken out the buildTest and samples targets from build.xml into
> buildTest.xml and buildSamples.xml
> 2. Componentized those big recursive "build everything" calls into
> specific test compilation xml files
> For example, the commands / code to build the sessions test is
> now located in a file, buildComponent.xml, in the directory
> test/sessions under the target "compile"
> 3. Instead, in the buildTest and buildSamples files, there now exists a
> non-complex variant of the ant ForeachTask, which finds all files
> named buildComponent.xml in test/ and samples/ respectively, and
> operates on them.
ok, an instance of the hypothetical <anton> task.
> This allows a new test / sample to be "plugged in" to the
test/
> and samples/ trees, and then be exectued upon, withouth
> having to muck with the master controlling files
nice.
> 4. If a single test or sample is needed to be instantiated directly (is
> a pre-req for another test or sample for example), then the
> developer simply adds a block as below into either buildTest.xml or
> buildSample.xml to enable the single component use of that
> specific test or sample
> <target name="session">
> <ant inheritAll="true"
> antfile="test/session/buildComponent.xml" target="compile"/>
> </target>
I dont like this style of running a build in a different base directory from
where the file lives, it causes confusion. There are some changes I need to
make to axis' build.xml to make the <ant dir= /> tactic work.
> 5. I am still working on this part. The basic jist is that if there is
> no internet (network failure, firewalled development environment, etc)
> the tests should not fail. I have created a property in the test.
> properties file which is test.runtest and is given a value of "LOCAL",
> "REMOTE", or "BOTH" to control this function. This is one of the
> parts that I will need some help with.
regarding online/offline, here is the test from ant1.5'd build file
<!-- test to see if we are online or not. can take a while when we are off
line, so
setting the property is a good shortcut-->
<target name="probe-offline">
<condition property="offline">
<or>
<isset property="offline"/>
<not>
<http url="http://www.apache.org/"/>
</not>
</or>
</condition>
<echo level="verbose" > offline=${offline}</echo>
</target>
if you run it with -Doffline=true then it is manually off-line, otherwise it
probes for apache and keeps going from there...this property is used to tune
some of the tests:
<patternset id="onlinetests">
<exclude name="**/GetTest.java" if="offline" />
</patternset>
This only works on ant1.5, as only it has the <http> condition, but it works
well. Personally I think any reworked Axis test framework should only use
ant1.5. as its basis; gump uses it after all
> 6. As in item 4, in order to "group" a bunch of tests, this can be
> implemented in two methods:
> A. Create a target task in the build<Test,Samples> file that
> instantiates them all
> B. Create a second ForeachTask loop that looks for files
called
> testBundle*.xml and exec each of these.
>
> As for your issue with "stopping on the first failure", I beleive that
> there is already a flag set up to ignoreFailures in the xml files, which
> will then exec all tests, and give you a wrap up of what did, or did not,
> work. Also, to that end, I will need to add a "common output" method for
> these componentized tests to allow them (if the user wishes) to have their
> output placed in either separate log file (reports/test-session.xml), a
> "wrap-up" log file (reports/test.xml), or both. Suggestions and input
here
> would be greatly appreciated.
We have just started discussing on ant-dev where we could go with a
'buildunit' framework; a way of running ant over a set of targets with the
output fed back into junit style XML report docs afterwards, that can be
<style>d into something reasonable.
The other trick would be to take your foreach task, add a failonerror attr,
and have it call ant onto the subbuilds with a new logger to log everything.
Or run ant from the toplevel with its xmllogger, and add a very fancy XSLT
to make something good from the results.
-steve
>
> Also, the buildComponent.xml are all a "template" type files, all with the
> same targets, the same dependancies, and the same ordering, all reading in
> common properties. In this way it should be much easier to trace through
> all of these tests, simply because they are a small bit easier to
> understand.
>
> Please, everyone feel free to comment openly on this. Tell me what you
> hate, and what you like, and where you need more information.
>
> Matt Seibert [EMAIL PROTECTED]
> IBM External: (512) 838-3656 Internal: 678-3656
>
>
> > Prompted by a note from Steve Loughran this morning, and being budged by
> > some of my teammates, I am putting in that I am willing to chip in on
the
> > restructuring.
> >
> > Steve,
> >
> > I have been working for about two weeks on the re-structuring of the
> test/
> > and samples/ compilation / execution process, being able to componentize
> > and instantiate any given test singularly, and making the system easier
> to
> > maintain and, hopefully, a little better organized.
> >
> > I would like to speak with you about how I foresee the build/execution
of
> > the test and samples structures changing in correlation to how you think
> > the overall build process should be changing. I would very much like
for
> > you to review my proposed changes, as well as see what your proposals
> are,
> > so as to produce the best possible output.
> >
> > Please feel free to ping me back. I am new to the axis community, and
am
> > not quite sure how we all go about sharing information......
>
> I am new to axis too, but worked on other apache projects. we can discuss
> stuff on the dev mail list as it stores the discussion and lets other
> people
> get involved; it "eliminates surprises"
>
> So, write up your proposed changes and see where you think we should go.
>
> Me, I think that as well as cleaning up the build process, we need to work
> on the test suite so it runs more like a big fat junit test; you run all
> the
> tests and get back an XML file of what worked, what didnt...to date the
way
> we go right now the first failure seems to stops the run.
>
> I dont know quite how to do that (yet), though I have some definate ideas,
> which revolve around how ant's unit tests run build files inside junit,
but
> with a bit more template driven generation of build files, and maybe even
> the junit test cases to call them.
>
> -steve
