Joe and Zach--
As we get closer to shipping a Beehive release, it would be great if we could get this controls test framework issue fixed. Still seems right to fail the Ant build when the TCH tests fail. This will still work with Cruisecontrol if you set a flag "cc.mode" when running in that environment. Would it be possible to address this in the next few weeks?
Thoughts?
Eddie
Eddie O'Neil wrote:
Definitely understand the cmd.exe issues. :)
But, isn't this just a matter of setting a flag for whether to fail a test run? It would be an addition to what we are doing now.
I'd propose that this work as follows:
- default TCH behavior is to fail the build in the case of a test failure / abort / etc. TCH does this by setting an Ant property, for example "test.failure" that can be used in Ant like:
<target name="drt.errors" if="test.failure"> <echo>!!!!! Errors or failures occurred running tests !!!!!</echo> <fail message="Controls TCH Tests Failed"/> </target>
- this behavior can be over-ridden in CruiseControl mode. The override will still print some failure message (or even set a property) but doesn't set the "test.failure" property so that the build fails.
The latter is the behavior that we have today; the former is just an addition to the TCH behavior / API. This would make TCH more like JUnit in the way that failures can be controlled through properties.
Thoughts?
Eddie
James Song wrote:
First, this is a windows only issue. Ant does not have this problem on other platform. There is no need to do String searching for failure detection on Linux/Unix.
Second, on Windows, someone mentioned python and perl works better than cmd.exe. Maybe cmd.exe could be switched to python or perl?
-James
-----Original Message----- From: Eddie O'Neil Sent: Wednesday, March 16, 2005 1:27 PM To: Beehive Developers Cc: Joe Pemberton; Zachary Smith Subject: Re: Beehive Check-in Test (Linux) Failed
Hm. I've seen CC machine configs that can handle this sort of thing,
so we should dig into how to fix it.
IMHO, it's pretty important to be very obvious about test failures when running in a shell, so we should look at addressing this.
Any other thoughts?
Eddie
James Song wrote:
I come across this issue when I set up a cc machine.
http://issues.apache.org/bugzilla/show_bug.cgi?id=30216
CC code boils down to the following:
<exec dir="." executable="cmd" failonerror="false" resultproperty="build.result">
The resultproperty does not return the correct value on Windows sometimes.
That is why I switched the failure detecting code to look for "Build Failed" from the log file.
-James
-----Original Message----- From: Joe Pemberton Sent: Wednesday, March 16, 2005 1:03 PM To: 'Beehive Developers'; Zachary Smith; James Song Subject: RE: Beehive Check-in Test (Linux) Failed
From what I understand, the reason the test run does not fail early
there is that for a CruiseControl run, it is useful to do the full
test
run (and not fail early if the controls tests have errors). After the full controls test build/run, CC greps for "BUILD FAILED", and fails
the
build accordingly.
James may know more about this..
-Joe
-----Original Message----- From: Eddie O'Neil Sent: Wednesday, March 16, 2005 12:58 PM To: Beehive Developers Subject: Re: Beehive Check-in Test (Linux) Failed
My take is that those three test aborts should have failed the
entire
Ant process at the point where the controls tests failed.
Without that, the tests fail and the build continues, and the Ant process can still end in BUILD SUCCESSFUL even if there were aborts.
Probably good to follow the JUnit / Ant model here -- fail early.
:)
Eddie
Joe Pemberton wrote:
Eddie, I'm not sure I follow you. I see the three aborts in the TCH run,
and
I see a BUILD FAILED at the end of the run. Also, the CC run itself failed.
-Joe -----Original Message----- From: Eddie O'Neil Sent: Wednesday, March 16, 2005 10:45 AM To: Beehive Developers Subject: Re: Beehive Check-in Test (Linux) Failed
A second-order problem is that there are three tests in the TCH
test
suite that were aborted (and even stack traced), but these didn't fail
the controls test suite. Look at this link and scroll down to the TCH
output:
http://beehive01.bea.com/downloads/20050316052618/general/build.out
Zach, can you take a look at this? We saw the same thing with some
controls test failures last week.
Eddie
Daryl Olander wrote:
So it appears that there is a break in the controls checkin tests
that
was caused by the same checkin from this morning.
http://svn.apache.org/viewcvs.cgi?rev=157742&view=rev
CC was broken at 5:26 this morning. There were two sets of failures, the WSM Drts failed in addition the controls check in test failed
http://beehive01.bea.com/downloads/20050316052618/controls/tch.cout
There is only a single checkin here.
On Wed, 16 Mar 2005 10:33:09 -0800 (PST), [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote:
View results here ->
http://beehive01.bea.com/cruisecontrol/buildresults?log=log2005031610210
0
