Re: Beehive Check-in Test (Linux) Failed

[Eddie O'Neil]

  Definitely understand the cmd.exe issues.  :)
  But, isn't this just a matter of setting a flag for whether to fail a 
test run?  It would be an addition to what we are doing now.

  I'd propose that this work as follows:
- default TCH behavior is to fail the build in the case of a test 
failure / abort / etc.  TCH does this by setting an Ant property, for 
example "test.failure" that can be used in Ant like:

<target name="drt.errors" if="test.failure">
    <echo>!!!!! Errors or failures occurred running tests !!!!!</echo>
    <fail message="Controls TCH Tests Failed"/>
</target>
- this behavior can be over-ridden in CruiseControl mode.  The override 
will still print some failure message (or even set a property) but 
doesn't set the "test.failure" property so that the build fails.

The latter is the behavior that we have today; the former is just an 
addition to the TCH behavior / API.  This would make TCH more like JUnit 
in the way that failures can be controlled through properties.

  Thoughts?
Eddie

James Song wrote:
First, this is a windows only issue. Ant does not have this problem on
other platform. There is no need to do String searching for failure
detection on Linux/Unix.
Second, on Windows, someone mentioned python and perl works better than
cmd.exe. Maybe cmd.exe could be switched to python or perl?
-James
-----Original Message-----
From: Eddie O'Neil 
Sent: Wednesday, March 16, 2005 1:27 PM
To: Beehive Developers
Cc: Joe Pemberton; Zachary Smith
Subject: Re: Beehive Check-in Test (Linux) Failed

   Hm.  I've seen CC machine configs that can handle this sort of thing,
so we should dig into how to fix it.
   IMHO, it's pretty important to be very obvious about test failures 
when running in a shell, so we should look at addressing this.

   Any other thoughts?
Eddie

James Song wrote:
I come across this issue when I set up a cc machine.
http://issues.apache.org/bugzilla/show_bug.cgi?id=30216
CC code boils down to the following:
<exec dir="." executable="cmd" failonerror="false"
           resultproperty="build.result">
The resultproperty does not return the correct value on Windows
sometimes.
That is why I switched the failure detecting code to look for "Build
Failed" from the log file.
-James
-----Original Message-----
From: Joe Pemberton 
Sent: Wednesday, March 16, 2005 1:03 PM
To: 'Beehive Developers'; Zachary Smith; James Song
Subject: RE: Beehive Check-in Test (Linux) Failed

From what I understand, the reason the test run does not fail early
there is that for a CruiseControl run, it is useful to do the full
test
run (and not fail early if the controls tests have errors).  After the
full controls test build/run, CC greps for "BUILD FAILED", and fails
the
build accordingly.
James may know more about this..
-Joe

-----Original Message-----
From: Eddie O'Neil 
Sent: Wednesday, March 16, 2005 12:58 PM
To: Beehive Developers
Subject: Re: Beehive Check-in Test (Linux) Failed

  My take is that those three test aborts should have failed the
entire
Ant process at the point where the controls tests failed.
  Without that, the tests fail and the build continues, and the Ant 
process can still end in BUILD SUCCESSFUL even if there were aborts.

  Probably good to follow the JUnit / Ant model here -- fail early.
  :)
Eddie

Joe Pemberton wrote:

Eddie,
I'm not sure I follow you.  I see the three aborts in the TCH run,
and

I see a BUILD FAILED at the end of the run.  Also, the CC run itself
failed.
-Joe 

-----Original Message-----
From: Eddie O'Neil 
Sent: Wednesday, March 16, 2005 10:45 AM
To: Beehive Developers
Subject: Re: Beehive Check-in Test (Linux) Failed

 A second-order problem is that there are three tests in the TCH
test 


suite that were aborted (and even stack traced), but these didn't fail

the controls test suite.  Look at this link and scroll down to the TCH

output:
 http://beehive01.bea.com/downloads/20050316052618/general/build.out
 Zach, can you take a look at this?  We saw the same thing with some

controls test failures last week.
Eddie

Daryl Olander wrote:

So it appears that there is a break in the controls checkin tests
that
was caused by the same checkin from this morning.
http://svn.apache.org/viewcvs.cgi?rev=157742&view=rev
CC was broken at 5:26 this morning.  There were two sets of failures,
the WSM Drts failed in addition the  controls check in test failed
http://beehive01.bea.com/downloads/20050316052618/controls/tch.cout
There is only a single checkin here.
On Wed, 16 Mar 2005 10:33:09 -0800 (PST), [EMAIL PROTECTED]
<[EMAIL PROTECTED]> wrote:


View results here ->

http://beehive01.bea.com/cruisecontrol/buildresults?log=log2005031610210
0