I agree with breaking temporarily a big compliance or function test suite.
I think we need to split our current "acceptance" test suite into two
branches (not necessarily right now but in a near future):
a) a set of basic Build Verification Tests that verify that our runtime
can run basic scenarios with Tomcat, Web Services, JSPs and POJOs, this
should take just a few minutes max to run
b) a much bigger Function Verification Test suite that will grow over
time and this is what we'll run before any release/distribution.
IMO what we have in our current acceptance test suite covers (a) right
now (in other words it looks really like a BVT to me) and I'd like to
try to keep it working at all times. For example if our runtime does not
even start in Tomcat, then there's really not much you can do with
it... and this will slow down all of us.
What do people in the group think?
--
Jean-Sebastien
Jeremy Boynes wrote:
Jean-Sebastien Delfino wrote:
- I think we need to establish a policy that the build should never
break and in particular the acceptance test suite should run before each
commit. The sooner we get a stable build the sooner all of us can
engage with all the work we have ahead of us.
I thought we split these to allow some stuff to break temporarily.
IOW, normal unit/integration tests would run automatically as part of a
normal build and were supposed to work for every commit. The goal is to
produce a build that is "good enough" for the developers to work with -
i.e. the default build works every time, no excuses.
The acceptance tests were meant to replicate scenarios users would see
and were meant to be run before any release/distribution e.g. before
posing an "unstable" build somewhere. Eventually these would grow (e.g.
to include compliance tests) and take so long to run that it would be
impractical to run them for every commit.
--
Jeremy
