Micheal J wrote:
> > Micheal J wrote:
> > > Juha-P Lindfors wrote:
> > > > On Fri, 23 Feb 2001, Phan Anh Tran wrote:
> > > > > 1. Are developers required to pass existing regression test
> > > > > before committing changes?
> > > >
> > > > Yes.
> > > >
> > > > > 2. Are developers required to develop a regression test
> > > > > (if one does not exist) and pass it before committing changes?
> > > >
> > > > No.
> > >
> > > Aha. A QA loop-hole.
> > > We can't have that now can we?
> >
> > That depends.
> >
> > What matters for the quality of the
> > product is end-to-end functionality
> > testing.
> > For this we have jbosstest.
>
> Disagree. It looks good is not the same as it *is* good.
I agree that it could be better. It also
seems to become better all the time.
> > Some regression tests belong in the
> > end-to-end functionality testing
> > category, and such should be added to
> > jbosstest in case it wasn't there
> > already. Here we may have a loop-hole,
> > but I haven't seen any problems
> > arising from it yet.
>
> The constant stream of commits, followed by an undo when something breaks
> and then a later re-commit along with a "I have tested it on blah, blah,
> blah and it shouldn't break anything else but..." comment is proof of that.
I agree that we see a lot of CVS commits
with broken code. And IMO we see way
too many.
But these commits always break one of
the jbosstest tests when committed.
So it looks to me like there could be
a problem with discipline: If all tests
do not pass, the code should not be
committed. (To be fair, not all tests
can always be passed. For example, the
CTS UserTransaction test does not pass
because this feature is not yet
implemented. But IMHO, programmers that
commit code to CVS should have a clear
understanding of all tests that do not
pass, and be !"#¤%& sure that it has
nothing to do with the code they are
committing.)
The loop-hole I was talking about comes
when a bug regression test for a fixed
bug belongs into the end-to-end test
category. Then it *should* be added to
the jbosstest test suite, but we do not
require that.
> > Other tests (like unit tests) are not
> > really done to improve the quality of
> > the final product, but to improve the
> > quality of the development process.
>
> Unit-testing (when done to properly) will help by improving our confidence
> in the quality of the code. If the tests define the contract the code is
> supposed to fill accurately then, we know that come rain or shine (or
> commits by whoever) if the any of the tests fail then something has been
> broken (or some test has been badly written).
I understand your point. The jbosstest
does only end-to-end testing, but may
still provide some (a lot) of confidence
in the code.
My problem here is probably that I have
never liked unit tests much. Subsystem
test maybe, but not the (IMHO braindead)
"for each class written, write a set of
tests that use this class in any possible
way".
Such fine granularity tests take a lot of
time to write. Time that IMHO could be
better used by writing subsystem tests,
end-to-end tests and properly documenting
the external interface of classes.
Think of the effort it would take to do
a *full* unit test on a class like
org.jboss.ejb.plugins.TxInterceptorBMT.
> Long term it improves our collective ability to improve the code. Just do it
> and run the tests...
Yes.
Some of this is in the jbosstest tests.
But IMO something else is more important
to have in place first: Simple internal
documentation in the form of code comments.
Way too many classes do not have any
comment describing what they are (except
for "<description>" added by some IDE),
and way too many methods and variables
also have no comment. This makes the code
hard to read, and makes it easier to
misunderstand something.
> The only problem I see is that the tests *have* to be written. And on an
> open source project it can be difficult to get people unfamilliar with the
> unit0testing process to get involved.
I guess this is why there is no
requirement for bug regression tests to
be written even if the test is end-to-end
and ought to be added to jbosstest.
> > But tests do not prove that a product
> > is bug-free. It only gives you a
> > certain confidence that whatever is
> > tested works as tested.
>
> Which is *infinitely* better than not testing at all...
>
> > IMHO, what *really* matters for the
> > quality of JBoss is the large number
> > of users that are using it in a lot
> > of different situations, with
> > different configurations and different
> > beans. A large part of these users
> > report when they have problems that
> > could be bugs in JBoss, and this
> > enables developers to fix things that
> > do not work in situations they have
> > not even thought about.
> > That does not only help to reduce the
> > number of bugs in JBoss, but also
> > makes JBoss more robust. And IMHO
> > robustness is just as important as
> > functionality for JBoss.
>
> This sound like a "write the code, don't test it, just release it" stance.
I am *not* advocating that.
> If there are any bugs, the users would find them soon enough when they try
> to use it. We kind of practise this now and it has served us so far but, I
> suspect we would have been much farther along if we didn't have the regular
> backtracks to (re-)fix something that a new commit has broken, then re-apply
> all the new stuff and hope nothing breaks this time...
I guess that anybody that commits without
running the jbosstest tests practice it...
> IMHO, functional tests and end-user acceptance tests compliment unit-tests.
> They do not replace them. In fact when a bug is found, we ask for a script
> or series of steps that lets us re-create the bug. A bug signature if you
> will.
>
> The reverse should also be true. A bug is due to the code breaking one or
> more contracts that it is supposed to fill. When we find the broken
> contracts, we should produce unit tests to ensure that later fixes/upgrades
> do not break that contract again. Every bug-fix should be accompanied by
> unit tests that can detect the (underlying cause of the) bug were it to
> reappear. A bug-fix signature if you will.
I see your point. Unit tests are for
testing the internal code contracts,
and for some parts of the JBoss code,
these have not even been documented
with a single line of comment. I think
we should try to at least document
these internal contracts first.
But for the end-user, the only thing
that matters is if the external
contracts are adhered to. The end-to-end
tests in jbosstest check this.
The point I was trying to make with
that last comment is that the large
user base of JBoss helps making JBoss
robust.
We simply cannot test JBoss in any
possible configuration, under any
possible operating system, with any
possible Java VM.
Here the feedback from users having
problems with particular setups
is extremely important.
JBoss is system-level software, and
to be useful system-level software
_must_ be robust. Bug-free is IMHO
not enough.
Best Regards,
Ole Husgaard.
--
--------------------------------------------------------------
To subscribe: [EMAIL PROTECTED]
To unsubscribe: [EMAIL PROTECTED]
List Help?: [EMAIL PROTECTED]
