I think that the principle of building a test case that demonstrates the
problem is a good practice. If there is a problem, them it is almost
always the case that the current test suite does not find the problem.
To ensure a fix, then *someone* has to build the test, which needs to be
added to the test suite. While I dont want to stop people filing
problems, it is fair to share around the creation of testcases.
Whether a test is a unit test or an integration test depends on the
problem and how it was found. You can never have too many tests...
Yours, Mike.
Jim Marino wrote:
On May 3, 2006, at 11:59 AM, Jean-Sebastien Delfino wrote:
Jim Marino wrote:
I tried to distinguish between commiters and non-commiters. For
*commiters* we should get into the habit of using unit tests to
demonstrate problems since I believe we want to encourage this among
non-comitters as well (note several Apache and many open source
projects do this, including many of the Geronimo subprojects). I'm
not saying showing how to reproduce the problem using a sample is
somehow "inappropriate", I'm just asking that commiters adopt a
default of using unit tests since it helps the project. Of course,
there are times when this is not easy to do, but when it is (which
was the case in the examples I referred to) it seems like a logical
request to make. Also, please note I did not mean to imply that the
bug fixer is free of the responsibility to write testcases
associated with the fix (actually I would argue that if they must
when checking in the fix).
Do you have an issue with this approach? If so, could you please
outline what specific concerns you have?
Jim
On May 3, 2006, at 8:43 AM, ant elder wrote:
I'm not sure I completely agree with this, if you make it hard to
report
problems then people don't report them. A bug reporter should have to
describe/provide just enough of a way to easily recreate a problem,
its down
to the bug fixer to write whatever regression and unit tests are
required by
the fix. If you can describe how to demonstrate a bug just by
running one of
our samples that's fine in my book.
...ant
On 4/25/06, James F Marino <[EMAIL PROTECTED]> wrote:
I've noticed several issues have been logged against the Java
container where entire Maven projects or samples are used to
demonstrate a problem. Sometimes this may be unavoidable, but I
would
like to propose that we get into the habit of submitting failing
testcases as a default instead. This is common practice in many
projects, and my reasons for suggesting this are:
- it helps diagnose the problem
- it provides verification when the issue is resolved
- it builds up our test suite which is woefully inadequate
- it is often less work than creating entire projects
Obviously, this can't be a literal policy but I believe it should be
"default" behavior for those directly involved in the project. For
non-commiters, we could encourage this behavior by noting that logged
issues with testcases generally get attention quicker.
Also, I would like to propose that while we may tolerate an issue
logged with a project which requires a particular IDE such as
IntelliJ
or Eclipse, or requires "main" to run from those outside the project,
commiters should never do this.
Comments?
Jim, could you please confirm what you mean by "test case" in this
context? Are you asking for a function test case? an integration test
case? or are you really talking about a unit test case?
This may depend on the issue but it is likely to be a unit or
integration test case. If we do not catch something there, it is
generally a problem with test coverage, although there are exceptions
to the rule (no pun intended). For example, if we can't use a unit or
integration test to check something, it's often (but not always)
reflects a design problem. An exception to this rule that comes to mind
are platform-related issues.
Also, we've been looking at adding to our web site some guideline
documents for contributors (exception handling, logging, coding
guidelines, submitting patches etc), can you point us to any
guideline document from the mentioned Apache projects describing this
approach?
Off the top of my head:
ActiveMQ: http://www.activemq.org/Contributing
Also, Spring (which obviously is not Apache).
I'm sure I can dig up others if I looked but I think those should
suffice. FYI on exception handling, Haleh was kind enough to revise
the original project exception handling document - I have a todo to
take a look at it.
Jim
--
Jean-Sebastien
