[
https://issues.apache.org/jira/browse/AMQ-5552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299612#comment-14299612
]
Hadrian Zbarcea commented on AMQ-5552:
--------------------------------------
I agree with some of the comments, I am mostly against this proposal. Will try
to explain why.
I agree for instance that running tests for hours unreliably gives a bad first
impression. Quite horrible, I'd think, and not only first impression, second
and third too.
This proposal only addresses releases from the title of this issue. Time is not
a problem as this is a one time excercise. Any entity/company who uses internal
builds have no problem running it overnight, however long it takes, as long as
the result is pretty much guaranteed. For us as a community is it enough to say
that it's a good release it passed the smoke testing? I would say not, the ASF
guidelines say the same thing. If the PMC votes an official release, we
guarantee that it actually works, which is demonstrated by the tests that pass.
Those that don't should be fixed or removed (if they serve no purpose).
I understand the need to be pragmatic. However, even if we run the tests with
-fae, if tests fail in a <module>, tests in all other modules that depend on it
will not be run. Therefore failing tests may hide other failing tests, very
hard or impractical to discover. There is another practical reason. At some
point, I introduced a regression while merging to a branch. One of the best
ways to identify the source is to use git bisect. Given the number of commits I
needed some 7 iterations. Now with activemq, where it takes some 10 mins for a
build skipping tests, plus running the test(s) you're interested, it would take
at least 15-20 min per iteration. While a significant chunk of time, at least
it's feasible. If the tests you start with are unreliable, it's a much, much
harder effort.
There are other arguments, community related.
I don't know what would constitute resolution of this issue. In my opinon
certainly not something that would be at odds with the ASF guidelines for
releases, or at least they way I understand them.
> introduce a smoke-test profile that is enabled by default and during
> release:prepare
> ------------------------------------------------------------------------------------
>
> Key: AMQ-5552
> URL: https://issues.apache.org/jira/browse/AMQ-5552
> Project: ActiveMQ
> Issue Type: Improvement
> Components: Distribution, Test Cases
> Affects Versions: 5.11.0
> Reporter: Gary Tully
> Labels: distribution, mvn, smoke, tests, validation
> Fix For: 5.12.0
>
>
> Users should be able to do $>mvn install on trunk or the source distribution
> and get a validated (smoke-tested) distribution in < 10 minutes.
> The smoke-test profile should be enabled for release:prepare
> At the moment, more than 3k tests are run, they are not reliable enough and
> the build is gone for a number of hours. This gives a bad first impression.
> Or course we should continue to improve the test suite but this has a totally
> different focus.
> The smoke-test profile takes a smart cross section of tests in each module
> that validate core functionality.
> It will be an interesting challenge to get the selection right; balancing
> typical use cases with coverage with speed etc.
> The tests should be:
> * representative of the module functionality
> * clean - no hard-coded ports and use only space on target
> * fast
> * reliable
> * can be run in parallel (maybe if it allows more tests to be run in the
> same time frame)
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
