I would agree with all those points
> On Jun 7, 2016, at 2:12 PM, Howard Pritchard <hpprit...@gmail.com> wrote:
>
> HI Ralph,
>
> We briefly discussed this some today. I would like to avoid the mini-MTT
> approach for PR checking.
> At the same time, one can also see why it might be useful from time to time
> to make changes to
> the script a given jenkins project runs on PRs.
>
> An idea we discussed was to have jenkins folks support a "stable" version of
> their jenkins script. If they would
> like to make changes, they would create an experimental, temporary jenkins
> project to run the new script.
> If the new project's script runs clean against open PRs, the new script can
> be swapped in to the
> original jenkins project. The experimental project could then be
> deactivated. If the new script showed failures in the
> open PRs, or against master or other branch, issues can be opened to track
> the problem(s) found by the
> script. The experimental, temporary jenkins project can continue to run, but
> its "failure" status can be ignored
> until the underlying bug(s) is fixed.
>
> I don't think it makes much sense to run a jenkins script against PRs if it
> fails when run against master.
> The purpose of jenkins PR testing is to trap new problems, not to keep
> reminding us there are problems
> with the underlying branch which the PR targets.
>
> Howard
>
>
> 2016-06-07 13:33 GMT-06:00 Ralph Castain <r...@open-mpi.org
> <mailto:r...@open-mpi.org>>:
> Hi folks
>
> I’m trying to get a handle on our use of Jenkins testing for PRs prior to
> committing them. When we first discussed this, it was my impression that our
> objective was to screen PRs to catch any errors caused by differences in
> environment and to avoid regressions. However, it appears that the tests keep
> changing without warning, leading to the impression that we are using Jenkins
> as a “mini-MTT” testing device.
>
> So I think we need to come to consensus on the purpose of the Jenkins
> testing. If it is to screen for regressions, then the tests need to remain
> stable. A PR that does not introduce any new problems might not address old
> ones, but that is no reason to flag it as an “error”.
>
> On the other hand, if the objective is to use Jenkins as a “mini-MTT”, then
> we need to agree on how/when a PR is ready to be merged. Insisting that
> nothing be merged until even a mini-MTT is perfectly clean is probably
> excessively prohibitive - it would require that the entire community (and not
> just the one proposing the PR) take responsibility for cleaning up the code
> base against any and all imposed tests.
>
> So I would welcome opinions on this: are we using Jenkins as a screening tool
> on changes, or as a test for overall correctness of the code base?
>
> Ralph
>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org <mailto:de...@open-mpi.org>
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> <https://www.open-mpi.org/mailman/listinfo.cgi/devel>
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/06/19087.php
> <http://www.open-mpi.org/community/lists/devel/2016/06/19087.php>
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: https://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2016/06/19088.php
