On 04/09/2012 3:44 PM, Terry Therneau wrote:
On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
> On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
>> On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.dun...@gmail.com> wrote:
>>
>>
>> >On 04/09/2012 8:20 AM, Terry Therneau wrote:
>> >>
>> >> On 09/04/2012 05:00 AM, r-devel-requ...@r-project.org wrote:
>> >> > The issue is not just about "CRAN" vs "off CRAN".
>> >> > It is good to think about a more general scheme of
>> >> > "light testing" vs "normal testing" vs "extensive testing",
>> >> > e.g., for the situation where the package implements
>> >> > (simulation/bootstrap/ ..) based inference, and the developer
>> >> > (but not only) should be able to run the extensive tests.
>> >> >
>> >> > Martin
>> >>
>> >> I agree with Martin. A mechanism to specify testing level would be the
>> >>best.
>> >> Then CRAN can choose to set that variable to "3" say, with level 1 for
>> >>extensive and 2 for
>> >> usual.
>> >> I'm quite willing to put up with the nuisance of print()
>> >>enclosures. I prefer it to
>> >> having yet another way to subvert the evaluation model.
>> >>
>> >> I'm a believer in testing everything possible in my packages, and
>> >>wear it it as a badge
>> >> of honor that the survival package has 4 lines of R code in the tests
>> >>directory for every
>> >> 3 in the R directory. But CRAN only needs to run a small subset of
>> >>this.
>> >
>> >We have a mechanism to specify testing level: the --as-cran flag. We
>> >could presumably make it more elaborate by adding other flags, or option
>> >levels, or whatever.
>> >
>> >What I think we shouldn't do is try to create an R-level test that says
>> >
>> > if (testingLevel() > 3) {
>> > doSomething
>> >}
>> >
>> >because tests can be turned on and off, individually. If testingLevel 3
>> >specified tests (A, B, C), then is our testingLevel higher if we are
>> >running tests (A, B, D, E, F, G)? Why not just test for the presence of
>> >whichever test is most relevant to that particular code block, e.g.
>> >
>> > if ("D" %in% tests()) {
>> > doSomething
>> >}
>>
>>
>> I would prefer the testingLevel() approach of the "D" %in% tests()
>> approach, since testingLevel() provides a natural way to add successively
>> greater test details without having to dig into the code to determine the
>> set of tests.
>
> I don't see how you could possibly calculate a single number in a reasonable
way. What is
> the number that should be returned for (A, B, D, E, F, G)?
>
> Duncan Murdoch
Duncan is leapfrogging ahead to another level that I hadn't thought of. An
example would
be to divide my survival package as "cox", "parametric", "survfit", "all", some
of whaich
overlap. Interesting idea, but beyond what I'd use. When I'm focused on a
detail I run
the test of interest directly, not through CMD check. For me low, med, high
intensity
suffices with as-cran invoking the first level and high including those that
take an
exceptionally long time.
If you went to an A, B, C, ... approach what would be the as-cran default?
Of course there is then the furthest level, which I recently instituted for
survival due
to the large number of dependencies: before posting a change download all the
other
dependent packages and run their tests too. It's an overnighter, and in that
case I'd
want level=high. Forewarned is forearmed.
I don't think I'm leapfrogging, I think you're asking about something
new. The thread was started by Spencer asking how to determine within
an example whether he was being tested by CRAN, because he didn't want
to run a long test there (it caused a timeout). This is an instance of
the general problem of tailoring example and test code to the tests
being run.
The testingLevel() function is supposed to be a way to know that a
certain level of testing is being done, to allow such tailoring.
However, I don't think it's practical. I think you can ask whether a
specific test is being run (my "D" %in% tests() example), but you can't
reasonably convert the set of tests chosen by a tester into a single number.
What I think you and Greg are talking about is something different. You
are asking that we set up more suites of tests, corresponding to
numerical levels. Currently we have two suites: the default, and the
--as-cran suite. But we also have completely customized suites, set by
users who want to check specific things. They can do that the way you
do (by calling the tests explicitly), or by setting environment
variables (as described in the Tools chapter of the R Internals manual).
Duncan Murdoch
______________________________________________
R-devel@r-project.org mailing list
https://stat.ethz.ch/mailman/listinfo/r-devel
