Re: [Rd] Running package tests and not stop on first fail
On Friday, 4 November 2016, Martin Maechler wrote: > > Dirk Eddelbuettel > > > on Fri, 4 Nov 2016 10:36:52 -0500 writes: > > > On 4 November 2016 at 16:24, Martin Maechler wrote: | My > > proposed name '--no-stop-on-error' was a quick shot; if | > > somebody has a more concise or better "English style" > > wording | (which is somewhat compatible with all the other > > options you see | from 'R CMD check --help'), | please > > speak up. > > > Why not keep it simple? The similar feature this most > > resembles is 'make -k' and its help page has > > >-k, --keep-going > > >Continue as much as possible after an > > error. While the target that failed, and those that > > depend on it, cannot be remade, the other dependencies of > > these targets can be processed all the same. > > Yes, that would be quite a bit simpler and nice in my view. > One may think it to be too vague, Mmn, I would agree on vagueness (and it breaks the pattern set by other flags of human-readability). Deep familiarity with make is probably not something we should ask of everyone who needs to test a package, too. I quite like stop-on-error=true (exactly the same as the previous suggestion but shaves off some characters by inverting the Boolean) notably from Brian Pedersen's mentioning that the examples are > already continued in any case if they lead to an error. > > Other opinions? > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel > [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
> Dirk Eddelbuettel > on Fri, 4 Nov 2016 10:36:52 -0500 writes: > On 4 November 2016 at 16:24, Martin Maechler wrote: | My > proposed name '--no-stop-on-error' was a quick shot; if | > somebody has a more concise or better "English style" > wording | (which is somewhat compatible with all the other > options you see | from 'R CMD check --help'), | please > speak up. > Why not keep it simple? The similar feature this most > resembles is 'make -k' and its help page has >-k, --keep-going >Continue as much as possible after an > error. While the target that failed, and those that > depend on it, cannot be remade, the other dependencies of > these targets can be processed all the same. Yes, that would be quite a bit simpler and nice in my view. One may think it to be too vague, notably from Brian Pedersen's mentioning that the examples are already continued in any case if they lead to an error. Other opinions? __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
> Brian G Peterson > on Fri, 4 Nov 2016 10:37:18 -0500 writes: > On Fri, 2016-11-04 at 16:24 +0100, Martin Maechler wrote: >> > Jan Gorecki > on Fri, 4 >> Nov 2016 11:20:37 + writes: >> >> > Martin, I submitted very simple patch on > >> https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 >> >> > Herve, While I like your idea, I prefer to keep my >> patch > simple, it is now exactly what Martin mentions. I >> think it > is a good start that can eventually be >> extended later for > what you are asking. >> >> I tend to agree; this seems indeed much easier than I >> anticipated. Thank you, Jan! >> >> I'm testing a version which uses the logical variable >> 'stop_on_error' rather than 'no_stop_on_error' (because >> !no_stop_on_error is hard to mentally parse quickly). >> >> My proposed name '--no-stop-on-error' was a quick shot; >> if somebody has a more concise or better "English style" >> wording (which is somewhat compatible with all the other >> options you see from 'R CMD check --help'), please speak >> up. > I might suggest > --stop-tests-on-error > with default=TRUE to match current functionality. Thank you, Brian. though that would be less concise and I think less matching the 'R CMD check' philosophy with many '--no-*' options to turn *off* defaults. Note that most options have no " = " part, because they are binary and I think that's easiest for use (when the 'binary' case is general enough). Also R CMD check --help ends saying "By default, all test sections are turned on." which does fit the use of all those '--no-*' options. OTOH, we also have '--ignore-vignettes' so we could consider --ignore-tests-errors ? > This might avoid any confusion related to the behavior of > continuing to run examples on error in R CMD check. You are quite right on that, indeed. Martin > Regards, > Brian __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
On Fri, 2016-11-04 at 16:24 +0100, Martin Maechler wrote: > > Jan Gorecki > > on Fri, 4 Nov 2016 11:20:37 + writes: > > > Martin, I submitted very simple patch on > > https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 > > > Herve, While I like your idea, I prefer to keep my patch > > simple, it is now exactly what Martin mentions. I think it > > is a good start that can eventually be extended later for > > what you are asking. > > I tend to agree; this seems indeed much easier than I > anticipated. Thank you, Jan! > > I'm testing a version which uses the logical variable > 'stop_on_error' rather than 'no_stop_on_error' (because > !no_stop_on_error is hard to mentally parse quickly). > > My proposed name '--no-stop-on-error' was a quick shot; if > somebody has a more concise or better "English style" wording > (which is somewhat compatible with all the other options you see > from 'R CMD check --help'), > please speak up. I might suggest --stop-tests-on-error with default=TRUE to match current functionality. This might avoid any confusion related to the behavior of continuing to run examples on error in R CMD check. Regards, Brian __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
On 4 November 2016 at 16:24, Martin Maechler wrote: | My proposed name '--no-stop-on-error' was a quick shot; if | somebody has a more concise or better "English style" wording | (which is somewhat compatible with all the other options you see | from 'R CMD check --help'), | please speak up. Why not keep it simple? The similar feature this most resembles is 'make -k' and its help page has -k, --keep-going Continue as much as possible after an error. While the target that failed, and those that depend on it, cannot be remade, the other dependencies of these targets can be processed all the same. Dirk -- http://dirk.eddelbuettel.com | @eddelbuettel | e...@debian.org __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
> Jan Gorecki > on Fri, 4 Nov 2016 11:20:37 + writes: > Martin, I submitted very simple patch on > https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 > Herve, While I like your idea, I prefer to keep my patch > simple, it is now exactly what Martin mentions. I think it > is a good start that can eventually be extended later for > what you are asking. I tend to agree; this seems indeed much easier than I anticipated. Thank you, Jan! I'm testing a version which uses the logical variable 'stop_on_error' rather than 'no_stop_on_error' (because !no_stop_on_error is hard to mentally parse quickly). My proposed name '--no-stop-on-error' was a quick shot; if somebody has a more concise or better "English style" wording (which is somewhat compatible with all the other options you see from 'R CMD check --help'), please speak up. Martin > Regards, Jan > On 3 November 2016 at 17:25, Hervé Pagès > wrote: >> >> Hi Martin, Jan, >> >> On 11/03/2016 03:45 AM, Martin Maechler wrote: Jan Gorecki on Tue, 1 Nov 2016 22:51:28 + writes: >>> >>> >>> > Hello community/devs, Is there an option to run >>> package > tests during R CMD check and not stop on first >>> error? I > know that testing frameworks (testhat and >>> others) can do > that but asking about just R and base >>> packages. Currently > when package check runs test >>> scripts in ./tests directory > it will stop after first >>> fail. Do you think it could be > optionally available >>> to continue to run tests after > failures? Regards, Jan >>> Gorecki >>> >>> I agree that this would be a useful option sometimes. >>> >>> So I would be supportive to get such an option, say, >>> >>> R CMD check --no-stop-on-error >> >> >> A couple of years ago the behavior of 'R CMD check' was >> changed to continue checking (e.g. the examples) after >> many types of errors, and to output a summary count of >> errors at the end if any have occurred. So >> --no-stop-on-error could easily be interpreted as an >> option that controls this behavior (and would also >> suggest that the default has been reverted back to what >> it was prior to R 3.2.0), rather than an option that >> specifically controls what should happen while running >> the tests. >> >> Cheers, H. >> >>> >>> into R if someone provided (relatively small) patches to >>> the R sources (i.e. subversion repos at >>> https://svn.r-project.org/R/trunk/ ). The relevant >>> source code should basically all be in >>> src/library/tools/R/testing.R >>> >>> Note that this may be complicated, also because >>> "parallel" checking is available in parts, via the >>> TEST_MC_CORES environment variable ((which is currently >>> only quickly documented in the 'R Administration ..' >>> manual)) >>> >>> >>> Martin Maechler ETH Zurich >>> >>> __ >>> R-devel@r-project.org mailing list >>> https://stat.ethz.ch/mailman/listinfo/r-devel >>> >> >> -- >> Hervé Pagès >> >> Program in Computational Biology Division of Public >> Health Sciences Fred Hutchinson Cancer Research Center >> 1100 Fairview Ave. N, M1-B514 P.O. Box 19024 Seattle, WA >> 98109-1024 >> >> E-mail: hpa...@fredhutch.org Phone: (206) 667-5791 Fax: >> (206) 667-1319 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] Running package tests and not stop on first fail
Martin, I submitted very simple patch on https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176 Herve, While I like your idea, I prefer to keep my patch simple, it is now exactly what Martin mentions. I think it is a good start that can eventually be extended later for what you are asking. Regards, Jan On 3 November 2016 at 17:25, Hervé Pagès wrote: > > Hi Martin, Jan, > > On 11/03/2016 03:45 AM, Martin Maechler wrote: >>> >>> Jan Gorecki >>> on Tue, 1 Nov 2016 22:51:28 + writes: >> >> >> > Hello community/devs, Is there an option to run package >> > tests during R CMD check and not stop on first error? I >> > know that testing frameworks (testhat and others) can do >> > that but asking about just R and base packages. Currently >> > when package check runs test scripts in ./tests directory >> > it will stop after first fail. Do you think it could be >> > optionally available to continue to run tests after >> > failures? Regards, Jan Gorecki >> >> I agree that this would be a useful option sometimes. >> >> So I would be supportive to get such an option, say, >> >>R CMD check --no-stop-on-error > > > A couple of years ago the behavior of 'R CMD check' was changed to > continue checking (e.g. the examples) after many types of errors, and > to output a summary count of errors at the end if any have occurred. > So --no-stop-on-error could easily be interpreted as an option that > controls this behavior (and would also suggest that the default has > been reverted back to what it was prior to R 3.2.0), rather than an > option that specifically controls what should happen while running > the tests. > > Cheers, > H. > >> >> into R if someone provided (relatively small) patches to the R >> sources (i.e. subversion repos at https://svn.r-project.org/R/trunk/ ). >> The relevant source code should basically all be in >> src/library/tools/R/testing.R >> >> Note that this may be complicated, also because "parallel" >> checking is available in parts, via the TEST_MC_CORES >> environment variable ((which is currently only quickly >> documented in the 'R Administration ..' manual)) >> >> >> Martin Maechler >> ETH Zurich >> >> __ >> R-devel@r-project.org mailing list >> https://stat.ethz.ch/mailman/listinfo/r-devel >> > > -- > Hervé Pagès > > Program in Computational Biology > Division of Public Health Sciences > Fred Hutchinson Cancer Research Center > 1100 Fairview Ave. N, M1-B514 > P.O. Box 19024 > Seattle, WA 98109-1024 > > E-mail: hpa...@fredhutch.org > Phone: (206) 667-5791 > Fax:(206) 667-1319 __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
