Re: [R] The Quality Accuracy of R
It would be possible to develop tools to develop code coverage statistics quantifying the percent of the code that the tests exercise. On Fri, Jan 23, 2009 at 10:04 AM, Muenchen, Robert A (Bob) muenc...@utk.edu wrote: Hi All, We have all had to face skeptical colleagues asking if software made by volunteers could match the quality and accuracy of commercially written software. Thanks to the prompting of a recent R-help thread, I read, R: Regulatory Compliance and Validation Issues, A Guidance Document for the Use of R in Regulated Clinical Trial Environments (http://www.r-project.org/doc/R-FDA.pdf). This is an important document, of interest to the general R community. The question of R's accuracy is such a frequent one, it would be beneficial to increase the visibility of the non-clinical information it contains. A document aimed at a general audience, entitled something like, R: Controlling Quality and Assuring Accuracy could be compiled from the these sections: 1. What is R? (section 4) 2. The R Foundation for Statistical Computing (section 3) 3. The Scope of this Guidance Document (section 2) 4. Software Development Life Cycle (section 6) Marc Schwartz, Frank Harrell, Anthony Rossini, Ian Francis and others did such a great job that very few words would need to change. The only addition I suggest is to mention how well R did in, Keeling Parvur's A comparative study of the reliability to nine statistical software packages, May 1, 2007 Computational Statistics Data Analysis, Vol.51, pp 3811-3831. Given the importance of this issue, I would like to see such a document added to the PDF manuals in R's Help. The document mentions (Sect. 6.3) that a set of validation tests, data and known results are available. It would be useful to have an option to run that test suite in every R installation, providing clear progress, Validating accuracy of t-tests...Validating accuracy of linear regression Whether or not people chose to run the tests, they would at least know that such tests are available. Back in my mainframe installation days, this step was part of many software installations and it certainly gave the impression that those were the companies that took accuracy seriously. Of course the other companies probably just ran their validation suite before shipping, but seeing it happen had a tremendous impact. I don't know how much this would add to download, but if it was too much, perhaps it could be implemented as a separate download. I hope these suggestions can help mitigate the concerns so many non-R users have. Cheers, Bob = Bob Muenchen (pronounced Min'-chen), Manager, Research Computing Support U of TN Office of Information Technology Stokely Management Center, Suite 200 916 Volunteer Blvd., Knoxville, TN 37996-0520 Voice: (865) 974-5230 FAX: (865) 974-4810 Email: muenc...@utk.edu Web: http://oit.utk.edu/research http://oit.utk.edu/scc Map to Office: http://www.utk.edu/maps Newsletter: http://listserv.utk.edu/archives/rcnews.html http://listserv.utk.edu/archives/statnews.html = [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] The Quality Accuracy of R
That's a great idea. I know of no commercial vendors who provide such detailed info. Bob -Original Message- From: Gabor Grothendieck [mailto:ggrothendi...@gmail.com] Sent: Monday, January 26, 2009 7:52 PM To: Muenchen, Robert A (Bob) Cc: R-help@r-project.org Subject: Re: [R] The Quality Accuracy of R It would be possible to develop tools to develop code coverage statistics quantifying the percent of the code that the tests exercise. On Fri, Jan 23, 2009 at 10:04 AM, Muenchen, Robert A (Bob) muenc...@utk.edu wrote: Hi All, We have all had to face skeptical colleagues asking if software made by volunteers could match the quality and accuracy of commercially written software. Thanks to the prompting of a recent R-help thread, I read, R: Regulatory Compliance and Validation Issues, A Guidance Document for the Use of R in Regulated Clinical Trial Environments (http://www.r-project.org/doc/R-FDA.pdf). This is an important document, of interest to the general R community. The question of R's accuracy is such a frequent one, it would be beneficial to increase the visibility of the non-clinical information it contains. A document aimed at a general audience, entitled something like, R: Controlling Quality and Assuring Accuracy could be compiled from the these sections: 1. What is R? (section 4) 2. The R Foundation for Statistical Computing (section 3) 3. The Scope of this Guidance Document (section 2) 4. Software Development Life Cycle (section 6) Marc Schwartz, Frank Harrell, Anthony Rossini, Ian Francis and others did such a great job that very few words would need to change. The only addition I suggest is to mention how well R did in, Keeling Parvur's A comparative study of the reliability to nine statistical software packages, May 1, 2007 Computational Statistics Data Analysis, Vol.51, pp 3811-3831. Given the importance of this issue, I would like to see such a document added to the PDF manuals in R's Help. The document mentions (Sect. 6.3) that a set of validation tests, data and known results are available. It would be useful to have an option to run that test suite in every R installation, providing clear progress, Validating accuracy of t-tests...Validating accuracy of linear regression Whether or not people chose to run the tests, they would at least know that such tests are available. Back in my mainframe installation days, this step was part of many software installations and it certainly gave the impression that those were the companies that took accuracy seriously. Of course the other companies probably just ran their validation suite before shipping, but seeing it happen had a tremendous impact. I don't know how much this would add to download, but if it was too much, perhaps it could be implemented as a separate download. I hope these suggestions can help mitigate the concerns so many non-R users have. Cheers, Bob = Bob Muenchen (pronounced Min'-chen), Manager, Research Computing Support U of TN Office of Information Technology Stokely Management Center, Suite 200 916 Volunteer Blvd., Knoxville, TN 37996-0520 Voice: (865) 974-5230 FAX: (865) 974-4810 Email: muenc...@utk.edu Web: http://oit.utk.edu/research http://oit.utk.edu/scc Map to Office: http://www.utk.edu/maps Newsletter: http://listserv.utk.edu/archives/rcnews.html http://listserv.utk.edu/archives/statnews.html = [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
Re: [R] The Quality Accuracy of R
Bob, Your point is well taken, but it also raises a number of issues (post-install testing to name one) for which the R-devel list would be more suitable. Could we move the discussion there? -Peter Muenchen, Robert A (Bob) wrote: Hi All, We have all had to face skeptical colleagues asking if software made by volunteers could match the quality and accuracy of commercially written software. Thanks to the prompting of a recent R-help thread, I read, R: Regulatory Compliance and Validation Issues, A Guidance Document for the Use of R in Regulated Clinical Trial Environments (http://www.r-project.org/doc/R-FDA.pdf). This is an important document, of interest to the general R community. The question of R's accuracy is such a frequent one, it would be beneficial to increase the visibility of the non-clinical information it contains. A document aimed at a general audience, entitled something like, R: Controlling Quality and Assuring Accuracy could be compiled from the these sections: 1. What is R? (section 4) 2. The R Foundation for Statistical Computing (section 3) 3. The Scope of this Guidance Document (section 2) 4. Software Development Life Cycle (section 6) Marc Schwartz, Frank Harrell, Anthony Rossini, Ian Francis and others did such a great job that very few words would need to change. The only addition I suggest is to mention how well R did in, Keeling Parvur's A comparative study of the reliability to nine statistical software packages, May 1, 2007 Computational Statistics Data Analysis, Vol.51, pp 3811-3831. Given the importance of this issue, I would like to see such a document added to the PDF manuals in R's Help. The document mentions (Sect. 6.3) that a set of validation tests, data and known results are available. It would be useful to have an option to run that test suite in every R installation, providing clear progress, Validating accuracy of t-tests...Validating accuracy of linear regression Whether or not people chose to run the tests, they would at least know that such tests are available. Back in my mainframe installation days, this step was part of many software installations and it certainly gave the impression that those were the companies that took accuracy seriously. Of course the other companies probably just ran their validation suite before shipping, but seeing it happen had a tremendous impact. I don't know how much this would add to download, but if it was too much, perhaps it could be implemented as a separate download. I hope these suggestions can help mitigate the concerns so many non-R users have. Cheers, Bob = Bob Muenchen (pronounced Min'-chen), Manager, Research Computing Support U of TN Office of Information Technology Stokely Management Center, Suite 200 916 Volunteer Blvd., Knoxville, TN 37996-0520 Voice: (865) 974-5230 FAX: (865) 974-4810 Email: muenc...@utk.edu Web: http://oit.utk.edu/research http://oit.utk.edu/scc Map to Office: http://www.utk.edu/maps Newsletter: http://listserv.utk.edu/archives/rcnews.html http://listserv.utk.edu/archives/statnews.html = [[alternative HTML version deleted]] __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code. -- O__ Peter Dalgaard Øster Farimagsgade 5, Entr.B c/ /'_ --- Dept. of Biostatistics PO Box 2099, 1014 Cph. K (*) \(*) -- University of Copenhagen Denmark Ph: (+45) 35327918 ~~ - (p.dalga...@biostat.ku.dk) FAX: (+45) 35327907 __ R-help@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-help PLEASE do read the posting guide http://www.R-project.org/posting-guide.html and provide commented, minimal, self-contained, reproducible code.
