Re: Automatically checking regtests (was: Re: Minor releases?)
On Mon, Aug 17, 2009 at 07:58:32PM -0300, Han-Wen Nienhuys wrote: On Mon, Aug 17, 2009 at 7:58 PM, Han-Wen Nienhuyshanw...@gmail.com wrote: Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. This is the wrong priority: this is the release manager's task, and in the ideal world, and the RM would continue the release if there are regression errors. I mean: he would stop the release process. Agreed for stable releases, but I disagree for devel stuff. The most important part of devel releases is to allow/encourage development. If the syntax changes, then nobody[1][2] can do doc work any more. Now, in an ideal world, everybody has Linux, has all the build tools installed, and knows how to use them. But we don't live in such a world -- it's a pain to compile lilypond even on OSX. (I used OSX for 4 years, and I can't recall *ever* compiling lilypond on it!) [1] ok, *I'm* obsessive enough to work on docs that I can't compile, and foolhardy enough to commit doc changes I haven't tested -- but few people will do the first, and nobody should do the second. [2] most of the more technically-inclined doc writers have shifted focus to programming. I heartily encourage this, since we need more programmers, but it *does* mean that our average doc writer... whatever that term might mean... is unable to compile lilypond himself. Therefore, if I think there's a need for a new binary (syntax change, major new feature, etc), then as long as the installers and docs compile, we should have the unstable release. Users will just have to learn that unstable release means unstable. :) Cheers, - Graham ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Automatically checking regtests (was: Re: Minor releases?)
[CC to -devel] (nobody checks the regression tests for each release, for example -- and that's trivially done with a web browser!) That reminds me of an idea I recently had: Wouldn't it be possible to automatically generate a sort of checksum for each regression-test output-file and compare it with the former releases? I know that every little change in the pdf appearance would lead to a warning then. But I'm sure there is a way to do not just md5sum regression-test-foo.pdf but create a difference score by comparing the two pdfs. Ideally, this would output to a list like: foo1.pdf - 0 foo2.pdf - 13 foo3.pdf - 142 foo4.pdf - 0 (...) Then the dev's can mark the foo3.pdf like Jerry R. Ehman's wow signal... ;) I mean, this would give a priority for checking, when e.g. values 100 are urgent to control. Cheers, Michael ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am Montag, 17. August 2009 16:08:36 schrieb Michael Käppler: [CC to -devel] (nobody checks the regression tests for each release, for example -- and that's trivially done with a web browser!) That reminds me of an idea I recently had: Wouldn't it be possible to automatically generate a sort of checksum for each regression-test output-file and compare it with the former releases? I know that every little change in the pdf appearance would lead to a warning then. But I'm sure there is a way to do not just md5sum regression-test-foo.pdf but create a difference score by comparing the two pdfs. Isn't this exactly what we already have (make test-baseline to create the reference version and then make test to compare the new output to the reference output...). For the results see http://lilypond.org/test/ The 2.13.3 results are at: http://lilypond.org/test/v2.13.3-0/compare-v2-13/index.html) Graham was referring to the fact that nobody seem to bother about looking at those automatically-created regression results before or after a release. Cheers, Reinhold - -- - -- Reinhold Kainhofer, reinh...@kainhofer.com, http://reinhold.kainhofer.com/ * Financial Actuarial Math., Vienna Univ. of Technology, Austria * http://www.fam.tuwien.ac.at/, DVR: 0005886 * LilyPond, Music typesetting, http://www.lilypond.org -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (GNU/Linux) iD8DBQFKiXR1TqjEwhXvPN0RAthBAKCLdUF/UdnNR/hre2Ux9TMrqG9X8QCgqBYK sbVjSztKwUjlvbNiPG/Tr4w= =hENM -END PGP SIGNATURE- ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
On Mon, Aug 17, 2009 at 05:17:08PM +0200, Reinhold Kainhofer wrote: Am Montag, 17. August 2009 16:08:36 schrieb Michael Käppler: (nobody checks the regression tests for each release, for example -- and that's trivially done with a web browser!) That reminds me of an idea I recently had: Wouldn't it be possible to automatically generate a sort of checksum for each regression-test output-file and compare it with the former releases? Isn't this exactly what we already have (make test-baseline to create the reference version and then make test to compare the new output to the reference output...). For the results see http://lilypond.org/test/ The 2.13.3 results are at: http://lilypond.org/test/v2.13.3-0/compare-v2-13/index.html) Yes. Right now there happen to be a lot, but I think there's generally around a dozen examples. Graham was referring to the fact that nobody seem to bother about looking at those automatically-created regression results before or after a release. Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. Cheers, - Graham ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
Graham Percival wrote Monday, August 17, 2009 10:35 PM On Mon, Aug 17, 2009 at 05:17:08PM +0200, Reinhold Kainhofer wrote: Am Montag, 17. August 2009 16:08:36 schrieb Michael Käppler: (nobody checks the regression tests for each release, for example -- and that's trivially done with a web browser!) That reminds me of an idea I recently had: Wouldn't it be possible to automatically generate a sort of checksum for each regression-test output-file and compare it with the former releases? Isn't this exactly what we already have (make test-baseline to create the reference version and then make test to compare the new output to the reference output...). For the results see http://lilypond.org/test/ The 2.13.3 results are at: http://lilypond.org/test/v2.13.3-0/compare-v2-13/index.html) Yes. Right now there happen to be a lot, but I think there's generally around a dozen examples. They're almost all due to the different beaming rules, and these are all changed for the better, IMHO. Graham was referring to the fact that nobody seem to bother about looking at those automatically-created regression results before or after a release. Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. I'd not looked at this before. It's very impressive! Since I usually download and install each release I'll commit to looking at it before I download and shouting if there is a serious difference. What in general is the base release used in the comparison? Is it the previous minor release or the previous stable release? Trevor Trevor ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
2009/8/17 Graham Percival gra...@percival-music.ca: Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. It probably doesn't help that several of the recent releases have broken test results: 2.12.2 only has the git head distance 2.13.0 has no previous versions to compare with 2.13.2 has no comparison against 2.13.1 2.13.3 only has results against 2.13.0 (should be 2.13.1/2) I always have a look at the test results whenever there's a new release, though any changes are usually quite familiar due to time spent running `make check'. :) Regards, Neil ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
On Mon, Aug 17, 2009 at 10:55:37PM +0100, Neil Puttock wrote: 2009/8/17 Graham Percival gra...@percival-music.ca: Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. It probably doesn't help that several of the recent releases have broken test results: 2.13.0 has no previous versions to compare with I think that's normal for a .0 release. 2.13.2 has no comparison against 2.13.1 2.13.3 only has results against 2.13.0 (should be 2.13.1/2) Oh, yet another problem with the lack of release steps. It'll probably take one or two more releases until I get it sorted out in CG 9.4. But this will be sorted out eventually. Cheers, - Graham ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
2009/8/17 Graham Percival gra...@percival-music.ca: I think that's normal for a .0 release. I assumed the same, but 2.12.0 has results against 2.11.66 (OK, that didn't officially exist since I made a mistake doing a version bump before 2.12) and 2.11.65. Oh, yet another problem with the lack of release steps. It'll probably take one or two more releases until I get it sorted out in CG 9.4. But this will be sorted out eventually. If I get the chance, I'll try to recreate the missing test results locally just in case we've missed anything important. Regards, Neil ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
On Mon, Aug 17, 2009 at 6:35 PM, Graham Percivalgra...@percival-music.ca wrote: Graham was referring to the fact that nobody seem to bother about looking at those automatically-created regression results before or after a release. Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. This is the wrong priority: this is the release manager's task, and in the ideal world, and the RM would continue the release if there are regression errors. -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
Re: Automatically checking regtests (was: Re: Minor releases?)
On Mon, Aug 17, 2009 at 7:58 PM, Han-Wen Nienhuyshanw...@gmail.com wrote: Yes. All it takes is bookmarking the site, checking it whenever there's a release, and reporting any broken examples. However, nobody is willing to commit to do this. 15 minutes whenever there's a release, which happens at most once every two weeks. This is the wrong priority: this is the release manager's task, and in the ideal world, and the RM would continue the release if there are regression errors. I mean: he would stop the release process. -- Han-Wen Nienhuys - han...@xs4all.nl - http://www.xs4all.nl/~hanwen ___ lilypond-user mailing list lilypond-user@gnu.org http://lists.gnu.org/mailman/listinfo/lilypond-user
