Re: Lilypond patchy and other Lilypond problems
On Fri, Apr 27, 2012 at 11:14:23AM +0200, David Kastrup wrote: Graham Percival gra...@percival-music.ca writes: Then we'll have hard numbers on which developers are abusing the process. I mean, sure, we all know whose patches tend to be great and whose patches tend to be problematic... but a completely automated, objective approach would remove any personal bias. And those who generated more negative karma with their work than the average horse in the stables near our house will get banished from contributing for two weeks? No; I'm expecting the Hawthorne effect to take care of it. Get real. When the cure is worse than the symptom, leave it alone. Well, that would be the question. If programmers know that there will be a record of any bad patch submissions, would they be less likely to contribute? Or would they be more likely to check their work before submitting it? I'm obviously hoping for the latter, but I suppose that the former is still a logical possibility. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Graham Percival gra...@percival-music.ca writes: On Fri, Apr 27, 2012 at 11:14:23AM +0200, David Kastrup wrote: Graham Percival gra...@percival-music.ca writes: Then we'll have hard numbers on which developers are abusing the process. I mean, sure, we all know whose patches tend to be great and whose patches tend to be problematic... but a completely automated, objective approach would remove any personal bias. And those who generated more negative karma with their work than the average horse in the stables near our house will get banished from contributing for two weeks? No; I'm expecting the Hawthorne effect to take care of it. Get real. When the cure is worse than the symptom, leave it alone. Well, that would be the question. If programmers know that there will be a record of any bad patch submissions, would they be less likely to contribute? Or would they be more likely to check their work before submitting it? I'm obviously hoping for the latter, but I suppose that the former is still a logical possibility. It's small fry. The really bad things are those that are prodded until they pass the tests rather than the committer's level of understanding. And those would create positive Karma points. In fact, if you have to do half a dozen of iterations before getting things actually right on the somewhat more than superficial level provided by our tests, you'll have gained lots of good Karma on the road. We need more human feedback. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
On Sat, Apr 28, 2012 at 10:13:50AM +0200, David Kastrup wrote: In fact, if you have to do half a dozen of iterations before getting things actually right on the somewhat more than superficial level provided by our tests, you'll have gained lots of good Karma on the road. I was thinking that each patch would generate its own karam, so each of those uploads that failed the tests would be -2 karma (or whatever the figure is), with only the final upload giving a +1. It's true that multiple rounds of patch, reviewer comment, patch, reviewer comment could potentially generate a karma windfall... but hey, since it's not like people can buy anything with karma, and as long as there's multiple rounds of not-automatic-failing patches, I'm fine with that. We need more human feedback. All my years of observing lilypond development suggests that saying we need xyz is a recipe for nothing happening. But hey, I'm not totally wedded to the idea of tracking karma, so meh. As long as I'm not personally playing nursemaid for people who don't run the basic tests, I don't mind if the patch-tester(s) want to keep on warning about flawed patches. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Graham Percival gra...@percival-music.ca writes: On Sat, Apr 28, 2012 at 10:13:50AM +0200, David Kastrup wrote: In fact, if you have to do half a dozen of iterations before getting things actually right on the somewhat more than superficial level provided by our tests, you'll have gained lots of good Karma on the road. I was thinking that each patch would generate its own karam, so each of those uploads that failed the tests would be -2 karma (or whatever the figure is), with only the final upload giving a +1. The point was patches that did not fail the upload, because in spite of being horribly broken, they were massaged long enough to past the test suite. We need more human feedback. All my years of observing lilypond development suggests that saying we need xyz is a recipe for nothing happening. I prefer nothing happening over the wrong things happening. Call me conservative. But hey, I'm not totally wedded to the idea of tracking karma, so meh. As long as I'm not personally playing nursemaid for people who don't run the basic tests, I don't mind if the patch-tester(s) want to keep on warning about flawed patches. We have very little nursemaid material here, I am afraid. Do you really think that somebody who misses registering the responses on his issue report for whatever reason (which caused the last temper drop by me) will get phased by an arbitrary _number_ being displayed on the issue page, based on some accumulating metric? And will people be more motivated if they figure that their own Karma life savings are minuscule compared to Mike's daily fluctuations? -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
On 28 April 2012 10:30, Graham Percival gra...@percival-music.ca wrote: As long as I'm not personally playing nursemaid for people who don't run the basic tests It seems that the whole talk about running tests is only because of me not running them before uploading my first patches because I didn't know that I should do it. Do you have any other examples of committers that did not run tests? I guess that you don't have. Łukasz ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
On 26 April 2012 20:41, Łukasz Czerwiński milimet...@gmail.com wrote: Mike, Graham and David wrote about more or less automatic running of tests and presenting only the results, possibly on an unused computer. I realised that I have a server on Dreamhost that probably could be such a computer - there is unlimited disk space and unlimited bandwidth (to some extend, I guess, but that will be enough for us). Now I'm trying to compile Lilypond on it - there are some libraries missing, I'm in progress of figuring out whether I can install it locally (it's a shared server, not a private one, so I don't have root on it). If yes and lilypond compiles, we could automatically pull git repo, run tests on it, pack the results and send an email with a link to them. There is Apache, PHP, MySQL there, so if you would like to do a website to present results directly on the server, it's possible :) I will finish installing Lilypond on a Dreamhost server soon, but because it's a shared server, I have some problems with overusing its resources. I've contacted Technical support and there are willing to help, e.g. giving me bigger limits or sth, BUT I must know one thing: How much CPU time and memory would regtests consume? How many times a day will they be run? Łukasz ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Hello, On 28 April 2012 23:18, Łukasz Czerwiński milimet...@gmail.com wrote: On 28 April 2012 10:30, Graham Percival gra...@percival-music.ca wrote: As long as I'm not personally playing nursemaid for people who don't run the basic tests It seems that the whole talk about running tests is only because of me not running them before uploading my first patches because I didn't know that I should do it. Do you have any other examples of committers that did not run tests? I guess that you don't have. Oh yes there have been lots. I know because I used to apply patches and run the tests manually. The most common problems were devs 'forget' they have a 'dirty' branch and upload their patches based on their own branches not master. So the patch doesn't apply. http://code.google.com/p/lilypond/issues/detail?id=1953 (comment #1) This can cause hunk failures or make problems (where someone has changed a file but not added it to their commit). This doesn't mean they didn't run tests, necessarily, but it means they didn't run tests against current master. James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Hello, How much CPU time and memory would regtests consume? Depends on how many CPUs you can allocate. You can run reg tests on 1 CPU or xCPUs. You explicitly state the number in your make command 'make -j7 CPU_COUNT=7' test will use 7 CPUs 'make test' just uses 1 CPU (even if you have more than 1) 7 CPUs on my desktop are at about 90-100% for approximately 10 minutes for a reg test. RAM doesn't go much above 1.5GB. 1 CPU will be about 100% for 30 minutes (maybe more - depends on the speed of the CPU). Make doc takes a lot longer. How many times a day will they be run? You can run them as many times as you like. You need one reg test per patch-new tracker item. James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Łukasz Czerwiński milimet...@gmail.com writes: On 28 April 2012 10:30, Graham Percival gra...@percival-music.ca wrote: As long as I'm not personally playing nursemaid for people who don't run the basic tests It seems that the whole talk about running tests is only because of me not running them before uploading my first patches because I didn't know that I should do it. More like uploading patches with known problems. The notehead merging code would have given wrong results with pretty much every test merging noteheads. It's not like you missed out on running the regressions test suite: you did not apparently run any file relevant to the patch, and ignored the reviews repeatedly. Do you have any other examples of committers that did not run tests? I guess that you don't have. Plenty. Overall, the discipline has increased a lot in recent months, but partly because the procedures have been streamlined, partly because I was the one running the tests and tend to be somewhat less linear in my response than a karma point system. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
- Original Message - From: Łukasz Czerwiński To: James Cc: m...@apollinemike.com ; k-ohara5...@oco.net ; David Kastrup ; lilypond-devel@gnu.org Sent: Thursday, April 26, 2012 7:41 PM Subject: Lilypond patchy and other Lilypond problems Still requires 'someone' to 'do' something and then say 'LGTM' and I don't know what the feed back has been with regard to the http://lilypond.org/doc/v2.15/Documentation/contributor/grand-regression-test-checking.html is this just not the same thing in essence? Woow, what's that:Â http://www.philholmes.net/lilypond/regtests/Â for? Is it for rating regression tests or for rating that particular result of a particular test run? Ĺukasz OK. Regtest checking can mean three things. 1) Between releases, comparing the output of the regtests with what they looked like before and flagging any differences as potential problems. I do this. 2) Doing the same thing for patches. Patchy does this, helped by a human, currently James and was David. 3) Checking that the regtests themselves are OK - well described and functional. The grand regtest checking project does this, hosted on my server and using my web code. We're doing well. If you log in and check a few, you can see the statistics. -- Phil Holmes ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Phil Holmes m...@philholmes.net writes: OK. Regtest checking can mean three things. 1) Between releases, comparing the output of the regtests with what they looked like before and flagging any differences as potential problems. I do this. 2) Doing the same thing for patches. Patchy does this, helped by a human, currently James and was David. There is no reason whatsoever that this should only be done by a single person. I would expect that _every_ person contributing more than two patches per month should be able, after uploading a patch, to be running test-patches.py and thus emptying the queue. This has the consequences that a) his own submission will get timely and qualified testing b) the queue does not build up c) it is not always the same people who get stuck with testing d) he can be more nonchalant about testing his submission in advance since a bad test upload mostly implies more work for himself, and then not all that much. -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
On Fri, Apr 27, 2012 at 10:28:54AM +0200, David Kastrup wrote: There is no reason whatsoever that this should only be done by a single person. Totally! I would expect that _every_ person contributing more than two patches per month should be able, after uploading a patch, to be running test-patches.py and thus emptying the queue. Could do, although I'd rather if somebody other than the original submitter gave the ok for a patch. That way there's a fresh pair of eyes looking at any differences, which would ideally make the original submitters be more clear in the commit message about any intentional changes to the regtests. b) the queue does not build up c) it is not always the same people who get stuck with testing d) he can be more nonchalant about testing his submission in advance since a bad test upload mostly implies more work for himself, and then not all that much. I really like those possibilities, though. I still think we should track Patchy responses, though. I mean, have a completely automated system which tracks your karma. For each patch, - fails to apply to master: -10 karma (with an option to cancel this penalty if master was updated after somebody submitted their patch) - fails to compile: -5 karma - has unintended regtest differences: -3 karma - has un-notified regtest differences which are accepted as ok after some discusion: -1 karma. (yes, we want to penalize people for not mentioning those differences up-front in the git commit message!) - passes test without problems: +1 karma Then we'll have hard numbers on which developers are abusing the process. I mean, sure, we all know whose patches tend to be great and whose patches tend to be problematic... but a completely automated, objective approach would remove any personal bias. - Graham ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Graham Percival gra...@percival-music.ca writes: Then we'll have hard numbers on which developers are abusing the process. I mean, sure, we all know whose patches tend to be great and whose patches tend to be problematic... but a completely automated, objective approach would remove any personal bias. And those who generated more negative karma with their work than the average horse in the stables near our house will get banished from contributing for two weeks? Get real. When the cure is worse than the symptom, leave it alone. There is an old adage for programmers: don't check for errors for which there is no sensible means of treatment. if (1 == 0) cerr Your CPU or compiler may be broken. endl; -- David Kastrup ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Lilypond patchy and other Lilypond problems
Wooow, a lot of emails were posted in the last 24 hours :) I'll try to comment all your important thoughts, but it's possible that I miss one or two... Anyway: On 26 April 2012 07:28, Graham Percival gra...@percival-music.ca wrote: Some people encourage new contributors. I encourage new contributors who want to work on administrative tasks. I try to discourage new programmers, precisely because they almost always end up in situations like yours. My situation isn't bad - it's more or less ok. I had had some minor problems and a little argue with David, because I didn't know that patchy is run for each patch set. Now I know it, my patch LGT patchy and we have a new patchy runner - James. I can continue on developing and patches will continue to be assessed. That's great, don't you think? :) Last January, I warned him that he should not try to recruit any new programmers unless he was willing to mentor them because it is very difficult for new programmers to get started. I think you have seen that my prediction was correct. I'm not an experienced developer, but I have a slight feeling that you are a bit exaggerating. If I'm wrong, correct me :) On 26 April 2012 08:55, m...@apollinemike.com m...@apollinemike.com wrote: I have a meeting in mid-May w/ the University of Paris VIII. They're donating a computer to LilyPond and I'll set patchy up on it. Wow, that's really, really nice. Why do they do so? Maybe my university could make a similar donation...? On 26 April 2012 09:05, James pkx1...@gmail.com wrote: I've run patchy-test just now for the three patches outstanding this morning. It's no a big deal, I've just never got round to running the patchy-test scripts (well since the scripts were very first created when I had trouble understanding them), so don't worry about patches David now, I'll pick up the slack here. Thanks, James! That's really great! :) Mike, Graham and David wrote about more or less automatic running of tests and presenting only the results, possibly on an unused computer. I realised that I have a server on Dreamhost that probably could be such a computer - there is unlimited disk space and unlimited bandwidth (to some extend, I guess, but that will be enough for us). Now I'm trying to compile Lilypond on it - there are some libraries missing, I'm in progress of figuring out whether I can install it locally (it's a shared server, not a private one, so I don't have root on it). If yes and lilypond compiles, we could automatically pull git repo, run tests on it, pack the results and send an email with a link to them. There is Apache, PHP, MySQL there, so if you would like to do a website to present results directly on the server, it's possible :) By now I'm trying to run successfully ./configure for guile - it requires some additional libraries, which require some other etc. On 26 April 2012 11:43, James pkx1...@gmail.com wrote: Still requires 'someone' to 'do' something and then say 'LGTM' and I don't know what the feed back has been with regard to the http://lilypond.org/doc/v2.15/Documentation/contributor/grand-regression-test-checking.html is this just not the same thing in essence? Woow, what's that: http://www.philholmes.net/lilypond/regtests/ for? Is it for rating regression tests or for rating that particular result of a particular test run? Łukasz ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Hello, 2012/4/26 Łukasz Czerwiński milimet...@gmail.com: ... On 26 April 2012 09:05, James pkx1...@gmail.com wrote: I've run patchy-test just now for the three patches outstanding this morning. It's no a big deal, I've just never got round to running the patchy-test scripts (well since the scripts were very first created when I had trouble understanding them), so don't worry about patches David now, I'll pick up the slack here. Thanks, James! That's really great! :) No problem, but it doesn't mean that you can just do some code and throw it up for review without ANY basic testing your side, it should apply to current tree and it should also pass a basic 'make'. I am sure you are aware of that courtesy. It is true that Patchy does this also and so catches the basic mistakes, but patchy's main goal is to take away the more intensive 'make test' from developers which can take a long time on less powerful machines. Also patchy testers are not (usually) programmers - like me for instance - so I'm not going to go into much detail; if patchy shows up regressions then that is easy to spot - that is a web page for me to see, but more subtle 'make' problems or 'patch apply' problems Like this this morning - see comment #1 http://code.google.com/p/lilypond/issues/detail?id=2498 are going to get a 'failed - patch doesn't apply' or 'failed to make' messages and not much else. Hence the need for some basic housekeeping on your side. Mike, Graham and David wrote about more or less automatic running of tests and presenting only the results, possibly on an unused computer. I realised that I have a server on Dreamhost that probably could be such a computer - there is unlimited disk space and unlimited bandwidth (to some extend, I guess, but that will be enough for us). Now I'm trying to compile Lilypond on it - there are some libraries missing, I'm in progress of figuring out whether I can install it locally (it's a shared server, not a private one, so I don't have root on it). Did you look at LilyDev? This is specifically aimed at LilyPond developers who don't have the time or inclination to set up their dev build. It's got pretty much all you need right there and yuo can be up and running in a few minutes (once you have it installed). LilyDev is a pre-built Ubuntu dist with all the dependencies. I run it in a VM (I use KVM at home but Virtual Box at work). It might be simpler. See: http://lilypond.org/doc/v2.15/Documentation/contributor/installing-lilydev The instructions have been updates significantly but I am sure you can understand how to install an OS using an ISO file. On 26 April 2012 11:43, James pkx1...@gmail.com wrote: Still requires 'someone' to 'do' something and then say 'LGTM' and I don't know what the feed back has been with regard to the http://lilypond.org/doc/v2.15/Documentation/contributor/grand-regression-test-checking.html is this just not the same thing in essence? Woow, what's that: http://www.philholmes.net/lilypond/regtests/ for? Is it for rating regression tests or for rating that particular result of a particular test run? Phil does a pixel comparison reg test between *releases* i.e. http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00078.html and has some programming experience so this is an offshoot of what he does with the project anyway and he offered this as a service, I am sure he will fill you in (I cannot find he original email I think he sent out). James ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Hello, On 26 April 2012 21:38, James pkx1...@gmail.com wrote: No problem, but it doesn't mean that you can just do some code and throw it up for review without ANY basic testing your side, it should apply to current tree and it should also pass a basic 'make'. Yes, before uploading a patch I make sure that it will apply and compile. Now I also know that I should be aware that patchy is run on each patch set, so I must check the result of it before uploading the next patch set. Mike, Graham and David wrote about more or less automatic running of tests and presenting only the results, possibly on an unused computer. I realised that I have a server on Dreamhost that probably could be such a computer - there is unlimited disk space and unlimited bandwidth (to some extend, I guess, but that will be enough for us). Now I'm trying to compile Lilypond on it - there are some libraries missing, I'm in progress of figuring out whether I can install it locally (it's a shared server, not a private one, so I don't have root on it). Did you look at LilyDev? This is specifically aimed at LilyPond developers who don't have the time or inclination to set up their dev build. It's got pretty much all you need right there and yuo can be up and running in a few minutes (once you have it installed). LilyDev is a pre-built Ubuntu dist with all the dependencies. I run it in a VM (I use KVM at home but Virtual Box at work). It might be simpler. See: http://lilypond.org/doc/v2.15/Documentation/contributor/installing-lilydev The instructions have been updates significantly but I am sure you can understand how to install an OS using an ISO file. Well, LilyDev won't help me - on the server exists an already installed system (Linux). As for Virtualbox, I believe, that without having admin rights I can't install it - correct me if I'm wrong. On 26 April 2012 11:43, James pkx1...@gmail.com wrote: Still requires 'someone' to 'do' something and then say 'LGTM' and I don't know what the feed back has been with regard to the http://lilypond.org/doc/v2.15/Documentation/contributor/grand-regression-test-checking.html is this just not the same thing in essence? Woow, what's that: http://www.philholmes.net/lilypond/regtests/ for? Is it for rating regression tests or for rating that particular result of a particular test run? Phil does a pixel comparison reg test between *releases* i.e. http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00078.html and has some programming experience so this is an offshoot of what he does with the project anyway and he offered this as a service, I am sure he will fill you in (I cannot find he original email I think he sent out). Do you mean Phil Holmes? Łukasz ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
Re: Lilypond patchy and other Lilypond problems
Hello, 2012/4/26 Łukasz Czerwiński milimet...@gmail.com: Well, LilyDev won't help me - on the server exists an already installed system (Linux). As for Virtualbox, I believe, that without having admin rights I can't install it - correct me if I'm wrong. Well I'm not a *NIX admin - I do have to use 'sudo' to install the program but not run it and I do need to add a vboxuser to allow USB support (and to stop the annoying warning even if I never use the USB support). There are lots of other virtualization platforms out there, I'd ask at your 'institution', if you don't already know, as from my own day-to-day working experience a lot of these places use Virtual Environments because it is so easy now to set up, XCP, Cirix XENserver, KVM, VMware ESX, Windows Hyper-V etc. In fact in many of these places set up a VM in preference to a /home dir because it is now so easy to manage a VM is just a 'big file' (no different in essence to a /home 'directory') - but I digress. Phil does a pixel comparison reg test between *releases* i.e. http://lists.gnu.org/archive/html/lilypond-devel/2010-11/msg00078.html and has some programming experience so this is an offshoot of what he does with the project anyway and he offered this as a service, I am sure he will fill you in (I cannot find he original email I think he sent out). Do you mean Phil Holmes? Yes. ___ lilypond-devel mailing list lilypond-devel@gnu.org https://lists.gnu.org/mailman/listinfo/lilypond-devel
