Alceu Rodrigues de Freitas Junior via cpan-testers-discuss <cpan-testers-discuss@perl.org> writes:
> Hello folks, > > In the past I already sent this for discussion, but didn't get much > attention. I hope I can get more feedback this time. > > We all create distroprefs to block distributions to run on our > smokers. Well, I cannot agree on this one. My primary goal for using distroprefs is to *enable* distributions to build and pass tests. Granted, I have one ugly big file called "01.DISABLED.yml" for blocking distributions, but this is mostly a short term scratchbook for misbehaving distributions. To enable distributions means that most of my distroprefs files try to set some knobs to make a distribution build & pass its tests. I do this for distributions which I need personally (my distroprefs files are not a smoker-only thing), or distributions which have reverse dependencies, so I am able to smoke these, too (especially important in times of larger upstream breakages, e.g. incompatible bleadperl changes). (Another tool for "enabling" is the CPAN.pm plugin CPAN::Plugin::Sysdeps, which installs system dependencies for CPAN distributions. Nothing fancy, this is just a hand-compiled mapping distribution -> package names.) > Although we can always publish those on something like > Github, it is not simple to keep sync with them. > > That's bad for several reasons: > > 1. We are not taking advantage of each other work of smoking test > distributions. It took me several months of work to go over the ~37K > distributions available on CPAN. This is a huge amount of resources > wasted considering that we don't have a "watchdog" or something like > to kill tests that were running for more than a hour, for instance. > It is almost funny that for many days I left the Smoker from running > unattended after hours just to find out the tests stopped after 15 > minutes I left. I guess that some of us indeed have watchdogs. A simple one is to setup a CPU limit (my smokers use one hour). If you're lucky and the misbehaving distribution has a busy endless loop, then the test will be automatically killed after an hour and a proper fail report is sent out. It's even quiet easy to recognize this, like in this report: http://www.cpantesters.org/cpan/report/3556e864-4490-11e7-a21f-d75903976bbd Typically, in such reports CPU usage is somewhat larger than 3600s, and exit code of one of the tests is 9 (SIGKILL). However, the just hanging ones without eating CPU need a real watchdog. I have an unpublished script which looks for inactivity on the tty. If there's no output for 200s, then the watchdog notices the process id and possible distribution name+version, but does not do anything. Only if the distribution is in a list of known problematic distributions+versions, then the watchdog actually kills the test script. This watchdog is not really ready for the public (data & code live in the same file, and there's only support for linux, freebsd, and windows), but if somebody is interested, I can share. To be killed by a watchdog means that typically a fail report is generated, which is quite OK. This way the author has a chance to see that something is problematic. As this watchdog does not really notifies me, I have another one which checks if a smoker had created recently a report, and if there was none for 2h (and the recent page on MetaCPAN says that there are newer distributions available), then I get an error message on my display. > 2. We cannot classify easily the distroprefs (for instance, I want to > use only distroprefs created for Linux or FreeBSD, or even for a > specific perl version) to download and use in our smokers. Actually it's possible to restrict distroprefs documents to operating systems and perl versions. You can specify match.perlconfig.osname and match.perlconfig.version (and actually more, everything listed in Config.pm may be matched against). > 3. We don't have any simple way to contact the distribution author to > let him/her know their respective distribution is blocked for some > reason. This can go the normal way: write a bug report. I made a habbit to note the ticket number in the files I used for blocks or automatic kills. This way I can also see if a distribution is lacking a bug report --- sometimes the problems are many and time is sparse, and I don't write immediately for every noticed problem a bug report. > 4. Kwalitee reports don't take into consideration that a distribution > is blocked. No opinion on this one. > 5. We can't easily implement policies like blocking distributions that > don't include any test files at all (we shouldn't have to download, > unpack and check that there aren't any test files). I don't think this is a big problem --- downloaded distributions are usually cached, so download has to be done only once, and the time to uselessly unpack such a distribution is probably much less than the time to write an entry in my block file. > 6. Not every CPAN contributor writes tests that allow execution of > tests in parallel, or even to run in parallel with another perl > executing the same tests at the same time. This would be an interesting thing to have, but I always thought that the specification should go into the distribution's META.* files (wasn't there already a proposal?). And CPAN.pm may inject changes into the META information via distroprefs files (at least this is possible for dependency information). > 7. Not every CPAN contributor writes tests that skips installation when > there aren't requirements in place like a specific OS. Why would we > want to smoke test a distribution that will only run on Win32 from a > UNIX-like based smoker? Actually I have in 01.DISABLED.yml some YAML documents for blocking distributions on alien operating systems, but these were mostly created in the early days of this file. Nowadays most authors do a check and emit an "OS unsupported" notice if needed. This reduces this problem to point 5 --- the distribution is needlessly downloaded und unpacked, but not a big problem for me. > > Due my experiments with OpenBSD running a CPAN::Reporter::Smoker, I > was able to fetch a set of 183 distroprefs and the vast majority of > them have the following comment: > > comment: Tests hang smoker > > This "default" comment is due a limitation of the program dblock (see > CPAN-Reporter-Smoker-OpenBSD) that I made available on CPAN. For some > cases, I tried to add something more meaningful: > > GRAVATTJ.Backup-EZ.yml: > > comment: Requires root access to test/install > > But at the end I did it manually. That could be easily fixed adding a > command line option, but that's just an example. Even if I implement > that, the author will not get anything automatically. > > There are some other cases that the distribution used a huge amount of > /tmp space Taking a lot of resources may or may not be a problem. Limited disk space is a constant problem for me, and some distroprefs are indeed blocking for this reason, but only on some of my machines --- match.perlconfig.myuname is used as a "proxy" for identifying the systems with limited resources. I think this is a problem: at which point this should be considered a problem? Maybe only the amount of disk space (in the build dir, in /tmp, in the final installation) should be noted, and smokers may decide if blocking is necessary. > and other that created directories on CPAN build_dir > parameter with permissions that didn't allowed removal after the > tests. This could be fixed with another cleanup script running as root. But yes, I also know this problem. > > I'm considering starting a project to share this kind of information > from a website with the following features: > > 1. Front end to allow a developer to search for anything blocked based > on this CPAN ID. > 2. REST interface to allow the same search (anonymous access) and > submission of distroprefs (authenticated) in JSON format. > 3. Send and e-mail to the author when something is received for his/her > distributions. > > For that, I reviewed the distroprefs specification (from the CPAN > module POD) and there aren't fields that I think would be required to > include into the YAML: > > * The OS name and version Which is already there, at least as regexp matches: match.perlconfig.osname (or archname) and match.perlconfig.osvers. > * The reporter ID Theoretically may be found via the source of the distroprefs file (i.e. git repository). > * Reason (we might define some standard values) > * perl details This is also possible: match.perlconfig.version, and everything else found in Config.pm > * creation date of the distroprefs Theoretically may be found in the git log. > * the version of the distribution that was being tested when the > distroprefs was created It's already possible to specify a distribution match with the version, at least using a regexp. My (conservative) approach is to never use matches without versions, unless it's clear that later distribution releases won't change anything about the problem. > > I guess I could declare multiple "comment" entries with that, but that > would be a hack at best. > > Since I'm not sure if anybody is working on something like that, I'm > hoping to get some feedback like suggestions, complains and maybe, who > knows, some code. :-) As I outlined, my perspective on distroprefs is different. What would be interesting for me is to attach some kind of "tags" to the files so other possibly interested parties could decide if the files are usable for them, possibly automatically. Possible tag values would be "private" vs. "public", "fixing" vs. "blocking", maybe others... Also interesting would be some kind of database where other testers observed hanging builds and tests --- this could be used for watchdog implementations. This could be a distroprefs-like system or something completely different. > Thanks! > > Alceu > Regards, Slaven -- Slaven Rezic - slaven <at> rezic <dot> de BBBike - route planner for cyclists in Berlin WWW version: http://www.bbbike.de Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net
