I would *love* a way for the cpanminus (cpanm) client to be able to use distroprefs. The #1 perl issue plaguing me at work right now is being able to control how each module in our dependency chain gets installed -- e.g. some of them need to skip tests (because of various failures), some of them need to be installed with PERL_USE_UNSAFE_INC=1, some of them need develop prereqs installed as well, etc. etc....
Also, I think that the distroprefs feature is underdocumented in general -- I've found that few people are aware of its existence. If it was used more widely, I suspect there would be better toolchain support for it. Pursuing this subject further is on my todo-list at QAH-Oslo this April. It would be awesome if others were interested as well, and we could figure out a list of action items and set a plan in motion. -ether On Sun, Jan 21, 2018 at 1:18 PM, Slaven Rezic <sla...@rezic.de> wrote: > Alceu Rodrigues de Freitas Junior via cpan-testers-discuss > <cpan-testers-discuss@perl.org> writes: > > > Hello folks, > > > > In the past I already sent this for discussion, but didn't get much > > attention. I hope I can get more feedback this time. > > > > We all create distroprefs to block distributions to run on our > > smokers. > > Well, I cannot agree on this one. My primary goal for using distroprefs > is to *enable* distributions to build and pass tests. Granted, I have > one ugly big file called "01.DISABLED.yml" for blocking distributions, > but this is mostly a short term scratchbook for misbehaving > distributions. > > To enable distributions means that most of my distroprefs files try to > set some knobs to make a distribution build & pass its tests. I do this > for distributions which I need personally (my distroprefs files are not > a smoker-only thing), or distributions which have reverse dependencies, > so I am able to smoke these, too (especially important in times of > larger upstream breakages, e.g. incompatible bleadperl changes). > > (Another tool for "enabling" is the CPAN.pm plugin > CPAN::Plugin::Sysdeps, which installs system dependencies for CPAN > distributions. Nothing fancy, this is just a hand-compiled mapping > distribution -> package names.) > > > Although we can always publish those on something like > > Github, it is not simple to keep sync with them. > > > > That's bad for several reasons: > > > > 1. We are not taking advantage of each other work of smoking test > > distributions. It took me several months of work to go over the ~37K > > distributions available on CPAN. This is a huge amount of resources > > wasted considering that we don't have a "watchdog" or something like > > to kill tests that were running for more than a hour, for instance. > > It is almost funny that for many days I left the Smoker from running > > unattended after hours just to find out the tests stopped after 15 > > minutes I left. > > I guess that some of us indeed have watchdogs. A simple one is to setup > a CPU limit (my smokers use one hour). If you're lucky and the > misbehaving distribution has a busy endless loop, then the test will be > automatically killed after an hour and a proper fail report is sent out. > It's even quiet easy to recognize this, like in this report: > http://www.cpantesters.org/cpan/report/3556e864-4490- > 11e7-a21f-d75903976bbd > Typically, in such reports CPU usage is somewhat larger than 3600s, and > exit code of one of the tests is 9 (SIGKILL). > > However, the just hanging ones without eating CPU need a real watchdog. > I have an unpublished script which looks for inactivity on the tty. If > there's no output for 200s, then the watchdog notices the process id and > possible distribution name+version, but does not do anything. Only if > the distribution is in a list of known problematic > distributions+versions, then the watchdog actually kills the test > script. This watchdog is not really ready for the public (data & code > live in the same file, and there's only support for linux, freebsd, and > windows), but if somebody is interested, I can share. > > To be killed by a watchdog means that typically a fail report is > generated, which is quite OK. This way the author has a chance to see > that something is problematic. > > As this watchdog does not really notifies me, I have another one which > checks if a smoker had created recently a report, and if there was none > for 2h (and the recent page on MetaCPAN says that there are newer > distributions available), then I get an error message on my display. > > > 2. We cannot classify easily the distroprefs (for instance, I want to > > use only distroprefs created for Linux or FreeBSD, or even for a > > specific perl version) to download and use in our smokers. > > Actually it's possible to restrict distroprefs documents to operating > systems and perl versions. You can specify match.perlconfig.osname and > match.perlconfig.version (and actually more, everything listed in > Config.pm may be matched against). > > > 3. We don't have any simple way to contact the distribution author to > > let him/her know their respective distribution is blocked for some > > reason. > > This can go the normal way: write a bug report. I made a habbit to note > the ticket number in the files I used for blocks or automatic kills. > This way I can also see if a distribution is lacking a bug report --- > sometimes the problems are many and time is sparse, and I don't write > immediately for every noticed problem a bug report. > > > 4. Kwalitee reports don't take into consideration that a distribution > > is blocked. > > No opinion on this one. > > > 5. We can't easily implement policies like blocking distributions that > > don't include any test files at all (we shouldn't have to download, > > unpack and check that there aren't any test files). > > I don't think this is a big problem --- downloaded distributions are > usually cached, so download has to be done only once, and the time to > uselessly unpack such a distribution is probably much less than the time > to write an entry in my block file. > > > 6. Not every CPAN contributor writes tests that allow execution of > > tests in parallel, or even to run in parallel with another perl > > executing the same tests at the same time. > > This would be an interesting thing to have, but I always thought that > the specification should go into the distribution's META.* files (wasn't > there already a proposal?). And CPAN.pm may inject changes into the META > information via distroprefs files (at least this is possible for > dependency information). > > > 7. Not every CPAN contributor writes tests that skips installation when > > there aren't requirements in place like a specific OS. Why would we > > want to smoke test a distribution that will only run on Win32 from a > > UNIX-like based smoker? > > Actually I have in 01.DISABLED.yml some YAML documents for blocking > distributions on alien operating systems, but these were mostly created > in the early days of this file. Nowadays most authors do a check and > emit an "OS unsupported" notice if needed. This reduces this problem to > point 5 --- the distribution is needlessly downloaded und unpacked, but > not a big problem for me. > > > > > Due my experiments with OpenBSD running a CPAN::Reporter::Smoker, I > > was able to fetch a set of 183 distroprefs and the vast majority of > > them have the following comment: > > > > comment: Tests hang smoker > > > > This "default" comment is due a limitation of the program dblock (see > > CPAN-Reporter-Smoker-OpenBSD) that I made available on CPAN. For some > > cases, I tried to add something more meaningful: > > > > GRAVATTJ.Backup-EZ.yml: > > > > comment: Requires root access to test/install > > > > But at the end I did it manually. That could be easily fixed adding a > > command line option, but that's just an example. Even if I implement > > that, the author will not get anything automatically. > > > > There are some other cases that the distribution used a huge amount of > > /tmp space > > Taking a lot of resources may or may not be a problem. Limited disk > space is a constant problem for me, and some distroprefs are indeed > blocking for this reason, but only on some of my machines --- > match.perlconfig.myuname is used as a "proxy" for identifying the > systems with limited resources. I think this is a problem: at which > point this should be considered a problem? Maybe only the amount of disk > space (in the build dir, in /tmp, in the final installation) should be > noted, and smokers may decide if blocking is necessary. > > > and other that created directories on CPAN build_dir > > parameter with permissions that didn't allowed removal after the > > tests. > > This could be fixed with another cleanup script running as root. But > yes, I also know this problem. > > > > > I'm considering starting a project to share this kind of information > > from a website with the following features: > > > > 1. Front end to allow a developer to search for anything blocked based > > on this CPAN ID. > > 2. REST interface to allow the same search (anonymous access) and > > submission of distroprefs (authenticated) in JSON format. > > 3. Send and e-mail to the author when something is received for his/her > > distributions. > > > > For that, I reviewed the distroprefs specification (from the CPAN > > module POD) and there aren't fields that I think would be required to > > include into the YAML: > > > > * The OS name and version > > Which is already there, at least as regexp matches: > match.perlconfig.osname (or archname) and match.perlconfig.osvers. > > > * The reporter ID > > Theoretically may be found via the source of the distroprefs file (i.e. > git repository). > > > * Reason (we might define some standard values) > > * perl details > > This is also possible: match.perlconfig.version, and everything else > found in Config.pm > > > * creation date of the distroprefs > > Theoretically may be found in the git log. > > > * the version of the distribution that was being tested when the > > distroprefs was created > > It's already possible to specify a distribution match with the version, > at least using a regexp. My (conservative) approach is to never use > matches without versions, unless it's clear that later distribution > releases won't change anything about the problem. > > > > > I guess I could declare multiple "comment" entries with that, but that > > would be a hack at best. > > > > Since I'm not sure if anybody is working on something like that, I'm > > hoping to get some feedback like suggestions, complains and maybe, who > > knows, some code. :-) > > As I outlined, my perspective on distroprefs is different. What would be > interesting for me is to attach some kind of "tags" to the files so > other possibly interested parties could decide if the files are usable > for them, possibly automatically. Possible tag values would be > "private" vs. "public", "fixing" vs. "blocking", maybe others... > > Also interesting would be some kind of database where other testers > observed hanging builds and tests --- this could be used for watchdog > implementations. This could be a distroprefs-like system or something > completely different. > > > Thanks! > > > > Alceu > > > > Regards, > Slaven > > -- > Slaven Rezic - slaven <at> rezic <dot> de > BBBike - route planner for cyclists in Berlin > WWW version: http://www.bbbike.de > Perl/Tk version for Unix and Windows: http://bbbike.sourceforge.net >