Hi Kenneth, The time I spent on it was not wasted, it was a good learning experience.
I had a look at the pull request for the easyblock, this is almost identical to my version (apart from somewhat different no_qa). My easyblock is also somewhat simpler since I set a number of options via the easyconfig, rather than in the easyblock. I have a slight problem with the easyconfigs though. Qt has a dependency on GLib, which is not installed on our compute nodes. I've an easyconfig for GLib, I've done a push to our fork. My easyconfig for Qt depends on it. I haven't pushed the Qt stuff yet, since I'd prefer to test whether it works properly. Something that I also modified are the easyconfigs for icc, ifort, impi and mkl. The current easyconfigs assume that the license file will be present in a directory in easybuild's home directory. I've added a test to check for INTEL_LICENSE_FILE, if set, it uses that, if not, it reverts to standard behavior. This allows for us to have our license files elsewhere, which we prefer. This too has been pushed to our fork. That might be useful for others as well. Best regards, -gjb- On Thu, Jun 27, 2013 at 10:54 PM, Kenneth Hoste <[email protected]>wrote: > Hi Geert-Jan, > > On 26 Jun 2013, at 13:50, Geert Jan BEX wrote: > > > Hi Kenneth, > > > > Thanks for the feedback. > > > > Meanwhile I got the run_cmd_qa problem solved. I needed an easyblock > because the configuration step prompts to agree with the license. I could > have sidestepped the issue by patching that out of the configuration > script, but that seems a bit unethical ;) The package is Qt, and it is a > prerequisite for UNIFY. > > There's already a pull request open for a Qt easyblock, see > https://github.com/hpcugent/easybuild-easyblocks/pull/210 (and > https://github.com/hpcugent/easybuild-easyconfigs/pull/294 for > easyconfigs with the goolf and ictce compiler toolchains). > > It's unfortunate that you were also spending time on it, but maybe you can > take that experience, take a look at the pull request and see if you have > any suggestions or comments. > > We like to have at least two pairs of eyes going over PRs (the one who > implemented it, and someone else), before we merge stuff in... > > > regards, > > Kenneth > > > > > > Adding the basic stuff to toolchains seems like a good idea. > > > > Best regards, -gjb- > > > > > > > > > > On Tue, Jun 25, 2013 at 6:56 PM, Kenneth Hoste <[email protected]> > wrote: > > Hi Geert-Jan, > > > > On 25 Jun 2013, at 16:51, Geert Jan BEX wrote: > > > > > Hi Easybuilders, > > > > > > A couple of questions: > > > • I'm running into trouble using run_cmd_qa in a custom > easyblock. The command's output is pretty huge, and after a while, the > maximum hitCount is exceeded. As I understand the inner workings of > run_cmd_qa, it keeps on reading bits of stdout/stderr from the command it > is executing, and it tries to match with the questions, answering them when > found. Failing that, it will check for matches in the no_qa list, and when > finding one, it resets hitCount, incrementing it otherwise. > > > So if I get this right, I've to provide sufficient no_qa elements so > that the whole of the output of the command is "covered", i.e., making sure > that there is a no_qa match very n lines? > > > > > > Your view on it is close to correct, but not fully. > > The hit count is only increased if the output remains the same and no > question could be matched against the end of the output. > > > > As soon as the output changes, the hitCount is reset (if not, it's a > bug), because progress was detected without needing to provide an answer. > > So, if some operation is taking a while, run_cmd_qa assumes it's a > question (i.e., the command is waiting for input), and will try to match a > question to the output. If it can't, it will die assuming something is > wrong. > > > > So, basically, yes, you'll have to try and come up with no_qa entries > (which can be regex patterns b.t.w.) that match any operation that is > printed at the end of the output for a while, but is not an actual question > (e.g. a compile or copy command). Up until now, we've never run into a > procedure in which this proves impractical (i.e. usually the no_qa list can > be kept quite small). > > > > Can you share an example of what you're running into? > > > > Also, out of curiosity: what is the custom easyblock for? > > > > > > > • There are a number of modules that are really basic, e.g., > git, CMake... Perhaps it would make sense to define a very minimal > toolchain (even one that simply uses the resident gcc) for these kind of > modules? That would allow to really minimize dependencies. Just a > question, maybe it is a dumb idea. > > > > That's one way, yes, but then you're depending on the system-provided > tools again, which is one of the things we want to avoid. > > Encouraging to provide modules for all dependencies makes > reproducibility more likely, both across different systems/OSs, and on the > same system with one or multiple minor/major OS updates in between. > > > > Relying on whatever (version of) tools the system provides is too likely > to change, which may result into builds that suddenly break (we've all seen > that). > > > > That being said, more stuff should be part of compiler toolchains, yes. > There is a ticket open for exactly that. Because tools/libs like zlib, > ncurses, binutils (ld, ar, ...) are (very) commonly dependencies, they > should be (and will be) part of the compiler toolchains we define. See > https://github.com/hpcugent/easybuild-framework/issues/644 . Maybe also > tar/gzip should be included. > > > > > > regards, > > > > Kenneth > > > > > > > > > > -- > > - Geert Jan BEX > > - HPC analyst/consultant Hasselt University & University of Leuven > > - email: [email protected] > > - homepage: http://alpha.uhasselt.be/~gjb/ > > - PGP public key: http://alpha.uhasselt.be/~gjb/pgp.txt > > --------------------------------------------------------- > > -- - Geert Jan BEX - HPC analyst/consultant Hasselt University & University of Leuven - email: [email protected] - homepage: http://alpha.uhasselt.be/~gjb/ - PGP public key: http://alpha.uhasselt.be/~gjb/pgp.txt ---------------------------------------------------------
