Hi Bart, On 02 Oct 2013, at 02:58, Bart Verleye wrote:
> Hi Kenneth, > > I will certainly pull everything I did...after some cleanup here. Don't worry about opening a pull request too soon. It's OK if you still need to do some cleanup, because the PR will be reviewed anyway, and it's not unlikely someone will have some remarks here and there. Also: by opening a (potentially incomplete) PR early, you avoid that other people are duplicated the work you're doing. If someone else is interested in ParaView, they'll hopefully notice someone else is already on it. That's a lot harder if you wait until the very last minute to open the PR... We have a "two pairs of eyes minimum" review policy, i.e. every PR has to be reviewed by someone other than the person opening it. The same goes for all stuff we do as well: we never merge something that hasn't been reviewed by someone else. This *really* helps in keeping everything nice and clean, sensible, etc. See https://github.com/hpcugent/easybuild/wiki/Contributing-back for more details. > For example, the toolchain version is probably not what it should be. Can you > point me to some explanation on how I should version it? It's better to > commit the file with the correct version name. Don't worry about opening a PR with a 'weird' version for the toolchain. Like I mentioned, the versioning scheme for toolchains is very ad hoc, simply because there is no way in which you can summarize multiple dependency versions into a single reasonable version. When we introduced the goolf toolchain, we applied the following reasoning to come up with the toolchain version: - 1.x, because we considered the toolchain ready for 'production' use (so no 0.x) - 1.4.x, with the '4' being picked because it matches the '4' in GCC 4.x and such that there's room for downgrading (e.g., using GCC 3.x, or a lower major version of any of the dependencies) - 1.4.10, with the '10' being picked such that several dependencies can have minor downgrades if needed, leaving room for combinations of minor downgrades (e.g. GCC 4.6.x or 4.5.x, OpenMPI 1.5.x, ...) The '1.1.0' toolchain version of goalf (our first non-ictce toolchain) doesn't match this scheme, and should be treated as the exception that confirm the rule. ;-) For the ictce versions, there's a long history. Basically, the major number is determined by a new major release of the compilers, and the lower numbers are bumped as updates are released. Suggesting a gmpolf version that matches this scheme requires the versions of the dependencies, but should stay close to the versioning we currently have for the cgmpolf toolchain, since only Clang is taking out... > The easyblock for Qt tests on libQtCore to be present. However, in Qt 5, this > will be libQt5Core. What is the way to go then? Delete the test from the qt > easyblock, or make a new block and directory for qt5 (i.e. treat it as > another program, rather than another version)? No, a version check should be added in the Qt easyblock that makes the sanity check for libQt5Core instead (look for uses of LooseVersion in existing easyblocks). Copy-pasting a Qt5 easyblock and only changing the libQt5Core things is overhead, and leads to too much duplicated code (even when deriving it from the existing Qt easyblock), which is a nightmare to maintain. > Let's try to set up a skype next week? My skype is bart.verleye. OK, let's discuss this off-list. regards, Kenneth > > Cheers, > Bart > > On 02/10/13 10:40, Kenneth Hoste wrote: >> Hi Bart, >> >> On 01 Oct 2013, at 22:48, Fotis Georgatos wrote: >> >>> Hi Bart, >>> >>> On Oct 1, 2013, at 10:31 PM, Bart Verleye wrote: >>>> I have installed a new toolchain, gmpolf-3.0.4. >> We would *love* to see a pull request being created for this, i.e. the new >> toolchain definition to the easybuild-framework repo, the easyconfig files >> for the actual toolchain version to the easybuild-easyconfigs repo. >> >> Are you up for that? >> >> The version number looks a bit odd to me, but I can't really tell since I >> have no view on the versions of the toolchain elements. >> >> And, I have to admit, the way we version toolchains is very ad >> hoc/unclear/undocumented, so I'm not blaming you. ;-) >> >> >>>> Now I want to install Paraview, which depends on quite a few other >>>> packages. >>> To save a few keystrokes, you can find pre-calculated templates for many >>> of the common building blocks over here (shameless rip from pkgsrc's >>> Makefiles): >>> https://github.com/fgeorgatos/easybuild.experimental/tree/master/contrib/pkgsrc/20130506 >>> (sorry to the rest for being repetitive, but you know how convenient this >>> can be...) >>> >> Paraview is new, i.e. we don't have support for that yet. >> So, if you end up getting it to work, again, please submit a pull request to >> include that support in the next EasyBuild release, >> to avoid that others lose time on the same problems you're solving (which is >> what EasyBuild is all about, cooperation!). >> >> >>>> During the installation, I came up with following comments: >>>> >>>> - --try-toolchain is not recursive. Is that 'not yet', or is there a >>>> reason behind that? >>> "Not yet", there is an open issue somewhere about it, AFAIK. >> It's "not yet", indeed. I agree this is a very desirable feature, but >> there's also a 'but'. >> >> First, let me try and explain why it's not there yet... Although you would >> expect this to be a feature that can be easily implemented, it in fact >> requires quite a bit of work. >> I started working on it once, and ended up spending a couple of hours on it, >> without a decent result (i.e., a huge ugly patch, and it didn't even work >> yet). >> It seems that the current codebase doesn't easily allow to support this, but >> it has been a while since I looked at it, so maybe I should just look into >> this again... >> We have an open issue for this, see >> https://github.com/hpcugent/easybuild-framework/issues/450. >> >> The 'but' is that --try-X is best effort, and is pretty likely to fail >> (depending on how different the toolchain you're 'trying' is from the >> current one). >> The implementation of this is pretty messy, or to use someone else's words >> "a steaming pile of crap" (he's right). >> >> So, I fully agree with you that this would be very useful, but it's less >> easy than it may seem at first. >> Anyway, I'll try and revisit this soon. Do leave a comment in issue #450 to >> indicate you really want to see this work. >> >> >>>> - --try-toolchain produces a eb file, but does not copy it to the lib >>>> directory at the end. Perhaps that could be added? >>> It's a bugture (contraption of bug-feature): manual work is expected. >>> In principle that requires heavy editing and is considered more of a >>> template. >> Are you expecting the easyconfig file to be added to the currently installed >> easybuild-easyconfigs package? >> That's a bit weird, since you're then modifying an installed part of EB, >> which has a particular version (which then no longer makes sense as >> reference point, by consequence). >> >> Actually, what we'd want you to do, is submit a pull request for the >> easyconfig(s) you obtained this way (if the build worked for you). >> That way, it will be included in the next version of EasyBuild, and other >> people can use them without needing to resort to --try-X. >> >> As Fotis already mentioned, the automagically tweaked easyconfig file(s) >> still need to be reviewed by a human (even if it worked) before they can be >> merged in... >> >> We're thinking about allowing people to configure EasyBuild such that pull >> requests are generated automagically whenever a build is successful >> and if the easyconfig file that was used is not available yet in our repos, >> but that's also a bit scary... ;-) >> >> >>>> - in one of my attemps, at the end of an installation, this error came up: >>>> (should only have 'files' and 'dirs' keys, values should be lists (at >>>> least one non-empty)). >>>> This could be checked before compilation as part of the eb-file sanity >>>> check. >>> IMHO, the error-level should then be non-zero; I believe then the right >>> thing will happen, no? >> Bart is right here, we could/should capture a problem like that a lot >> earlier, instead of performing the whole build procedure first, and then >> complain that the sanity check spec is not what we want it to be... >> >> Point taken: https://github.com/hpcugent/easybuild-framework/issues/703 >> >> >>>> During the installation of Qt, some "Project Messages" pop up, and the >>>> waiting time during the configuration is sometimes large. As a result, eb >>>> stops and prints: >>> [...] >>>> However, this message is not an error, and if I configure myself, without >>>> eb, the whole process runs through. How can I avoid eb from stopping on >>>> these messages? >>> Either the output has a keyword that makes EasyBuild tick (and requires >>> tuning), >>> or the error-level is incorrect or, there is a bug (eh, or all of the above >>> :). >>> Which case are we talking about here? >> The configure script for Qt is an interactive script (eek!), i.e. it asks >> questions and expects answers, see >> https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/q/qt.py >> . >> >> EasyBuild easily confuses messages that show up as the last line of output >> for a while for questions, and then fails when it can't figure out an answer >> to that 'question'. >> Hence, EB needs to be told which 'long-hanging' messages are *not* >> questions, via a list of regex patterns (see the no_qa part in the Qt >> easyblock). >> >> It seems that the current list of no_qa patterns in insufficient on your >> end, for some reason (maybe slower systems, whatever). >> >> Can you try and adjust the Qt easyblock such that these project messages are >> ignored by extending the no_qa list? >> If you get that to work, please open a PR to get that fix included. >> If not, let us know, and we'll help you out. >> >> The waiting time during configuration is large for Qt, I remember that from >> our installations. E.g., in our production build, the configure step took >> about 7 minutes. >> Are you saying the configuration is a lot faster if you run it manually >> (outside of EasyBuild)? Because that would surprise me, and would definitely >> point to a bug/problem. >> >> >> regards, >> >> Kenneth >> >> PS: It seems like you guys are really picking up on EasyBuild, great! >> If you think it's useful to have a conf call of some sort to quickly discuss >> a couple of things that are not entirely clear to you, please shout, it >> could be a huge time saver... >> I'm sure we can find a time that works well on both ends of the wire... >> > > -- > Centre for e-Research > Level G, Room 409-G21 > 24 SYMONDS ST > Auckland 1010 > New Zealand > +64 (0) 9 923 9740 ext 89740 >
