Thank you very much for the detailed response! I did not intend to be complaining about EasyBuild, and I am sorry if that is how my email came across.
I am trying to learn the mindset of the EasyBuild community so I can start to use it more efficiently and contribute the work we are doing. In this specific case, a colleague of mine was trying to build OpenCV, and so I am relaying his experiences second hand. Your email is very helpful to me in achieving an understanding of the community. I'll try to summarize points that I picked out individually: OS: We are using Ubuntu, and the inclusion of several CentOS-specific easyconfigs in the repo led me to believe OS was not a big compatibility issue. I don't know why I thought that. It is always an issue. It is my hope that we can help to adapt easyconfigs to include Ubuntu where there are compatibility issues. Isolated Builds: I cut my teeth in the RedHat world. For years I had to build in RHEL, and moving to Ubuntu was something of a revelation. The standard Debian build system does build an isolated chroot to ensure host OS packages don't creep in to your build (look at the pbuilder package in Ubuntu/Debian). EasyBuild has made great strides in implementing isolated builds without relying on a chroot/container type of system, and I think there is merit to that. [Side note on openssl] - It appears to be a Debian/Ubuntu issue: https://askubuntu.com/questions/523831/opensslconf-h-not-found/594486#594486 To the point of OpenSSL from OS packages, I had not thought of security updates as an issue, and if the easybuild community is not updating easyconfigs in response to security advisories (and why would you?), then that makes good sense. I suppose it comes to a push-pull between reproducing the easyconfig exactly and producing secure code. That is a decision I think each institution/author must make. --try-* commands and run-time easyconfigs: I have built many packages with --try-* commands, and it is a very useful feature. In fact, my workflow is typically to use an existing easyconfig with a --try-software-version to build a new version of a package, and only resort to easyconfig editing when that doesn't work. I believe I've even had some luck with major version changes, but I know that will be rare. Contributing successfully built easyconfigs back to the repo seems like a great direction, and I look forward to seeing what comes from --new-pr. For now, since I am not the only one using EB here, I will work on collecting our --try-* resultant easyconfigs. We are collecting our manually edited ones and will include them in our fork of the easyconfigs repo. When we initially implemented EasyBuild, there was confusion about what exactly it delivers. Some had hopes it could be used like a continuous integration system with regard to new versions of software - autobuild the new R on release day, for example. That is a herculean task. But I think that initial impression led us to an unrealistic expectation of how easy it would be to modify and write easyconfigs. As we move forward, there are a few things we are working on. We have a mandate to build newly released versions of R and Python as close to release day as possible. Also, we want to updated every library/module/extension in the easyconfig at that time. Using the APIs offered by pipy and cran, I think we have come close to scripting the creation of an updated easyconfig. Obviously that does not mean it builds, but you never know, it might! I am also working on documenting my experiences and constructing a sort of "Getting Started" guide on implementation in our environment (single EasyBuild location on shared NFS, multiple builders, many users, licensed compilers, toolchain advise, etc.). I hope to have something worthwhile in the end. This brings up several other questions I have, which I think best addressed in a separate email thread. I will start that now. Thank you for EasyBuild! Ben McGough System Administrator / Center IT - Scientific Computing / 206.667.7818 / [email protected] / Fred Hutch / Cures Start Here ----- Original Message ----- From: "Kenneth Hoste" <[email protected]> To: "easybuild" <[email protected]> Sent: Wednesday, January 20, 2016 11:35:47 AM Subject: Re: [easybuild] Easyconfig workflow example with questions Hi Ben, I'm sorry your experience hasn't been fluent/easy so far. On 20/01/16 19:21, McGough, Benjamin E wrote: > Goal: build OpenCV 3.1 with Intel 2015b toolchain (as that is the one we have > licensed). OpenCV is probably not an easy target, but you probably aren't working on it because you randomly picked it... :-) Bumping the major version (2 to 3) is also a sign of potential problems... There hasn't been a lot of OpenCV related activity in, but there has been some, albeit without high success rate (see the ratio of closed vs merged PRs at [1]). > We take the existing OpenCV 2.4.9 intel 2014b easyconfig and edit it with the > new version and toolchain (after trying with --try-toolchain and failing). What failed exactly with --try-toolchain? Did you get errors? Or did it fail to use what you wanted it to? Did you combine it with --robot? It's certainly not perfect (it's called 'try' for a reason), but it should save you quite a bit of time. You can grab hold of the easyconfig files that EB generates with -try-* through a construct like: eb OpenCV-2.4.9-intel-2014b.eb --try-toolchain-version=2015b -Dr --disable-cleanup-tmpdir > This then depends in each of the dependent packages under the new toolchain, > so easybuild builds those. > > During that build process, we discover that some dependencies may have > changed. JasPer is looking for libjpeg-turbo, but that is not depended in the > JasPer easyconfig, so we add that. So, here you had to add libjpeg-turbo as a dependency in the JasPer easyconfig? Did you also bump the version of JasPer in this process? I'm not sure I have a good view on what this problem is about. > CMake is depending on openssl (from the OS!) and cannot find opensslconf.h > as it has been moved from /usr/include to /usr/include/x86_64... . There's a good reason for CMake depending on the OS-provided OpenSSL by default: to try and ensure that security updates that become available through an OS update are picked up. If you provide OpenSSL through EasyBuild (which should work just fine), you'll likely never update it, an you'll be stuck with the same version of OpenSSL with that particular CMake build.. Depending on your setup/environment, that may be something you want to avoid (but it's a user/site-specific thing). > So now we have modified at least 3 easyconfigs, and the next step is to > depend openssl in CMake so it is not pulled from the OS, and modify the > easyconfig for that. The easyconfigs we include in EasyBuild are meant to be treated as 'examples'. Although a lot of them should work out-of-the-box (they do for us on SL6/CO7), there's too many variables we don't control (yet?) to ensure that each easyconfig will work on every possible system people try it on. Which OS are you working on? That's still a big factor. In this case, it seems that you're missing openssl-devel (or equivalent) on your system. Using an EB-provided OpenSSL should get you out of the woods here, but then you need to be aware of the possible security-related implications that has. > This is as far as we have gotten, but there are 8 more dependent packages, > not including Python, which I have yet to build successfully - different > story. One way to make better/faster progress is to start bottom-up rather than top-down. If you know you'll need Python, why not start there? And please, combine --try-toolchain with --robot, it should give you a big leap. Also, 8 dependencies is *nothing*. Take a look at this gem: a dependency graph for DOLFIN 1.6, which I was working on the last couple of weeks. You may need to zoom out a bit (144 nodes in total, including the toolchain)... http://users.ugent.be/~kehoste/DOLFIN-1.6.pdf This required several updates to existing easyconfigs, a handful of (simple) patches to fix build issues (mostly because of using Intel compilers, iirc), and a couple of tweaks to easyblocks because of changes in some of these components. It wasn't exactly a smooth ride, but I don't think you can blame EasyBuild here... It took me a couple of days, but *way* less than the (literally) months we spent on getting DOLFIN 1.0 supported... EasyBuild isn't magic: it definitely does *not* get you out of dependency hell, but it does take a whole bunch of work out of your hands. It's never going to be perfect though, certainly not when updating software versions is part of what you're working on. > This will lead to what I hope will be a reproducible software build. I hope you'll be able to reap the fruits of your labor soon. And if you do, please contribute back your easyconfigs, and report any issues that should/could be fixed upstream. Let others build on top of your hard work, like you're building on other people's work... (spoiler: I'm working on getting support for "eb --new-pr" include in the next EasyBuild release, which should make it significantly easier to contribute back; stay tuned for more information soon) > Am I following the recommended easyconfig workflow here? I assume the default > action should be to include dependent packages into easybuild in order to > produce the most reproducible easyconfig, right? The way we currently deal with dependencies is very 'ad hoc', unfortunately. If the configure script fails with errors that point out that certain dependencies could not be found, it's obvious. Sometimes it just continues and automagically disables components though, so you have to go in and check things in detail. Or, it picks up stuff from the OS on whichever system it was tested on, which makes dependencies to 'masked'. We don't have a good solution yet for this. One option is to rely on part of what the Nix package manager provides, some kind of "jail" in which the builds are performed, to avoid that stuff from the OS is being picked up without noticing. In your particular case, I'm afraid it's not going to be a smooth ride because of the major version upgrade for OpenCV... > Also, do the "--try-*" options have the ability to output an easyconfig after > the fact if the build was successful? For every successful installation, EB will copy the used easyconfig file to the installation directory (see the 'easybuild*' subdirectory), and to the easyconfigs archive/repository (cfr. http://easybuild.readthedocs.org/en/latest/Configuration.html#easyconfigs-repo). This is the case regardless of whether or not --try-* was used; in that case, EB will take the specified easyconfig, tweak it in /tmp, and then use that for the build. With the "-Dr --disable-tmpdir" approach mentioned above, you can get your hands on all tweaked easyconfigs. Please let us know if there's anything we can do to help: answer questions, provide suggestions to make your life easier, etc. Also, don't hesitate to open issues for features you feel are missing. Or even better, submit a pull request to implement them. ;-) regards, Kenneth [1] https://github.com/hpcugent/easybuild-easyconfigs/search?q=opencv&type=Issues&utf8=%E2%9C%93
