So I've tried out each path and have some additional questions about toolchain names and conventions. Leaving the original message below for reference and breaking out my follow-up on the approaches to help with discussion.
Patching GCC 4.7.2 in toolchain goolf-1.4.10: ---------------------------------------------------------- I've abandoned this approach. The patch applies fine, but there are more than the few texi parse errors listed in the GCC bug. This raises the question, is there a way to check tool versions from the system and report a dependency conflict if they are too new or old? I don't know exectly which tool is doing the parsing of the texi files in the GCC 4.7.2 build but it clearly doesn't tolerate the earlier documentation syntax. Trying builds with newer software or tools versions: ------------------------------------------------------------------- I tried the build with options like --try-toolchain-version=1.7.10 or --try-software-version=GCC,4.7.3 but this didn't seem to work well for a --robot build. The build still tried to use GCC 4.7.2. eb Cufflinks-2.2.1-goolf-1.4.10.eb --try-software-version=GCC,4.7.3 --robot I also tried creating a bumped toolchain that just incremented the GCC version and then using it. eb Cufflinks-2.2.1-goolf-1.4.10.eb --try-toolchain-version=1.4.11 --robot Are these arguments intended to address such situations or something else? Bumping the goolf tool chain to use GCC 4.7.3: -------------------------------------------------------------- Explicitly changing the easyconfig to use the goolf-1.4.11 with the bumped GCC release caused other problems. It seems a lot of the intermediary tools in the toolchain like OpenMPI, also need to be updated with such a change since they have an explicit dependency on 4.7.2 listed in their easyconfigs. [jpr@login001 ~]$ eb Cufflinks-2.2.1-goolf-1.4.10.eb --robot == temporary log file in case of crash /tmp/eb-pNsPag/easybuild-P1Dd_z.log == resolving dependencies ... ERROR: Irresolvable dependencies encountered: OpenMPI/1.6.4-GCC-4.7.3 [jpr@login001 ~]$ eb --buildpath=/var/tmp/jpr-build Cufflinks-2.2.1-goolf-1.4.10.eb --robot == temporary log file in case of crash /tmp/eb-AAwRM3/easybuild-PH1goQ.log == resolving dependencies ... ERROR: Irresolvable dependencies encountered: hwloc/1.6.2-GCC-4.7.3 I assume this comes from the dependency list in the goolf-1.4.10.eb easyconfig. Is there a way to avoid exposing what seems like a toolchain-internal dependency (GCC 4.7.3) to easyconfigs that sit above the base tool chain, ie. hide the dependency in the tool chain. Would it even make sense to do this? Bump Cufflinks to use a newer toolchain ----------------------------------------------------- This is the approach that worked in the end and relates to my pull request. https://github.com/hpcugent/easybuild-easyconfigs/pull/2058 Understanding the difference between goolf and foss toolchains: -------------------------------------------------------------------------------------- What is the packaging and naming philosophy behind goolf and foss toolchains? Is foss a newer preferred convention? Are they two distinct site models? Comparing two of the recent entries doesn't seem to show much substantive difference: $ diff g/goolf/goolf-1.7.20.eb f/foss/foss-2015a.eb 3,4c3,4 < name = 'goolf' < version = '1.7.20' --- > name = 'foss' > version = '2015a' 13c13 < comp_version = '4.8.4' --- > comp_version = '4.9.2' 30,31c30,31 < ('GCC', '4.8.4'), < ('OpenMPI', '1.8.4', '', comp), # part of gompi-1.6.20 --- > comp, > ('OpenMPI', '1.8.4', '', comp), Thanks for the pointers, ~jpr On 10/09/2015 05:08 PM, John-Paul Robinson wrote: > Thanks for all the helpful information and for the welcome aboard! > > On 10/09/2015 03:26 PM, Kenneth Hoste wrote: >> Hi John-Paul, >> >> On 09/10/15 21:43, John-Paul Robinson wrote: >>> Hi, >>> >>> I'm using EasyBuild 2.3.0 and are wanting to build Cufflinks. We tried >>> the Cufflinks-2.2.1-goolf-1.4.10.eb on CentOS and the build fails. >> How does it fail exactly? It works fine on our Scientific Linux 6 >> system. >> > The make is failing due to a syntax error in the documentation files. > > ../../gcc/doc/cppopts.texi:806: @itemx must follow @item > make[3]: *** [doc/cpp.info] Error 1 > make[3]: *** Waiting for unfinished jobs.... > <snip> > make[3]: *** [doc/gcc.info] Error 1 > make[3]: Leaving directory > `/home/jpr/.local/easybuild/build/GCC/4.7.2/dummy-dummy/gcc-4.7.2/obj/gcc' > make[2]: *** [all-stage1-gcc] Error 2 > make[2]: Leaving directory > `/home/jpr/.local/easybuild/build/GCC/4.7.2/dummy-dummy/gcc-4.7.2/obj' > make[1]: *** [stage1-bubble] Error 2 > make[1]: Leaving directory > `/home/jpr/.local/easybuild/build/GCC/4.7.2/dummy-dummy/gcc-4.7.2/obj' > make: *** [bootstrap] Error 2 > (at > > easybuild/2.3.0/software/EasyBuild/2.3.0/lib/python2.7/site-packages/easybuild_framework-2.3.0-py2.7.egg/easybuild/tools/run.py:360 > in parse_cmd_output) > == 2015-10-08 11:04:18,816 main.easyblock WARNING build failed > (first 300 chars): cmd " make -j 4 bootstrap " exited with > exitcode 2 and output: > echo stage3 > stage_final > > According to the GCC bug, this is the result of stricter parsing of > the texi files which catch an error that was previously ignored. > >>> Fixing the the build falls into one of several categories: >>> * patch GCC 4.7.2 (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=52903) >>> * update goolf-1.4.10 to use GCC 4.7.3 (fixes the bug in 4.7.2) >>> * create a new Cufflinks easyconfig based on goolf-1.7.20 >>> >>> As a learning exercise, I'm following each path. (BTW, I do plan to >>> submit pull requests once my builds are verified.) In doing so, I'm >>> wondering if there are there any recommended practices for updating and >>> easyconfig that has a bug, changing a compiler version dependency or >>> introducing new easyconfigs. >> >> I wouldn't start with backporting patches to older an compiler >> version and rebuilding the compiler, especially since there's a new >> minor release (4.7.3) for the same major compiler version (4.7). Just >> moving on to the newer minor (or even major) release of the compiler >> makes more sense, imho. >> >> We never update toolchain components without bumping the toolchain >> version in the easyconfigs we ship via the easybuild-easyconfigs >> repository, and I wouldn't recommend it either. Bumping a compiler >> (or any other component) should imply also bumping the toolchain >> version (that's the whole point of having a toolchain version imho). >> >> Rebuilding Cufflinks with a newer toolchain version makes most sense >> to me. >> And of course, a pull request for it makes sense. :-) >> > > Good to know. I do plan on adding the updated Cufflinks. Currently > working through an error finding the bam library. Trying to understand > how the EBROOT<tag> env vars get set. Will post a separate thread if I > can't figure it out shortly. > > I realize the bug in GCC 4.7.2 at this point is pretty old and it's > arguably better to move forward than fix the past. I was mainly > interested in understanding how to approach such scenarios if they are > current. > > BTW, I really like that are straight forward mechanisms within > Easybuild to address each of these issues, with a patch automatically > applied or updated easyconfig. > >> >>> I assume the git-hooks are designed to allow sites to build against >>> pending upstream commits using the "build against pull request >>> feature". >> You're referring to eb --from-pr here, right? >> >> It is mostly intended to easily pull requests to check whether >> they're good to merge. >> >> For new easyconfigs, it can also be used to build & install things >> not included in the latest release. >> But beware that later pull requests may modify the easyconfig before >> it gets included in a release (for whatever reason)... >> >>> I haven't seen much in the way of recommendations for handling the >>> local >>> environment to make contributing easier. For example, how to install >>> the easyconfig or blocks so that I can work against a git clone'd tree >>> (I'm currently using a symlink to the repo easyconfig dir inside my >>> personal easybuid env). >> >> There are a couple of ways to sit on top of the current 'develop' >> branches, but it also a risk (just so you're aware). >> >> Easyblocks and easyconfigs in the 'develop' branch may require >> features in the EasyBuild framework that are not included in the >> latest EasyBuild yet, and for all intents and purposes you should >> always consider 'develop' to be unstable. >> Although we are quite (very, even) careful when merging changes to >> the develop branch, it's always possible that bugs are introduced >> that only pop up during a thorough regression test (which is >> typically only done shortly before a new release). >> >> One way is to simply clone the different repositories (framework, >> easyblocks, easyconfigs), set $PYTHONPATH and $PATH, and use >> EasyBuild develop directly. >> >> Another option is to install the develop branches using >> easy_install/pip/whatever (for example, easy_install --prefix >> <prefix> >> https://github.com/hpcugent/easybuild-framework/archive/develop.tar.gz, >> after setting $PYTHONPATH and pre-creating the 'site-packages' dir). >> >> If you only care about easyconfigs, you can clone/download the >> develop branch, and use the --robot-paths configuration option (see >> http://easybuild.readthedocs.org/en/latest/Using_the_EasyBuild_command_line.html#controlling-the-robot-search-path) >> to make sure EasyBuild will pick up these easyconfigs. >> A simple "git pull" can make sure you're getting the latest >> easyconfigs that were reviewed/tested thoroughly before merging. >> >> To use 'develop' easyblocks by occasion, the --include-easyblocks >> configuration option can be very useful (see >> http://easybuild.readthedocs.org/en/latest/Including_additional_Python_modules.html#include-easyblocks). >> >> There's also a script available to set up the 'develop' clones, see >> http://easybuild.readthedocs.org/en/latest/Installation_Alternative.html#installation-of-latest-development-version-using-provided-script >> . >> > Thanks this helps. > > I'm less interested in sitting atop the develop branch than in a local > mods branch. Thanks for the reminder to actually pull the from a > release in the repo. > >>> Given that the easyconfigs are all distributed >>> from upstream in one big chunk vs. individual repos, I may have a need >>> to maintain local variations that don't make it into the upstream or >>> aren't appropriate for it. >>> >>> What approaches do you recommend for handling these situations? >> >> To maintain local variations, you can throw your easyconfigs in a >> separate directory, and list it first in --robot-paths. >> That way, your customized easyconfigs will be considered first, >> before other easyconfig files with the same name. >> >> Currently, we don't have a good way of keeping locally maintained >> copies of easyconfigs in sync with their equivalent 'upstream'. >> One idea has been to add support for 'easyconfig patch files' (.ebp), >> where you can just specify the customizations, rather than requiring >> you to copy the whole easyconfig file. But nobody has stepped up to >> implement this yet, so it remains just an idea. >> >> I hope this helps, do let us know if you have any more questions. >> >> Unless I'm mistaken, this is your first post to the list, so: welcome >> to the wonderful world of EasyBuild! >> >> > > Thanks again for the welcome and helpful feedback. > >> regards, >> >> Kenneth >
