[easybuild] [ANN] EasyBuild v2.1.0
Hello EasyBuilders, I'm pleased to announce the release of EasyBuild v2.1.0 [1]. There's no doubt about it: this is the best EasyBuild release so far (trust me, I checked). A detailed overview of the changes is available in the release notes [2], but here are the highlights: - support for only (re)generating the module file using --module-only [3] - support for generating module files in Lua syntax (requires Lmod) [4] - support for using external modules as dependencies [5] - support for installing modules for dependencies as hidden modules using --hide-deps [6] - experimental support for building/installing software on top of PrgEnv-* modules on Cray systems [7] We have also made two aspects of the EasyBuild configuration mechanism a bit more strict: an error will be thrown when EasyBuild detects that i) non-existing configuration files are specified (rather than just ignoring them), or i) $EASYBUILD-prefixed environment variables are defined that don't have a matching configuration option. In addition, we have deprecated using the 'log.error' method to report errors, because of problems when 3rd party libraries were being used together with EasyBuild. Instead of using log.error, an EasyBuildError should be raised (which will be logged automagically). We will keep supporting the use of 'log.error' until EasyBuild 3.0 (no plans yet w.r.t. timeframe), but we highly recommend to stop using it already in your easyblocks and other plugins to EasyBuild. See [8] for more details. Support for 27 new software packages has been added, including non-trivial ones like getdp, gmsh, Octave, TINKER, and Xmipp. To upgrade to EasyBuild v2.1.0, there are several options: (i) (re)bootstrap EasyBuild to obtain an EasyBuild module to load [9] (ii) install EasyBuild v2.1.0 with a previous version of EasyBuild, using the easyconfig file available in [10] (iii) install EasyBuild v2.1.0 from PyPI, using one of the standard Python installation tools (easy_install, pip, ...) [11] (iv) updating your Git working copies of the different EasyBuild repositories * make sure you also provide vsc-base v2.2.0 or more recent in your Python search path [12] Enjoy! regards, Kenneth [1] http://pypi.python.org/pypi/easybuild [2] http://easybuild.readthedocs.org/en/latest/Release_notes.html [3] http://easybuild.readthedocs.org/en/latest/Partial_installations.html#module-only [4] http://easybuild.readthedocs.org/en/latest/Configuration.html#module-syntax [5] http://easybuild.readthedocs.org/en/latest/Using_external_modules.html [6] http://easybuild.readthedocs.org/en/latest/Manipulating_dependencies.html#hide-deps [7] https://github.com/hpcugent/easybuild/wiki/EasyBuild-on-Cray [8] http://easybuild.readthedocs.org/en/latest/Deprecated-functionality.html#depr-error-reporting [9] http://easybuild.readthedocs.org/en/latest/Installation.html#bootstrapping-easybuild [10] https://github.com/hpcugent/easybuild-easyconfigs/pull/1581 [11] http://easybuild.readthedocs.org/en/latest/Installation_Alternative.html#standard-installation-of-latest-release [12] https://pypi.python.org/pypi/vsc-base
Re: [easybuild] Adding files to the repository
On 30/04/15 10:09, Backeljauw Franky wrote:
Hi Jens,
Op 30-apr.-2015, om 09:53 heeft Jens Timmerman
mailto:jens.timmer...@ugent.be>> het
volgende geschreven:
On 29/04/15 17:15, Backeljauw Franky wrote:
I have a small question: after unpacking a tar-file of a PythonPackage, I need
to add a file to its directory before starting the build process.
How can I do that with EasyBuild?
I tried with a patch as in “diff -ru -NB” but that tells me it cannot determine
the patch level…
Any ideas?
I've seen Kenneth do it like this
patches = [
'somepatch,patch',
['filetocopy', 'directory to copy it to, e.g, ./somedir']
]
This last construction worked:
patches = [‘build.conf’,’./‘]
This works, but it doesn't do what you think it does. :-)
What Jens was suggesting is this:
patches = [('build.conf', '.')]
That is: tell EasyBuild to *copy* the file build.conf to whatever the
current directory is at that time.
EasyBuild knows it should copy this file rather than try to treat it as
a patch file, because the filename doesn't end with .patch .
Since the default target is '.', this should be the same as:
patches = ['build.conf']
What you are doing is copying both 'build.conf' and '.' to the current
directory.
Apparently the Python equivalent of "cp . ." 'works' fine, but you just
got lucky there. ;-)
TL;DR: We need to get the various options for 'patches' documented,
there are other things too you can do (e.g., supply the patch level to
use for patch files, or the location where patch files should be applied
in).
K.
Re: [easybuild] Sanity test error for Python PyYAML
Hi Cormac,
For future reference:
* search for 'PyYAML' in the easyconfigs repository (use the search box
on top), see
https://github.com/hpcugent/easybuild-easyconfigs/search?q=pyyaml
* select one of the stand-alone easyconfigs, which include " options =
{'modulename': 'yaml'}".
This is not *exactly* what you're looking for, since you're including
PyYAML as an extension to Python here, but it's close.
Maybe you can figure out the problem yourself next time using this approach.
If not, don't hesitate to post your question to the mailing list, I'm
sure this can be useful to others too.
regards,
Kenneth
On 30/04/15 16:40, Garvey, Cormac T wrote:
Hi Ward,
That's exactly what I was looking for.
Cheers,
Cormac.
On Thu, Apr 30, 2015 at 1:50 AM, Ward Poelmans > wrote:
On Thu, Apr 30, 2015 at 2:22 AM, Garvey, Cormac T
mailto:cormac.gar...@inl.gov>> wrote:
> ('PyYAML', pyyamlversion, {
> 'source_urls':
> [('https://pypi.python.org/packages/source/P/PyYAML')
],
> }),
Use the 'modulename' option:
('PyYAML', pyyamlversion, {
'modulename': 'yaml',
'source_urls':
[('https://pypi.python.org/packages/source/P/PyYAML')
],
}),
Ward
--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory
Ph: 208-526-6294
Re: [easybuild] Sanity test error for Python PyYAML
Hi Ward,
That's exactly what I was looking for.
Cheers,
Cormac.
On Thu, Apr 30, 2015 at 1:50 AM, Ward Poelmans
wrote:
> On Thu, Apr 30, 2015 at 2:22 AM, Garvey, Cormac T
> wrote:
>
> > ('PyYAML', pyyamlversion, {
> > 'source_urls':
> > [('https://pypi.python.org/packages/source/P/PyYAML')],
> > }),
>
> Use the 'modulename' option:
> ('PyYAML', pyyamlversion, {
> 'modulename': 'yaml',
> 'source_urls': [('
> https://pypi.python.org/packages/source/P/PyYAML')],
> }),
>
> Ward
>
--
Cormac Garvey
HPC Software Consultant
Scientific Computing
Idaho National Laboratory
Ph: 208-526-6294
Re: [easybuild] Adding files to the repository
On Thu, Apr 30, 2015 at 10:06 AM, Backeljauw Franky
wrote:
> The main.filetools module seems to have a problem with the fact that the
> file build.conf does not exist at first…
Yes because it cannot derive the patch level for a non-existing file.
You will have to specify it manually:
patches = [('kwant-1.0.3-build-conf.patch', 1)]
Ward
Re: [easybuild] Adding files to the repository
Hi Jens, Op 30-apr.-2015, om 09:53 heeft Jens Timmerman mailto:jens.timmer...@ugent.be>> het volgende geschreven: On 29/04/15 17:15, Backeljauw Franky wrote: I have a small question: after unpacking a tar-file of a PythonPackage, I need to add a file to its directory before starting the build process. How can I do that with EasyBuild? I tried with a patch as in “diff -ru -NB” but that tells me it cannot determine the patch level… Any ideas? I've seen Kenneth do it like this patches = [ 'somepatch,patch', ['filetocopy', 'directory to copy it to, e.g, ./somedir'] ] This last construction worked: patches = [‘build.conf’,’./‘] — Thanks, Franky
Re: [easybuild] Adding files to the repository
Op 29-apr.-2015, om 17:23 heeft Ward Poelmans
mailto:ward.poelm...@ugent.be>> het volgende geschreven:
On Wed, Apr 29, 2015 at 5:15 PM, Backeljauw Franky
mailto:franky.backelj...@uantwerpen.be>> wrote:
I tried with a patch as in “diff -ru -NB” but that tells me it cannot determine
the patch level…
That should do the trick. Does the patch work if you try to apply it manually?
Autodetecting the patch level usually works but you can force it by
using a tuple.
Yep, I can apply the patch manually, but with EasyBuild it gives this:
== 2015-04-30 10:00:46,177 main.easyconfig.templates DEBUG name: versionprefix,
config:
== 2015-04-30 10:00:46,177 main.PythonPackage INFO Running method patch_step
part of step patch
== 2015-04-30 10:00:46,177 main.PythonPackage INFO Applying patch
kwant-1.0.3-build-conf.patch
== 2015-04-30 10:00:46,177 main.PythonPackage DEBUG Source index: 0; patch
level: 0; source path suffix: ; copy patch:
== 2015-04-30 10:00:46,178 main.PythonPackage DEBUG Determine begin path for
patch kwant-1.0.3-build-conf.patch:
/apps/antwerpen/easybuild/build/kwant/1.0.3/intel-2015a-Python-2.7.9/kwant-1.0.3
== 2015-04-30 10:00:46,178 main.PythonPackage DEBUG Applying patch {'path':
'/apps/antwerpen/easybuild/build/kwant-1.0.3-build-conf.patch', 'level': 0,
'name': 'kwant-1.0.3-build-conf.patch', 'md5':
'bf52a0a00f887a194e6f770e1db76e21', 'checksum': None} in path
/apps/antwerpen/easybuild/build/kwant/1.0.3/intel-2015a-Python-2.7.9/kwant-1.0.3
== 2015-04-30 10:00:46,178 main.filetools DEBUG No match found for
kwant-1.0.3/build.conf, trying next patched file...
== 2015-04-30 10:00:46,179 main.filetools ERROR EasyBuild crashed with an error
(at easybuild/tools/filetools.py:654 in apply_patch): Can't determine patch
level for patch /apps/antwerpen/easybuild/build/kwant-1.0.3-build-conf.patch
from directory
/apps/antwerpen/easybuild/build/kwant/1.0.3/intel-2015a-Python-2.7.9/kwant-1.0.3
== 2015-04-30 10:00:46,179 main.easyblock WARNING build failed (first 300
chars): EasyBuild crashed with an error (at easybuild/tools/filetools.py:654 in
apply_patch): Can't determine patch level for patch
/apps/antwerpen/easybuild/build/kwant-1.0.3-build-conf.patch from directory
/apps/antwerpen/easybuild/build/kwant/1.0.3/intel-2015a-Python-2.7.9/kwant-1.0.3
== 2015-04-30 10:00:46,179 main.PythonPackage INFO Closing log for application
name kwant version 1.0.3
The main.filetools module seems to have a problem with the fact that the file
build.conf does not exist at first…
For completeness, this is the patch file:
diff -ru -BN kwant-1.0.3.org/build.conf
kwant-1.0.3/build.conf
--- kwant-1.0.3.org/build.conf 1970-01-01
01:00:00.0 +0100
+++ kwant-1.0.3/build.conf 2015-02-04 23:48:00.0 +0100
@@ -0,0 +1,7 @@
+[mumps]
+libraries = zmumps mumps_common pord metis esmumps scotch scotcherr
+
+[lapack]
+libraries = mkl_intel_lp64 mkl_sequential mkl_core pthread m
+library_dirs = $EBROOTIMKL/mkl/lib/intel64
+extra_link_args = -Wl,-rpath=$EBROOTIMKL/mkl/lib/intel64
— Regards,
Franky
Re: [easybuild] Adding files to the repository
Hi On 29/04/15 17:15, Backeljauw Franky wrote: Hello, I have a small question: after unpacking a tar-file of a PythonPackage, I need to add a file to its directory before starting the build process. How can I do that with EasyBuild? I tried with a patch as in “diff -ru -NB” but that tells me it cannot determine the patch level… Any ideas? I've seen Kenneth do it like this patches = [ 'somepatch,patch', ['filetocopy', 'directory to copy it to, e.g, ./somedir'] ] — Many thanks for your reply, Franky Regards, Jens Timmerman
Re: [easybuild] Sanity test error for Python PyYAML
On Thu, Apr 30, 2015 at 2:22 AM, Garvey, Cormac T wrote:
> ('PyYAML', pyyamlversion, {
> 'source_urls':
> [('https://pypi.python.org/packages/source/P/PyYAML')],
> }),
Use the 'modulename' option:
('PyYAML', pyyamlversion, {
'modulename': 'yaml',
'source_urls': [('https://pypi.python.org/packages/source/P/PyYAML')],
}),
Ward
[easybuild] Sanity test error for Python PyYAML
Hello,
I am having difficulty installing the PyYAML python module
exts_list = [
('setuptools', '5.7', {
'source_urls': ['
http://pypi.python.org/packages/source/s/setuptools/'],
}),
('pip', '1.5.6', {
'source_urls': ['http://pypi.python.org/packages/source/p/pip/'],
}),
('nose', '1.3.3', {
'source_urls': ['http://pypi.python.org/packages/source/n/nose/'],
}),
('PyYAML', pyyamlversion, {
'source_urls': [('https://pypi.python.org/packages/source/P/PyYAML')
],
}),
('Yamlog', yamlogversion, {
'source_urls': [('https://pypi.python.org/packages/source/Y/Yamlog')
],
}),
('Scripy', scripyversion, {
'source_urls': [('https://pypi.python.org/packages/source/S/Scripy')
],
'patches': ['scripy_setup.py.patch'],
}),
('psycopg2', '2.5.3', {
'source_urls': ['http://pypi.python.org/packages/source/p/psycopg2/'
],
}),
('numpy', numpyversion, {
'source_urls': [('
http://sourceforge.net/projects/numpy/files/NumPy/%s' % numpyversion,
'download')],
'patches': [
'numpy-%s-mkl.patch' % numpyversion,
],
}),
('scipy', scipyversion, {
'source_urls': [('
http://sourceforge.net/projects/scipy/files/scipy/%s' % scipyversion,
'download')],
}),
('matplotlib', matplotlibversion, {
'source_urls': [('
http://sourceforge.net/projects/matplotlib/files/matplotlib/matplotlib-%s'
% matplotlibversion, 'download')],
}),
]
The PyYAML sanity test fails, because its testing
import PyYAML
Which does not exist.
from
EXTS_FILTER_PYTHON_PACKAGES = ('python -c "import %(ext_name)s"', "")
I would like to change the PyYAML test to
import yaml
What is the easiest way to do this in Easybuild?
Thanks
Cormac.
