On 18/03/16 16:00, Pablo Escobar Lopez wrote:
2016-03-18 15:11 GMT+01:00 Fotis Georgatos <[email protected] <mailto:[email protected]>>:ie. we might be able to stop most of the network activity in user-space, w/out requiring blockage at root level (there are some caveats, agreed), by rewiring certain calls. thoughts?Not sure if adding and external dependency to easybuild and using LD_PRELOAD is a good idea. I think Kenneth was looking at a solution using python but I don't remember the details. I think he should decide about this.
What I had in mind is specifying a different (broken) PyPI URL, cfr. 'index-url' at https://pythonhosted.org/setuptools/easy_install.html#command-line-options
Of course, this approach is very specific to installing Python packages... K.
F. -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # Yelling in a CERN forum ------------------------------------------------------------------------ *From:* [email protected] <mailto:[email protected]> [[email protected] <mailto:[email protected]>] on behalf of Pablo Escobar Lopez [[email protected] <mailto:[email protected]>] *Sent:* Friday, March 18, 2016 14:50 *To:* [email protected] <mailto:[email protected]> *Subject:* Re: [easybuild] Using easybuild to manage different Python toolstacks as Alan suggests I think python bundle is the way to go for what you want. I would also suggest that if you want to assure reproducibility you should test your bundles in a machine without internet access. Some python libraries download extra dependencies during installation so if you run your easyconfig in a machine with internet access you won't notice which extra deps have been downloaded during the installation and if you run the same easyconfig in the future maybe different versions of the dependencies can be downloaded. If you try the easyconfig without internet access you would get an error in case any extra dependency needs to be download so you can check the error log and add the missing dependency to your bundle. The only procedure I know now to address this issue is by try&error in a machine without internet access. Some development is being done to address this in easybuild in a cleaner way but nothing available in the main easybuild release yet as far as I know. regards, Pablo. 2016-03-18 14:42 GMT+01:00 [email protected] <mailto:[email protected]> <[email protected] <mailto:[email protected]>>: Thanks, Alan, I'll look into this following your numba example. I'm just wondering, shouldn't for example the numpy version be part of the toolchain definition? Libraries like pandas and scipy use the numpy C stuff, if I remember correctly, so that a pandas build with numpy 1.9 is different from a pandas build with numpy 1.10. Not sure if numba from your example should also suffer from this or not. Cheers, Andreas. Alan O'Cais writes: > Dear Andreas, > > I was actually going to look into this soon. What you are describing (I think) is a Python package bundle for the SciPy Stack<http://www.scipy.org/stackspec.html>. An example of what that might look like is >https://github.com/hpcugent/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/n/numba/numba-0.22.1-intel-2015b-Python-2.7.11.eb > The Python dependency is given by the version suffix and the blas dependency is provided by the toolchain itself (a toolchain is compiler+mpi+blas/lapack/fftw) > > You could work on this yourself based on that example and I'd be happy to help if you run into issues. > > Alan > > On 18 March 2016 at 14:20, <[email protected] <mailto:[email protected]><mailto:[email protected] <mailto:[email protected]>>> wrote: > I recently stumbled across easybuild, it's an amazing piece of > software -- thanks for making this available! > > I'm wondering if there is any documentation / hints available for > managing different version of the Python scientific toolstack with > easybuild. I'm thinking of toolchains like > "py27-np19-sp016-pd017-openblas" -- you get my point. So basically I'm > looking at using easybuild as a replacement for virtualenvs / Anaconda > envs. > > Is there anything I need to consider for this use case? > > Cheers, > Andreas. > > -- > Dr. Andreas Hilboll > > Center for Marine Environmental Sciences (MARUM) > - AND - > Institute of Environmental Physics (IUP) > > University of Bremen > > U3145 > Otto-Hahn-Allee 1 > D-28359 Bremen > Germany > > +49(0)421 218 62133 <tel:%2B49%280%29421%20218%2062133><tel:%2B49%280%29421%20218%2062133> (phone) > +49(0)421 218 98 62133 <tel:%2B49%280%29421%20218%2098%2062133><tel:%2B49%280%29421%20218%2098%2062133> (fax) > http://www.iup.uni-bremen.de/~hilboll <http://www.iup.uni-bremen.de/%7Ehilboll> -- Dr. Andreas Hilboll Center for Marine Environmental Sciences (MARUM) - AND - Institute of Environmental Physics (IUP) University of Bremen U3145 Otto-Hahn-Allee 1 D-28359 Bremen Germany +49(0)421 218 62133 <tel:%2B49%280%29421%20218%2062133> (phone) +49(0)421 218 98 62133 <tel:%2B49%280%29421%20218%2098%2062133> (fax) http://www.iup.uni-bremen.de/~hilboll <http://www.iup.uni-bremen.de/%7Ehilboll>-- Pablo Escobar LópezHPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
