2016-03-18 15:11 GMT+01:00 Fotis Georgatos <[email protected]>:
> > ie. we might be able to stop most of the network activity in user-space, > w/out requiring blockage at root level (there are some caveats, agreed), > by rewiring certain calls. thoughts? > > Not sure if adding and external dependency to easybuild and using LD_PRELOAD is a good idea. I think Kenneth was looking at a solution using python but I don't remember the details. I think he should decide about this. > F. > > -- > > echo "sysadmin know better bash than english" | sed s/min/mins/ \ > | sed 's/better bash/bash better/' # Yelling in a CERN forum > > ------------------------------ > *From:* [email protected] [[email protected]] > on behalf of Pablo Escobar Lopez [[email protected]] > *Sent:* Friday, March 18, 2016 14:50 > *To:* [email protected] > *Subject:* Re: [easybuild] Using easybuild to manage different Python > toolstacks > > as Alan suggests I think python bundle is the way to go for what you want. > > I would also suggest that if you want to assure reproducibility you should > test your bundles in a machine without internet access. Some python > libraries download extra dependencies during installation so if you run > your easyconfig in a machine with internet access you won't notice which > extra deps have been downloaded during the installation and if you run the > same easyconfig in the future maybe different versions of the dependencies > can be downloaded. If you try the easyconfig without internet access you > would get an error in case any extra dependency needs to be download so you > can check the error log and add the missing dependency to your bundle. The > only procedure I know now to address this issue is by try&error in a > machine without internet access. Some development is being done to address > this in easybuild in a cleaner way but nothing available in the main > easybuild release yet as far as I know. > > regards, > Pablo. > > 2016-03-18 14:42 GMT+01:00 [email protected] <[email protected]>: > >> Thanks, Alan, >> >> I'll look into this following your numba example. >> >> I'm just wondering, shouldn't for example the numpy version be part of >> the toolchain definition? Libraries like pandas and scipy use the numpy >> C stuff, if I remember correctly, so that a pandas build with numpy 1.9 >> is different from a pandas build with numpy 1.10. Not sure if numba >> from your example should also suffer from this or not. >> >> Cheers, >> Andreas. >> >> >> >> Alan O'Cais writes: >> >> > Dear Andreas, >> > >> > I was actually going to look into this soon. What you are describing (I >> think) is a Python package bundle for the SciPy Stack< >> http://www.scipy.org/stackspec.html>. An example of what that might look >> like is >> > >> https://github.com/hpcugent/easybuild-easyconfigs/blob/develop/easybuild/easyconfigs/n/numba/numba-0.22.1-intel-2015b-Python-2.7.11.eb >> > The Python dependency is given by the version suffix and the blas >> dependency is provided by the toolchain itself (a toolchain is >> compiler+mpi+blas/lapack/fftw) >> > >> > You could work on this yourself based on that example and I'd be happy >> to help if you run into issues. >> > >> > Alan >> > >> > On 18 March 2016 at 14:20, <[email protected]<mailto: >> [email protected]>> wrote: >> > I recently stumbled across easybuild, it's an amazing piece of >> > software -- thanks for making this available! >> > >> > I'm wondering if there is any documentation / hints available for >> > managing different version of the Python scientific toolstack with >> > easybuild. I'm thinking of toolchains like >> > "py27-np19-sp016-pd017-openblas" -- you get my point. So basically I'm >> > looking at using easybuild as a replacement for virtualenvs / Anaconda >> > envs. >> > >> > Is there anything I need to consider for this use case? >> > >> > Cheers, >> > Andreas. >> > >> > -- >> > Dr. Andreas Hilboll >> > >> > Center for Marine Environmental Sciences (MARUM) >> > - AND - >> > Institute of Environmental Physics (IUP) >> > >> > University of Bremen >> > >> > U3145 >> > Otto-Hahn-Allee 1 >> > D-28359 Bremen >> > Germany >> > >> > +49(0)421 218 62133<tel:%2B49%280%29421%20218%2062133> (phone) >> > +49(0)421 218 98 62133<tel:%2B49%280%29421%20218%2098%2062133> (fax) >> > http://www.iup.uni-bremen.de/~hilboll >> >> >> -- >> Dr. Andreas Hilboll >> >> Center for Marine Environmental Sciences (MARUM) >> - AND - >> Institute of Environmental Physics (IUP) >> >> University of Bremen >> >> U3145 >> Otto-Hahn-Allee 1 >> D-28359 Bremen >> Germany >> >> +49(0)421 218 62133 (phone) >> +49(0)421 218 98 62133 (fax) >> http://www.iup.uni-bremen.de/~hilboll >> > > > > -- > Pablo Escobar López > HPC systems engineer > sciCORE, University of Basel > SIB Swiss Institute of Bioinformatics > http://scicore.unibas.ch > -- Pablo Escobar López HPC systems engineer sciCORE, University of Basel SIB Swiss Institute of Bioinformatics http://scicore.unibas.ch
