On Wed, 4 Mar 2015 at 16:06 -0000, Kenneth Hoste wrote: > Unfortunately, there's no easy way to prevent this auto-downloading > and installing dependencies when installing a Python package with > "python setup.py install".
Excuse the following paragraph of ranting, easybuild is still looking like a good and useful tool. I'm an old curmudgeon who does not like some of the trends on the Internet as a whole. I find it disgusting that a lot of modern software will just arbitrarily download things from the Internet. Perl started doing some of this late in it's life and is somewhat controllable. Python seems to have embraced this early in it's life. I believe that software builders really need to know and explicitly state/request all of the libraries their software uses. Not doing this can build large, complex or other long term issues into software without anyone knowing. This issue is part of why I support our systems not having a direct Internet connection. > There are some ways (e.g., doing the build in a jail-like > environment), but they're not trivial. My initial hope was that easybuild supported building software outside of the live environment (mock, chroot, fakeroot). This is not the case and I can live with it for now but I hope this is/becomes part of the future direction. > I'll see what I can do to get six and ecdsa properly included where > needed, as dependencies for dateutil and paramiko, resp., thanks for > bringing this up. thanks. > Just to clarify: the easyconfigs that are part of the EasyBuild > installation are meant to be treated as examples (which does not > mean they should contain inconsistencies like the one you reported). This is also where I find the current easybuild process a little awkward. I generally prefer examples to be out of the main execution path. i.e. If these are really examples, they should not be used automatically. If they are a default configuration, they should work and be maintained (which is a lot of work for 2000+ configurations). I presume this can be addressed by not installing the easyconfig package, but I would still like to have the examples handy for reference. It is a little more unclear how easyblocks are to be treated. So far I have not had to look at or deal with any of the easyblocks. Again, easybuild is very helpful, the current documented quick start process and built in easyconfigs are very helpful to prove the usefulness of easybuild and get started with a couple of applications. This is allowing me to fairly quickly build an initial test installation of a large set of applications. After things settle down I expect to have a consistent collection of easyconfig files in the ebfiles_repo directory. I do notice that these files have an extra buildstats clause in them which causes minor difficulties diffing my original files against the built versions to ensure consistency. > So, feel free to adjust easyconfigs to your liking for your local > installations. This is what I'm doing and I expect to spend considerable time building our first production set of applications. Easybuild does cover a large portion of our user software needs. At this time I have no need/desire to address any application unless it already has an example easyconfig (in the future I do expect to start adding other or unique software). A follow on issue is when to update our private configuration with new versions of software. I hope this or the HPCBIOS project can provide some backend support for this. > > I also see a python-dateutil-2.1-goolf-1.4.10-Python-2.7.3.eb > > configuration file. It looks like this would be redundant with > > the current easybuild files, but could serve as a good example for > > users building python addons for individual use. > > This one mainly serves as an example, yes. > > However, there are other example of similar easyconfigs that do make > sense. > > See numpy-1.7.1-goolf-1.4.10-Python-2.7.3.eb for example. The numpy > that is included in Python-2.7.3-goolf-1.4.10.eb is older (v1.6.1), > so a seperate easyconfig file for numpy allows to install a newer > version of numpy than the one that comes with the Python module. I did see that numpy and scipy and a few others were already included in the Python build. I didn't notice the version numbers differences but it is good to see support for other versions available. Can these modules be loaded on top of a python already containing older versions? What is current thinking on whether things like scipy should be built into the python build or be separate modules? For now I'm planning to build a single consistent set of software (with just the goolf toolchain). Every 6-8 months or so would be an updated release with freshened software. My expectation is that most users should be able to use any fairly recent version of things in our easybuild software set. I don't plan to add new software until it has reasonable maturity. Any user needing a different version would be on their own. It sounds like they could build their own easybuild based add on packages but I don't plan to immediately support. My real hope (but not expectation) is that scientific software developers will improve their processes to better address compatibility between software versions. > > These changes just allow building to complete successfully. I > > have not started testing any applications beyond the build > > process. > > Some builds include tests, like the numpy easyblock that checks the > performance of numpy.dot in an attempt to ensure that numpy was > correctly linked with BLAS (see > https://github.com/hpcugent/easybuild-easyblocks/blob/master/easybuild/easyblocks/n/numpy.py#L197). > And that's beside the unit test suite which is also run. > > Lately, we've been paying more attention to including (simple) tests > in the build procedure as executed by EasyBuild, but we should do > this more, where possible. This is good and I do notice test cases being run in the log files. I'm still getting used to the log files and am finding some issues in their usability, but that is subject for a separate discussing at another time. Other things I plan to do are building documentation, demonstration, example and performance cases for each major software package. In part, this becomes site specific but I also hope for some support from the HPCBIOS and similar efforts. Stuart -- I've never been lost; I was once bewildered for three days, but never lost! -- Daniel Boone
