Hi Ken et al, please, while you are at it, could you check that whatever the outcome it can really run stand-alone in off-line mode, without asking online for “distribute*"? (or, make that a tuneable)
>1yr ago, while I experimented on delivering EB in a module-less and offline >environment [1], I remember I had to spin-up a local web-server to feed distribute_setup.py or such. If I recall right, the first stage would not need it but the next stage then required it (!), so some extra hackery was required to overcome the dependency. I had to find out first-hand because I was at the bottom of the pit, right here [2]. The good news are that, yes, it worked! I’d agree this would be a fine subject for a hackathon.... t.b.c. F. [1] https://github.com/fgeorgatos/easybuild.experimental/tree/master/users/fgeorgatos/eb_boot [2] http://www.tunneltalk.com/images/Channel-Tunnel/Geological-section-of-the-53km-long-Channel.jpg On Jun 13, 2016, at 1:12 PM, Kenneth Hoste <[email protected]> wrote: > Hi Stuart, > > I patched the EasyBuildMeta easyblock so that the vsc-install source tarball > is also picked up, which should result into supporting offline installation > of EasyBuild-with-EasyBuild (note: the bootstrap script currently still > requires to be online). > > Can you give this a try? > > You can download the updated easyblock from [1] (see [2] for raw link), and > use the updated easyconfig from [3] (see [4] for raw link), > and then run this on an offline system: > > eb EasyBuild-2.8.1.eb --include-easyblocks $PWD/easybuildmeta.py > > Please let me know how that turns out. > > regards, > > Kenneth > > [1] https://github.com/hpcugent/easybuild-easyblocks/pull/957 > [2] > https://raw.githubusercontent.com/boegel/easybuild-easyblocks/vsc_install_easybuildmeta/easybuild/easyblocks/e/easybuildmeta.py > [3] https://github.com/hpcugent/easybuild-easyconfigs/pull/3203 > [4] > https://raw.githubusercontent.com/boegel/easybuild-easyconfigs/eb281_vsc_install/easybuild/easyconfigs/e/EasyBuild/EasyBuild-2.8.1.eb > > > On 11/06/16 22:44, Kenneth Hoste wrote: >> Hi Stuart, >> >> Again, thanks for the feedback! >> >> On 11/06/16 20:22, Stuart Barkley wrote: >>> I'm having another issue freshening up my easybuild environment. >>> >>> My cluster is not directly Internet connected and compute nodes do not >>> have any Internet access at all. In the past I've been able to build >>> easybuild based programs once I've externally fetched the sources and >>> placed them in a source tree. >>> >>> With 2.8.1 (and I think starting with 2.7.0) I'm now unable to rebuild >>> easybuild itself into its final place. There seems to be at least one >>> failing connection attempt in the easybuild build process itself. >>> >>> Hand transcribed output from the error log: >>> >>> == testing... >>> == installimg... >>> == FAILED: Installation ended unsuccessfully...300 chars: cmd " >>> /usr/bin/python setup.py install --prefix.... >>> >>> Hand transcribed portion of the --keep-log log file: >>> >>> ERROR cmd " /usr/bin/python setup.py install --prefix.... >>> This is based on vsc.install.shared_setup 0.10.6 >>> ... >>> Searching for vsc-install>=0.10.1 >>> Reading http://pypi.python.org/simple/vsc-install/ >>> WARN: Download error: [errno -2] name or service not found..... >>> >>> On my external VM I see the following file/directory being fetched: >>> >>> GET "http://pypi.python.org/simple/vsc-install/"; "Python-urllib/2.6 >>> distribute/0.6.10" >>> >>> It only seems to read the directory and not download any actual files. >>> On the non-Internet connected system this seems to be the cause of the >>> installation abort. >>> >>> Is vsc-install required? It doesn't actually seem to download it. I >>> thought it was actually packaged in easybuild itself. I think there >>> have been issues in the past with vsc-base requirements. I don't see >>> a CentOS RPM that provides this functionality (but may have missed >>> it). >> >> vsc-install is currently a runtime requirement for vsc-base, although only >> for 'obscure' reasons. >> >> It is a dependency for EasyBuild itself as well, but only to run the unit >> tests that are included in the EasyBuild installation (cfr. >> http://easybuild.readthedocs.io/en/latest/Unit-tests.html). >> >> So yes, vsc-install is required, and the installation process will download >> it (albeit indirectly via setuptools, which will try and pull it from PyPI). >> >> >>> I have tried placing the current vsc-install source tarball in my >>> sources directory at various places that seem likely but haven't found >>> a path which works (v/, v/vsc-install/, in a python modules >>> directory). Is there somewhere I can drop the source file to work >>> around this problem? >> >> Currently, that won't work, because vsc-install isn't listed as a direct >> dependency of EasyBuild; it's pulled in indirectly as a dependency of >> vsc-base by whichever Python install tool that is used (e.g. setuptools when >> installing EB with EB). >> >> Of course, we should fix this. >> >> I've opened https://github.com/hpcugent/easybuild-framework/issues/1803 to >> track this. >> >> BTW: source tarballs should go into the sourcepath for the 'parent' project; >> so in this case, you should place the source tarballs for vsc-base (and soon >> vsc-install) into <sourcepath>/e/EasyBuild/. >> >>> Please excuse the vagueness of this report. Getting machine readable >>> data out from my build system is very awkward at the present time. >>> >>> ======== >>> >>> For reference (in the past) I have had reasonable success with build >>> on a non-Internet connected system with the following process (again >>> waving hands a little bit since I'm not on my actual build system at >>> the moment): >>> >>> I build a scratch KVM Based virtual machine with Internet access. I >>> can then download and execute the easybuild bootstrap script. >>> Alternatively, I can copy over an already installed easybuild >>> installation and use it. >>> >>> I can then use the --stop=fetch (along with the occasional --robot and >>> --force options) to get easybuild to download most source distribution >>> files. >>> >>> I always use a constant --sourcepath option so I accumulate a cache of >>> downloaded sources which I can transfer to the build system. >>> >>> This works for most of the sources, however there are a couple of >>> exceptions where this doesn't work (I think these actually apply even >>> if you are Internet connected): >>> >>> - Certain files require a manual download, possibly accepting a >>> license agreement through a web interface. In some cases finding the >>> download link is difficult and easybuild could help by giving an >>> accurate starting link (some do, some are missing or wrong). >> >> We usually try to provide information on where to fetch sources from in the >> easyconfig file itself, above the 'sources' line. >> >>> - Occasionally a file distribution location has changed and the .eb >>> files needs to be updated (or the source file manually downloaded). >> >> Please report issues like these via >> https://github.com/hpcugent/easybuild-easyconfigs/issues . >> >>> - Source distribution files "disappear". I keep a long term cache of >>> all source distribution files I've previously downloaded. >> There have been discussions on a central 'archive' set up by the EasyBuild >> community for things software that does this, but it hasn't materialized >> yet. Redistributing sources and the possible legal aspects involved with >> that certainly haven't helped... >> >> We do try an reach out to upstream to remedy this though. Sometimes we fail >> (*cough* Bioconductor *cough*), but sometimes it works out (e.g. MVAPICH2, >> cfr. https://github.com/hpcugent/easybuild-easyconfigs/issues/973). >> >>> - Other than the above, I don't think I've run into anything which >>> actually requires Internet access at build time if you have all the >>> source tarballs. Anything needing Internet access at build/install >>> time is not supplying complete sources and I consider broken. >> >> I agree, and I don't like it either that EasyBuild itself currently falls >> into this category. >> >> There are examples I can come up with though that require online access >> because they download stuff at build time (ParaView, Amber, ...). >> >> >>> Even with having source tarballs, you may not have the actual sources. >>> A disturbing number of distribution tarballs are now coming with >>> precompiled binaries and not true sources. >> >> Any examples of that? >> >> >> regards, >> >> Kenneth > cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
