Hi Cormac, (replies inline)
On 24 Dec 2013, at 19:14, Garvey, Cormac T wrote: > Hi Kenneth, > > Here is the structure of my private easyblocks, easyconfigs and toolchains. (I trimmed down the below to the important bits) > > └── easybuild > ├── easyblocks > │ ├── __init__.py > │ ├── mcnp5.py > │ ├── mcnp5.pyc > ├── easyconfigs > │ ├── ... > ├── __init__.py > ├── tbb.py > └── tbb.pyc > > > > When I remove __init__.py I can use my private easyblocks. > When I execute "eb --list-easyblocks" I do see my private easyblocks. > > garvct@eos:~> eb --list-easyblocks > EasyBlock ... > | |-- EB_MCNP5 ... > My PYTHONPATH looks like > > echo $PYTHONPATH > > /hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages:/hpc-common/software/easybuild/inl-easyconfigs I just played around with this, and you're right: there doesn't need to be a __init__.py present in the easybuild directory for your own easyblocks to be available to EasyBuild. It turns out the __init__.py is only required when you need to import stuff directory from that namespace. Imports deeper down do work: (no __init__.py in the easybuild dir at first in the example below) $ touch $HOME/my_easyblocks/easybuild/foo.py $ touch$HOME/my_easyblocks/easybuild/easyblocks/foo.py $ python -c "import easybuild.easyblocks.foo" $ python -c "from easybuild.easyblocks import foo" $ python -c "import easybuild.foo" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named foo $ touch $HOME/my_easyblocks/easybuild/__init__.py $ python -c "import easybuild.foo" This surprised us, but it works, so good! :-) I've adjusted the documentation at https://github.com/hpcugent/easybuild/wiki/Setting-up-your-own-easyblocks-repository to reflect this, thanks a lot for bringing this forward and reporting back. > If I include the path of the easyconfig files which come with EB, then EB > cannot find my private easyconfig files? > > e.g > echo $PYTHONPATH > > /hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages:/hpc-common/software/easybuild/1.9.0/gcc/opt:/hpc-common/software/easybuild/inl-easyconfigs > > eb MVAPICH2-1.7-GCC-4.7.3-inl.eb > == temporary log file in case of crash /tmp/easybuild-1iSTLA.log > == ERROR: Can't find path /home/garvct/MVAPICH2-1.7-GCC-4.7.3-inl.eb $PYTHONPATH is only used by --robot for finding easyconfig files, not for the files directory specified on the command line. In my view, changing that would create too much confusion/surprises when using easyconfig files like "test.eb" for example. If you happen to be in a directory without a test.eb file in it (by accident), EasyBuild might succeed in finding a test.eb somewhere else by searching $PYTHONPATH, which could lead to very surprising situations. For --robot, that's a different story, because in that case it's looking for a file with a very particular filename, so it's unlikely it finds and uses something incorrect (especially because EB will complain if the robot found an easyconfig file that doesn't provide what its filename suggests it does). The files you specify on the command line should be found at the (relative or absolute) path you're specifying; if not, EB will stop right there. Do let us know if you feel differently about this... regards, Kenneth > > Thanks, > Cormac. > > > On Mon, Dec 23, 2013 at 10:41 AM, Kenneth Hoste <[email protected]> > wrote: > Hi Cormac, > > On 19 Dec 2013, at 23:47, Garvey, Cormac T wrote: > > > Hi All, > > On the Easybuild wiki page "Setting up own easyblock repository", it > > instructs to > > put __init__.py in the easybuild directory (e.g touch > > $MYEBDIR/easybuild/__init__.py). > > > > When I do that I get the following easybuild error. > > > > Traceback (most recent call last): > > File > > "/hpc-common/software/easybuild/1.9.0/gcc/opt/lib/python2.7/site-packages/easybuild/main.py", > > line 80, in <module> > > from easybuild.tools.build_log import EasyBuildError, print_msg, > > print_error, print_warning > > ImportError: No module named tools.build_log > > > > Removing __init__.py from the $MYEBDIR/easybuild directory works (i.e My > > easyblocks can co-exist with the default Easybuild > > easyblocks.) > > When you say things work when you simply remove the __init__.py, is EasyBuils > still able to find your 'private' easyblocks? > I.e., does "eb --list-easyblocks" include your easyblocks as well (they > really shouldn't, I would be really surprised if they are)? > > The __init__.py really needs to be there to declare the 'easybuild' > namespace, such that 'from easybuild.easyblocks.foo import EB_Foo' can work > (with Foo a private easyblock). > > There is one nasty detail though, and I guess you're being bitten by this. > The path with your private easyblocks needs to be added *last* to > $PYTHONPATH, rather than first. > This is also described on the "Setting up your own easyblock repository" wiki > page at > https://github.com/hpcugent/easybuild/wiki/Setting-up-your-own-easyblocks-repository: > > "Note: You need to add the path to your easyblocks at the end, not at the > beginning (see below as to why)." > > The reason why is also explained on that page, but it boils down to use > flattening the easyblocks namespace from e.g. 'easybuild.easyblocks.g.gcc' to > 'easybuid,easyblocks.gcc'. > This trick only works *once* though, and only when your easybuild.easyblocks > namespace is first in $PYTHONPATH. > > I hope this helps, do let us know how this works out. > > > regards, > > Kenneth > > > > -- > Cormac Garvey > HPC Software Consultant > Scientific Computing > Idaho National Laboratory > Ph: 208-526-6294 >
