Re: [easybuild] building mxnet

[Ward Poelmans]

On 27-06-17 11:41, Jure Pečar wrote:
> On Mon, 8 May 2017 12:27:55 +0200

> Just haswell remaining ... and boom. Sanity check for python fails with:
> 
> == 2017-06-27 11:05:27,003 extension.py:181 WARNING Extension: MXNet failed 
> to install, cmd 
> '/g/easybuild/x86_64/CentOS/7/haswell/software/Python/2.7.12-foss-2016b/bin/python
>  -c "import mxnet"' (stdin: None) output: terminate called after throwing an 
> instance of 'std::system_error'
>   what():  Operation not permitted
> 
> Which is something I'm seeing for the first time. Google tells me that is is 
> somehow related to -pthread or a bug in the code itself. Both seems unlikely 
> since builds for different architectures passed.
> 
> Any hint how to figure out this one?

Unfortunately, no. You might try running it with strace but I'm not sure
if that will tell you much. You can also try a newer version of MXNet?
If it's a bug in de code, it might have been fixed already.

We've build it on haswell without issue.


Ward

Re: [easybuild] building mxnet

[Jure Pečar]

On Mon, 8 May 2017 12:27:55 +0200
Ward Poelmans  wrote:

> OK, they did something I didn't expect. Apparently commit 0d64855f741e04
> for nnvm was also added to mxnet itself. This makes that EB downloads
> the wrong tarball...
> 
> I've changed the commit I use of nnvm for now. I will look into the
> framework so we can specify from which url which tarball should be
> downloaded.

This now works fine. I've build mxnet from eb 3.3.0 for nehalem and sandybridge 
and sanity checks are passing, yay! :)

Just haswell remaining ... and boom. Sanity check for python fails with:

== 2017-06-27 11:05:27,003 extension.py:181 WARNING Extension: MXNet failed to 
install, cmd 
'/g/easybuild/x86_64/CentOS/7/haswell/software/Python/2.7.12-foss-2016b/bin/python
 -c "import mxnet"' (stdin: None) output: terminate called after throwing an 
instance of 'std::system_error'
  what():  Operation not permitted

Which is something I'm seeing for the first time. Google tells me that is is 
somehow related to -pthread or a bug in the code itself. Both seems unlikely 
since builds for different architectures passed.

Any hint how to figure out this one?


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401

Re: [easybuild] building mxnet

[Ward Poelmans]

On 02-05-17 17:13, Jure Pečar wrote:
> On Tue, 2 May 2017 15:36:27 +0200
> Ward Poelmans  wrote:
> 
>> Hi Jure,
>>
>> There are now an easyblock [1] and easyconfig [2] for MXNet.
>>
>> Can you give them a spin?
> 
> Nice ... I'll have to learn from this example how to handle submodules :)
> 
> 0d64855f741e04.tar.gz I find downloaded in the sources dir has a hash of 
> 4d83e41d97419ce04e8394dbb8893042, not fb04cb1694f1e66591a31a18bc92b451 ... 
> any idea why?

OK, they did something I didn't expect. Apparently commit 0d64855f741e04
for nnvm was also added to mxnet itself. This makes that EB downloads
the wrong tarball...

I've changed the commit I use of nnvm for now. I will look into the
framework so we can specify from which url which tarball should be
downloaded.

Ward

Re: [easybuild] building mxnet

[Jure Pečar]

On Tue, 2 May 2017 15:36:27 +0200
Ward Poelmans  wrote:

> Hi Jure,
> 
> There are now an easyblock [1] and easyconfig [2] for MXNet.
> 
> Can you give them a spin?

Nice ... I'll have to learn from this example how to handle submodules :)

0d64855f741e04.tar.gz I find downloaded in the sources dir has a hash of 
4d83e41d97419ce04e8394dbb8893042, not fb04cb1694f1e66591a31a18bc92b451 ... any 
idea why?

Then I get this:

== temporary log file in case of crash /tmp/eb-Z536yA/easybuild-eYSQ3h.log
== processing EasyBuild easyconfig 
/g/its/home/pecar/eb/MXNet-0.9.3-foss-2016b-Python-2.7.12.eb
== building and installing MXNet/0.9.3-foss-2016b-Python-2.7.12-R-3.3.1...
== fetching files...
== creating build dir, resetting environment...
== unpacking...
== FAILED: Installation ended unsuccessfully (build directory: 
/tmp/eb-build/MXNet/0.9.3/foss-2016b-Python-2.7.12-R-3.3.1): build failed 
(first 300 chars): Failed to find/isolate MXNet source directory: 
['/tmp/eb-build/MXNet/0.9.3/foss-2016b-Python-2.7.12-R-3.3.1/mxnet-0d64855f741e0482be7a3dfecde05f290bfec85d',
 '/tmp/eb-build/MXNet/0.9.3/foss-2016b-Python-2.7.12-R-3.3.1/mxnet-0.9.3']

Which I don't fully, since both of these two directories are present. 
Could be related to different 0d64855f741e04.tar.gz I'm getting?


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401

Re: [easybuild] building mxnet

[Ward Poelmans]

On 25-01-17 11:51, Jure Pečar wrote:
> 
> Hi all,
> 
> I'm trying to build mxnet thru EasyBuild. It's quite a challenge ...

Hi Jure,

There are now an easyblock [1] and easyconfig [2] for MXNet.

Can you give them a spin?

[1] https://github.com/hpcugent/easybuild-easyconfigs/pull/4346
[2] https://github.com/hpcugent/easybuild-easyblocks/pull/1135

Ward

Re: [easybuild] building mxnet

[Balázs Hajgató]

I have https://github.com/hpcugent/easybuild-easyconfigs/pull/4127 
(WIP), the core lib is done. I have to do the python bindings then it is 
done. The next one will be the R.




On 15/02/2017 09:24, Jure Pečar wrote:

On Tue, 14 Feb 2017 19:54:11 +0100
Kenneth Hoste  wrote:


Balázs (who is now also subscribed to this list) was working on mxnet
during the hackathon, maybe he has some useful information on this...

I now have it mostly done (the ugly and half manual way), there's just one 
detail I need to figure out to get R extension working properly:
https://github.com/dmlc/mxnet/issues/4870





--
HPC/VSC Support and System Administration
Computing Center
ULB/VUB
Avenue Adolphe Buyllaan 91 - CP 197
1050 Brussels
Belgium

Re: [easybuild] building mxnet

[Kenneth Hoste]

Hi Jurij,

On 06/02/2017 14:55, Jure Pečar wrote:

On Sat, 28 Jan 2017 17:43:42 +0100
Kenneth Hoste  wrote:


If there are releases available for each of the submodules, you can make
EB pulls those in, and then you could reassemble things in the build dir
using symlinks via 'preconfigopts' for example in the easyconfig?

That would be one way to automate this while retaining control over
which sources are actually used...
  
Interesting idea. Will play with it when I get some time.
  

You could just get that R library built via the extensions mechanism in
EasyBuild, and just make it part of the same installation?

I'm exploring this now ... and have a bit of a chicken-and-egg issue.

If you check mxnet docs at 
http://mxnet.io/get_started/ubuntu_setup.html#install-the-mxnet-package-for-r
you'll see that the steps to install the R-package of mxnet are:
* build mxnet
* Rscript install_deps (which installs about 11 additional R packages beside 
the ones listed in R eb)
* make rpkg (which installs mxnet R package and only then creates a tarball ... 
and tarball is created by R from mxnet:::mxnet.export ... grrr)
* at that point I have a tarball that I can push to sources dir and use as a 
source of R extension

I don't see how I could simply map this to existing easybuild facilities.

Maybe I can tear apart the "make rpkg" thing and try to reimplement it in .eb 
file ... But still I have to get those required R packages installed first. Any 
suggestions?

Maybe we can discuss this on Thursday ...


Balázs (who is now also subscribed to this list) was working on mxnet 
during the hackathon, maybe he has some useful information on this...



regards,

Kenneth

Re: [easybuild] building mxnet

[Jure Pečar]

On Sat, 28 Jan 2017 17:43:42 +0100
Kenneth Hoste  wrote:

> If there are releases available for each of the submodules, you can make 
> EB pulls those in, and then you could reassemble things in the build dir 
> using symlinks via 'preconfigopts' for example in the easyconfig?
> 
> That would be one way to automate this while retaining control over 
> which sources are actually used...
 
Interesting idea. Will play with it when I get some time.
 
> You could just get that R library built via the extensions mechanism in 
> EasyBuild, and just make it part of the same installation?

I'm exploring this now ... and have a bit of a chicken-and-egg issue.

If you check mxnet docs at 
http://mxnet.io/get_started/ubuntu_setup.html#install-the-mxnet-package-for-r
you'll see that the steps to install the R-package of mxnet are:
* build mxnet
* Rscript install_deps (which installs about 11 additional R packages beside 
the ones listed in R eb)
* make rpkg (which installs mxnet R package and only then creates a tarball ... 
and tarball is created by R from mxnet:::mxnet.export ... grrr)
* at that point I have a tarball that I can push to sources dir and use as a 
source of R extension

I don't see how I could simply map this to existing easybuild facilities.

Maybe I can tear apart the "make rpkg" thing and try to reimplement it in .eb 
file ... But still I have to get those required R packages installed first. Any 
suggestions?

Maybe we can discuss this on Thursday ...


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401

Re: [easybuild] building mxnet

[Kenneth Hoste]

On 25/01/2017 15:28, Jure Pečar wrote:

On Wed, 25 Jan 2017 12:04:19 +0100
Alan O'Cais  wrote:


Hi Jure,

What exactly is the list of everything required? I'd be surprised if they 
excluded necessary sources from their releases. Perhaps you just need to add 
the appropriate dependencies to your  easyconfig and tell mxnet how to find 
them (if it can't auto-detect them)?

Hm ... not sure if packaging every submodule as a eb project and loadable 
module would work ... since make itself fails if it can't find files in 
appropriate subdirs.

Right now I'm downloading the whole thing manually and stuffing the tarball 
into eb sources directory where it picks it up and doesn't try to go online and 
download it. But I would like to automate this on the long term ...


If there are releases available for each of the submodules, you can make 
EB pulls those in, and then you could reassemble things in the build dir 
using symlinks via 'preconfigopts' for example in the easyconfig?


That would be one way to automate this while retaining control over 
which sources are actually used...




Next question.
Mxnet comes with a subfolder "R-package" which has sources to build R module. 
Is there a way to make this available as a separate module from the same eb file?

Right now I'm thinking of rolling a tarball  with just that folder as a source 
for additional eb file built around Rpackage easyblock. Would that be accepted?


That would be one way, but why do it as a separate module?

You could just get that R library built via the extensions mechanism in 
EasyBuild, and just make it part of the same installation?


A bit like how we install recent versions of matplotlib (see [1]), a 
major difference being that there only stuff via extensions is being 
installed, nothing else.



regards,

Kenneth


[1] 
https://github.com/hpcugent/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/m/matplotlib/matplotlib-1.5.3-intel-2016b-Python-3.5.2.eb

Re: [easybuild] building mxnet

[Jure Pečar]

On Wed, 25 Jan 2017 12:04:19 +0100
Alan O'Cais  wrote:

> Hi Jure,
> 
> What exactly is the list of everything required? I'd be surprised if they 
> excluded necessary sources from their releases. Perhaps you just need to add 
> the appropriate dependencies to your  easyconfig and tell mxnet how to find 
> them (if it can't auto-detect them)?

Hm ... not sure if packaging every submodule as a eb project and loadable 
module would work ... since make itself fails if it can't find files in 
appropriate subdirs.

Right now I'm downloading the whole thing manually and stuffing the tarball 
into eb sources directory where it picks it up and doesn't try to go online and 
download it. But I would like to automate this on the long term ...


Next question.
Mxnet comes with a subfolder "R-package" which has sources to build R module. 
Is there a way to make this available as a separate module from the same eb 
file?

Right now I'm thinking of rolling a tarball  with just that folder as a source 
for additional eb file built around Rpackage easyblock. Would that be accepted? 


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401

Re: [easybuild] building mxnet

[Alan O'Cais]

Hi Jure,

What exactly is the list of everything required? I'd be surprised if they 
excluded necessary sources from their releases. Perhaps you just need to add 
the appropriate dependencies to your  easyconfig and tell mxnet how to find 
them (if it can't auto-detect them)?

Alan

On 25 January 2017 at 11:51, Jure Pečar 
> wrote:

Hi all,

I'm trying to build mxnet thru EasyBuild. It's quite a challenge ...

First question:

mxnet docs tell you to do git clone https://github.com/dmlc/mxnet.git ~/mxnet 
--recursive to download the source. If I try to specify
sources = ['%(name)s-%(version)s.tar.gz']
source_urls = ['https://github.com/dmlc/mxnet/archive']
the tarball downloaded does not include everything required to build mxnet.

>From the number of scripts floating around online that automate downloading of 
>all submodules (just one example: 
>http://minhajuddin.com/2016/01/10/how-to-get-a-git-archive-including-submodules/)
> I conclude that github api does not offer the ability we want. What is the 
>preferred EasyBuild way to download source organized in this way?


--

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401



--
Dr. Alan O'Cais
E-CAM Software Manager
Juelich Supercomputing Centre
Forschungszentrum Juelich GmbH
52425 Juelich, Germany

Phone: +49 2461 61 5213
Fax: +49 2461 61 6656
E-mail: a.oc...@fz-juelich.de
WWW:http://www.fz-juelich.de/ias/jsc/EN




Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt

[easybuild] building mxnet

[Jure Pečar]

Hi all,

I'm trying to build mxnet thru EasyBuild. It's quite a challenge ...

First question:

mxnet docs tell you to do git clone https://github.com/dmlc/mxnet.git ~/mxnet 
--recursive to download the source. If I try to specify 
sources = ['%(name)s-%(version)s.tar.gz']
source_urls = ['https://github.com/dmlc/mxnet/archive']
the tarball downloaded does not include everything required to build mxnet.

>From the number of scripts floating around online that automate downloading of 
>all submodules (just one example: 
>http://minhajuddin.com/2016/01/10/how-to-get-a-git-archive-including-submodules/)
> I conclude that github api does not offer the ability we want. What is the 
>preferred EasyBuild way to download source organized in this way?


-- 

  Jurij Pečar
  HPC Engineer, IT Operations, IT Services
  EMBL Heidelberg, Meyerhofstraße 1, 69117, Heidelberg, Germany
  Room 13-401