Re: [easybuild] install dir on gpfs
On Jun 21, 2017, at 5:20 PM, Heywood, Todd wrote: > And Easybuid-built GCC-4.9.2 or earlier works. I think you changed things up > for 4.9.3 and later since GCCore and binutils-2.25 appeared in the process at > that point. This invites for differential debugging of multiple possible directions: * renaming the gcc-4.9.2 tarball to gcc-4.9.3 is a cheap and interesting experiment * try with/without binutils; fake that build via a nearly empty tarball or something that builds alike * try different versions of binutils; renaming the tarball files is often a cheap way to do that * try the same binutils version as the one on the underlying filesystem; result? * binutils build options could possibly be a factor as well (after, what is different from underlying OS?) Love these problems but they are a time sink. F. -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
Re: [easybuild] install dir on gpfs
Yes, GCC 4.9.3 built manually works fine: download gcc-4.9.3.tar.gz, configure, make , make install. And Easybuid-built GCC-4.9.2 or earlier works. I think you changed things up for 4.9.3 and later since GCCore and binutils-2.25 appeared in the process at that point. From: Kenneth Hoste mailto:kenneth.ho...@ugent.be>> Date: Wednesday, June 21, 2017 at 12:01 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>>, Todd Heywood mailto:heyw...@cshl.edu>> Subject: Re: [easybuild] install dir on gpfs On 21/06/2017 17:13, Heywood, Todd wrote: Changing the build directory to /tmp/eb-build got easybuild working again for me. The install directory is on GPFS. However, GCC compilers >= 4.9.3 built by Easybuild give the problem I described when the code being compiled is on GPFS. But not when that code is not on GPFS. Also built GCC 4.9.3 directly, without Easybuild, on GPFS, and that works fine Wait, are you saying that an EB-built GCC 4.9.3 is broken on GPFS, but one you built manually works fine? That's something to look into imho (although the expression "needle in a haystack" comes to mind for that one...). K. FYI. From: mailto:easybuild-requ...@lists.ugent.be><mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>> Date: Wednesday, June 21, 2017 at 9:45 AM To: Fotis Georgatos mailto:fo...@mail.cern.ch><mailto:fo...@mail.cern.ch>>, "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>> Subject: Re: [easybuild] install dir on gpfs Thanks for the replies, everyone. We have opened a ticket with DDN, through whom we have GPFS. It is version 4.2.1 I’m going to try a few things, like configure the build directory to be under /tmp. Funny thing is that GCC 4.9.2 works fine, but 4.9.3 and later do not. GCC 4.9.3 and later are built differently, right? With GCC/*-binutils-*. From: Fotis Georgatos mailto:fo...@mail.cern.ch><mailto:fo...@mail.cern.ch><mailto:fo...@mail.cern.ch>> Date: Wednesday, June 21, 2017 at 4:16 AM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>> Cc: Todd Heywood mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> Subject: Re: [easybuild] install dir on gpfs Hi, I feel for you :-p as a matter of fact, any parallel fs chimes in with its own quirks during a build process. More than a dozen years ago, GPFS/2.2 would not support symlinks from a given size and above, go figure! For this and few other reasons, I consider pointing EASYBUILD_BUILDPATH towards /dev/shm or /tmp or so, a totally essential move. I’d rather get a failure fi. because /dev/shm is full, than risk having variating builds across two different (versions of) parallel filesystems. Truth be told, there is a `make install` step which may still ruin the day; I am not isolating that yet, but it is not entirely impossible: use a container or RPM to deliver, then use that towards your final destination. Indeed, that’s a project of its own. On Jun 20, 2017, at 9:52 PM, Heywood, Todd mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> wrote: You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Finally, this seems to be yet another example of the tension between metadata consistency and need for speed. What is a matter of wonder though is, why the compiler version is important! It could be that the way the compiler opens the output file for writing makes some kind of difference. (weird.. strace is your friend) Thanks for reporting this! F. On Jun 21, 2017, at 6:52 AM, Kenneth Hoste mailto:kenneth.ho...@ugent.be><mailto:kenneth.ho...@ugent.be><mailto:kenneth.ho...@ugent.be>> wrote: Hi Todd, We ran into this issue too a while ago, but it wasn't this extreme, I was only able to reproduce this using OCaml... We reported this back to IBM, and we eventually applied a fix (GPFS update, I think, in Aug'16), that resolved the problem for us. I can try to get more information on this if it would be useful for you. regards, Kenneth On 20/06/2017 22:52, Heywood, Todd wrote: Has anyone i
Re: [easybuild] install dir on gpfs
On 21/06/2017 17:13, Heywood, Todd wrote: Changing the build directory to /tmp/eb-build got easybuild working again for me. The install directory is on GPFS. However, GCC compilers >= 4.9.3 built by Easybuild give the problem I described when the code being compiled is on GPFS. But not when that code is not on GPFS. Also built GCC 4.9.3 directly, without Easybuild, on GPFS, and that works fine Wait, are you saying that an EB-built GCC 4.9.3 is broken on GPFS, but one you built manually works fine? That's something to look into imho (although the expression "needle in a haystack" comes to mind for that one...). K. FYI. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Wednesday, June 21, 2017 at 9:45 AM To: Fotis Georgatos mailto:fo...@mail.cern.ch>>, "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: Re: [easybuild] install dir on gpfs Thanks for the replies, everyone. We have opened a ticket with DDN, through whom we have GPFS. It is version 4.2.1 I’m going to try a few things, like configure the build directory to be under /tmp. Funny thing is that GCC 4.9.2 works fine, but 4.9.3 and later do not. GCC 4.9.3 and later are built differently, right? With GCC/*-binutils-*. From: Fotis Georgatos mailto:fo...@mail.cern.ch><mailto:fo...@mail.cern.ch>> Date: Wednesday, June 21, 2017 at 4:16 AM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>> Cc: Todd Heywood mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> Subject: Re: [easybuild] install dir on gpfs Hi, I feel for you :-p as a matter of fact, any parallel fs chimes in with its own quirks during a build process. More than a dozen years ago, GPFS/2.2 would not support symlinks from a given size and above, go figure! For this and few other reasons, I consider pointing EASYBUILD_BUILDPATH towards /dev/shm or /tmp or so, a totally essential move. I’d rather get a failure fi. because /dev/shm is full, than risk having variating builds across two different (versions of) parallel filesystems. Truth be told, there is a `make install` step which may still ruin the day; I am not isolating that yet, but it is not entirely impossible: use a container or RPM to deliver, then use that towards your final destination. Indeed, that’s a project of its own. On Jun 20, 2017, at 9:52 PM, Heywood, Todd mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> wrote: You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Finally, this seems to be yet another example of the tension between metadata consistency and need for speed. What is a matter of wonder though is, why the compiler version is important! It could be that the way the compiler opens the output file for writing makes some kind of difference. (weird.. strace is your friend) Thanks for reporting this! F. On Jun 21, 2017, at 6:52 AM, Kenneth Hoste mailto:kenneth.ho...@ugent.be><mailto:kenneth.ho...@ugent.be>> wrote: Hi Todd, We ran into this issue too a while ago, but it wasn't this extreme, I was only able to reproduce this using OCaml... We reported this back to IBM, and we eventually applied a fix (GPFS update, I think, in Aug'16), that resolved the problem for us. I can try to get more information on this if it would be useful for you. regards, Kenneth On 20/06/2017 22:52, Heywood, Todd wrote: Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
Re: [easybuild] install dir on gpfs
Changing the build directory to /tmp/eb-build got easybuild working again for me. The install directory is on GPFS. However, GCC compilers >= 4.9.3 built by Easybuild give the problem I described when the code being compiled is on GPFS. But not when that code is not on GPFS. Also built GCC 4.9.3 directly, without Easybuild, on GPFS, and that works fine FYI. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Wednesday, June 21, 2017 at 9:45 AM To: Fotis Georgatos mailto:fo...@mail.cern.ch>>, "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: Re: [easybuild] install dir on gpfs Thanks for the replies, everyone. We have opened a ticket with DDN, through whom we have GPFS. It is version 4.2.1 I’m going to try a few things, like configure the build directory to be under /tmp. Funny thing is that GCC 4.9.2 works fine, but 4.9.3 and later do not. GCC 4.9.3 and later are built differently, right? With GCC/*-binutils-*. From: Fotis Georgatos mailto:fo...@mail.cern.ch><mailto:fo...@mail.cern.ch>> Date: Wednesday, June 21, 2017 at 4:16 AM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be><mailto:easybuild@lists.ugent.be>> Cc: Todd Heywood mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> Subject: Re: [easybuild] install dir on gpfs Hi, I feel for you :-p as a matter of fact, any parallel fs chimes in with its own quirks during a build process. More than a dozen years ago, GPFS/2.2 would not support symlinks from a given size and above, go figure! For this and few other reasons, I consider pointing EASYBUILD_BUILDPATH towards /dev/shm or /tmp or so, a totally essential move. I’d rather get a failure fi. because /dev/shm is full, than risk having variating builds across two different (versions of) parallel filesystems. Truth be told, there is a `make install` step which may still ruin the day; I am not isolating that yet, but it is not entirely impossible: use a container or RPM to deliver, then use that towards your final destination. Indeed, that’s a project of its own. On Jun 20, 2017, at 9:52 PM, Heywood, Todd mailto:heyw...@cshl.edu><mailto:heyw...@cshl.edu>> wrote: You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Finally, this seems to be yet another example of the tension between metadata consistency and need for speed. What is a matter of wonder though is, why the compiler version is important! It could be that the way the compiler opens the output file for writing makes some kind of difference. (weird.. strace is your friend) Thanks for reporting this! F. On Jun 21, 2017, at 6:52 AM, Kenneth Hoste mailto:kenneth.ho...@ugent.be><mailto:kenneth.ho...@ugent.be>> wrote: Hi Todd, We ran into this issue too a while ago, but it wasn't this extreme, I was only able to reproduce this using OCaml... We reported this back to IBM, and we eventually applied a fix (GPFS update, I think, in Aug'16), that resolved the problem for us. I can try to get more information on this if it would be useful for you. regards, Kenneth On 20/06/2017 22:52, Heywood, Todd wrote: Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
Re: [easybuild] install dir on gpfs
Thanks for the replies, everyone. We have opened a ticket with DDN, through whom we have GPFS. It is version 4.2.1 I’m going to try a few things, like configure the build directory to be under /tmp. Funny thing is that GCC 4.9.2 works fine, but 4.9.3 and later do not. GCC 4.9.3 and later are built differently, right? With GCC/*-binutils-*. From: Fotis Georgatos mailto:fo...@mail.cern.ch>> Date: Wednesday, June 21, 2017 at 4:16 AM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Cc: Todd Heywood mailto:heyw...@cshl.edu>> Subject: Re: [easybuild] install dir on gpfs Hi, I feel for you :-p as a matter of fact, any parallel fs chimes in with its own quirks during a build process. More than a dozen years ago, GPFS/2.2 would not support symlinks from a given size and above, go figure! For this and few other reasons, I consider pointing EASYBUILD_BUILDPATH towards /dev/shm or /tmp or so, a totally essential move. I’d rather get a failure fi. because /dev/shm is full, than risk having variating builds across two different (versions of) parallel filesystems. Truth be told, there is a `make install` step which may still ruin the day; I am not isolating that yet, but it is not entirely impossible: use a container or RPM to deliver, then use that towards your final destination. Indeed, that’s a project of its own. On Jun 20, 2017, at 9:52 PM, Heywood, Todd mailto:heyw...@cshl.edu>> wrote: You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Finally, this seems to be yet another example of the tension between metadata consistency and need for speed. What is a matter of wonder though is, why the compiler version is important! It could be that the way the compiler opens the output file for writing makes some kind of difference. (weird.. strace is your friend) Thanks for reporting this! F. On Jun 21, 2017, at 6:52 AM, Kenneth Hoste mailto:kenneth.ho...@ugent.be>> wrote: Hi Todd, We ran into this issue too a while ago, but it wasn't this extreme, I was only able to reproduce this using OCaml... We reported this back to IBM, and we eventually applied a fix (GPFS update, I think, in Aug'16), that resolved the problem for us. I can try to get more information on this if it would be useful for you. regards, Kenneth On 20/06/2017 22:52, Heywood, Todd wrote: Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
Re: [easybuild] install dir on gpfs
Hi, I feel for you :-p as a matter of fact, any parallel fs chimes in with its own quirks during a build process. More than a dozen years ago, GPFS/2.2 would not support symlinks from a given size and above, go figure! For this and few other reasons, I consider pointing EASYBUILD_BUILDPATH towards /dev/shm or /tmp or so, a totally essential move. I’d rather get a failure fi. because /dev/shm is full, than risk having variating builds across two different (versions of) parallel filesystems. Truth be told, there is a `make install` step which may still ruin the day; I am not isolating that yet, but it is not entirely impossible: use a container or RPM to deliver, then use that towards your final destination. Indeed, that’s a project of its own. On Jun 20, 2017, at 9:52 PM, Heywood, Todd wrote: > You can see the issue here: > > > xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out > > -bash: ./a.out: cannot execute binary file > > xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Finally, this seems to be yet another example of the tension between metadata consistency and need for speed. What is a matter of wonder though is, why the compiler version is important! It could be that the way the compiler opens the output file for writing makes some kind of difference. (weird.. strace is your friend) Thanks for reporting this! F. On Jun 21, 2017, at 6:52 AM, Kenneth Hoste wrote: > Hi Todd, > > We ran into this issue too a while ago, but it wasn't this extreme, I was > only able to reproduce this using OCaml... > > We reported this back to IBM, and we eventually applied a fix (GPFS update, I > think, in Aug'16), that resolved the problem for us. > > I can try to get more information on this if it would be useful for you. > > > regards, > > Kenneth > > On 20/06/2017 22:52, Heywood, Todd wrote: >> Has anyone installed easybuild on GPFS, not NFS? We just migrated our >> easybuild installation to GPFS, and now when we use an Easybuild-built >> compiler to build software (on GPFS), the “configure” step fails when tests >> the compiler to create an executable, and tests whether the executable >> works. You can see the issue here: >> >> >> xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out >> >> -bash: ./a.out: cannot execute binary file >> >> xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later >> >> Hello World!heywood@bnbmgmt2:~/tmp$ >> >> xyz@bnbmgmt2:~/tmp$ >> >> xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out >> >> a.out: data >> >> xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later >> >> a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically >> linked (uses shared libs), for GNU/Linux 2.6.18, not stripped >> >> xyz@bnbmgmt2:~/tmp$ >> >> Thanks, >> >> T >> > cheers, Fotis -- echo "sysadmin know better bash than english" | sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum
Re: [easybuild] install dir on gpfs
Hi Todd, We ran into this issue too a while ago, but it wasn't this extreme, I was only able to reproduce this using OCaml... We reported this back to IBM, and we eventually applied a fix (GPFS update, I think, in Aug'16), that resolved the problem for us. I can try to get more information on this if it would be useful for you. regards, Kenneth On 20/06/2017 22:52, Heywood, Todd wrote: Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
Re: [easybuild] install dir on gpfs
On 21/06/17 06:52, Heywood, Todd wrote: > xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out > > a.out: data > > xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later > > a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically > linked (uses shared libs), for GNU/Linux 2.6.18, not stripped Which version of GPFS? Have you opened a PMR with IBM about this? Good luck! Chris -- Christopher SamuelSenior Systems Administrator Melbourne Bioinformatics - The University of Melbourne Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
Re: [easybuild] install dir on gpfs
e.g.
$ cat /sw/hprc/mods/all/EasyBuild-terra/0.lua
whatis([[Description: EasyBuild environment for terra.tamu.edu]])
if (mode() == "load") then
if ( not (isloaded("EasyBuild") ) ) then
load ("EasyBuild")
end
end
pushenv("EASYBUILD_PREFIX","/sw/eb")
pushenv("EASYBUILD_BUILDPATH","/tmp/"..os.getenv("USER").."/easybuild/")
pushenv("EASYBUILD_DEPRECATED","2.0")
pushenv("EASYBUILD_INCLUDE_EASYBLOCKS",
"/sw/eb/hprc/ec-dev/easyblocks/*.py")
pushenv("EASYBUILD_INSTALLPATH_MODULES", "/sw/eb/mods")
pushenv("EASYBUILD_INSTALLPATH_SOFTWARE", "/sw/eb/sw")
pushenv("EASYBUILD_MINIMAL_TOOLCHAINS", "1")
pushenv("EASYBUILD_MODULES_TOOL","Lmod")
pushenv("EASYBUILD_READ_ONLY_INSTALLDIR", "1")
pushenv("EASYBUILD_REPOSITORYPATH","/sw/eb/ebfiles_repo/terra")
pushenv("EASYBUILD_ROBOT","/sw/eb/ebfiles_repo_cleaned/terra")
pushenv("EASYBUILD_SOURCEPATH","/sw/eb/sources")
pushenv("EASYBUILD_TMPDIR","/tmp/"..os.getenv("USER").."/tmp/")
pushenv("EASYBUILD_TMP_LOGDIR","/tmp/"..os.getenv("USER").."/logs/")
-- pushenv("EASYBUILD_USE_EXISTING_MODULES", "1")
pushenv("LMOD_IGNORE_CACHE","1")
pushenv("INTEL_LICENSE_FILE","/sw/hprc/local/etc/licenses/intel")
pushenv("EB_ANSYS_LICENSE_SERVER", "hprc-license.tamu.edu")
pushenv("EB_ANSYS_LICENSE_SERVER_PORT", "2325:1055")
pushenv("PGROUPD_LICENSE_FILE",
"/sw/hprc/local/etc/licenses/PGI/license.dat")
help([[
Easybuild environment for terra.tamu.edu
]])
On 06/20/2017 04:30 PM, Jack Perdue wrote:
We have three clusters running off of three
GSS/GPFS storage systems (one serving two
clusters and two serving the other cluster).
Haven't seen issues with using EasyBuild
on GPFS here. However, I do use:
EASYBUILD_TMPDIR=/tmp/j-perdue/tmp/
EASYBUILD_BUILDPATH=/tmp/j-perdue/easybuild/
EASYBUILD_TMP_LOGDIR=/tmp/j-perdue/logs/
while building so that might matter (I don't
build on GPFS too much traffic too slow).
Jack Perdue
Lead Systems Administrator
High Performance Research Computing
TAMU Division of Research
j-per...@tamu.eduhttp://hprc.tamu.edu
HPRC Helpdesk: h...@hprc.tamu.edu
On 06/20/2017 04:13 PM, Heywood, Todd wrote:
P.p.s I just noticed the issue does not occur for GCC/4.9.2 but does
occur for GCC/4.9.3-2.25. This is consistent.
heywood@bnbmgmt2:~/tmp$ module --ignore_cache av
Rebuilding cache, please wait ... (written to file) done.
--
/sonas-hs/it/hpc/home/easybuild/install_prod/modules/all/Core
---
GCC/4.4.6GCC/4.9.3-binutils-2.25
GNU/4.9.3-2.25foss/2015a
gompi/1.4.10-opt-hpcgompi/2015b
GCC/4.8.3 (D)GCC/4.9.3-2.25
SRA-Toolkit/2.8.1-3-centos_linux64foss/2015b
gompi/2014b (D)gompi/2016a
GCC/4.9.2GCCcore/4.9.3 foss/2014b
(D)foss/2016a gompi/2015a goolf/1.4.10-opt-hpc
Where:
(D): Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible modules
matching any of the "keys".
heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.2
heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out
a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV),
dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not
stripped
heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.3-2.25
heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out
a.out: data
heywood@bnbmgmt2:~/tmp$
From: Todd Heywood mailto:heyw...@cshl.edu>>
Date: Tuesday, June 20, 2017 at 5:04 PM
To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>"
mailto:easybuild@lists.ugent.be>>
Subject: Re: [easybuild] install dir on gpfs
P.s. This issue does NOT occur if using g++ from a GCC built outside
of the Easybuild framework.
From:
mailto:easybuild-requ...@lists.ugent.be>>
on behalf of Todd Heywood mailto:heyw...@cshl.edu>>
Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>"
mailto:easybuild@lists.ugent.be>>
Date: Tuesday, June 20, 2017 at 4:52 PM
To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>"
mailto:easybuild@lists.ugent.be>>
Subject: [easybuild] install dir on gpfs
Has anyone installed easybuild on GPFS, not NFS? We just migrated our
easybuild installation to GPFS, and now when we use an
Easybuild-built compiler
Re: [easybuild] install dir on gpfs
We have three clusters running off of three GSS/GPFS storage systems (one serving two clusters and two serving the other cluster). Haven't seen issues with using EasyBuild on GPFS here. However, I do use: EASYBUILD_TMPDIR=/tmp/j-perdue/tmp/ EASYBUILD_BUILDPATH=/tmp/j-perdue/easybuild/ EASYBUILD_TMP_LOGDIR=/tmp/j-perdue/logs/ while building so that might matter (I don't build on GPFS too much traffic too slow). Jack Perdue Lead Systems Administrator High Performance Research Computing TAMU Division of Research j-per...@tamu.eduhttp://hprc.tamu.edu HPRC Helpdesk: h...@hprc.tamu.edu On 06/20/2017 04:13 PM, Heywood, Todd wrote: P.p.s I just noticed the issue does not occur for GCC/4.9.2 but does occur for GCC/4.9.3-2.25. This is consistent. heywood@bnbmgmt2:~/tmp$ module --ignore_cache av Rebuilding cache, please wait ... (written to file) done. -- /sonas-hs/it/hpc/home/easybuild/install_prod/modules/all/Core --- GCC/4.4.6GCC/4.9.3-binutils-2.25GNU/4.9.3-2.25 foss/2015agompi/1.4.10-opt-hpcgompi/2015b GCC/4.8.3 (D)GCC/4.9.3-2.25 SRA-Toolkit/2.8.1-3-centos_linux64foss/2015bgompi/2014b (D)gompi/2016a GCC/4.9.2GCCcore/4.9.3 foss/2014b (D)foss/2016agompi/2015a goolf/1.4.10-opt-hpc Where: (D): Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.2 heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.3-2.25 heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data heywood@bnbmgmt2:~/tmp$ From: Todd Heywood mailto:heyw...@cshl.edu>> Date: Tuesday, June 20, 2017 at 5:04 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: Re: [easybuild] install dir on gpfs P.s. This issue does NOT occur if using g++ from a GCC built outside of the Easybuild framework. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Tuesday, June 20, 2017 at 4:52 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: [easybuild] install dir on gpfs Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
Re: [easybuild] install dir on gpfs
Todd, We have all EasyBuild built software on GPFS and I have never experienced that kind of issue. Unfortunately I do not have any suggestion on where to start looking at since I am not much a storage expert. -- Davide Vanzo, PhD Application Developer Advanced Computing Center for Research and Education (ACCRE) Vanderbilt University - Hill Center 201 (615)-875-9137 www.accre.vanderbilt.edu On Jun 20 2017, at 4:07 pm, Heywood, Todd wrote: P.s. This issue does NOT occur if using g++ from a GCC built outside of the Easybuild framework. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Tuesday, June 20, 2017 at 4:52 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: [easybuild] install dir on gpfs Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
Re: [easybuild] install dir on gpfs
P.p.s I just noticed the issue does not occur for GCC/4.9.2 but does occur for GCC/4.9.3-2.25. This is consistent. heywood@bnbmgmt2:~/tmp$ module --ignore_cache av Rebuilding cache, please wait ... (written to file) done. -- /sonas-hs/it/hpc/home/easybuild/install_prod/modules/all/Core --- GCC/4.4.6GCC/4.9.3-binutils-2.25GNU/4.9.3-2.25 foss/2015agompi/1.4.10-opt-hpcgompi/2015b GCC/4.8.3 (D)GCC/4.9.3-2.25 SRA-Toolkit/2.8.1-3-centos_linux64foss/2015bgompi/2014b (D)gompi/2016a GCC/4.9.2GCCcore/4.9.3 foss/2014b (D)foss/2016agompi/2015a goolf/1.4.10-opt-hpc Where: (D): Default Module Use "module spider" to find all possible modules. Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys". heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.2 heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped heywood@bnbmgmt2:~/tmp$ module load GCC/4.9.3-2.25 heywood@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data heywood@bnbmgmt2:~/tmp$ From: Todd Heywood mailto:heyw...@cshl.edu>> Date: Tuesday, June 20, 2017 at 5:04 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: Re: [easybuild] install dir on gpfs P.s. This issue does NOT occur if using g++ from a GCC built outside of the Easybuild framework. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Tuesday, June 20, 2017 at 4:52 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: [easybuild] install dir on gpfs Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
Re: [easybuild] install dir on gpfs
P.s. This issue does NOT occur if using g++ from a GCC built outside of the Easybuild framework. From: mailto:easybuild-requ...@lists.ugent.be>> on behalf of Todd Heywood mailto:heyw...@cshl.edu>> Reply-To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Date: Tuesday, June 20, 2017 at 4:52 PM To: "easybuild@lists.ugent.be<mailto:easybuild@lists.ugent.be>" mailto:easybuild@lists.ugent.be>> Subject: [easybuild] install dir on gpfs Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
[easybuild] install dir on gpfs
Has anyone installed easybuild on GPFS, not NFS? We just migrated our easybuild installation to GPFS, and now when we use an Easybuild-built compiler to build software (on GPFS), the “configure” step fails when tests the compiler to create an executable, and tests whether the executable works. You can see the issue here: xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; ./a.out -bash: ./a.out: cannot execute binary file xyz@bnbmgmt2:~/tmp$ ./a.out # it work a couple seconds later Hello World!heywood@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ xyz@bnbmgmt2:~/tmp$ g++ hello.cpp ; file a.out a.out: data xyz@bnbmgmt2:~/tmp$ file a.out # gives expected output a couple seconds later a.out: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, not stripped xyz@bnbmgmt2:~/tmp$ Thanks, T
