Re: [easybuild] Re: Configuring ELPA according to the cpu
Hi Michael, That would indeed be great. Let us keep in touch, so we don’t do it both :-) Jakob > On 11 Jan 2019, at 16:19, Micael Oliveira wrote: > > Hi, > > I'm planning on writing a custom easyblock for ELPA precisely to tackle this > problem, so I guess I'm the person mentioned in Alan's email, although I'm > still not sure I can attend the EasyBuild user meeting. > > My idea was precisely to use the FFTW easyblock as a starting point. > > Best regards, > > Micael > > On 1/11/19 3:34 PM, Kenneth Hoste wrote: >> Dear Jakob, >> I'm not sure there's much EasyBuild can do about this, unless I'm missing >> something. >> You can build using different sets of configuration options in a single >> installation, but I don't think that's of much use here. >> The best solution indeed seems to be to write an easyblock for ELPA that >> checks which CPU capabilities are supported, and then builds ELPA with the >> appropriate configure options. >> We have some of that already in custom easyblocks, see for example the FFTW >> easyblock: >> https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/f/fftw.py#L124 >> . >> This really is beyond what's possible in an easyconfig file, you need custom >> code to deal with this... >> regards, >> Kenneth >> On 11/01/2019 12:48, Jakob Schiøtz wrote: >>> Dear all, >>> >>> I did not get any response to this before Christmas. Does anybody have any >>> ideas that might help me? >>> >>> Best regards >>> >>> Jakob >>> On 18 Dec 2018, at 15:00, Jakob Schiøtz wrote: Hi, I am trying to build the newest version of ELPA (Eigenvalue SoLvers for Petaflop-Applications, https://elpa.mpcdf.mpg.de/). It looks like ELPA builds a number of “kernels” that are chosen at runtime depending on the capabilities of the CPU, for example SSE, AVX, AVX2 or AVX512. Unfortunately, with the foss toolchain the compiler refuses to build these kernels if the CPU does not support them (the Intel toolchain happily builds them). This means that I have to configure ELPA with --disable-avx2 or --disable-avx if the CPU does not support these instructions, and --enable-avx512 if the CPU does support that. Is there some way to do this easily in an easyconfig? Maybe a compiler flag that allows using “AVX2 gcc intrinsics” in the code, without otherwise turning on AVX2 instructions (as that would cause the rest of the code to fail if AVX2 is not supported). Or do I have to write some complicated stuff into an easyblock? And if I do the latter, do you know how to portably detect the capabilities of the CPU? It seems somewhat silly that the ELPA configure script tests if these instructions are supported and fails if one does not manually disable the unsupported versions instead of just doing the right thing. But that is beyond my control. Best regards Jakob -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/ >>> >>> -- >>> Jakob Schiøtz, professor, Ph.D. >>> Department of Physics >>> Technical University of Denmark >>> DK-2800 Kongens Lyngby, Denmark >>> http://www.fysik.dtu.dk/~schiotz/ >>> >>> >>> -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/
Re: [easybuild] Python links to libncurses, not libncursesw
Dear Jakob, On 18/12/2018 14:10, Jakob Schiøtz wrote: Dear all, One of my colleagues has a problem with the default Python build. The curses module does not properly support unicode because the Python executable is linked with libncurses instead of libncursesw. Do you know if that is something that is easily fixable, or would fixing this break something else? Symptom: ~$ module load Python/3.6.6-foss-2018b ~$ python -c "import curses; curses.wrapper(lambda scr: scr.get_wch)" Traceback (most recent call last): File "", line 1, in File "/home/modules/software/Python/3.6.6-foss-2018b/lib/python3.6/curses/__init__.py", line 94, in wrapper return func(stdscr, *args, **kwds) File "", line 1, in AttributeError: '_curses.window' object has no attribute 'get_wch' On other machines (e.g. my Macbook), this does not produce an error. Linking to libncurses.a is currently hardcoded in the Python easyblock, so that would have to be changed definitely in one way or another. It seems like ncursesw can be seen as a drop-in replacement for ncurses with better locale support, based on https://invisible-island.net/ncurses/ncurses.faq.html#growing_features . But it's unclear how to best handle this w.r.t. backward compatibility. Do we change the Python easyblock to (always) prefer linking to ncursesw if it's available, and use ncurses only as a fallback? Or do we add a custom easyconfig parameter to specify preference of ncursesw vs ncurses? Or maybe something else? regards, Kenneth
Re: [easybuild] Re: Configuring ELPA according to the cpu
Hi, I'm planning on writing a custom easyblock for ELPA precisely to tackle this problem, so I guess I'm the person mentioned in Alan's email, although I'm still not sure I can attend the EasyBuild user meeting. My idea was precisely to use the FFTW easyblock as a starting point. Best regards, Micael On 1/11/19 3:34 PM, Kenneth Hoste wrote: Dear Jakob, I'm not sure there's much EasyBuild can do about this, unless I'm missing something. You can build using different sets of configuration options in a single installation, but I don't think that's of much use here. The best solution indeed seems to be to write an easyblock for ELPA that checks which CPU capabilities are supported, and then builds ELPA with the appropriate configure options. We have some of that already in custom easyblocks, see for example the FFTW easyblock: https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/f/fftw.py#L124 . This really is beyond what's possible in an easyconfig file, you need custom code to deal with this... regards, Kenneth On 11/01/2019 12:48, Jakob Schiøtz wrote: Dear all, I did not get any response to this before Christmas. Does anybody have any ideas that might help me? Best regards Jakob On 18 Dec 2018, at 15:00, Jakob Schiøtz wrote: Hi, I am trying to build the newest version of ELPA (Eigenvalue SoLvers for Petaflop-Applications, https://elpa.mpcdf.mpg.de/). It looks like ELPA builds a number of “kernels” that are chosen at runtime depending on the capabilities of the CPU, for example SSE, AVX, AVX2 or AVX512. Unfortunately, with the foss toolchain the compiler refuses to build these kernels if the CPU does not support them (the Intel toolchain happily builds them). This means that I have to configure ELPA with --disable-avx2 or --disable-avx if the CPU does not support these instructions, and --enable-avx512 if the CPU does support that. Is there some way to do this easily in an easyconfig? Maybe a compiler flag that allows using “AVX2 gcc intrinsics” in the code, without otherwise turning on AVX2 instructions (as that would cause the rest of the code to fail if AVX2 is not supported). Or do I have to write some complicated stuff into an easyblock? And if I do the latter, do you know how to portably detect the capabilities of the CPU? It seems somewhat silly that the ELPA configure script tests if these instructions are supported and fails if one does not manually disable the unsupported versions instead of just doing the right thing. But that is beyond my control. Best regards Jakob -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/ -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/
Re: [easybuild] Re: Configuring ELPA according to the cpu
Dear Jakob, I'm not sure there's much EasyBuild can do about this, unless I'm missing something. You can build using different sets of configuration options in a single installation, but I don't think that's of much use here. The best solution indeed seems to be to write an easyblock for ELPA that checks which CPU capabilities are supported, and then builds ELPA with the appropriate configure options. We have some of that already in custom easyblocks, see for example the FFTW easyblock: https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/f/fftw.py#L124 . This really is beyond what's possible in an easyconfig file, you need custom code to deal with this... regards, Kenneth On 11/01/2019 12:48, Jakob Schiøtz wrote: Dear all, I did not get any response to this before Christmas. Does anybody have any ideas that might help me? Best regards Jakob On 18 Dec 2018, at 15:00, Jakob Schiøtz wrote: Hi, I am trying to build the newest version of ELPA (Eigenvalue SoLvers for Petaflop-Applications, https://elpa.mpcdf.mpg.de/). It looks like ELPA builds a number of “kernels” that are chosen at runtime depending on the capabilities of the CPU, for example SSE, AVX, AVX2 or AVX512. Unfortunately, with the foss toolchain the compiler refuses to build these kernels if the CPU does not support them (the Intel toolchain happily builds them). This means that I have to configure ELPA with --disable-avx2 or --disable-avx if the CPU does not support these instructions, and --enable-avx512 if the CPU does support that. Is there some way to do this easily in an easyconfig? Maybe a compiler flag that allows using “AVX2 gcc intrinsics” in the code, without otherwise turning on AVX2 instructions (as that would cause the rest of the code to fail if AVX2 is not supported). Or do I have to write some complicated stuff into an easyblock? And if I do the latter, do you know how to portably detect the capabilities of the CPU? It seems somewhat silly that the ELPA configure script tests if these instructions are supported and fails if one does not manually disable the unsupported versions instead of just doing the right thing. But that is beyond my control. Best regards Jakob -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/ -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/
[easybuild] Re: Configuring ELPA according to the cpu
Dear all, I did not get any response to this before Christmas. Does anybody have any ideas that might help me? Best regards Jakob > On 18 Dec 2018, at 15:00, Jakob Schiøtz wrote: > > Hi, > > I am trying to build the newest version of ELPA (Eigenvalue SoLvers for > Petaflop-Applications, https://elpa.mpcdf.mpg.de/). It looks like ELPA > builds a number of “kernels” that are chosen at runtime depending on the > capabilities of the CPU, for example SSE, AVX, AVX2 or AVX512. > > Unfortunately, with the foss toolchain the compiler refuses to build these > kernels if the CPU does not support them (the Intel toolchain happily builds > them). This means that I have to configure ELPA with --disable-avx2 or > --disable-avx if the CPU does not support these instructions, and > --enable-avx512 if the CPU does support that. > > Is there some way to do this easily in an easyconfig? Maybe a compiler flag > that allows using “AVX2 gcc intrinsics” in the code, without otherwise > turning on AVX2 instructions (as that would cause the rest of the code to > fail if AVX2 is not supported). > > Or do I have to write some complicated stuff into an easyblock? And if I do > the latter, do you know how to portably detect the capabilities of the CPU? > > It seems somewhat silly that the ELPA configure script tests if these > instructions are supported and fails if one does not manually disable the > unsupported versions instead of just doing the right thing. But that is > beyond my control. > > Best regards > > Jakob > > -- > Jakob Schiøtz, professor, Ph.D. > Department of Physics > Technical University of Denmark > DK-2800 Kongens Lyngby, Denmark > http://www.fysik.dtu.dk/~schiotz/ > > > -- Jakob Schiøtz, professor, Ph.D. Department of Physics Technical University of Denmark DK-2800 Kongens Lyngby, Denmark http://www.fysik.dtu.dk/~schiotz/
