[Numpy-discussion] PR, extended site.cfg capabilities
Dear all, I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) extra_compile_args: Adds extra compile flags to the compilation extra_link_args: Adds extra flags when linking to libraries Note that this PR will fix a lot of issues down the line. Specifically all software which utilises numpy's distutils will benefit from this. As an example, I have successfully set runtime_library_dirs for site.cfg in numpy, where scipy, petsc4py, pygsl, slepc4py utilises these flags and this enables me to create environments without the need for LD_LIBRARY_PATH. The other options simply adds to the flexibility of the compilation to test different optimisations etc. For instance my OpenBLAS section looks like this: [openblas] library_dirs = /opt/openblas/0.2.13/gnu-4.9.2/lib include_dirs = /opt/openblas/0.2.13/gnu-4.9.2/include runtime_library_dirs = /opt/openblas/0.2.13/gnu-4.9.2/lib I hope this can be of use to somebody else than me :) Feel free to test it and provide feedback! -- Kind regards Nick ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PR, extended site.cfg capabilities
On 02/24/2015 02:16 PM, Nick Papior Andersen wrote: Dear all, I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. I do not fully understand the purpose of these changes. Can you give some more detailed use cases? So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) LD_LIBRARY_PATH should not be used during compilation, this is a runtime flag that numpy.distutils has no control over. Can you explain in more detail what you intend to do with this flag? extra_compile_args: Adds extra compile flags to the compilation extra flags to which compilation? site.cfg lists libraries that already are compiled. The only influence compiler flags could have is for header only libraries that profit from some flags. But numpy has no support for such libraries currently. E.g. cblas.h (which is just a header with signatures) is bundled with numpy. I guess third parties may make use of this, an example would be good. extra_link_args: Adds extra flags when linking to libraries This flag may be useful. It could be used to pass options required during linking, like -Wl,--no-as-needed that is sometimes needed to link with gsl. Possibly also useful for link time optimizations. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PR, extended site.cfg capabilities
On 02/24/2015 02:31 PM, Julian Taylor wrote: On 02/24/2015 02:16 PM, Nick Papior Andersen wrote: Dear all, I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. I do not fully understand the purpose of these changes. Can you give some more detailed use cases? I think I understand better now, so this is intended as a site.cfg equivalent (and possibly more portable) variant of the environment variables that control these options? e.g. runtime_lib_dirs would be equivalent to LD_RUN_PATH env. variable and build_ext --rpath and the compile_extra_opts equivalent to the OPT env variable? So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) LD_LIBRARY_PATH should not be used during compilation, this is a runtime flag that numpy.distutils has no control over. Can you explain in more detail what you intend to do with this flag? extra_compile_args: Adds extra compile flags to the compilation extra flags to which compilation? site.cfg lists libraries that already are compiled. The only influence compiler flags could have is for header only libraries that profit from some flags. But numpy has no support for such libraries currently. E.g. cblas.h (which is just a header with signatures) is bundled with numpy. I guess third parties may make use of this, an example would be good. extra_link_args: Adds extra flags when linking to libraries This flag may be useful. It could be used to pass options required during linking, like -Wl,--no-as-needed that is sometimes needed to link with gsl. Possibly also useful for link time optimizations. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PR, extended site.cfg capabilities
2015-02-24 14:31 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com: On 02/24/2015 02:16 PM, Nick Papior Andersen wrote: Dear all, I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. I do not fully understand the purpose of these changes. Can you give some more detailed use cases? So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) LD_LIBRARY_PATH should not be used during compilation, this is a runtime flag that numpy.distutils has no control over. Can you explain in more detail what you intend to do with this flag? Yes, but in my case I almost never set LD_LIBRARY_PATH, instead I link with the runtime library directory so that LD_LIBRARY_PATH need not be set. Consider this output from linalg/lapack_lite.so $ echo $LD_LIBRARY_PATH $ldd lapack_lite.so libopenblas.so.0 = not found $ echo $LD_LIBRARY_PATH /path/to/openblas/lib $ldd lapack_lite.so libopenblas.so.0 = /path/to/openblas/lib/libopenblas.so.0 However, if I compile numpy with runtime_library_dirs = /path/to/openblas/lib in the openblas section, then the output would be $ echo $LD_LIBRARY_PATH $ldd lapack_lite.so libopenblas.so.0 = /path/to/openblas/lib/libopenblas.so.0 I.e. screw-ups in LD_LIBRARY_PATHS can be circumvented. extra_compile_args: Adds extra compile flags to the compilation extra flags to which compilation? site.cfg lists libraries that already are compiled. The only influence compiler flags could have is for header only libraries that profit from some flags. But numpy has no support for such libraries currently. E.g. cblas.h (which is just a header with signatures) is bundled with numpy. I guess third parties may make use of this, an example would be good. The way I see distutils in numpy is that it extends the generic distutils package so that packages relying on numpy can compile their software the way they want. In some of the extra software I work with, using numpy's distutils to link to lapack/blas is easy, but adding specific compilation flags to sources is not so easy (requires editing the compiler sources). Also, when numpy compiles the lapack_litemodules.c it does so by the generic flags in the compilers specified in the numpy distribution, however now it also uses the flags provided in extra_compile_args from the lapack section of site.cfg. In that regard I would not consider numpy as having no support as some packages does in fact use it. extra_link_args: Adds extra flags when linking to libraries This flag may be useful. It could be used to pass options required during linking, like -Wl,--no-as-needed that is sometimes needed to link with gsl. Possibly also useful for link time optimizations. Exactly, the runtime_library_dirs can be considered a shorthand for: extra_link_args = -Wl,-rpath=dir1 -Wl,-rpath=dir2 So you might consider it superfluous, but the intrinsic distutils package allows both abstractions, so why not allow them both? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion I hope this clarified a bit. Thanks for the questions. -- Kind regards Nick ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PR, extended site.cfg capabilities
2015-02-24 14:56 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com: On 02/24/2015 02:31 PM, Julian Taylor wrote: On 02/24/2015 02:16 PM, Nick Papior Andersen wrote: Dear all, I have initiated a PR-5597 https://github.com/numpy/numpy/pull/5597, which enables the reading of new flags from the site.cfg file. @rgommers requested that I posted some information on this site, possibly somebody could test it on their setup. I do not fully understand the purpose of these changes. Can you give some more detailed use cases? I think I understand better now, so this is intended as a site.cfg equivalent (and possibly more portable) variant of the environment variables that control these options? e.g. runtime_lib_dirs would be equivalent to LD_RUN_PATH env. variable and build_ext --rpath and the compile_extra_opts equivalent to the OPT env variable? Yes, but with the flexibility of each library (section). Instead of globally using the env's. And also that the site.cfg file is used in scipy which does not force the user to build numpy AND scipy with build_ext --rpath, etc. So the PR basically enables reading these extra options in each section: runtime_library_dirs : Add runtime library directories to the shared libraries (overrides the dreaded LD_LIBRARY_PATH) LD_LIBRARY_PATH should not be used during compilation, this is a runtime flag that numpy.distutils has no control over. Can you explain in more detail what you intend to do with this flag? extra_compile_args: Adds extra compile flags to the compilation extra flags to which compilation? site.cfg lists libraries that already are compiled. The only influence compiler flags could have is for header only libraries that profit from some flags. But numpy has no support for such libraries currently. E.g. cblas.h (which is just a header with signatures) is bundled with numpy. I guess third parties may make use of this, an example would be good. extra_link_args: Adds extra flags when linking to libraries This flag may be useful. It could be used to pass options required during linking, like -Wl,--no-as-needed that is sometimes needed to link with gsl. Possibly also useful for link time optimizations. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Kind regards Nick ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSoC'15 - mentors ideas
On Mon, Feb 23, 2015 at 11:49 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers ralf.gomm...@gmail.com wrote: Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. We could add a benchmark project for numpy that would build off the work Pauli is doing in Scipy. That would be easy to intermediate I think, as the programming bits might be easy, but coming up with the benchmarks would be more difficult. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSoC'15 - mentors ideas
On 02/24/2015 05:41 PM, Charles R Harris wrote: On Mon, Feb 23, 2015 at 11:49 PM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. We could add a benchmark project for numpy that would build off the work Pauli is doing in Scipy. That would be easy to intermediate I think, as the programming bits might be easy, but coming up with the benchmarks would be more difficult. we already have decent set of benchmarks in yaroslavs setup: http://yarikoptic.github.io/numpy-vbench/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSoC'15 - mentors ideas
On Tue, Feb 24, 2015 at 10:15 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 02/24/2015 05:41 PM, Charles R Harris wrote: On Mon, Feb 23, 2015 at 11:49 PM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. We could add a benchmark project for numpy that would build off the work Pauli is doing in Scipy. That would be easy to intermediate I think, as the programming bits might be easy, but coming up with the benchmarks would be more difficult. we already have decent set of benchmarks in yaroslavs setup: http://yarikoptic.github.io/numpy-vbench/ Are you suggesting that we delsteal/del copy those (my thought), or that we don't need a project? I note that Pauli is using Air Speed Velocity instead of vbench. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSoC'15 - mentors ideas
Not sure if this is a full GSoC but it would be good to get the benchmarks into the numpy repository, so we can start asking people who submit optimizations to submit new benchmarks as part of the PR (just like other changes require tests). On Feb 24, 2015 10:29 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Tue, Feb 24, 2015 at 10:15 AM, Julian Taylor jtaylor.deb...@googlemail.com wrote: On 02/24/2015 05:41 PM, Charles R Harris wrote: On Mon, Feb 23, 2015 at 11:49 PM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers ralf.gomm...@gmail.com mailto:ralf.gomm...@gmail.com wrote: Hi all, It's time to start preparing for this year's Google Summer of Code. There is actually one urgent thing to be done (before 19.00 UTC today), which is to get our ideas page in decent shape. It doesn't have to be final, but there has to be enough on there for the organizers to judge it. This page is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be reworking it and linking it from the PSF page today, but if you already have new ideas please add them there. See https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this year's template for adding a new idea. The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. We could add a benchmark project for numpy that would build off the work Pauli is doing in Scipy. That would be easy to intermediate I think, as the programming bits might be easy, but coming up with the benchmarks would be more difficult. we already have decent set of benchmarks in yaroslavs setup: http://yarikoptic.github.io/numpy-vbench/ Are you suggesting that we delsteal/del copy those (my thought), or that we don't need a project? I note that Pauli is using Air Speed Velocity instead of vbench. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
