subject:"\[Numpy\-discussion\] Default builds of OpenBLAS development branch are now fork safe"

Hi,

On Wed, Mar 26, 2014 at 1:41 PM, Nathaniel Smith n...@pobox.com wrote:
On Wed, Mar 26, 2014 at 7:34 PM, Julian Taylor
jtaylor.deb...@googlemail.com wrote:
as for using openblas by default in binary builds, no.
pthread openblas build is now fork safe which is great but it is still
not reliable enough for a default.
E.g. the current latest release 0.2.8 still has one crash bug on
dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3].
git head has the former four fixed bug still has wrong results for cgemv.
The not so old 0.2.8 also fixed whole bunch more crashes and wrong
result issues (crashes on QR, uninitialized data use in dgemm, ...).
None of the fixes received unit tests, so I am somewhat pessimistic that
it will improve, especially as the maintainer is dissertating (is that
the right word?) and most of the code is assembler code only few people
can write (it is simply not required anymore, we have code generators
and intrinsics for that).

Openblas is great if you do not have the patience to build ATLAS and
only use a restricted set of functionality and platforms you can easily
test.
Currently it is in my opinion not suitable for a general purpose library
like numpy.

Those problems you list are pretty damning, but neither is it
reasonable to expect everyone to manually build ATLAS on every machine
they use (or their students use, or...) :-(. So what other options do
we have for general purpose builds? Give up and use MKL? How's
eigen-blas doing these days? (I guess from skimming their docs they
use OpenMP?)

I see it should be possible to build a full blas and partial lapack
library with eigen [1] [2].

Does anyone know how their performance compares to MKL or the
reference implementations?

Carl - have you tried building eigen with your custom tool chain?

Cheers,

Matthew

[1] http://eigen.tuxfamily.org/index.php?title=3.0
[2]
http://stackoverflow.com/questions/20441851/build-numpy-with-eigen-instead-of-atlas-or-openblas
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Matthew Brett matthew.br...@gmail.com wrote:

 I see it should be possible to build a full blas and partial lapack
 library with eigen [1] [2].

Eigen has a licensing issue as well, unfortunately, MPL2. 

E.g. it requires recipients to be informed of the MPL requirements (cf.
impossible with pip install numpy). 

Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Fri, Mar 28, 2014 at 3:31 PM, Alan G Isaac alan.is...@gmail.com wrote:
 On 3/28/2014 10:54 AM, Sturla Molden wrote:
 Eigen has a licensing issue as well, unfortunately, MPL2.

 E.g. it requires recipients to be informed of the MPL requirements (cf.
 impossible with pip install numpy).

 Eigen chose MPL2 with the intent that Eigen be usable by
 all projects.
 http://eigen.tuxfamily.org/index.php?title=Licensing_FAQ
 If you are correct in your interpretation, it may be worth
 raising the issue and requesting the needed accommodation.

The authors of Eigen are familiar with our policy on this matter. See
the thread following this email:

  http://mail.scipy.org/pipermail/numpy-discussion/2010-January/047958.html

The change from LGPL to MPL2 isn't relevant to our policy. Both have
more restrictions and conditions than the BSD license.

-- 
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-28 Thread Alan G Isaac

On 3/28/2014 10:54 AM, Sturla Molden wrote:
 Eigen has a licensing issue as well, unfortunately, MPL2.

 E.g. it requires recipients to be informed of the MPL requirements (cf.
 impossible with pip install numpy).


Eigen chose MPL2 with the intent that Eigen be usable by
all projects.
http://eigen.tuxfamily.org/index.php?title=Licensing_FAQ
If you are correct in your interpretation, it may be worth
raising the issue and requesting the needed accommodation.

Alan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Fri, Mar 28, 2014 at 2:54 PM, Sturla Molden sturla.mol...@gmail.com wrote:
 Matthew Brett matthew.br...@gmail.com wrote:

 I see it should be possible to build a full blas and partial lapack
 library with eigen [1] [2].

 Eigen has a licensing issue as well, unfortunately, MPL2.

 E.g. it requires recipients to be informed of the MPL requirements (cf.
 impossible with pip install numpy).

That's not the relevant condition. That's easily taken care of by
including the MPL2 license text in the binary alongside numpy's BSD
license text. This is no different than numpy's BSD license itself,
which requires that the license text be included. It's not like people
can't distribute any MPL2 project on PyPI just because pip doesn't
print out the license before installing.

The extra-BSD conditions of the MPL2 are sections 3.1 and 3.2.

-- 
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Fri, Mar 28, 2014 at 4:58 PM, Robert Kern robert.k...@gmail.com wrote:
 On Fri, Mar 28, 2014 at 2:54 PM, Sturla Molden sturla.mol...@gmail.com 
 wrote:
 Matthew Brett matthew.br...@gmail.com wrote:

 I see it should be possible to build a full blas and partial lapack
 library with eigen [1] [2].

 Eigen has a licensing issue as well, unfortunately, MPL2.

 E.g. it requires recipients to be informed of the MPL requirements (cf.
 impossible with pip install numpy).

 That's not the relevant condition. That's easily taken care of by
 including the MPL2 license text in the binary alongside numpy's BSD
 license text. This is no different than numpy's BSD license itself,
 which requires that the license text be included. It's not like people
 can't distribute any MPL2 project on PyPI just because pip doesn't
 print out the license before installing.

 The extra-BSD conditions of the MPL2 are sections 3.1 and 3.2.

Those requirements just say that in addition to including the MPL2
license text, we also have to include a notice saying where the source
code is available, i.e. the package would have to somewhere include a
link to eigen.org.

https://www.mozilla.org/MPL/2.0/FAQ.html#distribute-my-binaries

I'm not sure why this would be a problem.

-n

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

On Fri, Mar 28, 2014 at 8:58 AM, Robert Kern robert.k...@gmail.com wrote:
 On Fri, Mar 28, 2014 at 2:54 PM, Sturla Molden sturla.mol...@gmail.com 
 wrote:
 Matthew Brett matthew.br...@gmail.com wrote:

 I see it should be possible to build a full blas and partial lapack
 library with eigen [1] [2].

 Eigen has a licensing issue as well, unfortunately, MPL2.

 E.g. it requires recipients to be informed of the MPL requirements (cf.
 impossible with pip install numpy).

 That's not the relevant condition. That's easily taken care of by
 including the MPL2 license text in the binary alongside numpy's BSD
 license text. This is no different than numpy's BSD license itself,
 which requires that the license text be included. It's not like people
 can't distribute any MPL2 project on PyPI just because pip doesn't
 print out the license before installing.

 The extra-BSD conditions of the MPL2 are sections 3.1 and 3.2.

Thanks for thinking this through.

If I read you right, your opinion is that there would be no problem
including Eigen binaries with Numpy.

License here: http://www.mozilla.org/MPL/2.0/

Section 3.1 - if we distribute Eigen source, it has to be under the
MPL; so that doesn't apply to binaries.

Section 3.2 a) says that if we distribute binaries, we have to point
to the original source (e.g. link on pypi page)
Section 3.2 b) says we can distribute the executable code under any
license as long as it doesn't restrict the user's access to Eigen
source.  I think that means there is no problem distributing the
binaries under the BSD license.  Nathaniel's link is the relevant one
: http://www.mozilla.org/MPL/2.0/FAQ.html#distribute-my-binaries

So - is Eigen our best option for optimized blas / lapack binaries on
64 bit Windows?

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Matthew Brett matthew.br...@gmail.com wrote:

 Does anyone know how their performance compares to MKL or the
 reference implementations?

http://eigen.tuxfamily.org/index.php?title=Benchmark 

http://gcdart.blogspot.de/2013/06/fast-matrix-multiply-and-ml.html


Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Matthew Brett matthew.br...@gmail.com wrote:

 So - is Eigen our best option for optimized blas / lapack binaries on
 64 bit Windows?

Maybe not:

http://gcdart.blogspot.de/2013/06/fast-matrix-multiply-and-ml.html

With AVX the difference is possibly even larger.


Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On Fri, Mar 28, 2014 at 8:01 PM, Sturla Molden sturla.mol...@gmail.com wrote:
 Matthew Brett matthew.br...@gmail.com wrote:

 So - is Eigen our best option for optimized blas / lapack binaries on
 64 bit Windows?

 Maybe not:

 http://gcdart.blogspot.de/2013/06/fast-matrix-multiply-and-ml.html

 With AVX the difference is possibly even larger.

But if we rule out closed-source BLAS, and we rule out OpenBLAS
because of our distrusting its accuracy, and we aren't going to
recompile ATLAS on every machine, then Eigen is the only library they
tested that is even an option for us.

It would be nice to see some comparison between our actual options --
Eigen, generically compiled ATLAS, anything else?

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

It's only a problem in that the binary will not be BSD, and we do need to
communicate that appropriately. It will contain a significant component
that is MPL2 licensed. The terms that force us to include the link to the
Eigen source that we used forces downstream redistributors of the binary to
do the same. Now, of all the copyleft licenses, this is certainly the most
friendly, but it is not BSD.
On Mar 28, 2014 6:43 PM, Nathaniel Smith n...@pobox.com wrote:

 On Fri, Mar 28, 2014 at 4:58 PM, Robert Kern robert.k...@gmail.com
 wrote:
  On Fri, Mar 28, 2014 at 2:54 PM, Sturla Molden sturla.mol...@gmail.com
 wrote:
  Matthew Brett matthew.br...@gmail.com wrote:
 
  I see it should be possible to build a full blas and partial lapack
  library with eigen [1] [2].
 
  Eigen has a licensing issue as well, unfortunately, MPL2.
 
  E.g. it requires recipients to be informed of the MPL requirements (cf.
  impossible with pip install numpy).
 
  That's not the relevant condition. That's easily taken care of by
  including the MPL2 license text in the binary alongside numpy's BSD
  license text. This is no different than numpy's BSD license itself,
  which requires that the license text be included. It's not like people
  can't distribute any MPL2 project on PyPI just because pip doesn't
  print out the license before installing.
 
  The extra-BSD conditions of the MPL2 are sections 3.1 and 3.2.

 Those requirements just say that in addition to including the MPL2
 license text, we also have to include a notice saying where the source
 code is available, i.e. the package would have to somewhere include a
 link to eigen.org.

 https://www.mozilla.org/MPL/2.0/FAQ.html#distribute-my-binaries

 I'm not sure why this would be a problem.

 -n

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

On Fri, Mar 28, 2014 at 11:56 AM, Sturla Molden sturla.mol...@gmail.com wrote:
 Matthew Brett matthew.br...@gmail.com wrote:

 Does anyone know how their performance compares to MKL or the
 reference implementations?

 http://eigen.tuxfamily.org/index.php?title=Benchmark

I don't know how relevant these are to our case. If I understand
correctly, the usual use of Eigen, as in these benchmarks, is to use
the Eigen headers to get fast code via C++ templating.

Because they know some of us need this, Eigen can also build a more
standard blas / lapack library to link against, but I presume this
will stop Eigen templating doing lots of clever tricks with the
operations, and therefore slow it down.  Happy to be corrected though.

 http://gcdart.blogspot.de/2013/06/fast-matrix-multiply-and-ml.html

I think this page does not use the Eigen blas libraries either [1]

Also - this is on a massive linux machine (48 core and 66GB RAM).
He's done a great job of showing what he did though.

The problem for us is:

We can't use MKL, ACML [2]
atlas is very difficult to compile on 64 bit windows, and has some
technical limitations on 64 bit [3]

So I think we're down to openblas and eigen for 64-bit windows. Does
anyone disagree?

Cheers,

Matthew

[1] : 
https://github.com/gcdart/dense-matrix-mult/blob/master/EIGEN/compile_eigen.sh
[2] : 
http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2013/12/ACML_June_24_2010_v2.pdf
[3] : http://math-atlas.sourceforge.net/atlas_install/node57.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

On Fri, Mar 28, 2014 at 12:26 PM, Robert Kern robert.k...@gmail.com wrote:
 It's only a problem in that the binary will not be BSD, and we do need to
 communicate that appropriately. It will contain a significant component that
 is MPL2 licensed. The terms that force us to include the link to the Eigen
 source that we used forces downstream redistributors of the binary to do the
 same. Now, of all the copyleft licenses, this is certainly the most
 friendly, but it is not BSD.

I think the binary would be BSD because of section 3.2:

You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter the
recipients' rights in the Source Code Form under this License.

I think this is specifically saying - as long as our license (BSD)
does not try and limit access to Eigen source, we can distribute our
binary under our license.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

On 28 Mar 2014 20:26, Robert Kern robert.k...@gmail.com wrote:

 It's only a problem in that the binary will not be BSD, and we do need to
communicate that appropriately. It will contain a significant component
that is MPL2 licensed. The terms that force us to include the link to the
Eigen source that we used forces downstream redistributors of the binary to
do the same. Now, of all the copyleft licenses, this is certainly the most
friendly, but it is not BSD.

AFAICT, the only way redistributers could violate the MPL would be if they
unpacked our binary and deleted the license file. But this would also be a
violation of the BSD. The only difference in terms of requirements on
redistributors between MPL and BSD seems to be exactly *which* text you
include in your license file.

I don't know if Eigen is a good choice on technical grounds (or even a
possible one - has anyone ever actually compiled numpy against it?), but
this license thing just doesn't seem like an important issue to me, if the
alternative is not providing useful binaries.

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

The BSD license alters the recipient's rights. BSD binaries can be
redistributed without pointing to the sources.
On Mar 28, 2014 7:33 PM, Matthew Brett matthew.br...@gmail.com wrote:

 Hi,

 On Fri, Mar 28, 2014 at 12:26 PM, Robert Kern robert.k...@gmail.com
 wrote:
  It's only a problem in that the binary will not be BSD, and we do need to
  communicate that appropriately. It will contain a significant component
 that
  is MPL2 licensed. The terms that force us to include the link to the
 Eigen
  source that we used forces downstream redistributors of the binary to do
 the
  same. Now, of all the copyleft licenses, this is certainly the most
  friendly, but it is not BSD.

 I think the binary would be BSD because of section 3.2:

 You may distribute such Executable Form under the terms of this
 License, or sublicense it under different terms, provided that the
 license for the Executable Form does not attempt to limit or alter the
 recipients' rights in the Source Code Form under this License.

 I think this is specifically saying - as long as our license (BSD)
 does not try and limit access to Eigen source, we can distribute our
 binary under our license.

 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

No, the license does not contain a pointer to the Eigen sources, which is
required.

https://bitbucket.org/eigen/eigen/src/fabd880592ac3343713cc07e7287098afd0f18ca/COPYING.MPL2?at=default
On Mar 28, 2014 7:34 PM, Nathaniel Smith n...@pobox.com wrote:

On 28 Mar 2014 20:26, Robert Kern robert.k...@gmail.com wrote:

It's only a problem in that the binary will not be BSD, and we do need
to communicate that appropriately. It will contain a significant component
that is MPL2 licensed. The terms that force us to include the link to the
Eigen source that we used forces downstream redistributors of the binary to
do the same. Now, of all the copyleft licenses, this is certainly the most
friendly, but it is not BSD.

AFAICT, the only way redistributers could violate the MPL would be if they
unpacked our binary and deleted the license file. But this would also be a
violation of the BSD. The only difference in terms of requirements on
redistributors between MPL and BSD seems to be exactly *which* text you
include in your license file.

I don't know if Eigen is a good choice on technical grounds (or even a
possible one - has anyone ever actually compiled numpy against it?), but
this license thing just doesn't seem like an important issue to me, if the
alternative is not providing useful binaries.

-n

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-28 Thread Andrea Gavana

On 28 March 2014 19:56, Sturla Molden  wrote:

 Matthew Brett matthew.br...@gmail.com wrote:

  Does anyone know how their performance compares to MKL or the
  reference implementations?

 http://eigen.tuxfamily.org/index.php?title=Benchmark



Very, very funny and twisted approach to legend-ordering-in-a-plot
approach. Maybe someone more knowledgeable can explain the ordering of the
labels in the plot legends, as after a while they don't make any sense -
neither lexicographically nor performance-wise.


Andrea.

Imagination Is The Only Weapon In The War Against Reality.
http://www.infinity77.net

# - #
def ask_mailing_list_support(email):

if mention_platform_and_version() and include_sample_app():
send_message(email)
else:
install_malware()
erase_hard_drives()
# - #
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Of course, that's besides the point. Yes,  pretty much everyone that likes
the BSD license of numpy will be okay with the minimal burdens the MPL2
lays on them. The problem is that we need to properly communicate that
license. The PyPI page is not adequate to that task, in my opinion. I have
no problem with the project distributing such binaries anywhere else. But
then, I have no problem with the project distributing MKL binaries
elsewhere either.
On Mar 28, 2014 7:34 PM, Nathaniel Smith n...@pobox.com wrote:

 On 28 Mar 2014 20:26, Robert Kern robert.k...@gmail.com wrote:
 
  It's only a problem in that the binary will not be BSD, and we do need
 to communicate that appropriately. It will contain a significant component
 that is MPL2 licensed. The terms that force us to include the link to the
 Eigen source that we used forces downstream redistributors of the binary to
 do the same. Now, of all the copyleft licenses, this is certainly the most
 friendly, but it is not BSD.

 AFAICT, the only way redistributers could violate the MPL would be if they
 unpacked our binary and deleted the license file. But this would also be a
 violation of the BSD. The only difference in terms of requirements on
 redistributors between MPL and BSD seems to be exactly *which* text you
 include in your license file.

 I don't know if Eigen is a good choice on technical grounds (or even a
 possible one - has anyone ever actually compiled numpy against it?), but
 this license thing just doesn't seem like an important issue to me, if the
 alternative is not providing useful binaries.

 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Yes, because they're distributing source. But *our* license file could
contain the text of the BSD, the text of the MPL, and the text Eigen
source is available at http://eigen.org.;

If the only problem with eigen turns out to be that we have to add a line
of text to a file then I think we can probably manage this somehow.

-n
On 28 Mar 2014 20:40, Robert Kern robert.k...@gmail.com wrote:

No, the license does not contain a pointer to the Eigen sources, which is
required.

https://bitbucket.org/eigen/eigen/src/fabd880592ac3343713cc07e7287098afd0f18ca/COPYING.MPL2?at=default
On Mar 28, 2014 7:34 PM, Nathaniel Smith n...@pobox.com wrote:

On 28 Mar 2014 20:26, Robert Kern robert.k...@gmail.com wrote:

AFAICT, the only way redistributers could violate the MPL would be if
they unpacked our binary and deleted the license file. But this would also
be a violation of the BSD. The only difference in terms of requirements on
redistributors between MPL and BSD seems to be exactly *which* text you
include in your license file.

-n

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Nathaniel Smith n...@pobox.com wrote:

 If the only problem with eigen turns out to be that we have to add a line
 of text to a file then I think we can probably manage this somehow.

We would also have to compile Eigen-BLAS for various architectures and CPU
counts. It is not adaptive like MKL or OpenBLAS.

Sturla

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

On Fri, Mar 28, 2014 at 12:57 PM, Robert Kern robert.k...@gmail.com wrote:
 Of course, that's besides the point. Yes,  pretty much everyone that likes
 the BSD license of numpy will be okay with the minimal burdens the MPL2 lays
 on them. The problem is that we need to properly communicate that license.
 The PyPI page is not adequate to that task, in my opinion. I have no problem
 with the project distributing such binaries anywhere else. But then, I have
 no problem with the project distributing MKL binaries elsewhere either.

 On Mar 28, 2014 7:34 PM, Nathaniel Smith n...@pobox.com wrote:

 On 28 Mar 2014 20:26, Robert Kern robert.k...@gmail.com wrote:
 
  It's only a problem in that the binary will not be BSD, and we do need
  to communicate that appropriately. It will contain a significant component
  that is MPL2 licensed. The terms that force us to include the link to the
  Eigen source that we used forces downstream redistributors of the binary to
  do the same. Now, of all the copyleft licenses, this is certainly the most
  friendly, but it is not BSD.

 AFAICT, the only way redistributers could violate the MPL would be if they
 unpacked our binary and deleted the license file. But this would also be a
 violation of the BSD. The only difference in terms of requirements on
 redistributors between MPL and BSD seems to be exactly *which* text you
 include in your license file.

I don't think even that would violate the MPL.  The MPL says only that
we - the distributors of binary code from an MPL project - must do
this (3.1) ... inform recipients of the Executable Form how they can
obtain a copy of such Source Code Form.  It doesn't say we have to
require the recipient to do the same [1], and it doesn't say that has
to be in our license.  I don't think it can mean that, because
otherwise it would not make sense to say (in section 3.2) that we can
sublicense it under different terms, provided that the license for
the Executable Form does not attempt to limit or alter the recipients'
rights in the Source Code Form under this License..  The unmodified
standard BSD license does not alter the recipients rights to the
source code form of Eigen.

 I don't know if Eigen is a good choice on technical grounds (or even a
 possible one - has anyone ever actually compiled numpy against it?), but
 this license thing just doesn't seem like an important issue to me, if the
 alternative is not providing useful binaries.

Am I correct in thinking we are all agreeing that it would be OK to
distribute binary wheels for numpy from pypi, with compiled Eigen?

See you,

Matthew

[1]  It's important to understand that the condition to distribute
files under the MPL's terms only applies to the party that first
creates and distributes the Larger Work.
https://www.gnu.org/licenses/license-list.html
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

On Fri, Mar 28, 2014 at 1:28 PM, Sturla Molden sturla.mol...@gmail.com wrote:
 Nathaniel Smith n...@pobox.com wrote:

 If the only problem with eigen turns out to be that we have to add a line
 of text to a file then I think we can probably manage this somehow.

 We would also have to compile Eigen-BLAS for various architectures and CPU
 counts. It is not adaptive like MKL or OpenBLAS.

Yes, I guess we currently have no idea how bad a default Eigen would be.

We also have the soft constraint that any choice we make should also
work for building scipy binaries - so adequate lapack coverage.

I believe that means lapack_lite is not an option?

So I guess the options are:

* eigen - could it be slow?
* openblas - could it be buggy?
* reference blas / lapack [1] [2] [3]

In [2] someone seems to be getting very good performance from the
reference implementation.

I guess we need to benchmark these guys on some standard systems, and
decide how bad the performance / stability has to be before it's
better not to provide binaries at all.

Cheers,

Matthew

[1] http://icl.cs.utk.edu/lapack-for-windows/lapack/
[2] http://ylzhao.blogspot.com/2013/10/blas-lapack-precompiled-binaries-for.html
[3] http://www.fi.muni.cz/~xsvobod2/misc/lapack/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

I thought OpenBLAS is usually used with reference lapack?
On 28 Mar 2014 22:16, Matthew Brett matthew.br...@gmail.com wrote:

 Hi,

 On Fri, Mar 28, 2014 at 1:28 PM, Sturla Molden sturla.mol...@gmail.com
 wrote:
  Nathaniel Smith n...@pobox.com wrote:
 
  If the only problem with eigen turns out to be that we have to add a
 line
  of text to a file then I think we can probably manage this somehow.
 
  We would also have to compile Eigen-BLAS for various architectures and
 CPU
  counts. It is not adaptive like MKL or OpenBLAS.

 Yes, I guess we currently have no idea how bad a default Eigen would be.

 We also have the soft constraint that any choice we make should also
 work for building scipy binaries - so adequate lapack coverage.

 I believe that means lapack_lite is not an option?

 So I guess the options are:

 * eigen - could it be slow?
 * openblas - could it be buggy?
 * reference blas / lapack [1] [2] [3]

 In [2] someone seems to be getting very good performance from the
 reference implementation.

 I guess we need to benchmark these guys on some standard systems, and
 decide how bad the performance / stability has to be before it's
 better not to provide binaries at all.

 Cheers,

 Matthew

 [1] http://icl.cs.utk.edu/lapack-for-windows/lapack/
 [2]
 http://ylzhao.blogspot.com/2013/10/blas-lapack-precompiled-binaries-for.html
 [3] http://www.fi.muni.cz/~xsvobod2/misc/lapack/
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-28 Thread Olivier Grisel

2014-03-28 22:18 GMT+01:00 Nathaniel Smith n...@pobox.com:
 I thought OpenBLAS is usually used with reference lapack?

I am no longer sure myself. Debian  thus Ubuntu seem to be only
packaging the BLAS part of OpenBLAS for the libblas.so symlink and
uses the reference implementation of lapack for the liblapack.so
symlink.

I observed a sparse leastsqr bug when linking scipy against the full
OpenBLAS under OSX that I could not reproduce under with the openblas
+ lapack combo shipped Ubuntu so there might be a difference. But that
could also be caused by a version / platform discrepancy between my
two setups.

-- 
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-28 Thread Julian Taylor

On 28.03.2014 22:38, Olivier Grisel wrote:
 2014-03-28 22:18 GMT+01:00 Nathaniel Smith n...@pobox.com:
 I thought OpenBLAS is usually used with reference lapack?
 
 I am no longer sure myself. Debian  thus Ubuntu seem to be only
 packaging the BLAS part of OpenBLAS for the libblas.so symlink and
 uses the reference implementation of lapack for the liblapack.so
 symlink.

You can link the reference lapack with any library providing a BLAS
compatible API/ABI. ATLAS and OpenBlas are ABI compatible with reference
BLAS which allows replacing the library via LD_PRELOAD or debian
alternatives without recompiling.

Both ATLAS and OpenBlas provide a subset of optimized lapack functions,
but they are optional. On Debian/Ubuntu you can install ATLAS lapack but
then you are also forced to use ATLAS blas.

I am not familiar with how relevant the optimized parts of lapack for
the general use case.

 
 I observed a sparse leastsqr bug when linking scipy against the full
 OpenBLAS under OSX that I could not reproduce under with the openblas
 + lapack combo shipped Ubuntu so there might be a difference. But that
 could also be caused by a version / platform discrepancy between my
 two setups.
 

what kind of a bug? wrong result or crash? Which target did openblas use
when compiling on macos? or was it a dynamic build? (see the name of the
built static library)
The adaptive nature of OpenBLAS can make reproducing issues tricky, I
don't think there is a way to force it to use a certain kernel at
runtime besides recompiling for a different target.

If you have a testcase I'd be interested in it, as I'm trying to get
openblas into a decent shape for ubuntu 14.04.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-28 Thread Olivier Grisel

2014-03-28 22:55 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com:
 On 28.03.2014 22:38, Olivier Grisel wrote:
 2014-03-28 22:18 GMT+01:00 Nathaniel Smith n...@pobox.com:
 I thought OpenBLAS is usually used with reference lapack?

 I am no longer sure myself. Debian  thus Ubuntu seem to be only
 packaging the BLAS part of OpenBLAS for the libblas.so symlink and
 uses the reference implementation of lapack for the liblapack.so
 symlink.

 You can link the reference lapack with any library providing a BLAS
 compatible API/ABI. ATLAS and OpenBlas are ABI compatible with reference
 BLAS which allows replacing the library via LD_PRELOAD or debian
 alternatives without recompiling.

 Both ATLAS and OpenBlas provide a subset of optimized lapack functions,
 but they are optional. On Debian/Ubuntu you can install ATLAS lapack but
 then you are also forced to use ATLAS blas.

 I am not familiar with how relevant the optimized parts of lapack for
 the general use case.


 I observed a sparse leastsqr bug when linking scipy against the full
 OpenBLAS under OSX that I could not reproduce under with the openblas
 + lapack combo shipped Ubuntu so there might be a difference. But that
 could also be caused by a version / platform discrepancy between my
 two setups.


 what kind of a bug? wrong result or crash?

Here it is: https://github.com/scikit-learn/scikit-learn/issues/2986

I have not found the time to investigate yet.

 Which target did openblas use
 when compiling on macos? or was it a dynamic build? (see the name of the
 built static library)

I think I used target=NEHALEM that time (but not 100% sure).

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Nathaniel Smith n...@pobox.com wrote:

 I thought OpenBLAS is usually used with reference lapack?

It is.

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-27 Thread Olivier Grisel

2014-03-26 16:27 GMT+01:00 Olivier Grisel olivier.gri...@ensta.org:
 Hi Carl,

 I installed Python 2.7.6 64 bits on a windows server instance from
 rackspace cloud and then ran get-pip.py and then could successfully
 install the numpy and scipy wheel packages from your google drive
 folder. I tested dot products and scipy.linalg.svd and they work as
 expected.

 Then I uncompressed your mingw toolchain in c:\mingw, put c:\mingw\bin
 in my PATH and tried to build the scikit-learn git master with it,
 however it fails with:

 building 'sklearn.__check_build._check_build' extension
 compiling C sources
 C compiler: gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes

 compile options: '-D__MSVCRT_VERSION__=0x0900
 -Ic:\Python27\lib\site-packages\numpy\core\include
 -Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python2
 7\include -Ic:\Python27\PC -c'
 gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes
 -D__MSVCRT_VERSION__=0x0900
 -Ic:\Python27\lib\site-packages\numpy\core\include
 -Ic:\Python27\lib\site-
 packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c
 sklearn\__check_build\_check_build.c -o
 build\temp.win-amd64-2.7\Release\sklearn\__check_b
 uild\_check_build.o
 Found executable c:\mingw\bin\gcc.exe
 gcc -shared -Wl,-gc-sections -Wl,-s
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
 -Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -Lbuild
 \temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
 build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x3):
 undefined reference to `__imp__Py_NoneStruct'
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x1ca):
 undefined reference to `__imp__PyThreadState_Current'
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x405):
 undefined reference to `__imp_PyExc_ImportError'
 c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.8.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.
 o: bad reloc address 0x0 in section `.data'
 collect2.exe: error: ld returned 1 exit status
 error: Command gcc -shared -Wl,-gc-sections -Wl,-s
 build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
 -Lc:\Python27\libs -Lc:\Python27\PCbui
 ld\amd64 -Lbuild\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
 build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd failed
 with exit status 1

Ignore that, I had forgotten to copy the libpython17.a file in
c:\Python27\libs on that instance.

Building scikit-learn works with the static toolchain. I have failing
tests but those are probably not related to the toolchain.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-27 Thread josef . pktd

On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel
olivier.gri...@ensta.org wrote:
 My understanding of Carl's effort is that the long term goal is to
 have official windows whl packages for both numpy and scipy published
 on PyPI with a builtin BLAS / LAPACK implementation so that users can
 do `pip install scipy` under windows and get something that just works
 without have to install any compiler (fortran or C) nor any additional
 library manually.

 Most windows users are beginners and you cannot really expect them to
 understand how to build the whole scipy stack from source.

 The current solution (executable setup installers) is not optimal as
 it requires Administrator rights to run, does not resolve dependencies
 as pip does and cannot be installed in virtualenvs.

as small related point:

The official installers can be used to install in virtualenv
The way I do it:
Run the superpack, official installer, wait until it extracts the
correct (SSE) install exe, then cancel
Then easy_install the install exe file that has been extracted to the
temp folder into the virtualenv.

I don't remember if the extraction already requires admin rights, but
I think not.
easy_install doesn't require any, IIRC.

Josef


 If we can build numpy / scipy whl packages for windows with the Atlas
 dlls then fine embedded in the numpy package then good. It does not
 need to be the
 fastest BLAS / LAPACK lib in my opinion. Just something that works.

 The problem with ATLAS is that you need to select the number of thread
 at build time AFAIK. But we could set it to a reasonable default (e.g.
 4 threads) for the default windows package.

 --
 Olivier
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-27 Thread Olivier Grisel

2014-03-27 14:55 GMT+01:00  josef.p...@gmail.com:
 On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel
 olivier.gri...@ensta.org wrote:
 My understanding of Carl's effort is that the long term goal is to
 have official windows whl packages for both numpy and scipy published
 on PyPI with a builtin BLAS / LAPACK implementation so that users can
 do `pip install scipy` under windows and get something that just works
 without have to install any compiler (fortran or C) nor any additional
 library manually.

 Most windows users are beginners and you cannot really expect them to
 understand how to build the whole scipy stack from source.

 The current solution (executable setup installers) is not optimal as
 it requires Administrator rights to run, does not resolve dependencies
 as pip does and cannot be installed in virtualenvs.

 as small related point:

 The official installers can be used to install in virtualenv
 The way I do it:
 Run the superpack, official installer, wait until it extracts the
 correct (SSE) install exe, then cancel
 Then easy_install the install exe file that has been extracted to the
 temp folder into the virtualenv.

 I don't remember if the extraction already requires admin rights, but
 I think not.
 easy_install doesn't require any, IIRC.

Hackish but interesting. Maybe the extraction can be done with generic
tools like winzip?

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-27 Thread josef . pktd

On Thu, Mar 27, 2014 at 9:59 AM, Olivier Grisel
olivier.gri...@ensta.org wrote:
 2014-03-27 14:55 GMT+01:00  josef.p...@gmail.com:
 On Wed, Mar 26, 2014 at 5:17 PM, Olivier Grisel
 olivier.gri...@ensta.org wrote:
 My understanding of Carl's effort is that the long term goal is to
 have official windows whl packages for both numpy and scipy published
 on PyPI with a builtin BLAS / LAPACK implementation so that users can
 do `pip install scipy` under windows and get something that just works
 without have to install any compiler (fortran or C) nor any additional
 library manually.

 Most windows users are beginners and you cannot really expect them to
 understand how to build the whole scipy stack from source.

 The current solution (executable setup installers) is not optimal as
 it requires Administrator rights to run, does not resolve dependencies
 as pip does and cannot be installed in virtualenvs.

 as small related point:

 The official installers can be used to install in virtualenv
 The way I do it:
 Run the superpack, official installer, wait until it extracts the
 correct (SSE) install exe, then cancel
 Then easy_install the install exe file that has been extracted to the
 temp folder into the virtualenv.

 I don't remember if the extraction already requires admin rights, but
 I think not.
 easy_install doesn't require any, IIRC.

 Hackish but interesting. Maybe the extraction can be done with generic
 tools like winzip?

I tried to open and unzip with WinRAR but couldn't make sense of the content.

BTW: easy_install for other installers like matplotlib also works
nicely for virtualenv



However, the official installers are only for 32-bit python, and I
appreciate all the efforts to modernize the numpy and scipy builds.

Josef


 --
 Olivier
 http://twitter.com/ogrisel - http://github.com/ogrisel
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Olivier Grisel

Hi Carl,

I installed Python 2.7.6 64 bits on a windows server instance from
rackspace cloud and then ran get-pip.py and then could successfully
install the numpy and scipy wheel packages from your google drive
folder. I tested dot products and scipy.linalg.svd and they work as
expected.

Then I uncompressed your mingw toolchain in c:\mingw, put c:\mingw\bin
in my PATH and tried to build the scikit-learn git master with it,
however it fails with:

building 'sklearn.__check_build._check_build' extension
compiling C sources
C compiler: gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes

compile options: '-D__MSVCRT_VERSION__=0x0900
-Ic:\Python27\lib\site-packages\numpy\core\include
-Ic:\Python27\lib\site-packages\numpy\core\include -Ic:\Python2
7\include -Ic:\Python27\PC -c'
gcc -DMS_WIN64 -O2 -msse -msse2 -Wall -Wstrict-prototypes
-D__MSVCRT_VERSION__=0x0900
-Ic:\Python27\lib\site-packages\numpy\core\include
-Ic:\Python27\lib\site-
packages\numpy\core\include -Ic:\Python27\include -Ic:\Python27\PC -c
sklearn\__check_build\_check_build.c -o
build\temp.win-amd64-2.7\Release\sklearn\__check_b
uild\_check_build.o
Found executable c:\mingw\bin\gcc.exe
gcc -shared -Wl,-gc-sections -Wl,-s
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
-Lc:\Python27\libs -Lc:\Python27\PCbuild\amd64 -Lbuild
\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x3):
undefined reference to `__imp__Py_NoneStruct'
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x1ca):
undefined reference to `__imp__PyThreadState_Current'
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o:_check_build.c:(.text+0x405):
undefined reference to `__imp_PyExc_ImportError'
c:/mingw/bin/../lib/gcc/x86_64-w64-mingw32/4.8.2/../../../../x86_64-w64-mingw32/bin/ld.exe:
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.
o: bad reloc address 0x0 in section `.data'
collect2.exe: error: ld returned 1 exit status
error: Command gcc -shared -Wl,-gc-sections -Wl,-s
build\temp.win-amd64-2.7\Release\sklearn\__check_build\_check_build.o
-Lc:\Python27\libs -Lc:\Python27\PCbui
ld\amd64 -Lbuild\temp.win-amd64-2.7 -lpython27 -lmsvcr90 -o
build\lib.win-amd64-2.7\sklearn\__check_build\_check_build.pyd failed
with exit status 1

Furthermore, when I try to introspect the blas information on this box I get:

In [1]: import scipy
C:\Python27\lib\site-packages\numpy\core\__init__.py:6: Warning: Numpy
64bit experimental build with Mingw-w64 and OpenBlas. Use with care.
  from . import multiarray
OpenBLAS : Your OS does not support AVX instructions. OpenBLAS is
using Barcelona kernels as a fallback, which may give poorer
performance.

In [2]: scipy.show_config()
umfpack_info:
  NOT AVAILABLE
lapack_opt_info:
libraries = ['openblas', 'openblas']
library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
language = f77
blas_opt_info:
libraries = ['openblas', 'openblas']
library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
language = f77
openblas_info:
libraries = ['openblas', 'openblas']
library_dirs = ['D:/devel/mingw64static/x86_64-w64-mingw32/lib']
language = f77
blas_mkl_info:
  NOT AVAILABLE

In [3]: from numpy.distutils.system_info import get_info

In [4]: get_info('blas_opt')
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:576:
UserWarning: Specified path
D:/devel/mingw64static/x86_64-w64-mingw32/lib is invalid.
  warnings.warn('Specified path %s is invalid.' % d)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1522: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
  warnings.warn(AtlasNotFoundError.__doc__)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1531: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
  warnings.warn(BlasNotFoundError.__doc__)
C:\Python27\lib\site-packages\numpy\distutils\system_info.py:1534: UserWarning:
Blas (http://www.netlib.org/blas/) sources not found.
Directories to search for the sources can be specified in the
numpy/distutils/site.cfg file (section [blas_src]) or by setting
the BLAS_SRC environment variable.
  warnings.warn(BlasSrcNotFoundError.__doc__)
Out[4]: {}

Would it make sense to embed the blas and lapack header files as part
of this numpy wheel and make numpy.distutils.system_info return the
lib and include folder pointing to the embedded libopenblas.dll and
header files so has to make third

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Julian Taylor

On 26.03.2014 16:27, Olivier Grisel wrote:
 Hi Carl,
 
 I installed Python 2.7.6 64 bits on a windows server instance from
 rackspace cloud and then ran get-pip.py and then could successfully
 install the numpy and scipy wheel packages from your google drive
 folder. I tested dot products and scipy.linalg.svd and they work as
 expected.
 

 
 Would it make sense to embed the blas and lapack header files as part
 of this numpy wheel and make numpy.distutils.system_info return the
 lib and include folder pointing to the embedded libopenblas.dll and
 header files so has to make third party libraries directly buildable
 against those?
 

as for using openblas by default in binary builds, no.
pthread openblas build is now fork safe which is great but it is still
not reliable enough for a default.
E.g. the current latest release 0.2.8 still has one crash bug on
dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3].
git head has the former four fixed bug still has wrong results for cgemv.
The not so old 0.2.8 also fixed whole bunch more crashes and wrong
result issues (crashes on QR, uninitialized data use in dgemm, ...).
None of the fixes received unit tests, so I am somewhat pessimistic that
it will improve, especially as the maintainer is dissertating (is that
the right word?) and most of the code is assembler code only few people
can write (it is simply not required anymore, we have code generators
and intrinsics for that).

Openblas is great if you do not have the patience to build ATLAS and
only use a restricted set of functionality and platforms you can easily
test.
Currently it is in my opinion not suitable for a general purpose library
like numpy.

I don't have any objections to adding get_info(openblas) if that does
not work yet. Patches welcome.

[0] https://github.com/xianyi/OpenBLAS/issues/304
[1] https://github.com/xianyi/OpenBLAS/issues/333
[2] https://github.com/xianyi/OpenBLAS/issues/340
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Nathaniel Smith

On Wed, Mar 26, 2014 at 7:34 PM, Julian Taylor
jtaylor.deb...@googlemail.com wrote:
 as for using openblas by default in binary builds, no.
 pthread openblas build is now fork safe which is great but it is still
 not reliable enough for a default.
 E.g. the current latest release 0.2.8 still has one crash bug on
 dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3].
 git head has the former four fixed bug still has wrong results for cgemv.
 The not so old 0.2.8 also fixed whole bunch more crashes and wrong
 result issues (crashes on QR, uninitialized data use in dgemm, ...).
 None of the fixes received unit tests, so I am somewhat pessimistic that
 it will improve, especially as the maintainer is dissertating (is that
 the right word?) and most of the code is assembler code only few people
 can write (it is simply not required anymore, we have code generators
 and intrinsics for that).

 Openblas is great if you do not have the patience to build ATLAS and
 only use a restricted set of functionality and platforms you can easily
 test.
 Currently it is in my opinion not suitable for a general purpose library
 like numpy.

Those problems you list are pretty damning, but neither is it
reasonable to expect everyone to manually build ATLAS on every machine
they use (or their students use, or...) :-(. So what other options do
we have for general purpose builds? Give up and use MKL? How's
eigen-blas doing these days? (I guess from skimming their docs they
use OpenMP?)

-- 
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Julian Taylor

On 26.03.2014 21:41, Nathaniel Smith wrote:
 On Wed, Mar 26, 2014 at 7:34 PM, Julian Taylor
 jtaylor.deb...@googlemail.com wrote:
 as for using openblas by default in binary builds, no.
 pthread openblas build is now fork safe which is great but it is still
 not reliable enough for a default.
 E.g. the current latest release 0.2.8 still has one crash bug on
 dgemv[1], and wrong results zherk/zer2[2] and dgemv/cgemv[3].
 git head has the former four fixed bug still has wrong results for cgemv.
 The not so old 0.2.8 also fixed whole bunch more crashes and wrong
 result issues (crashes on QR, uninitialized data use in dgemm, ...).
 None of the fixes received unit tests, so I am somewhat pessimistic that
 it will improve, especially as the maintainer is dissertating (is that
 the right word?) and most of the code is assembler code only few people
 can write (it is simply not required anymore, we have code generators
 and intrinsics for that).

 Openblas is great if you do not have the patience to build ATLAS and
 only use a restricted set of functionality and platforms you can easily
 test.
 Currently it is in my opinion not suitable for a general purpose library
 like numpy.
 
 Those problems you list are pretty damning, but neither is it
 reasonable to expect everyone to manually build ATLAS on every machine
 they use (or their students use, or...) :-(. So what other options do
 we have for general purpose builds? Give up and use MKL? How's
 eigen-blas doing these days? (I guess from skimming their docs they
 use OpenMP?)
 

I don't think general purpose builds need to have perfect performance.
we should provide something that works and allow users to tune it when
required. The slower general purpose build is also a great testcase to
verify that the tuned build works for your problem.

I didn't notice this is a reply to third party provided win64 binaries
with openblas. I though it was about official numpy binaries with
openblas again.
Third party binaries using openblas are great, especially these that
seem to warn that this is experimental.
It helps ironing out the kinks of openblas. Thanks for providing them.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Olivier Grisel

My understanding of Carl's effort is that the long term goal is to
have official windows whl packages for both numpy and scipy published
on PyPI with a builtin BLAS / LAPACK implementation so that users can
do `pip install scipy` under windows and get something that just works
without have to install any compiler (fortran or C) nor any additional
library manually.

Most windows users are beginners and you cannot really expect them to
understand how to build the whole scipy stack from source.

The current solution (executable setup installers) is not optimal as
it requires Administrator rights to run, does not resolve dependencies
as pip does and cannot be installed in virtualenvs.

If we can build numpy / scipy whl packages for windows with the Atlas
dlls then fine embedded in the numpy package then good. It does not
need to be the
fastest BLAS / LAPACK lib in my opinion. Just something that works.

The problem with ATLAS is that you need to select the number of thread
at build time AFAIK. But we could set it to a reasonable default (e.g.
4 threads) for the default windows package.

-- 
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Julian Taylor

On 26.03.2014 22:17, Olivier Grisel wrote:
 
 The problem with ATLAS is that you need to select the number of thread
 at build time AFAIK. But we could set it to a reasonable default (e.g.
 4 threads) for the default windows package.
 

You have to set the number of threads at build time with OpenBlas too.
At runtime it then selects the number of online cpus but limited to the
build time maximum.
It defaults to the maximum of the machine it was built on. You need to
explicitly override that for generic binaries. (I think debian binaries
uses 64 which is probably reasonable for non MIC systems)
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-03-26 Thread Olivier Grisel

2014-03-26 22:31 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com:
 On 26.03.2014 22:17, Olivier Grisel wrote:

 The problem with ATLAS is that you need to select the number of thread
 at build time AFAIK. But we could set it to a reasonable default (e.g.
 4 threads) for the default windows package.


 You have to set the number of threads at build time with OpenBlas too.
 At runtime it then selects the number of online cpus but limited to the
 build time maximum.
 It defaults to the maximum of the machine it was built on. You need to
 explicitly override that for generic binaries. (I think debian binaries
 uses 64 which is probably reasonable for non MIC systems)

Yes, the official windows binary for OpenBLAS is also build with a
maximum number of threads of 64 (NUM_THREADS=64) which I find
reasonable since the runtime number of threads will be capped by the
actual number of cores.

For ATLAS I don't think there is a runtime cap, or is there?

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-21 Thread Olivier Grisel

2014-02-20 23:56 GMT+01:00 Carl Kleffner cmkleff...@gmail.com:
 Hi,

 2014-02-20 23:17 GMT+01:00 Olivier Grisel olivier.gri...@ensta.org:

 I had a quick look (without running the procedure) but I don't
 understand some elements:

 - apparently you never tell in the numpy's site.cfg nor the scipy.cfg
 to use the openblas lib nor set the
 library_dirs: how does numpy.distutils know that it should dynlink
 against numpy/core/libopenblas.dll


 numpy's site.cfg is something like: (64 bit)

 [openblas]
 libraries = openblas
 library_dirs = D:/devel/mingw64static/x86_64-w64-mingw32/lib
 include_dirs = D:/devel/mingw64static/x86_64-w64-mingw32/include

 or (32 bit)

 [openblas]
 libraries = openblas
 library_dirs = D:/devel32/mingw32static/i686-w64-mingw32/lib
 include_dirs = D:/devel32/mingw32static/i686-w64-mingw32/include

Thanks, what I don't understand is how the libopenblas.dll will be
found at runtime. Is it a specific feature of windows? For instance
how would the scipy/linalg/_*.so file know that the libopenblas.dll
can be found in $PYTHONPATH/numpy/core?

 Please adapt the paths of course and apply the patches to numpy.


 - how to you deal with the link to the following libraries:
 libgfortran.3.dll
 libgcc_s.1.dll
 libquadmath.0.dll

 You won't need them. I build the toolchain statically. Thus you don't have
 to mess up with GCC runtime libs. You can check the dependencies with MS
 depends or with ntldd (included in the toolchain)

Great! I did not know it was possible. I guess that if we want to
replicate that for Linux and Mac we will have to also build custom
static GCC toolchains as well. Is there a good reference doc somewhere
on how to do so? When googling I only find posts by people who cannot
make their toolchain build statically correctly.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 Thread Julian Taylor

On Thu, Feb 20, 2014 at 1:25 AM, Nathaniel Smith n...@pobox.com wrote:
 Hey all,

 Just a heads up: thanks to the tireless work of Olivier Grisel, the OpenBLAS
 development branch is now fork-safe when built with its default threading
 support. (It is still not thread-safe when built using OMP for threading and
 gcc, but this is not the default.)

 Gory details: https://github.com/xianyi/OpenBLAS/issues/294

 Check it out - if it works you might want to consider lobbying your favorite
 distro to backport it.


debian unstable and the upcoming ubuntu 14.04 are already fixed.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 11:32 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com:
 On Thu, Feb 20, 2014 at 1:25 AM, Nathaniel Smith n...@pobox.com wrote:
 Hey all,

 Just a heads up: thanks to the tireless work of Olivier Grisel, the OpenBLAS
 development branch is now fork-safe when built with its default threading
 support. (It is still not thread-safe when built using OMP for threading and
 gcc, but this is not the default.)

 Gory details: https://github.com/xianyi/OpenBLAS/issues/294

 Check it out - if it works you might want to consider lobbying your favorite
 distro to backport it.


 debian unstable and the upcoming ubuntu 14.04 are already fixed.

Nice!

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 Thread Sturla Molden

Will this mean NumPy, SciPy et al. can start using OpenBLAS in the
official binary packages, e.g. on Windows and Mac OS X? ATLAS is slow and
Accelerate conflicts with fork as well. 

Will dotblas be built against OpenBLAS? AFAIK, it is only buit against
ATLAS or MKL, not any other BLAS, but it should just be a matter of
changing the build/link process.

Sturla



Nathaniel Smith n...@pobox.com wrote:
 Hey all,
 
 Just a heads up: thanks to the tireless work of Olivier Grisel, the
 OpenBLAS development branch is now fork-safe when built with its default
 threading support. (It is still not thread-safe when built using OMP for
 threading and gcc, but this is not the default.)
 
 Gory details: a
 href=https://github.com/xianyi/OpenBLAS/issues/294;https://github.com/xianyi/OpenBLAS/issues/294/a
 
 Check it out - if it works you might want to consider lobbying your
 favorite distro to backport it.
 
 -n
 
 ___ NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org a
 href=http://mail.scipy.org/mailman/listinfo/numpy-discussion;http://mail.scipy.org/mailman/listinfo/numpy-discussion/a

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 14:28 GMT+01:00 Sturla Molden sturla.mol...@gmail.com:
 Will this mean NumPy, SciPy et al. can start using OpenBLAS in the
 official binary packages, e.g. on Windows and Mac OS X? ATLAS is slow and
 Accelerate conflicts with fork as well.

This what I would like to do personnally. Ideally as a distribution of
wheel packages

To do so I built the current develop branch of OpenBLAS with:

  make USE_OPENMP=0 NUM_THREAD=32 NO_AFFINITY=1
  make PREFIX=/opt/OpenBLAS-noomp install

Then I added a site.cfg file in the numpy source folder with the lines:

  [openblas]
  libraries = openblas
  library_dirs = /opt/OpenBLAS-noomp/lib
  include_dirs = /opt/OpenBLAS-noomp/include


 Will dotblas be built against OpenBLAS?

Yes:

  $ ldd numpy/core/_dotblas.so
  linux-vdso.so.1 =  (0x7fff24d04000)
  libopenblas.so.0 = /opt/OpenBLAS-noomp/lib/libopenblas.so.0
(0x7f432882f000)
  libc.so.6 = /lib/x86_64-linux-gnu/libc.so.6 (0x7f4328449000)
  libm.so.6 = /lib/x86_64-linux-gnu/libm.so.6 (0x7f432814c000)
  libpthread.so.0 = /lib/x86_64-linux-gnu/libpthread.so.0
(0x7f4327f2f000)
  libgfortran.so.3 = /usr/lib/x86_64-linux-gnu/libgfortran.so.3
(0x7f4327c18000)
  /lib64/ld-linux-x86-64.so.2 (0x7f43298d3000)
  libquadmath.so.0 = /usr/lib/x86_64-linux-gnu/libquadmath.so.0
(0x7f43279e1000)


However when testing this I noticed the following strange slow import
and memory usage in an IPython session:

 import os, psutil
 psutil.Process(os.getpid()).get_memory_info().rss / 1e6
20.324352
 %time import numpy
CPU times: user 1.95 s, sys: 1.3 s, total: 3.25 s
Wall time: 530 ms
 psutil.Process(os.getpid()).get_memory_info().rss / 1e6
349.507584

The libopenblas.so file is just 14MB so I don't understand how I could
get those 330MB from.

It's even worst when using static linking (libopenblas.a instead of
libopenblas.so under linux).

With Atlas or MKL I get import times under 50 ms and the memory
overhead of the numpy import is just ~15MB.

I would be very interested in any help on this:

- can you reproduce this behavior?
- do you have an idea of a possible cause?
- how to investigate?

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

FYI: to build scipy against OpenBLAS I used the following site.cfg at
the root of my scipy source folder:

[DEFAULT]
library_dirs = /opt/OpenBLAS-noomp/lib:/usr/local/lib
include_dirs = /opt/OpenBLAS-noomp/include:/usr/local/include

[blas_opt]
libraries = openblas

[lapack_opt]
libraries = openblas

But this is unrelated to the previous numpy memory pattern as it
occurs independendly of scipy.

-- 
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Hi,

some days ago I put some preliminary mingw-w64 binaries and code based on
python2.7 on my google drive to discuss it with Matthew Brett. Maybe its
time for a broader discussion. IMHO it is ready for testing but not for
consumption.

url:
https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1kusp=sharing

contains:

(1) patches used

  numpy.patch
  scipy.patch


(2) 64 bit GCC toolchain
amd64/
 mingw-w64-toolchain-static_amd64-gcc-4.8.2_vc90_rev-20140131.7z
 libpython27.a

(3) numpy-1.8.0 linked against OpenBLAS
amd64/numpy-1.8.0/
  numpy-1.8.0.win-amd64-py2.7.exe
  numpy-1.8.0-cp27-none-win_amd64.whl
  numpy_amd64_fcompiler.log
  numpy_amd64_build.log
  numpy_amd64_test.log
  _numpyconfig.h
  config.h

(4) scipy-0.13.3 linked against OpenBLAS
amd64/scipy-0.13.3/
  scipy-0.13.3.win-amd64-py2.7.exe
  scipy-0.13.3-cp27-none-win_amd64.whl
  scipy_amd64_fcompiler.log
  scipy_amd64_build.log
  scipy_amd64_build_cont.log
  scipy_amd64_test._segfault.log
  scipy_amd64_test.log


(5) 32 bit GCC toolchain
win32/
  mingw-w64-toolchain-static_win32-gcc-4.8.2_vc90_rev-20140131.7z
  libpython27.a

(6) numpy-1.8.0 linked against OpenBLAS
win32/numpy-1.8.0/
  numpy-1.8.0.win32-py2.7.exe
  numpy-1.8.0-cp27-none-win32.whl
  numpy_win32_fcompiler.log
  numpy_win32_build.log
  numpy_win32_test.log
  _numpyconfig.h
  config.h

(7) scipy-0.13.3 linked against OpenBLAS
win32/scipy-0.13.3/
  scipy-0.13.3.win32-py2.7.exe
  scipy-0.13.3-cp27-none-win32.whl
  scipy_win32_fcompiler.log
  scipy_win32_build.log
  scipy_win32_build_cont.log
  scipy_win32_test.log

Summary to compile numpy:

(1) mingw\bin and python should be in the PATH. Choose 32 bit or 64 bit
architecture.
(2) copy libpython27.a to python\libs
 check, that python\libs does not contain libmsvcr90.a
(3) apply numpy.patch
(4) copy libopenblas.dll from mingw\bin to numpy\core
of course don't ever mix 32bit and 64 bit code
(5) create a site.cfg in the numpy folder with the absolute path to the
mingw import
files/header files. I copied the openblas header files, importlibs into
the GCC toolchain.
(6) create a mingw distutils.cfg file
(7) test the configuration
python setup.py config_fc --verbose
and
python setup.py build --help-fcompiler
(8) build
python setup.py build --fcompiler=gnu95
(9) make a distro
python setup.py bdist --format=wininst
(10) make a wheel
 wininst2wheel numpy-1.8.0.win32-py2.7.exe  (for 32 bit)
(11) install
 wheel install numpy-1.8.0-cp27-none-win32.whl
(12) import numpy; numpy.test()

Summary to compile scipy:

(1) apply scipy.patch
(2) python setup.py build --fcompiler=gnu95
and a second time
python setup.py build --fcompiler=gnu95
(3) python setup.py bdist --format=wininst
(4) install
(5) import scipy; scipy.test()

Hints:

(1) libpython import file:

The libpython27.a import files has been generated with gendef and dlltool
according to the recommendations on the mingw-w64 faq  site. It is
essential to not use import libraries from anywhere, but create it with the
tools in the GCC toolchain. The GCC toolchains contains correct generated
mscvrXX import files per default.

(2) OpenBLAS:

the openblas DLL must be copied to numpy/core before building numpy. All
Blas and Lapack code will be linked dynamically to this DLL. Because of
this the overall distro size gets much smaller compared to numpy-MKL or
scipy-MKL. It is not necessary to add numpy/core to the path! (at least on
my machine). To load libopenblas.dll to the process space it is only
necessary to import numpy - nothing else. libopenblas.dll is linked against
the msvcr90.dll, just like python. The DLL itself is a fat binary
containing all optimized kernels for all supported platforms. DLL, headers
and import files have been included into the toolchain.

(3) mingw-w64 toolchain:

In short it is an extended version of the 'recommended' mingw-builds
toolchain with some minor patches and customizations. I used
https://github.com/niXman/mingw-builds for my build. It is a 'statically'
build, thus all gcc related runtimes are linked statically into the
resulting binaries.

(4) Results:

Some FAILS - see corresp. log-files. I got a segfault with scipy.test() (64
bit) with multithreaded OpenBLAS (test_ssygv_1) but not in single threaded
mode. Due to time constraints I didn't made further tests right now.

Regards

Carl


2014-02-20 14:28 GMT+01:00 Sturla Molden sturla.mol...@gmail.com:

 Will this mean NumPy, SciPy et al. can start using OpenBLAS in the
 official binary packages, e.g. on Windows and Mac OS X? ATLAS is slow and
 Accelerate conflicts with fork as well.

 Will dotblas be built against OpenBLAS? AFAIK, it is only buit against
 ATLAS or MKL, not any other BLAS, but it should just be a matter of
 changing the build/link process.

 Sturla



 Nathaniel Smith n...@pobox.com wrote:
  Hey all,
 
  Just a heads up: thanks to the tireless work of Olivier Grisel, the
  OpenBLAS development branch is now fork-safe when built with its

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Thanks for sharing, this is all very interesting.

Have you tried to have a look at the memory usage and import time of
numpy when linked against libopenblas.dll?

-- 
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 Thread Julian Taylor

On Thu, Feb 20, 2014 at 3:50 PM, Olivier Grisel
olivier.gri...@ensta.org wrote:
 Thanks for sharing, this is all very interesting.

 Have you tried to have a look at the memory usage and import time of
 numpy when linked against libopenblas.dll?

 --

this is probably caused by the memory warmup
it can be disabled with NO_WARMUP=1 in some configuration file.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

looked at the taskmanager there is not much difference to numpy-MKL. I
didn't made any qualified measurements however.

Carl


2014-02-20 15:50 GMT+01:00 Olivier Grisel olivier.gri...@ensta.org:

 Thanks for sharing, this is all very interesting.

 Have you tried to have a look at the memory usage and import time of
 numpy when linked against libopenblas.dll?

 --
 Olivier
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

good point, I didn't used this option.

Carl


2014-02-20 16:01 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com:

 On Thu, Feb 20, 2014 at 3:50 PM, Olivier Grisel
 olivier.gri...@ensta.org wrote:
  Thanks for sharing, this is all very interesting.
 
  Have you tried to have a look at the memory usage and import time of
  numpy when linked against libopenblas.dll?
 
  --

 this is probably caused by the memory warmup
 it can be disabled with NO_WARMUP=1 in some configuration file.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

2014-02-20 16:01 GMT+01:00 Julian Taylor jtaylor.deb...@googlemail.com:

 this is probably caused by the memory warmup
 it can be disabled with NO_WARMUP=1 in some configuration file.

This was it, I now get:

 import os, psutil
 psutil.Process(os.getpid()).get_memory_info().rss / 1e6
20.324352
 %time import numpy
CPU times: user 84 ms, sys: 464 ms, total: 548 ms
Wall time: 59.3 ms
 psutil.Process(os.getpid()).get_memory_info().rss / 1e6
27.906048

Thanks for the tip.

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

I had a quick look (without running the procedure) but I don't
understand some elements:

- apparently you never tell in the numpy's site.cfg nor the scipy.cfg
to use the openblas lib nor set the
library_dirs: how does numpy.distutils know that it should dynlink
against numpy/core/libopenblas.dll

- how to you deal with the link to the following libraries:
libgfortran.3.dll
libgcc_s.1.dll
libquadmath.0.dll

If MinGW is installed on the system I assume that the linker will find
them. But would it work when the wheel packages are installed on a
system that does not have MinGW installed?

Best,

-- 
Olivier
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe