Re: [Numpy-discussion] Is numpy.test() supposed to be multithreaded?

[Chris Barker - NOAA Federal]

> Now the user is writing back to say, "my test code is fast now, but
> numpy.test() is still about three times slower than  don't manage>".  When I watch htop as numpy.test() executes, sure enough,
> it's using one core
>

* if numpy.test() is supposed to be using multiple cores, why isn't it,
> when we've established with other test code that it's now using multiple
> cores?
>

Some numpy.linalg functions (like np.dot) will be using multiple cores, but
np.linalg.test() takes only ~1% of the time of the full test suite.
Everything else will be running single core. So your observations are not
surprising.


Though why it would run slower on one box than another comparable box is a
mystery...

-CHB
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Is numpy.test() supposed to be multithreaded?

[Ralf Gommers]

On Tue, Jun 28, 2016 at 10:36 PM, Michael Ward  wrote:

> Heya, I'm not a numbers guy, but I maintain servers for scientists and
> researchers who are.  Someone pointed out that our numpy installation on a
> particular server was only using one core.  I'm unaware of the who/how the
> previous version of numpy/OpenBLAS were installed, so I installed them from
> scratch, and confirmed that the users test code now runs on multiple cores
> as expected, drastically increasing performance time.
>
> Now the user is writing back to say, "my test code is fast now, but
> numpy.test() is still about three times slower than  don't manage>".  When I watch htop as numpy.test() executes, sure enough,
> it's using one core.  Now I'm not sure if that's the expected behavior or
> not.  Questions:
>
> * if numpy.test() is supposed to be using multiple cores, why isn't it,
> when we've established with other test code that it's now using multiple
> cores?
>

Some numpy.linalg functions (like np.dot) will be using multiple cores, but
np.linalg.test() takes only ~1% of the time of the full test suite.
Everything else will be running single core. So your observations are not
surprising.

Cheers,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Is numpy.test() supposed to be multithreaded?

[Michael Ward]

Heya, I'm not a numbers guy, but I maintain servers for scientists and 
researchers who are.  Someone pointed out that our numpy installation on 
a particular server was only using one core.  I'm unaware of the who/how 
the previous version of numpy/OpenBLAS were installed, so I installed 
them from scratch, and confirmed that the users test code now runs on 
multiple cores as expected, drastically increasing performance time.


Now the user is writing back to say, "my test code is fast now, but 
numpy.test() is still about three times slower than we don't manage>".  When I watch htop as numpy.test() executes, sure 
enough, it's using one core.  Now I'm not sure if that's the expected 
behavior or not.  Questions:


* if numpy.test() is supposed to be using multiple cores, why isn't it, 
when we've established with other test code that it's now using multiple 
cores?


* if numpy.test() is not supposed to be using multiple cores, what could 
be the reason that the performance is drastically slower than another 
server with a comparable CPU, when the user's test code performs 
comparably?


For what it's worth, the users "test" code which does run on multiple 
cores is as simple as:


size=4000
a = np.random.random_sample((size,size))
b = np.random.random_sample((size,size))
x = np.dot(a,b)

Whereas this uses only one core:

numpy.test()

---

OpenBLAS 0.2.18 was basically just compiled with "make", nothing special 
to it.  Numpy 1.11.0 was installed from source (python setup.py 
install), using a site.cfg file to point numpy to the new OpenBLAS.


Thanks,
Mike
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] ANN: Bokeh 0.12 Released

[Bryan Van de Ven]

Hi all,

On behalf of the Bokeh team, I am pleased to announce the release of version 
0.12.0 of Bokeh!

This release was a major update, and was focused on areas of layout and 
styling, new JavaScript APIs for BokehJS, and improvements to the Bokeh Server. 
But there were many additional improvements in other areas as well. Rather than 
try to describe all the changes here, I encourage every one to check out the 
new project blog:

https://bokeh.github.io/blog/2016/6/28/release-0-12/

which has details as well as live demonstrations. And as always, see the 
CHANGELOG and Release Notes for full details.

If you are using Anaconda/miniconda, you can install it with conda:

conda install bokeh

Alternatively, you can also install it with pip:

pip install bokeh

Full information including details about how to use and obtain BokehJS are at:

http://bokeh.pydata.org/en/0.12.0/docs/installation.html

Issues, enhancement requests, and pull requests can be made on the Bokeh Github 
page: https://github.com/bokeh/bokeh

Documentation is available at http://bokeh.pydata.org/en/0.12.0

Questions can be directed to the Bokeh mailing list: bo...@continuum.io or the 
Gitter Chat room: https://gitter.im/bokeh/bokeh

Thanks,

Bryan Van de Ven
Continuum Analytics
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Ralf Gommers]

On Tue, Jun 28, 2016 at 5:50 PM, Chris Barker  wrote:

>
>
>
>
> > This doesn't really matter too much imho, we have to support Accelerate
>> > either way.
>>
>
> do we? -- so if we go OpenBlas, and someone want to do a simple build from
> source, what happens? Do they get accelerate?
>

Indeed, unless they go through the effort of downloading a separate BLAS
and LAPACK, and figuring out how to make that visible to numpy.distutils.
Very few users will do that.


> or would we ship OpenBlas source itself?
>

Definitely don't want to do that.


> or would they need to install OpenBlas some other way?
>

Yes, or MKL, or ATLAS, or BLIS. We have support for all these, and that's a
good thing. Making a uniform choice for our official binaries on various
OSes doesn't reduce the need or effort for supporting those other options.


>
> >> >> Faster to fix bugs with good support from main developer.  No
>> >> >> multiprocessing crashes for Python 2.7.
>>
>
> this seems to be the compelling one.
>
> How does the performance compare?
>

For most routines performance seems to be comparable, and both are much
better than ATLAS. When there's a significant difference, I have the
impression that OpenBLAS is more often the slower one (example:
https://github.com/xianyi/OpenBLAS/issues/533).

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Chris Barker]

On Tue, Jun 28, 2016 at 8:15 AM, Matthew Brett 
wrote:

> > dropping the dual arch stuff is probably the way to go - the 32-bit
> builds
> > make very little sense these days.
>
> Yes, that's true, but as you know, the OSX system and Python.org
> Pythons are still dual arch, so technically a matching wheel should
> also be dual arch.


but as they say, practicality beat purity...

It's not clear yet whether 3.6 will be built dual arch at this point, but
in any case, no one is going to go back and change the builds on 2.7 or 3.4
or 3.5 

But that doesn't mean we necessarily need to support dual arch downstream.
Personally, I"d drop it and see if anyone screams.

Though it's actually a bit tricky, at least with my knowledge to build a 64
bit only extension against the dual-arch build. At least the only way I
figured out was to hack the install. ( I did this a while back when I
needed a 32bit-only build -- ironic?)


> This doesn't really matter too much imho, we have to support Accelerate
> > either way.
>

do we? -- so if we go OpenBlas, and someone want to do a simple build from
source, what happens? Do they get accelerate? or would we ship OpenBlas
source itself? or would they need to install OpenBlas some other way?

>> >> Faster to fix bugs with good support from main developer.  No
> >> >> multiprocessing crashes for Python 2.7.
>

this seems to be the compelling one.

How does the performance compare?

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR(206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115   (206) 526-6317   main reception

chris.bar...@noaa.gov
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Matthew Brett]

Hi,

On Tue, Jun 28, 2016 at 7:33 AM, Ralf Gommers  wrote:
>
>
> On Tue, Jun 28, 2016 at 2:55 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Tue, Jun 28, 2016 at 5:25 AM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Mon, Jun 27, 2016 at 9:46 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> I just succeeded in getting an automated dual arch build of numpy and
>> >> scipy, using OpenBLAS.  See the last three build jobs in these two
>> >> build matrices:
>> >>
>> >> https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119
>> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673
>> >>
>> >> Tests are passing on 32 and 64-bit.
>> >>
>> >> I didn't upload these to the usual Rackspace container at
>> >> wheels.scipy.org to avoid confusion.
>> >>
>> >> So, I guess the question now is - should we switch to shipping
>> >> OpenBLAS wheels for the next release of numpy and scipy?  Or should we
>> >> stick with the Accelerate framework that comes with OSX?
>> >>
>> >> In favor of the Accelerate build : faster to build, it's what we've
>> >> been doing thus far.
>
>
> Faster to build isn't really an argument right? Should be the same build
> time except for building OpenBLAS itself once per OpenBLAS version. And only
> applies to building wheels for releases - nothing changes for source builds
> done by users on OS X. If build time ever becomes a real issue, then
> dropping the dual arch stuff is probably the way to go - the 32-bit builds
> make very little sense these days.

Yes, that's true, but as you know, the OSX system and Python.org
Pythons are still dual arch, so technically a matching wheel should
also be dual arch.   I agree that we're near the point where there's
near zero likelihood that the 32-bit arch will ever get exercised.

> What we've been doing thus far - that is the more important argument.
> There's a risk in switching, we may encounter new bugs or lose some
> performance in particular functions.
>
>>
>> >>
>> >> In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK
>> >> library cross platform,
>
>
> This doesn't really matter too much imho, we have to support Accelerate
> either way.
>
>>
>> when we have the Windows builds working.
>> >> Faster to fix bugs with good support from main developer.  No
>> >> multiprocessing crashes for Python 2.7.
>
>
> This is probably the main reason to make the switch, if we decide to do
> that.
>
>>
>>   I'm still a bit nervous about OpenBLAS, see
>> > https://github.com/scipy/scipy/issues/6286. That was with version
>> > 0.2.18,
>> > which is pretty recent.
>>
>> Well - we are committed to OpenBLAS already for the Linux wheels, so
>> if that failure was due to an error in OpenBLAS, we'll have to report
>> it and get it fixed / fix it ourselves upstream.
>
>
> Indeed. And those wheels have been downloaded a lot already, without any
> issues being reported.
>
> I'm +0 on the proposal - the risk seems acceptable, but the reasons to make
> the switch are also not super compelling.

I guess I'm about +0.5 (multiprocessing, simplifying mainstream blas /
lapack support) - I'm floating it now because I hadn't got the build
machinery working before.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Ralf Gommers]

On Tue, Jun 28, 2016 at 2:55 PM, Matthew Brett 
wrote:

> Hi,
>
> On Tue, Jun 28, 2016 at 5:25 AM, Charles R Harris
>  wrote:
> >
> >
> > On Mon, Jun 27, 2016 at 9:46 PM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> I just succeeded in getting an automated dual arch build of numpy and
> >> scipy, using OpenBLAS.  See the last three build jobs in these two
> >> build matrices:
> >>
> >> https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119
> >> https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673
> >>
> >> Tests are passing on 32 and 64-bit.
> >>
> >> I didn't upload these to the usual Rackspace container at
> >> wheels.scipy.org to avoid confusion.
> >>
> >> So, I guess the question now is - should we switch to shipping
> >> OpenBLAS wheels for the next release of numpy and scipy?  Or should we
> >> stick with the Accelerate framework that comes with OSX?
> >>
> >> In favor of the Accelerate build : faster to build, it's what we've
> >> been doing thus far.
>

Faster to build isn't really an argument right? Should be the same build
time except for building OpenBLAS itself once per OpenBLAS version. And
only applies to building wheels for releases - nothing changes for source
builds done by users on OS X. If build time ever becomes a real issue, then
dropping the dual arch stuff is probably the way to go - the 32-bit builds
make very little sense these days.

What we've been doing thus far - that is the more important argument.
There's a risk in switching, we may encounter new bugs or lose some
performance in particular functions.


> >>
> >> In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK
> >> library cross platform,
>

This doesn't really matter too much imho, we have to support Accelerate
either way.


> when we have the Windows builds working.
> >> Faster to fix bugs with good support from main developer.  No
> >> multiprocessing crashes for Python 2.7.
>

This is probably the main reason to make the switch, if we decide to do
that.


>   I'm still a bit nervous about OpenBLAS, see
> > https://github.com/scipy/scipy/issues/6286. That was with version
> 0.2.18,
> > which is pretty recent.
>
> Well - we are committed to OpenBLAS already for the Linux wheels, so
> if that failure was due to an error in OpenBLAS, we'll have to report
> it and get it fixed / fix it ourselves upstream.
>

Indeed. And those wheels have been downloaded a lot already, without any
issues being reported.

I'm +0 on the proposal - the risk seems acceptable, but the reasons to make
the switch are also not super compelling.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Matthew Brett]

Hi,

On Tue, Jun 28, 2016 at 5:25 AM, Charles R Harris
 wrote:
>
>
> On Mon, Jun 27, 2016 at 9:46 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> I just succeeded in getting an automated dual arch build of numpy and
>> scipy, using OpenBLAS.  See the last three build jobs in these two
>> build matrices:
>>
>> https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119
>> https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673
>>
>> Tests are passing on 32 and 64-bit.
>>
>> I didn't upload these to the usual Rackspace container at
>> wheels.scipy.org to avoid confusion.
>>
>> So, I guess the question now is - should we switch to shipping
>> OpenBLAS wheels for the next release of numpy and scipy?  Or should we
>> stick with the Accelerate framework that comes with OSX?
>>
>> In favor of the Accelerate build : faster to build, it's what we've
>> been doing thus far.
>>
>> In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK
>> library cross platform, when we have the Windows builds working.
>> Faster to fix bugs with good support from main developer.  No
>> multiprocessing crashes for Python 2.7.
>
>
> I'm still a bit nervous about OpenBLAS, see
> https://github.com/scipy/scipy/issues/6286. That was with version 0.2.18,
> which is pretty recent.

Well - we are committed to OpenBLAS already for the Linux wheels, so
if that failure was due to an error in OpenBLAS, we'll have to report
it and get it fixed / fix it ourselves upstream.

Cheers,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Accelerate or OpenBLAS for numpy / scipy wheels?

[Charles R Harris]

On Mon, Jun 27, 2016 at 9:46 PM, Matthew Brett 
wrote:

> Hi,
>
> I just succeeded in getting an automated dual arch build of numpy and
> scipy, using OpenBLAS.  See the last three build jobs in these two
> build matrices:
>
> https://travis-ci.org/matthew-brett/numpy-wheels/builds/140388119
> https://travis-ci.org/matthew-brett/scipy-wheels/builds/140684673
>
> Tests are passing on 32 and 64-bit.
>
> I didn't upload these to the usual Rackspace container at
> wheels.scipy.org to avoid confusion.
>
> So, I guess the question now is - should we switch to shipping
> OpenBLAS wheels for the next release of numpy and scipy?  Or should we
> stick with the Accelerate framework that comes with OSX?
>
> In favor of the Accelerate build : faster to build, it's what we've
> been doing thus far.
>
> In favor of OpenBLAS build : allows us to commit to one BLAS / LAPACK
> library cross platform, when we have the Windows builds working.
> Faster to fix bugs with good support from main developer.  No
> multiprocessing crashes for Python 2.7.
>

I'm still a bit nervous about OpenBLAS, see
https://github.com/scipy/scipy/issues/6286. That was with version 0.2.18,
which is pretty recent.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion