Re: [Numpy-discussion] Removal of some numpy files

[David Cournapeau]

tools/win32build is used to build the so-called superpack installers, which
we don't build anymore AFAIK

tools/numpy-macosx-installer is used to build the .dmg for numpy (also not
used anymore AFAIK).

On Sat, Feb 25, 2017 at 3:21 PM, Charles R Harris  wrote:

> Hi All,
>
> While looking through the numpy tools directory I noticed some scripts
> that look outdated that might be candidates for removal:
>
>1. tools/numpy-macosx-installer/
>2. tools/win32build/
>
> Does anyone know if either of those are stlll relevant?
>
> Cheers,
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.3, scipy 0.18.1, MSVC 2015 and crashes in complex functions

[David Cournapeau]

Indeed. I wrongly assumed that since gholke's wheels did not crash, they
did not run into that issue.

That sounds like an ABI issue, since I suspect intel math library supports
C99 complex numbers. I will add info on that issue then,

David

On Mon, Jan 23, 2017 at 11:46 AM, Evgeni Burovski <
evgeny.burovs...@gmail.com> wrote:

> Related to https://github.com/scipy/scipy/issues/6336?
> 23.01.2017 14:40 пользователь "David Cournapeau" <courn...@gmail.com>
> написал:
>
>> Hi there,
>>
>> While building the latest scipy on top of numpy 1.11.3, I have noticed
>> crashes while running the scipy test suite, in scipy.special (e.g. in
>> scipy.special hyp0f1 test).. This only happens on windows for python 3.5
>> (where we use MSVC 2015 compiler).
>>
>> Applying some violence to distutils, I re-built numpy/scipy with debug
>> symbols, and the debugger claims that crashes happen inside scipy.special
>> ufunc cython code, when calling clog or csqrt. I first suspected a compiler
>> bug, but disabling those functions in numpy, to force using our own
>> versions in npymath, made the problem go away.
>>
>> I am a bit suspicious about the whole thing as neither conda's or
>> gholke's wheel crashed. Has anybody else encountered this ?
>>
>> David
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Numpy 1.11.3, scipy 0.18.1, MSVC 2015 and crashes in complex functions

[David Cournapeau]

Hi there,

While building the latest scipy on top of numpy 1.11.3, I have noticed
crashes while running the scipy test suite, in scipy.special (e.g. in
scipy.special hyp0f1 test).. This only happens on windows for python 3.5
(where we use MSVC 2015 compiler).

Applying some violence to distutils, I re-built numpy/scipy with debug
symbols, and the debugger claims that crashes happen inside scipy.special
ufunc cython code, when calling clog or csqrt. I first suspected a compiler
bug, but disabling those functions in numpy, to force using our own
versions in npymath, made the problem go away.

I am a bit suspicious about the whole thing as neither conda's or gholke's
wheel crashed. Has anybody else encountered this ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NumPy 1.12.0 release

[David Cournapeau]

On Wed, Jan 18, 2017 at 11:43 AM, Julian Taylor <
jtaylor.deb...@googlemail.com> wrote:

> The version of gcc used will make a large difference in some places.
> E.g. the AVX2 integer ufuncs require something around 4.5 to work and in
> general the optimization level of gcc has improved greatly since the
> clang competition showed up around that time. centos 5 has 4.1 which is
> really ancient.
> I though the wheels used newer gccs also on centos 5?
>

I don't know if it is mandatory for many wheels, but it is possilbe to
build w/ gcc 4.8 at least, and still binary compatibility with centos 5.X
and above, though I am not sure about the impact on speed.

It has been quite some time already that building numpy/scipy with gcc 4.1
causes troubles with errors and even crashes anyway, so you definitely want
to use a more recent compiler in any case.

David


> On 18.01.2017 08:27, Nathan Goldbaum wrote:
> > I've seen reports on the anaconda mailing list of people seeing similar
> > speed ups when they compile e.g. Numpy with a recent gcc. Anaconda has
> > the same issue as manylinux in that they need to use versions of GCC
> > available on CentOS 5.
> >
> > Given the upcoming official EOL for CentOS5, it might make sense to
> > think about making a pep for a CentOS 6-based manylinux2 docker image,
> > which will allow compiling with a newer GCC.
> >
> > On Tue, Jan 17, 2017 at 9:15 PM Jerome Kieffer  > > wrote:
> >
> > On Tue, 17 Jan 2017 08:56:42 -0500
> >
> > Neal Becker >
> wrote:
> >
> >
> >
> > > I've installed via pip3 on linux x86_64, which gives me a wheel.
> My
> >
> > > question is, am I loosing significant performance choosing this
> > pre-built
> >
> > > binary vs. compiling myself?  For example, my processor might have
> > some more
> >
> > > features than the base version used to build wheels.
> >
> >
> >
> > Hi,
> >
> >
> >
> > I have done some benchmarking (%timeit) for my code running in a
> >
> > jupyter-notebook within a venv installed with pip+manylinux wheels
> >
> > versus ipython and debian packages (on the same computer).
> >
> > I noticed the debian installation was ~20% faster.
> >
> >
> >
> > I did not investigate further if those 20% came from the manylinux (I
> >
> > suspect) or from the notebook infrastructure.
> >
> >
> >
> > HTH,
> >
> > --
> >
> > Jérôme Kieffer
> >
> >
> >
> > ___
> >
> > NumPy-Discussion mailing list
> >
> > NumPy-Discussion@scipy.org 
> >
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Dropping sourceforge for releases.

[David Cournapeau]

+1 from me.

If we really need some distribution on top of github/pypi, note that
bintray (https://bintray.com/) is free for OSS projects, and is a much
better experience than sourceforge.

David

On Sun, Oct 2, 2016 at 12:02 AM, Charles R Harris  wrote:

> Hi All,
>
> Ralf has suggested dropping sourceforge as a NumPy release site. There was
> discussion of doing that some time back but we have not yet done it. Now
> that we put wheels up on PyPI for all supported architectures source forge
> is not needed. I note that there are still some 15,000 downloads a week
> from the site, so it is still used.
>
> Thoughts?
>
> Chuck
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Changing FFT cache to a bounded LRU cache

[David Cournapeau]

On Tue, May 31, 2016 at 10:36 PM, Sturla Molden 
wrote:

> Joseph Martinot-Lagarde  wrote:
>
> > The problem with FFTW is that its license is more restrictive (GPL), and
> > because of this may not be suitable everywhere numpy.fft is.
>
> A lot of us use NumPy linked with MKL or Accelerate, both of which have
> some really nifty FFTs. And the license issue is hardly any worse than
> linking with them for BLAS and LAPACK, which we do anyway. We could extend
> numpy.fft to use MKL or Accelerate when they are available.
>

That's what we used to do in scipy, but it was a PITA to maintain. Contrary
to blas/lapack, fft does not have a standard API, hence exposing a
consistent API in python, including data layout involved quite a bit of
work.

It is better to expose those through 3rd party APIs.

David


> Sturla
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Windows wheels, built, but should we deploy?

[David Cournapeau]

On Fri, Mar 4, 2016 at 4:42 AM, Matthew Brett 
wrote:

> Hi,
>
> Summary:
>
> I propose that we upload Windows wheels to pypi.  The wheels are
> likely to be stable and relatively easy to maintain, but will have
> slower performance than other versions of numpy linked against faster
> BLAS / LAPACK libraries.
>
> Background:
>
> There's a long discussion going on at issue github #5479 [1], where
> the old problem of Windows wheels for numpy came up.
>
> For those of you not following this issue, the current situation for
> community-built numpy Windows binaries is dire:
>
> * We have not so far provided windows wheels on pypi, so `pip install
> numpy` on Windows will bring you a world of pain;
> * Until recently we did provide .exe "superpack" installers on
> sourceforge, but these became increasingly difficult to build and we
> gave up building them as of the latest (1.10.4) release.
>
> Despite this, popularity of Windows wheels on pypi is high.   A few
> weeks ago, Donald Stufft ran a query for the binary wheels most often
> downloaded from pypi, for any platform [2] . The top five most
> downloaded were (n_downloads, name):
>
> 6646,
> numpy-1.10.4-cp27-none-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl
> 5445, cryptography-1.2.1-cp27-none-win_amd64.whl
> 5243, matplotlib-1.4.0-cp34-none-win32.whl
> 5241, scikit_learn-0.15.1-cp34-none-win32.whl
> 4573, pandas-0.17.1-cp27-none-win_amd64.whl
>
> So a) the OSX numpy wheel is very popular and b) despite the fact that
> we don't provide a numpy wheel for Windows, matplotlib, sckit_learn
> and pandas, that depend on numpy, are the 3rd, 4th and 5th most
> downloaded wheels as of a few weeks ago.
>
> So, there seems to be a large appetite for numpy wheels.
>
> Current proposal:
>
> I have now built numpy wheels, using the ATLAS blas / lapack library -
> the build is automatic and reproducible [3].
>
> I chose ATLAS to build against, rather than, say OpenBLAS, because
> we've had some significant worries in the past about the reliability
> of OpenBLAS, and I thought it better to err on the side of
> correctness.
>
> However, these builds are relatively slow for matrix multiply and
> other linear algebra routines compared numpy built against OpenBLAS or
> MKL (which we cannot use because of its license) [4].   In my very
> crude array test of a dot product and matrix inversion, the ATLAS
> wheels were 2-3 times slower than MKL.  Other benchmarks on Julia
> found about the same result for ATLAS vs OpenBLAS on 32-bit bit, but a
> much bigger difference on 64-bit (for an earlier version of ATLAS than
> we are currently using) [5].
>
> So, our numpy wheels likely to be stable and give correct results, but
> will be somewhat slow for linear algebra.
>

I would not worry too much about this: at worst, this gives us back the
situation where we were w/ so-called superpack, which have been successful
in the past to spread numpy use on windows.

My main worry is whether this locks us into ATLAS  for a long time because
of package depending on numpy blas/lapack (scipy, scikit learn). I am not
sure how much this is the case.

David


>
> I propose that we upload these ATLAS wheels to pypi.  The upside is
> that this gives our Windows users a much better experience with pip,
> and allows other developers to build Windows wheels that depend on
> numpy.  The downside is that these will not be optimized for
> performance on modern processors.  In order to signal that, I propose
> adding the following text to the numpy pypi front page:
>
> ```
> All numpy wheels distributed from pypi are BSD licensed.
>
> Windows wheels are linked against the ATLAS BLAS / LAPACK library,
> restricted to SSE2 instructions, so may not give optimal linear
> algebra performance for your machine. See
> http://docs.scipy.org/doc/numpy/user/install.html for alternatives.
> ```
>
> In a way this is very similar to our previous situation, in that the
> superpack installers also used ATLAS - in fact an older version of
> ATLAS.
>
> Once we are up and running with numpy wheels, we can consider whether
> we should switch to other BLAS libraries, such as OpenBLAS or BLIS
> (see [6]).
>
> I'm posting here hoping for your feedback...
>
> Cheers,
>
> Matthew
>
>
> [1] https://github.com/numpy/numpy/issues/5479
> [2] https://gist.github.com/dstufft/1dda9a9f87ee7121e0ee
> [3] https://ci.appveyor.com/project/matthew-brett/np-wheel-builder
> [4] http://mingwpy.github.io/blas_lapack.html#intel-math-kernel-library
> [5] https://github.com/numpy/numpy/issues/5479#issuecomment-185033668
> [6] https://github.com/numpy/numpy/issues/7372
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org

Re: [Numpy-discussion] PyData Madrid

[David Cournapeau]

On Sat, Feb 20, 2016 at 5:26 PM, Kiko  wrote:

>
>
> 2016-02-20 17:58 GMT+01:00 Ralf Gommers :
>
>>
>>
>> On Wed, Feb 17, 2016 at 9:46 PM, Sebastian Berg <
>> sebast...@sipsolutions.net> wrote:
>>
>>> On Mi, 2016-02-17 at 20:59 +0100, Jaime Fernández del Río wrote:
>>> > Hi all,
>>> >
>>> > I just found out there is a PyData Madrid happening in early April,
>>> > and it would feel wrong not to go, it being my hometown and all.
>>> >
>>> > Aside from the usual "Who else is going? We should meet!" I was also
>>> > thinking of submitting a proposal for a talk.  My idea was to put
>>> > something together on "The future of NumPy indexing" and use it as an
>>> > opportunity to raise awareness and hopefully gather feedback from
>>> > users on the proposed changes, in sort of a "if the mountain won't
>>> > come to Muhammad" type of thing.
>>> >
>>>
>>> I guess you do know my last name means mountain in german? But if
>>> Muhammed might come, I should really improve my arabic ;).
>>>
>>> In any case sounds good to me if you like to do it, I don't think I
>>> will go, though it sounds nice.
>>>
>>
>> Sounds like a good idea to me too. I like both the concrete topic, as
>> well as just having a talk on Numpy at a PyData conference. In general
>> there are too few (if any) talks on Numpy and other core libraries at
>> PyData and Scipy confs I think.
>>
>
> +1.
>
> It would be great a numpy talk from a core developer. BTW, C4P closes
> tomorrow!!!
>
> Jaime, if you come to Madrid you know you have some beers waiting for you.
>
> Disclaimer, I'm one of co-organizers of the PyData Madrid.
>

Since when does one need disclaimer when offering beers ? That would make
for a dangerous precedent :)

David

>
> Best.
>
>
>> Ralf
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] building NumPy with gcc if Python was built with icc?!?

[David Cournapeau]

On Tue, Feb 16, 2016 at 7:39 PM, BERGER Christian 
wrote:

> Hi All,
>
>
>
> Here's a potentially dumb question: is it possible to build NumPy with
> gcc, if python was built with icc?
>
> Right now, the build is failing in the toolchain check phase, because gcc
> doesn't know how to handle icc-specific c flags (like -fp-model, prec-sqrt,
> ...)
>
> In our environment we're providing an embedded python that our customers
> should be able to use and extend with 3rd party modules (like numpy).
> Problem is that our sw is built using icc, but we don't want to force our
> customers to do the same and we also don't want to build every possible 3rd
> party module for our customers.
>

If you are the one providing python, your best bet is to post process your
python to strip the info from Intel-specific options. The process is
convoluted, but basically:

- at configure time, python makes a difference between CFLAGS and OPTS, and
will store both in your Makefile
- when python is installed, it will parse the Makefile and generate some
dict written in the python stdlib as _sysconfigdata.py

_sysconfigdata.py is what's used by distutils to build C extensions.

This is only valid on Unix/cygwin, if you are on windows, the process is
completely different.

David


>
> Thanks for your help,
>
> Christian
>
>
>
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential and/or
> privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[David Cournapeau]

On Fri, Jan 15, 2016 at 9:56 PM, Steve Waterbury 
wrote:

> On 01/15/2016 04:08 PM, Benjamin Root wrote:
>
>> So, again, I love conda for what it can do when it works well. I only
>> take exception to the notion that it can address *all* problems, because
>> there are some problems that it just simply isn't properly situated for.
>>
>
> Actually, I would say you didn't mention any ... ;)  The issue is
> not that it "isn't properly situated for" (whatever that means)
> the problems you describe, but that -- in the case you mention,
> for example -- no one has conda-packaged those solutions yet.
>
> FWIW, our sysadmins and I use conda for django / apache / mod_wsgi
> sites and we are very happy with it.  IMO, compiling mod_wsgi in
> the conda environment and keeping it up is trivial compared to the
> awkwardnesses introduced by using pip/virtualenv in those cases.
>
> We also use conda for sites with nginx and the conda-packaged
> uwsgi, which works great and even permits the use of a separate
> env (with, if necessary, different versions of django, etc.)
> for each application.  No need to set up an entire VM for each app!
> *My* sysadmins love conda -- as soon as they saw how much better
> than pip/virtualenv it was, they have never looked back.
>
> IMO, conda is by *far* the best packaging solution the python
> community has ever seen (and I have been using python for more
> than 20 years).  I too have been stunned by some of the resistance
> to conda that one sometimes sees in the python packaging world.
> I've had a systems package maintainer tell me "it solves a
> different problem [than pip]" ... hmmm ... I would say it
> solves the same problem *and more*, *better*.  I attribute
> some of the conda-ignoring to "NIH" and, to some extent,
> possibly defensiveness (I would be defensive too if I had been
> working on pip as long as they had when conda came along ;).
>

Conda and pip solve some of the same problems, but pip also does quite a
bit more than conda (and vice et versa, as conda also acts akin
rvm-for-python). Conda works so well because it supports a subset of what
pip does: install things from binaries. This is the logical thing to do
when you want to distribute binaries because in the python world, since for
historical reasons, metadata in the general python lang are dynamic by the
very nature of setup.py.

For a long time, pip worked from sources instead of binaries (this is
actually the reason why it was started following easy_install), and thus
had to cope w/ those dynamic metadata. It also has to deal w/ building
packages and the whole distutils/setuptools interoperability mess. Conda
being solely a packaging solution can sidestep all this complexity (which
is again a the logical and smart thing to do if what you care is
deployment).

Having pip understand conda packages is a non trivial endeavour: since
conda packages are relocatable, they are not compatible with the usual
python interpreters, and setuptools metadata is neither a subset or a
superset of conda metadata. Regarding conda/pip interoperability, there are
things that conda could (and IMO should) do, such as writing the expected
metadata in site-packages (PEP 376). Currently, conda does not recognize
packages installed by pip (because it does not implement PEP 376 and co),
so if you do a "pip install ." of a package, it will likely break existing
package if present.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Defining a base linux-64 environment [was: Should I use pip install numpy in linux?]

[David Cournapeau]

On Sat, Jan 9, 2016 at 12:20 PM, Julian Taylor <
jtaylor.deb...@googlemail.com> wrote:

> On 09.01.2016 12:52, Robert McGibbon wrote:
> > Hi all,
> >
> > I went ahead and tried to collect a list of all of the libraries that
> > could be considered to constitute the "base" system for linux-64. The
> > strategy I used was to leverage off the work done by the folks at
> > Continuum by searching through their pre-compiled binaries
> > from https://repo.continuum.io/pkgs/free/linux-64/ to find shared
> > libraries that were dependened on (according to ldd)  that were not
> > accounted for by the declared dependencies that each package made known
> > to the conda package manager.
> >
>
> do those packages use ld --as-needed for linking?
> there are a lot libraries in that list that I highly doubt are directly
> used by the packages.
>

It is also a common problem when building packages without using a "clean"
build environment, as it is too easy to pick up dependencies accidentally,
especially for autotools-based packages (unless one uses pbuilder or
similar tools).

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should I use pip install numpy in linux?

[David Cournapeau]

On Sat, Jan 9, 2016 at 12:12 PM, Julian Taylor <
jtaylor.deb...@googlemail.com> wrote:

> On 09.01.2016 04:38, Nathaniel Smith wrote:
> > On Fri, Jan 8, 2016 at 7:17 PM, Nathan Goldbaum 
> wrote:
> >> Doesn't building on CentOS 5 also mean using a quite old version of gcc?
> >
> > Yes. IIRC CentOS 5 ships with gcc 4.4, and you can bump that up to gcc
> > 4.8 by using the Redhat Developer Toolset release (which is gcc +
> > special backport libraries to let it generate RHEL5/CentOS5-compatible
> > binaries). (I might have one or both of those version numbers slightly
> > wrong.)
> >
> >> I've never tested this, but I've seen claims on the anaconda mailing
> list of
> >> ~25% slowdowns compared to building from source or using system
> packages,
> >> which was attributed to building using an older gcc that doesn't
> optimize as
> >> well as newer versions.
> >
> > I'd be very surprised if that were a 25% slowdown in general, as
> > opposed to a 25% slowdown on some particular inner loop that happened
> > to neatly match some new feature in a new gcc (e.g. something where
> > the new autovectorizer kicked in). But yeah, in general this is just
> > an inevitable trade-off when it comes to distributing binaries: you're
> > always going to pay some penalty for achieving broad compatibility as
> > compared to artisanally hand-tuned binaries specialized for your
> > machine's exact OS version, processor, etc. Not much to be done,
> > really. At some point the baseline for compatibility will switch to
> > "compile everything on CentOS 6", and that will be better but it will
> > still be worse than binaries that target CentOS 7, and so on and so
> > forth.
> >
>
> I have over the years put in one gcc specific optimization after the
> other so yes using an ancient version will make many parts significantly
> slower. Though that is not really a problem, updating a compiler is easy
> even without redhats devtoolset.
>
> At least as far as numpy is concerned linux binaries should not be a
> very big problem. The only dependency where the version matters is glibc
> which has updated its interfaces we use (in a backward compatible way)
> many times.
> But here if we use a old enough baseline glibc (e.g. centos5 or ubuntu
> 10.04) we are fine at reasonable performance costs, basically only
> slower memcpy.
>
> Scipy on the other hand is a larger problem as it contains C++ code.
> Linux systems are now transitioning to C++11 which is binary
> incompatible in parts to the old standard. There a lot of testing is
> necessary to check if we are affected.
> How does Anaconda deal with C++11?
>

For canopy packages, we use the RH devtoolset w/ gcc 4.8.X, and statically
link the C++ stdlib.

It has worked so far for the few packages requiring C++11 and gcc > 4.4
(llvm/llvmlite/dynd), but that's not a solution I am a fan of myself, as
the implications are not always very clear.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop providing official win32 downloads (for now)

[David Cournapeau]

On Tue, Dec 22, 2015 at 7:11 PM, Chris Barker  wrote:

> On Mon, Dec 21, 2015 at 10:05 PM, Ralf Gommers 
> wrote:
>
>>
 There's a good chance that many downloads are from unsuspecting users
>> with a 64-bit Python, and they then just get an unhelpful "cannot find
>> Python" error from the installer.
>>
>
> could be -- hard to know.
>
>
>> At least until we have binary wheels on PyPi.
>>>
>>> What's up with that, by the way?
>>>
>>
>> I expect those to appear in 2016, built with MinGW-w64 and OpenBLAS.
>>
>
> nice. Anyway, I do think it's important to have a "official" easy way to
> get numpy for pyton.org pythons.
>
> numpy does/can/should see a lot of use outside the "scientific computing"
> community. And people are wary of dependencies. people should be able to
> use numpy in their projects, without requiring that their users start all
> over with Anaconda or ???
>
> The ideal is for "pip install" to "just work" -- sound sike we're getting
> there.
>
> BTW, we've been wary of putting a 32 bit wheel up 'cause of the whole
> "what processor features to require" issue, but if we think it's OK to drop
> the binary installer altogether, I can't see the harm in putting a 32 bit
> SSE2 wheel up.
>
> Any way to know how many people are running 32 bit Python on Windows these
> days??
>

I don't claim we are representative of the whole community, but as far as
canopy is concerned, it is still a significant platform. That's the only 32
bit platform we still support (both linux and osx 32 bits were < 1 % of our
downloads)

David


>
> -CHB
>
>
>
>
>
>
>
>
>
>
>
>
>
>>
>> Ralf
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
>
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR(206) 526-6959   voice
> 7600 Sand Point Way NE   (206) 526-6329   fax
> Seattle, WA  98115   (206) 526-6317   main reception
>
> chris.bar...@noaa.gov
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Fast vectorized arithmetic with ~32 significant digits under Numpy

[David Cournapeau]

On Fri, Dec 11, 2015 at 4:22 PM, Anne Archibald  wrote:

> Actually, GCC implements 128-bit floats in software and provides them as
> __float128; there are also quad-precision versions of the usual functions.
> The Intel compiler provides this as well, I think, but I don't think
> Microsoft compilers do. A portable quad-precision library might be less
> painful.
>
> The cleanest way to add extended precision to numpy is by adding a
> C-implemented dtype. This can be done in an extension module; see the
> quaternion and half-precision modules online.
>

We actually used __float128 dtype as an example of how to create a custom
dtype for a numpy C tutorial we did w/ Stefan Van der Walt a few years ago
at SciPy.

IIRC, one of the issue to make it more than a PoC was that numpy hardcoded
things like long double being the higest precision, etc... But that may has
been fixed since then.

David

> Anne
>
> On Fri, Dec 11, 2015, 16:46 Charles R Harris 
> wrote:
>
>> On Fri, Dec 11, 2015 at 6:25 AM, Thomas Baruchel 
>> wrote:
>>
>>> From time to time it is asked on forums how to extend precision of
>>> computation on Numpy array. The most common answer
>>> given to this question is: use the dtype=object with some arbitrary
>>> precision module like mpmath or gmpy.
>>> See
>>> http://stackoverflow.com/questions/6876377/numpy-arbitrary-precision-linear-algebra
>>> or
>>> http://stackoverflow.com/questions/21165745/precision-loss-numpy-mpmath
>>> or
>>> http://stackoverflow.com/questions/15307589/numpy-array-with-mpz-mpfr-values
>>>
>>> While this is obviously the most relevant answer for many users because
>>> it will allow them to use Numpy arrays exactly
>>> as they would have used them with native types, the wrong thing is that
>>> from some point of view "true" vectorization
>>> will be lost.
>>>
>>> With years I got very familiar with the extended double-double type
>>> which has (for usual architectures) about 32 accurate
>>> digits with faster arithmetic than "arbitrary precision types". I even
>>> used it for research purpose in number theory and
>>> I got convinced that it is a very wonderful type as long as such
>>> precision is suitable.
>>>
>>> I often implemented it partially under Numpy, most of the time by trying
>>> to vectorize at a low-level the libqd library.
>>>
>>> But I recently thought that a very nice and portable way of implementing
>>> it under Numpy would be to use the existing layer
>>> of vectorization on floats for computing the arithmetic operations by
>>> "columns containing half of the numbers" rather than
>>> by "full numbers". As a proof of concept I wrote the following file:
>>> https://gist.github.com/baruchel/c86ed748939534d8910d
>>>
>>> I converted and vectorized the Algol 60 codes from
>>> http://szmoore.net/ipdf/documents/references/dekker1971afloating.pdf
>>> (Dekker, 1971).
>>>
>>> A test is provided at the end; for inverting 100,000 numbers, my type is
>>> about 3 or 4 times faster than GMPY and almost
>>> 50 times faster than MPmath. It should be even faster for some other
>>> operations since I had to create another np.ones
>>> array for testing this type because inversion isn't implemented here
>>> (which could of course be done). You can run this file by yourself
>>> (maybe you will have to discard mpmath or gmpy if you don't have it).
>>>
>>> I would like to discuss about the way to make available something
>>> related to that.
>>>
>>> a) Would it be relevant to include that in Numpy ? (I would think to
>>> some "contribution"-tool rather than including it in
>>> the core of Numpy because it would be painful to code all ufuncs; on the
>>> other hand I am pretty sure that many would be happy
>>> to perform several arithmetic operations by knowing that they can't use
>>> cos/sin/etc. on this type; in other words, I am not
>>> sure it would be a good idea to embed it as an every-day type but I
>>> think it would be nice to have it quickly available
>>> in some way). If you agree with that, in which way should I code it (the
>>> current link only is a "proof of concept"; I would
>>> be very happy to code it in some cleaner way)?
>>>
>>> b) Do you think such attempt should remain something external to Numpy
>>> itself and be released on my Github account without being
>>> integrated to Numpy?
>>>
>>
>> I think astropy does something similar for time and dates. There has also
>> been some talk of adding a user type for ieee 128 bit doubles. I've looked
>> once for relevant code for the latter and, IIRC, the available packages
>> were GPL :(.
>>
>> Chuck
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[David Cournapeau]

On Fri, Dec 4, 2015 at 11:06 AM, Nathaniel Smith <n...@pobox.com> wrote:

> On Fri, Dec 4, 2015 at 1:27 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> > I would be in favour of dropping 3.3, but not 2.6 until it becomes too
> > cumbersome to support.
> >
> > As a data point, as of april, 2.6 was more downloaded than all python 3.X
> > versions together when looking at pypi numbers:
> > https://caremad.io/2015/04/a-year-of-pypi-downloads/
>
> I'm not sure what's up with those numbers though -- they're *really*
> unrepresentative of what we see for numpy otherwise. E.g. they show
> 3.X usage as ~5%, but for numpy, 3.x usage has risen past 25%.
> (Source: 'vanity numpy', looking at OS X wheels b/c they're
> per-version and unpolluted by CI download spam. Unfortunately this
> doesn't provide numbers for 2.6 b/c we don't ship 2.6 binaries.) For
> all we know all those 2.6 downloads are travis builds testing projects
> on 2.6 to make sure they keep working because there are so many 2.6
> downloads on pypi :-). Which isn't an argument for dropping 2.6
> either, I just wouldn't put much weight on that blog post either
> way...
>

I agree pypi is only one data point. The proportion is also package
dependent (e.g. django had higher proportion of python 3.X). It is just
that having multiple data points is often more useful than guesses

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] When to stop supporting Python 2.6?

[David Cournapeau]

I would be in favour of dropping 3.3, but not 2.6 until it becomes too
cumbersome to support.

As a data point, as of april, 2.6 was more downloaded than all python 3.X
versions together when looking at pypi numbers:
https://caremad.io/2015/04/a-year-of-pypi-downloads/

David

On Thu, Dec 3, 2015 at 11:03 PM, Jeff Reback  wrote:

> pandas is going to drop
> 2.6 and 3.3 next release at end of Jan
>
> (3.2 dropped in 0.17, in October)
>
>
>
> I can be reached on my cell 917-971-6387
> > On Dec 3, 2015, at 6:00 PM, Bryan Van de Ven 
> wrote:
> >
> >
> >> On Dec 3, 2015, at 4:59 PM, Eric Firing  wrote:
> >>
> >> Chuck,
> >>
> >> I would support dropping the old versions now.  As a related data
> point, matplotlib is testing master on 2.7, 3.4, and 3.5--no more 2.6 and
> 3.3.
> >
> > Ditto for Bokeh.
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead

[David Cournapeau]

On Tue, Oct 27, 2015 at 2:31 PM, Edison Gustavo Muenz <
edisongust...@gmail.com> wrote:

> I'm sorry if this is out-of-topic, but I'm curious on why nobody mentioned
> Conda yet.
>

Conda is a binary distribution system, whereas we are talking about
installing from sources. You will need a way to install things when
building a conda package in any case

David


> Is there any particular reason for not using it?
>
> On Tue, Oct 27, 2015 at 11:48 AM, James E.H. Turner 
> wrote:
>
>> Apparently it is not well known that if you have a Python project
>>> source tree (e.g., a numpy checkout), then the correct way to install
>>> it is NOT to type
>>>
>>>python setup.py install   # bad and broken!
>>>
>>> but rather to type
>>>
>>>pip install .
>>>
>>
>> Though I haven't studied it exhaustively, it always seems to me that
>> pip is bad & broken, whereas python setup.py install does what I
>> expect (even if it's a mess internally). In particular, when
>> maintaining a distribution of Python packages, you try to have some
>> well-defined, reproducible build from source tarballs and then you
>> find that pip is going off and downloading stuff under the radar
>> without being asked (etc.). Stopping that can be a pain & I always
>> groan whenever some package insists on using pip. Maybe I don't
>> understand it well enough but in this role its dependency handling
>> is an unnecessary complication with no purpose. Just a comment that
>> not every installation is someone trying to get numpy on their
>> laptop...
>>
>> Cheers,
>>
>> James.
>>
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.10.1 released.

[David Cournapeau]

On Wed, Oct 14, 2015 at 5:38 PM, Nathaniel Smith  wrote:

> On Oct 14, 2015 9:15 AM, "Chris Barker"  wrote:
> >
> > On Mon, Oct 12, 2015 at 9:27 AM, Charles R Harris <
> charlesr.har...@gmail.com> wrote:
> >>
> >> * Compiling with msvc9 or msvc10 for 32 bit Windows now requires SSE2.
> >>   This was the easiest fix for what looked to be some miscompiled code
> when
> >>   SSE2 was not used.
> >
> >
> > Note that there is discusion right now on pyton-dev about requireing
> SSE2 for teh python.org build of python3.5 -- it does now, so it's fine
> for third party pacakges to also require it. But there is some talk of
> removing that requirement -- still a lot of old machines around, I guess --
> particular at schools and the like.
>
> Note that the 1.10.1 release announcement is somewhat misleading --
> apparently the affected builds have actually required SSE2 since numpy 1.8,
> and the change here just makes it even more required. I'm not sure if this
> is all 32 bit builds or only ones using msvc that have been needing SSE2
> all along. The change in 1.10.1 only affects msvc, which is not what most
> people are using (IIUC Enthought Canopy uses msvc, but the pypi, gohlke,
> and Anaconda builds don't).
>
> I'm actually not sure if anyone even uses the 32 bit builds at all :-)
>

I cannot divulge exact figures for downloads, but for us at Enthought,
windows 32 bits is in the same ballpark as OS X and Linux (64 bits) in
terms of proportion, windows 64 bits being significantly more popular.

Linux 32 bits and OS X 32 bits have been in the 1 % range each of our
downloads for a while (we recently stopped support for both).

David

> > Ideally, any binary wheels on PyPi should be compatible with the
> python.org builds -- so not require SSE2, if the python.org builds don't.
> >
> > Though we had this discussion a while back -- and numpy could, and maybe
> should require more -- did we ever figure out a way to get a meaningful
> message to the user if they try to run an SSE2 build on a machine without
> SSE2?
>
> It's not that difficult in principle, just someone has to do it :-).
>
> -n
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[David Cournapeau]

On Tue, Oct 6, 2015 at 8:04 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Tue, Oct 6, 2015 at 11:52 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> >
> >
> > On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith <n...@pobox.com> wrote:
> >>
> >> [splitting this off into a new thread]
> >>
> >> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau <courn...@gmail.com>
> >> wrote:
> >> [...]
> >> > I also agree the current situation is not sustainable -- as we
> discussed
> >> > privately before, cythonizing numpy.core is made quite more
> complicated
> >> > by
> >> > this. I have myself quite a few issues w/ cythonizing the other parts
> of
> >> > umath. I would also like to support the static link better than we do
> >> > now
> >> > (do we know some static link users we can contact to validate our
> >> > approach
> >> > ?)
> >> >
> >> > Currently, what we have in numpy core is the following:
> >> >
> >> > numpy.core.multiarray -> compilation units in
> numpy/core/src/multiarray/
> >> > +
> >> > statically link npymath
> >> > numpy.core.umath -> compilation units in numpy/core/src/umath +
> >> > statically
> >> > link npymath/npysort + some shenanigans to use things in
> >> > numpy.core.multiarray
> >>
> >> There are also shenanigans in the other direction - supposedly umath
> >> is layered "above" multiarray, but in practice there are circular
> >> dependencies (see e.g. np.set_numeric_ops).
> >
> > Indeed, I am not arguing about merging umath and multiarray.
>
> Oh, okay :-).
>
> >> > I would suggest to have a more layered approach, to enable both
> 'normal'
> >> > build and static build, without polluting the public namespace too
> much.
> >> > This is an approach followed by most large libraries (e.g. MKL), and
> is
> >> > fairly flexible.
> >> >
> >> > Concretely, we could start by putting more common functionalities (aka
> >> > the
> >> > 'core' library) into its own static library. The API would be
> considered
> >> > private to numpy (no stability guaranteed outside numpy), and every
> >> > exported
> >> > symbol from that library would be decorated appropriately to avoid
> >> > potential
> >> > clashes (e.g. '_npy_internal_').
> >>
> >> I don't see why we need this multi-layered complexity, though.
> >
> >
> > For several reasons:
> >
> >  - when you want to cythonize either extension, it is much easier to
> > separate it as cython for CPython API, C for the rest.
>
> I don't think this will help much, because I think we'll want to have
> multiple cython files, and that we'll probably move individual
> functions between being implemented in C and Cython (including utility
> functions). So that means we need to solve the problem of mixing C and
> Cython files inside a single library.
>

Separating the pure C code into static lib is the simple way of achieving
the same goal. Essentially, you write:

# implemented in npyinternal.a
_npy_internal_foo()

# implemented in merged_multiarray_umath.pyx
cdef PyArray_Foo(...):
# use _npy_internal_foo()

then our merged_multiarray_umath.so is built by linking the .pyx and the
npyinternal.a together. IOW, the static link is internal.

Going through npyinternal.a instead of just linking .o from pure C and
Cython together gives us the following:

 1. the .a can just use normal linking strategies instead of the awkward
capsule thing. Those are easy to get wrong when using cython as you may end
up with multiple internal copies of the wrapped object inside capsule,
causing hard to track bugs (this is what we wasted most of the time on w/
Stefan and Kurt during ds4ds)
 2. the only public symbols in .a are the ones needed by the cython
wrapping, and since those are decorated with npy_internal, clashes are
unlikely to happen
 3. since most of the code is already in .a internally, supporting the
static linking should be simpler since the only difference is how you
statically link the cython-generated code. Because of 1, you are also less
likely to cause nasty surprises when putting everything together.

When you cythonize umath/multiarray, you need to do most of the underlying
work anyway

I don't really care if the files are in the same directory or not, we can
keep things as they are now.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[David Cournapeau]

On Thu, Oct 8, 2015 at 8:47 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Oct 8, 2015 06:30, "David Cournapeau" <courn...@gmail.com> wrote:
> >
> [...]
> >
> > Separating the pure C code into static lib is the simple way of
> achieving the same goal. Essentially, you write:
> >
> > # implemented in npyinternal.a
> > _npy_internal_foo()
> >
> > # implemented in merged_multiarray_umath.pyx
> > cdef PyArray_Foo(...):
> > # use _npy_internal_foo()
> >
> > then our merged_multiarray_umath.so is built by linking the .pyx and the
> npyinternal.a together. IOW, the static link is internal.
> >
> > Going through npyinternal.a instead of just linking .o from pure C and
> Cython together gives us the following:
> >
> >  1. the .a can just use normal linking strategies instead of the awkward
> capsule thing. Those are easy to get wrong when using cython as you may end
> up with multiple internal copies of the wrapped object inside capsule,
> causing hard to track bugs (this is what we wasted most of the time on w/
> Stefan and Kurt during ds4ds)
>
> Check out Stéfan's branch -- it just uses regular linking to mix cython
> and C.
>
I know, we worked on this together after all ;)

My suggested organisation is certainly not mandatory, I was not trying to
claim otherwise, sorry if that was unclear.

At that point, I guess the consensus is that I have to prove my suggestion
is useful. I will take a few more hours to submit a PR with the umath
conversion (maybe merging w/ the work from Stéfan). I discovered on my
flight back that you can call PyModule_Init multiple times for a given
module, which is useful while we do the transition C->Cython for the module
initialization (it is not documented as possible, so I would not to rely on
it for long either).

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] reorganizing numpy internal extensions (was: Re: Should we drop support for "one file" compilation mode?)

[David Cournapeau]

On Tue, Oct 6, 2015 at 7:30 PM, Nathaniel Smith <n...@pobox.com> wrote:

> [splitting this off into a new thread]
>
> On Tue, Oct 6, 2015 at 3:00 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> [...]
> > I also agree the current situation is not sustainable -- as we discussed
> > privately before, cythonizing numpy.core is made quite more complicated
> by
> > this. I have myself quite a few issues w/ cythonizing the other parts of
> > umath. I would also like to support the static link better than we do now
> > (do we know some static link users we can contact to validate our
> approach
> > ?)
> >
> > Currently, what we have in numpy core is the following:
> >
> > numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/
> +
> > statically link npymath
> > numpy.core.umath -> compilation units in numpy/core/src/umath +
> statically
> > link npymath/npysort + some shenanigans to use things in
> > numpy.core.multiarray
>
> There are also shenanigans in the other direction - supposedly umath
> is layered "above" multiarray, but in practice there are circular
> dependencies (see e.g. np.set_numeric_ops).
>

Indeed, I am not arguing about merging umath and multiarray.


> > I would suggest to have a more layered approach, to enable both 'normal'
> > build and static build, without polluting the public namespace too much.
> > This is an approach followed by most large libraries (e.g. MKL), and is
> > fairly flexible.
> >
> > Concretely, we could start by putting more common functionalities (aka
> the
> > 'core' library) into its own static library. The API would be considered
> > private to numpy (no stability guaranteed outside numpy), and every
> exported
> > symbol from that library would be decorated appropriately to avoid
> potential
> > clashes (e.g. '_npy_internal_').
>
> I don't see why we need this multi-layered complexity, though.
>

For several reasons:

 - when you want to cythonize either extension, it is much easier to
separate it as cython for CPython API, C for the rest.
 - if numpy.core.multiarray.so is built as cython-based .o + a 'large' C
static library, it should become much simpler to support static link.
 - maybe that's just personal, but I find the whole multiarray + umath
quite beyond manageable in terms of intertwined complexity. You may argue
it is not that big, and we all have different preferences in terms of
organization, but if I look at the binary size of multiarray + umath, it is
quite larger than the median size of the .so I have in my /usr/lib.

I am also hoping that splitting up numpy.core in separate elements that
communicate through internal APIs would make participating into numpy
easier.

We could also swap the argument: assuming it does not make the build more
complex, and that it does help static linking, why not doing it ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Mon, Oct 5, 2015 at 11:26 PM, Nathaniel Smith  wrote:

> Hi all,
>
> For a long time, NumPy has supported two different ways of being compiled:
>
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
>
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
>
> Supporting these two different build modes creates a drag on
> development progress; in particular Stefan recently ran into this in
> this experiments with porting parts of the NumPy internals to Cython:
>   https://github.com/numpy/numpy/pull/6408
> (I suspect the particular problem he's running into can be fixed b/c
> so far he only has one .pyx file, but I also suspect that it will be
> impossible to support "one file" mode once we have multiple .pyx
> files.)
>
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors. If all we can
> get are rumors, then eventually we have to risk breaking things just
> to force anyone who cares to actually show up and explain what they
> need so we can support it properly :-).
>

Assuming one of the rumour is related to some comments I made some time
(years ?) earlier, the context was the ability to hide exported symbols. As
you know, the issue is not to build extensions w/ multiple compilation
units, but sharing functionalities between them without sharing them
outside the extension. I am just reiterating that point so that we all
discuss under the right context :)

I also agree the current situation is not sustainable -- as we discussed
privately before, cythonizing numpy.core is made quite more complicated by
this. I have myself quite a few issues w/ cythonizing the other parts of
umath. I would also like to support the static link better than we do now
(do we know some static link users we can contact to validate our approach
?)

Currently, what we have in numpy core is the following:

numpy.core.multiarray -> compilation units in numpy/core/src/multiarray/ +
statically link npymath
numpy.core.umath -> compilation units in numpy/core/src/umath + statically
link npymath/npysort + some shenanigans to use things in
numpy.core.multiarray

I would suggest to have a more layered approach, to enable both 'normal'
build and static build, without polluting the public namespace too much.
This is an approach followed by most large libraries (e.g. MKL), and is
fairly flexible.

Concretely, we could start by putting more common functionalities (aka the
'core' library) into its own static library. The API would be considered
private to numpy (no stability guaranteed outside numpy), and every
exported symbol from that library would be decorated appropriately to avoid
potential clashes (e.g. '_npy_internal_').

FWIW, that has always been my intention to go toward this when I split up
multiarray/umath into multiple .c files and extracted out npymath.

cheers,
David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 12:07 PM, Antoine Pitrou <solip...@pitrou.net> wrote:

> On Tue, 6 Oct 2015 11:00:30 +0100
> David Cournapeau <courn...@gmail.com> wrote:
> >
> > Assuming one of the rumour is related to some comments I made some time
> > (years ?) earlier, the context was the ability to hide exported symbols.
> As
> > you know, the issue is not to build extensions w/ multiple compilation
> > units, but sharing functionalities between them without sharing them
> > outside the extension.
>
> Can't you use the visibility attribute with gcc for this?
>

We do that already for gcc, I think the question was whether every platform
supported this or not (and whether we should care).


> Other Unix compilers probably provide something similar. The issue
> doesn't exist on Windows by construction.
>
>
> https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes
>
> By the way, external packages may reuse the npy_* functions, so I would
> like them not the be hidden :-)
>

The npy_ functions in npymath were designed to be exported. Those would
stay that way.

David

>
> Regards
>
> Antoine.
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith  wrote:

> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou 
> wrote:
> > On Tue, 6 Oct 2015 09:40:43 -0700
> > Nathaniel Smith  wrote:
> >>
> >> If you need some npy_* function it'd be much better to let us know
> >> what it is and let us export it in an intentional way, instead of just
> >> relying on whatever stuff we accidentally exposed?
> >
> > Ok, we seem to be using only the complex math functions (npy_cpow and
> > friends, I could make a complete list if required).
>
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?
>

It is possible (and documented) to use any of the npy_ symbols from npymath
from outside numpy:
http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension

The design is not perfect (I was young and foolish :) ), but it has worked
fairly well and has been used in at least scipy since the 1.4/1.5 days IIRC
(including windows).

David


>
> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
>
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.
>
> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
>
> Maybe this new package would also be a good place for exposing these
> wrappers?
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> >
> >
> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <n...@pobox.com> wrote:
> >>
> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solip...@pitrou.net>
> >> wrote:
> >> > On Tue, 6 Oct 2015 09:40:43 -0700
> >> > Nathaniel Smith <n...@pobox.com> wrote:
> >> >>
> >> >> If you need some npy_* function it'd be much better to let us know
> >> >> what it is and let us export it in an intentional way, instead of
> just
> >> >> relying on whatever stuff we accidentally exposed?
> >> >
> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
> >> > friends, I could make a complete list if required).
> >>
> >> And how are you getting at them? Are you just relying the way that on
> >> ELF systems, if two libraries are loaded into the same address space
> >> then they automatically get access to each other's symbols, even if
> >> they aren't linked to each other? What do you do on Windows?
> >
> >
> > It is possible (and documented) to use any of the npy_ symbols from
> npymath
> > from outside numpy:
> >
> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
> >
> > The design is not perfect (I was young and foolish :) ), but it has
> worked
> > fairly well and has been used in at least scipy since the 1.4/1.5 days
> IIRC
> > (including windows).
>
> Okay, so just to confirm, it looks like this does indeed implement the
> static linking thing I just suggested (so perhaps I am also young and
> foolish ;-)) -- from looking at the output of get_info("npymath"), it
> seems to add -I.../numpy/core/include to the compiler flags, add
> -lnpymath -L.../numpy/core/lib to the linker flags, and then
> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>

Yes, I was not trying to argue otherwise. If you thought I was, blame it on
my poor English (which sadly does not get better as I get less young...).

My proposal is to extend this technique for *internal* API, but with the
following differences:
 * the declarations are not put in any public header
 * we don't offer any way to link to this library, and name it something
scary enough that people would have to be foolish (young or not) to use it.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> > The npy_ functions in npymath were designed to be exported. Those would
> stay
> > that way.
>
> If we want to export these then I vote that we either:
> - use the usual API export mechanism, or else
> - provide a static library for people to link to, instead of trying to
> do runtime binding. (I.e. drop it in some known place, and then
> provide some functions for extension modules to find it at build time
> -- similar to how np.get_include() works.)
>

Unless something changed, that's more or less how it works already (npymath
is used in scipy, for example, which was one of the rationale for writing
it in the first place !).

You access the compilation/linking issues through the numpy distutils
get_info function.

David


> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:51 PM, David Cournapeau <courn...@gmail.com> wrote:

>
>
> On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <n...@pobox.com> wrote:
>
>> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <courn...@gmail.com>
>> wrote:
>> > The npy_ functions in npymath were designed to be exported. Those would
>> stay
>> > that way.
>>
>> If we want to export these then I vote that we either:
>> - use the usual API export mechanism, or else
>> - provide a static library for people to link to, instead of trying to
>> do runtime binding. (I.e. drop it in some known place, and then
>> provide some functions for extension modules to find it at build time
>> -- similar to how np.get_include() works.)
>>
>
> Unless something changed, that's more or less how it works already
> (npymath is used in scipy, for example, which was one of the rationale for
> writing it in the first place !).
>
> You access the compilation/linking issues through the numpy distutils
> get_info function.
>

And my suggestion is to use a similar mechanism for multiarray and umath,
so that in the end the exported Python C API is just a thin layer on top of
the underlying static library. That would make cython and I suspect static
linking quite a bit easier. The API between the low layer and python C API
of multiarray/umath would be considered private and outside any API/ABI
stability.

IOW, it would be an internal change, and should not cause visible changes
to the users, except that some _npy_private_ symbols would be exported (but
you would be crazy to use them, and the prototype declarations would not be
available when you install numpy anyway). Think of those as the internal
driver API/ABI of Linux or similar.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 6:18 PM, David Cournapeau <courn...@gmail.com> wrote:

>
>
> On Tue, Oct 6, 2015 at 6:14 PM, Nathaniel Smith <n...@pobox.com> wrote:
>
>> On Tue, Oct 6, 2015 at 10:10 AM, David Cournapeau <courn...@gmail.com>
>> wrote:
>> >
>> >
>> > On Tue, Oct 6, 2015 at 6:07 PM, Nathaniel Smith <n...@pobox.com> wrote:
>> >>
>> >> On Tue, Oct 6, 2015 at 10:00 AM, Antoine Pitrou <solip...@pitrou.net>
>> >> wrote:
>> >> > On Tue, 6 Oct 2015 09:40:43 -0700
>> >> > Nathaniel Smith <n...@pobox.com> wrote:
>> >> >>
>> >> >> If you need some npy_* function it'd be much better to let us know
>> >> >> what it is and let us export it in an intentional way, instead of
>> just
>> >> >> relying on whatever stuff we accidentally exposed?
>> >> >
>> >> > Ok, we seem to be using only the complex math functions (npy_cpow and
>> >> > friends, I could make a complete list if required).
>> >>
>> >> And how are you getting at them? Are you just relying the way that on
>> >> ELF systems, if two libraries are loaded into the same address space
>> >> then they automatically get access to each other's symbols, even if
>> >> they aren't linked to each other? What do you do on Windows?
>> >
>> >
>> > It is possible (and documented) to use any of the npy_ symbols from
>> npymath
>> > from outside numpy:
>> >
>> http://docs.scipy.org/doc/numpy-dev/reference/c-api.coremath.html#linking-against-the-core-math-library-in-an-extension
>> >
>> > The design is not perfect (I was young and foolish :) ), but it has
>> worked
>> > fairly well and has been used in at least scipy since the 1.4/1.5 days
>> IIRC
>> > (including windows).
>>
>> Okay, so just to confirm, it looks like this does indeed implement the
>> static linking thing I just suggested (so perhaps I am also young and
>> foolish ;-)) -- from looking at the output of get_info("npymath"), it
>> seems to add -I.../numpy/core/include to the compiler flags, add
>> -lnpymath -L.../numpy/core/lib to the linker flags, and then
>> .../numpy/core/lib contains only libnpymath.a, so it's static linking.
>>
>
> Yes, I was not trying to argue otherwise. If you thought I was, blame it
> on my poor English (which sadly does not get better as I get less young...).
>
> My proposal is to extend this technique for *internal* API, but with the
> following differences:
>  * the declarations are not put in any public header
>  * we don't offer any way to link to this library, and name it something
> scary enough that people would have to be foolish (young or not) to use it.
>

I am stupid: we of course do not even ship that internal library, it would
just be linked into multiarray/umath and never installed or part of binary
packages.

David

>
> David
>
>
>> -n
>>
>> --
>> Nathaniel J. Smith -- http://vorpus.org
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[David Cournapeau]

On Tue, Oct 6, 2015 at 5:58 PM, Nathaniel Smith <n...@pobox.com> wrote:

> On Tue, Oct 6, 2015 at 9:51 AM, David Cournapeau <courn...@gmail.com>
> wrote:
> >
> > On Tue, Oct 6, 2015 at 5:44 PM, Nathaniel Smith <n...@pobox.com> wrote:
> >>
> >> On Tue, Oct 6, 2015 at 4:46 AM, David Cournapeau <courn...@gmail.com>
> >> wrote:
> >> > The npy_ functions in npymath were designed to be exported. Those
> would
> >> > stay
> >> > that way.
> >>
> >> If we want to export these then I vote that we either:
> >> - use the usual API export mechanism, or else
> >> - provide a static library for people to link to, instead of trying to
> >> do runtime binding. (I.e. drop it in some known place, and then
> >> provide some functions for extension modules to find it at build time
> >> -- similar to how np.get_include() works.)
> >
> > Unless something changed, that's more or less how it works already
> (npymath
> > is used in scipy, for example, which was one of the rationale for
> writing it
> > in the first place !).
>
> Okay... in fact multiarray.so right now *does* export tons and tons of
> random junk into the global symbol namespace (on systems like Linux
> that do have a global symbol namespace), so it isn't obvious whether
> people are asking for that to continue :-). I'm just specifically
> saying that we should try to get this back down to the 1 exported
> symbol.
>
> (Try:
>objdump -T $(python -c 'import numpy;
> print(numpy.core.multiarray.__file__)')
> This *should* print 1 line... I currently get ~700. numpy.core.umath
> is similar.)
>
>
I think this overestimates the amount by quite a bit, since you see GLIBC
symbols, etc... I am using nm -Dg --defined-only $(python -c 'import numpy;
print(numpy.core.multiarray.__file__)')instead.

I see around 290 symboles: the npy_ from npymath don't bother me, but the
ones from npysort do. We should at least prefix those with npy_ (I don't
think npysort API has ever been publisher in our header like npymath was ?)

David

-n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] composition of the steering council (was Re: Governance model request)

[David Cournapeau]

od). [1]
>
> More to the point of the actual members:
>
> So to say, I feel the council members have to try to be *directly*
> active and see being active as a necessary *commitment* (i.e. also try
> to travel to meetings). This will always be a difficult judgment of
> course, but there is no help to it. The current definitions imply this.
> And two years seems fine. It is not that short, at least unless someone
> stops contributing very abruptly which I do not think is that usual. I
> will weight in to keep the current times but do not feel very strongly.
>
> About using the commit log to seed, I think there are some old term
> contributers (David Cournapeau maybe?), who never stopped doing quite a
> bit but may not have merge commits. However, I think we can start of
> with what we had, then I would hope Chuck and maybe Ralf can fill in the
> blanks.
>

AFAIK, I still have merge commits. I am actually doing a bit of numpy
development ATM, so I would prefer keeping them, but I won't fight it
either.

David


>
> About the size, I think if we get too many -- if that is possible -- we
> should just change the governance at that time to be not veto based
> anymore. This is something to keep in mind, but probably does not need
> to be formalized.
>
> - Sebastian
>
>
> [1] Sorry to "footnote" this, but I think I am probably rudely repeating
> myself and frankly do **not want this to be discussed**. It is just to
> try to be fully clear where I come from:
> Until SciPy 2015, I could list many people on this list who have shown
> more direct involvement in numpy then Travis since I joined and have no
> affiliation to numpy. If Travis had been new to the community at the
> time, I would be surprised if I would even recognize his name.
> I know this is only half the picture and Travis already mentioned
> another side, but this is what I mostly saw even if it may be a harsh
> and rude assessment.
>
>
> >
> > Chuck
> >
> >
> >
> >
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> https://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5?

[David Cournapeau]

On Mon, Sep 14, 2015 at 9:18 AM, Matthew Brett 
wrote:

> Hi,
>
> I'm just building numpy 1.9.2 for Python 3.5 (just released).
>
> In order to get the tests to pass on Python 3.5, I need to cherry pick
> commit 7d6aa8c onto the 1.9.2 tag position.
>
> Does anyone object to me uploading a wheel built from this patched
> version to pypi as 1.9.2 for Python 3.5 on OSX?   It would help to get
> the ball rolling for Python 3.5 binary wheels.
>

Why not releasing this as 1.9.3 ? It does not need to be a full release
(with binaries and all), but having multiple sources for a given tag is
confusing.

David


>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Cythonizing some of NumPy

[David Cournapeau]

On Tue, Sep 1, 2015 at 8:16 AM, Nathaniel Smith <n...@pobox.com> wrote:

> On Sun, Aug 30, 2015 at 2:44 PM, David Cournapeau <courn...@gmail.com>
> wrote:
> > Hi there,
> >
> > Reading Nathaniel summary from the numpy dev meeting, it looks like
> there is
> > a consensus on using cython in numpy for the Python-C interfaces.
> >
> > This has been on my radar for a long time: that was one of my rationale
> for
> > splitting multiarray into multiple "independent" .c files half a decade
> ago.
> > I took the opportunity of EuroScipy sprints to look back into this, but
> > before looking more into it, I'd like to make sure I am not going astray:
> >
> > 1. The transition has to be gradual
>
> Yes, definitely.
>
> > 2. The obvious way I can think of allowing cython in multiarray is
> modifying
> > multiarray such as cython "owns" the PyMODINIT_FUNC and the module
> > PyModuleDef table.
>
> The seems like a plausible place to start.
>
> In the longer run, I think we'll need to figure out a strategy to have
> source code divided over multiple .pyx files (for the same reason we
> want multiple .c files -- it'll just be impossible to work with
> otherwise). And this will be difficult for annoying technical reasons,
> since we definitely do *not* want to increase the API surface exposed
> by multiarray.so, so we will need to compile these multiple .pyx and
> .c files into a single module, and have them talk to each other via
> internal interfaces. But Cython is currently very insistent that every
> .pyx file should be its own extension module, and the interface
> between different files should be via public APIs.
>
> I spent some time poking at this, and I think it's possible but will
> take a few kluges at least initially. IIRC the tricky points I noticed
> are:
>
> - For everything except the top-level .pyx file, we'd need to call the
> generated module initialization functions "by hand", and have a bit of
> utility code to let us access the symbol tables for the resulting
> modules
>
> - We'd need some preprocessor hack (or something?) to prevent the
> non-main module initialization functions from being exposed at the .so
> level (like 'cdef extern from "foo.h"', 'foo.h' re#defines
> PyMODINIT_FUNC to remove the visibility declaration)
>
> - By default 'cdef' functions are name-mangled, which is annoying if
> you want to be able to do direct C calls between different .pyx and .c
> files. You can fix this by adding a 'public' declaration to your cdef
> function. But 'public' also adds dllexport stuff which would need to
> be hacked out as per above.
>
> I think the best strategy for this is to do whatever horrible things
> are necessary to get an initial version working (on a branch, of
> course), and then once that's done assess what changes we want to ask
> the cython folks for to let us eliminate the gross parts.
>

Agreed.

Regarding multiple cython .pyx and symbol pollution, I think it would be
fine to have an internal API with the required prefix (say `_npy_cpy_`) in
a core library, and control the exported symbols at the .so level. This is
how many large libraries work in practice (e.g. MKL), and is a model well
understood by library users.

I will start the cythonize process without caring about any of that though:
one large .pyx file, and everything build together by putting everything in
one .so. That will avoid having to fight both cython and distutils at the
same time :)

David

>
> (Insisting on compiling everything into the same .so will probably
> also help at some point in avoiding Cython-Related Binary Size Blowup
> Syndrome (CRBSBS), because the masses of boilerplate could in
> principle be shared between the different files. I think some modern
> linkers are even clever enough to eliminate this kind of duplicate
> code automatically, since C++ suffers from a similar problem.)
>
> > 3. We start using cython for the parts that are mostly menial refcount
> work.
> > Things like functions in calculation.c are obvious candidates.
> >
> > Step 2 should not be disruptive, and does not look like a lot of work:
> there
> > are < 60 methods in the table, and most of them should be fairly
> > straightforward to cythonize. At worse, we could just keep them as is
> > outside cython and just "export" them in cython.
> >
> > Does that sound like an acceptable plan ?
> >
> > If so, I will start working on a PR to work on 2.
>
> Makes sense to me!
>
> -n
>
> --
> Nathaniel J. Smith -- http://vorpus.org
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Cythonizing some of NumPy

[David Cournapeau]

Hi there,

Reading Nathaniel summary from the numpy dev meeting, it looks like there
is a consensus on using cython in numpy for the Python-C interfaces.

This has been on my radar for a long time: that was one of my rationale for
splitting multiarray into multiple independent .c files half a decade
ago. I took the opportunity of EuroScipy sprints to look back into this,
but before looking more into it, I'd like to make sure I am not going
astray:

1. The transition has to be gradual
2. The obvious way I can think of allowing cython in multiarray is
modifying multiarray such as cython owns the PyMODINIT_FUNC and the
module PyModuleDef table.
3. We start using cython for the parts that are mostly menial refcount
work. Things like functions in calculation.c are obvious candidates.

Step 2 should not be disruptive, and does not look like a lot of work:
there are  60 methods in the table, and most of them should be fairly
straightforward to cythonize. At worse, we could just keep them as is
outside cython and just export them in cython.

Does that sound like an acceptable plan ?

If so, I will start working on a PR to work on 2.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

[David Cournapeau]

Thanks for the good summary Nathaniel.

Regarding dtype machinery, I agree casting is the hardest part. Unless the
code has changed dramatically, this was the main reason why you could not
make most of the dtypes separate from numpy codebase (I tried to move the
datetime dtype out of multiarray into a separate C extension some years
ago). Being able to separate the dtypes from the multiarray module would be
an obvious way to drive the internal API change.

Regarding the use of cython in numpy, was there any discussion about the
compilation/size cost of using cython, and talking to the cython team to
improve this ? Or was that considered acceptable with current cython for
numpy. I am convinced cleanly separating the low level parts from the
python C API plumbing would be the single most important thing one could do
to make the codebase more amenable.

David




On Tue, Aug 25, 2015 at 9:58 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Tue, Aug 25, 2015 at 1:00 PM, Travis Oliphant tra...@continuum.io
 wrote:

 Thanks for the write-up Nathaniel.   There is a lot of great detail and
 interesting ideas here.

 I've am very eager to understand how to help NumPy and the wider
 community move forward however I can (my passions on this have not changed
 since 1999, though what I myself spend time on has changed).

 There are a lot of ways to think about approaching this, though.   It's
 hard to get all the ideas on the table, and it was unfortunate we couldn't
 get everybody wyho are core NumPy devs together in person to have this
 discussion as there are still a lot of questions unanswered and a lot of
 thought that has gone into other approaches that was not brought up or
 represented in the meeting (how does Numba fit into this, what about
 data-shape, dynd, memory-views and Python type system, etc.).   If NumPy
 becomes just an interface-specification, then why don't we just do that
 *outside* NumPy itself in a way that doesn't jeopardize the stability of
 NumPy today.These are some of the real questions I have.   I will try
 to write up my thoughts in more depth soon, but  I won't be able to respond
 in-depth right now.   I just wanted to comment because Nathaniel said I
 disagree which is only partly true.

 The three most important things for me are 1) let's make sure we have
 representation from as wide of the community as possible (this is really
 hard), 2) let's look around at the broader community and the prior art that
 is happening in this space right now and 3) let's not pretend we are going
 to be able to make all this happen without breaking ABI compatibility.
 Let's just break ABI compatibility with NumPy 2.0 *and* have as much
 fidelity with the API and semantics of current NumPy as possible (though
 there will be some changes necessary long-term).

 I don't think we should intentionally break ABI if we can avoid it, but I
 also don't think we should spend in-ordinate amounts of time trying to
 pretend that we won't break ABI (for at least some people), and most
 importantly we should not pretend *not* to break the ABI when we actually
 do.We did this once before with the roll-out of date-time, and it was
 really un-necessary. When I released NumPy 1.0, there were several
 things that I knew should be fixed very soon (NumPy was never designed to
 not break ABI).Those problems are still there.Now, that we have
 quite a bit better understanding of what NumPy *should* be (there have been
 tremendous strides in understanding and community size over the past 10
 years), let's actually make the infrastructure we think will last for the
 next 20 years (instead of trying to shoe-horn new ideas into a 20-year old
 code-base that wasn't designed for it).

 NumPy is a hard code-base.  It has been since Numeric days in 1995. I
 could be wrong, but my guess is that we will be passed by as a community if
 we don't seize the opportunity to build something better than we can build
 if we are forced to use a 20 year old code-base.

 It is more important to not break people's code and to be clear when a
 re-compile is necessary for dependencies.   Those to me are the most
 important constraints. There are a lot of great ideas that we all have
 about what we want NumPy to be able to do. Some of this are pretty
 transformational (and the more exciting they are, the harder I think they
 are going to be to implement without breaking at least the ABI). There
 is probably some CAP-like theorem around
 Stability-Features-Speed-of-Development (pick 2) when it comes to Open
 Source Software development and making feature-progress with NumPy *is
 going* to create in-stability which concerns me.

 I would like to see a little-bit-of-pain one time with a NumPy 2.0,
 rather than a constant pain because of constant churn over many years
 approach that Nathaniel seems to advocate.   To me NumPy 2.0 is an
 ABI-breaking release that is as API-compatible as possible and whose
 

Re: [Numpy-discussion] Proposal to remove the Bento build.

[David Cournapeau]

On Wed, Aug 19, 2015 at 1:22 AM, Nathaniel Smith n...@pobox.com wrote:

 On Tue, Aug 18, 2015 at 4:15 PM, David Cournapeau courn...@gmail.com
 wrote:
  If everybody wants to remove bento, we should remove it.

 FWIW, I don't really have an opinion either way on bento versus
 distutils, I just feel that we shouldn't maintain two build systems
 unless we're actively planning to get rid of one of them, and for
 several years now we haven't really been learning anything by keeping
 the bento build working, nor has there been any movement towards
 switching to bento as the one-and-only build system, or even a clear
 consensus that this would be a good thing. (Obviously distutils and
 numpy.distutils are junk, so that's a point in bento's favor, but it
 isn't *totally* cut and dried -- we know numpy.distutils works and we
 have to maintain it regardless for backcompat, while bento doesn't
 seem to have any activity upstream or any other users...).

 So I'd be totally in favor of adding bento back later if/when such a
 plan materializes; I just don't think it makes sense to keep
 continuously investing effort into it just in case such a plan
 materializes later.

  Regarding single file builds, why would it help for static builds ? I
  understand it would make things slightly easier to have one .o per
  extension, but it does not change the fundamental process as the exported
  symbols are the same in the end ?

 IIUC they aren't: with the multi-file build we control exported
 symbols using __attribute__((visibility(hidden)) or equivalent,
 which hides symbols from the shared object export table, but not from
 other translation units that are statically linked. So if you want to
 statically link cpython and numpy, you need some other way to let
 numpy .o files see each others's symbols without exposing them to
 cpython's .o files,


It is less a problem than in shared linking because you can detect the
conflicts at linking time (instead of loading time).


and the single-file build provides one mechanism
 to do that: make the numpy symbols 'static' and then combine them all
 into a single translation unit.

 I would love to be wrong about this though. The single file build is
 pretty klugey :-).


I know, it took me a while to split the files to go out of single file
build in the first place :)

David



 -n

 --
 Nathaniel J. Smith -- http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Proposal to remove the Bento build.

[David Cournapeau]

If everybody wants to remove bento, we should remove it.

Regarding single file builds, why would it help for static builds ? I
understand it would make things slightly easier to have one .o per
extension, but it does not change the fundamental process as the exported
symbols are the same in the end ?

David

On Tue, Aug 18, 2015 at 9:07 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,
  .
 I'm bringing up this topic again on account of the discussion at
 https://github.com/numpy/numpy/pull/6199. The proposal is to stop
 (trying) to support the Bento build system for Numpy and remove it. Votes
 and discussion welcome.

 Along the same lines, Pauli has suggested removing the single file builds,
 but Nathaniel has pointed out that it may be the only way to produce static
 python + numpy builds. If anyone does that or has more information about
 it, please comment.

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy-vendor vcvarsall.bat problem.

[David Cournapeau]

Which command exactly did you run to have that error ? Normally, the code
in msvc9compiler should not be called if you call the setup.py with the
mingw compiler as expected by distutils

On Fri, Aug 7, 2015 at 12:19 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Thu, Aug 6, 2015 at 5:11 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:



 On Thu, Aug 6, 2015 at 4:22 PM, David Cournapeau courn...@gmail.com
 wrote:

 Sorry if that's obvious, but do you have Visual Studio 2010 installed ?

 On Thu, Aug 6, 2015 at 11:17 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Anyone know how to fix this? I've run into it before and never got it
 figured out.

 [192.168.121.189:22] out:   File
 C:\Python34\lib\distutils\msvc9compiler.py, line 259, in query_vcvarsall
 [192.168.121.189:22] out:
 [192.168.121.189:22] out: raise DistutilsPlatformError(Unable to
 find vcvarsall.bat)
 [192.168.121.189:22] out:
 [192.168.121.189:22] out: distutils.errors.DistutilsPlatformError:
 Unable to find vcvarsall.bat

 Chuck



 I'm running numpy-vendor, which is running wine. I think it is all mingw
 with a few installed dll's. The error is coming from the Python distutils
 as part of `has_cblas`.


 It's not impossible that we have changed the build somewhere along the
 line.

 Chuck


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy-vendor vcvarsall.bat problem.

[David Cournapeau]

Sorry if that's obvious, but do you have Visual Studio 2010 installed ?

On Thu, Aug 6, 2015 at 11:17 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Anyone know how to fix this? I've run into it before and never got it
 figured out.

 [192.168.121.189:22] out:   File
 C:\Python34\lib\distutils\msvc9compiler.py, line 259, in query_vcvarsall
 [192.168.121.189:22] out:
 [192.168.121.189:22] out: raise DistutilsPlatformError(Unable to
 find vcvarsall.bat)
 [192.168.121.189:22] out:
 [192.168.121.189:22] out: distutils.errors.DistutilsPlatformError: Unable
 to find vcvarsall.bat

 Chuck


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Verify your sourceforge windows installer downloads

[David Cournapeau]

IMO, this really begs the question on whether we still want to use
sourceforge at all. At this point I just don't trust the service at all
anymore.

Could we use some resources (e.g. rackspace ?) to host those files ? Do we
know how much traffic they get so estimate the cost ?

David

On Thu, May 28, 2015 at 9:46 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 hi,
 It has been reported that sourceforge has taken over the gimp
 unofficial windows downloader page and temporarily bundled the
 installer with unauthorized adware:
 https://plus.google.com/+gimp/posts/cxhB1PScFpe

 As NumPy is also distributing windows installers via sourceforge I
 recommend that when you download the files you verify the downloads
 via the checksums in the README.txt before using them. The README.txt
 is clearsigned with my gpg key so it should be safe from tampering.
 Unfortunately as I don't use windows I cannot give any advice on how
 to do the verifcation on these platforms. Maybe someone familar with
 available tools can chime in.

 I have checked the numpy downloads and they still match what I
 uploaded, but as sourceforge does redirect based on OS and geolocation
 this may not mean much.

 Cheers,
 Julian Taylor
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Verify your sourceforge windows installer downloads

[David Cournapeau]

On Fri, May 29, 2015 at 2:00 AM, Andrew Collette andrew.colle...@gmail.com
wrote:

  Here is their lame excuse:
 
 
 https://sourceforge.net/blog/gimp-win-project-wasnt-hijacked-just-abandoned/
 
  It probably means this:
 
  If NumPy installers are moved away from Sourceforge, they will set up a
  mirror and load the mirrored installers with all sorts of crapware. It is
  some sort of racket the mob couldn't do better.

 I noticed that like most BSD-licensed software, NumPy's license
 includes this clause:

 Neither the name of the NumPy Developers nor the names of any
 contributors may be used to endorse or promote products derived from
 this software without specific prior written permission.

 There's an argument to be made that SF isn't legally permitted to
 distribute poisoned installers under the name NumPy without
 permission.  I recall a similar dust-up a while ago about Standard
 Markdown using the name Markdown; the original author (John Gruber)
 took action and got them to change the name.

 In any case I've always been surprised that NumPy is distributed
 through SourceForge, which has been sketchy for years now. Could it
 simply be hosted on PyPI?


They don't accept arbitrary binaries like SF does, and some of our
installer formats can't be uploaded there.

David



 Andrew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.diag(np.dot(A, B))

[David Cournapeau]

On Fri, May 22, 2015 at 5:39 PM, Mathieu Blondel math...@mblondel.org
wrote:

 Hi,

 I often need to compute the equivalent of

 np.diag(np.dot(A, B)).

 Computing np.dot(A, B) is highly inefficient if you only need the diagonal
 entries. Two more efficient ways of computing the same thing are

 np.sum(A * B.T, axis=1)

 and

 np.einsum(ij,ji-i, A, B).

 The first can allocate quite a lot of temporary memory.
 The second can be quite cryptic for someone not familiar with einsum.
 I assume that einsum does not compute np.dot(A, B), but I haven't verified.

 Since this is is quite a recurrent pattern, I was wondering if it would be
 worth adding a dedicated function to NumPy and SciPy's sparse module. A
 possible name would be diagdot. The best performance would be obtained
 when A is C-style and B fortran-style.


Does your implementation use BLAS, or is just a a wrapper around einsum ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[David Cournapeau]

On Sat, Apr 4, 2015 at 4:25 AM, Nathaniel Smith n...@pobox.com wrote:

 IIRC there allegedly exist platforms where separate compilation doesn't
 work right? I'm happy to get rid of it if no one speaks up to defend such
 platforms, though, we can always add it back later. One case was for
 statically linking numpy into the interpreter, but I'm skeptical about how
 much we should care about that case, since that's already a hacky kind of
 process and there are simple alternative hacks that could be used to strip
 the offending symbols.

 Depends on how much it lets us simplify things, I guess. Would we get to
 remove all the no-export attributes on everything?


No, the whole point of the no-export is to support the separate compilation
use case.

David


 On Apr 3, 2015 8:01 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Fri, Apr 3, 2015 at 9:00 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Hi All,

 Just to raise the question if these two options should be removed at
 some point? The current default value for both is 0, so we have separate
 compilation and relaxed strides checking by default.


 Oops, default value is 1, not 0.

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] IDE's for numpy development?

[David Cournapeau]

On Wed, Apr 1, 2015 at 7:43 PM, Charles R Harris charlesr.har...@gmail.com
wrote:



 On Wed, Apr 1, 2015 at 11:55 AM, Sturla Molden sturla.mol...@gmail.com
 wrote:

 Charles R Harris charlesr.har...@gmail.com wrote:

  I'd be
  interested in information from anyone with experience in using such an
 IDE
  and ideas of how Numpy might make using some of the common IDEs easier.
 
  Thoughts?

 I guess we could include project files for Visual Studio (and perhaps
 Eclipse?), like Python does. But then we would need to make sure the
 different build systems are kept in sync, and it will be a PITA for those
 who do not use Windows and Visual Studio. It is already bad enough with
 Distutils and Bento. I, for one, would really prefer if there only was one
 build process to care about. One should also note that a Visual Studio
 project is the only supported build process for Python on Windows. So they
 are not using this in addition to something else.

 Eclipse is better than Visual Studio for mixed Python and C development.
 It
 is also cross-platform.

 cmake needs to be mentioned too. It is not fully integrated with Visual
 Studio, but better than having multiple build processes.


 Mark chose cmake for DyND because it supported Visual Studio projects.
 OTOH, he said it was a PITA to program.


I concur on that:  For the 350+ packages we support at Enthought, cmake has
been a higher pain point than any other build tool (that is including
custom ones). And we only support mainstream platforms.

But the real question for me is what does visual studio support mean ? Does
it really mean solution files ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] PyCon?

[David Cournapeau]

I'll be there as well, though I am still figuring out when exactly .

On Wed, Feb 18, 2015 at 1:07 AM, Nathaniel Smith n...@pobox.com wrote:

 Hi all,

 It looks like I'll be at PyCon this year. Anyone else? Any interest in
 organizing a numpy sprint?

 -n

 --
 Nathaniel J. Smith -- http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] missing FloatingPointError for numpy on cygwin64

[David Cournapeau]

Hi Sebastian,

I think you may be one of the first person to report using cygwin 64. I
think it makes sense to support that platform as it is becoming more common.

Could you report the value of `sys.platform` on cygwin64 ? The first place
I would look for cygwin-related FPU issues is there:
https://github.com/numpy/numpy/blob/master/numpy/core/setup.py#L638

David

On Sat, Jan 31, 2015 at 9:53 PM, Sebastien Gouezel 
sebastien.goue...@univ-rennes1.fr wrote:

 Dear all,

 I tried to use numpy (version 1.9.1, installed by `pip install numpy`)
 on cygwin64. I encountered the following weird bug:

   import numpy
   with numpy.errstate(all='raise'):
 ...print 1/float64(0.0)
 inf

 I was expecting a FloatingPointError, but it didn't show up. Curiously,
 with different numerical types (all intxx, or float128), I indeed get
 the FloatingPointError.

 Same thing with the most recent git version, or with 1.7.1 provided as a
 precompiled package by cygwin. This behavior does not happen on cygwin32
 (I always get the FloatingPointError there).

 I wonder if there is something weird with my config, or if this is a
 genuine reproducible bug. If so, where should I start looking if I want
 to fix it? (I don't know anything about numpy's code)

 Sebastien

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] npymath on Windows

[David Cournapeau]

On Sun, Dec 28, 2014 at 1:59 AM, Matthew Brett matthew.br...@gmail.com
wrote:

 Hi,

 Sorry for this ignorant email, but we got confused trying to use
 'libnpymath.a' from the mingw builds of numpy:

 We were trying to link against the mingw numpy 'libnpymath.a' using
 Visual Studio C, but this give undefined symbols from 'libnpymath.a'
 like this:


This is not really supported. You should avoid mixing compilers when
building C extensions using numpy C API. Either all mingw, or all MSVC.

David



 npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol
 _atanf referenced in function _npy_atanf
 npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol
 _acosf referenced in function _npy_acosf
 npymath.lib(npy_math.o) : error LNK2019: unresolved external symbol
 _asinf referenced in function _npy_asinf

 (see :
 http://nipy.bic.berkeley.edu/builders/dipy-bdist32-33/builds/73/steps/shell_6/logs/stdio
 )

 npymath.lib from Christophe Gohlke's (MSVC compiled) numpies does not
 give such an error.  Sure enough, 'npymath.lib' shows these lines from
 `dumpbin /all npymath.lib`:

   0281  REL32  4F  asinf
   0291  REL32  51  acosf
   02A1  REL32  53  atanf

 whereas `dumpbin /all libnpymath.a` shows these kinds of lines:

  08E5  REL32  86  _asinf
  08F5  REL32  85  _acosf
  0905  REL32  84  _atanf

 As far as I can see, 'acosf' is defined in the msvc runtime library.
 I guess that '_acosf' is defined in some mingw runtime library?   Is
 there any way of making a npymath library that will pick up the msvc
 math and so may work with both msvc and mingw?

 Sorry again if that's a dumb question,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [SciPy-Dev] ANN: Scipy 0.14.1 release candidate 1

[David Cournapeau]

I built that rc on top of numpy 1.8.1 and MKL, and it worked on every
platform we support @ Enthought.

I saw a few test failures on linux and windows 64 bits, but those were
there before or are precisions issues.

I also tested when run on top of numpy 1.9.1 (but still built against
1.8.1), w/ similar results.

Thanks for all the hard work,
David

On Sun, Dec 14, 2014 at 10:29 PM, Pauli Virtanen p...@iki.fi wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Dear all,

 We have finished preparing the Scipy 0.14.1 release candidate 1.
 If no regressions turn up, the final release is planned within the
 following weeks.

 The 0.14.1 release will be a bugfix-only release, addressing the
 following issues:

 - - gh-3630 NetCDF reading results in a segfault
 - - gh-3631 SuperLU object not working as expected for complex matrices
 - - gh-3733 Segfault from map_coordinates
 - - gh-3780 Segfault when using CSR/CSC matrix and uint32/uint64
 - - gh-3781 Fix omitted types in sparsetools typemaps
 - - gh-3802 0.14.0 API breakage: _gen generators are missing from
 scipy.stats.distributions API
 - - gh-3805 ndimge test failures with numpy 1.10
 - - gh-3812 == sometimes wrong on csr_matrix
 - - gh-3853 Many scipy.sparse test errors/failures with numpy 1.9.0b2
 - - gh-4084 Fix exception declarations for Cython 0.21.1 compatibility
 - - gh-4093 Avoid a memory error in splev(x, tck, der=k)
 - - gh-4104 Workaround SGEMV segfault in Accelerate (maintenance 0.14.x)
 - - gh-4143 Fix ndimage functions for large data
 - - gh-4149 Bug in expm for integer arrays
 - - gh-4154 Ensure that the 'size' argument of PIL's 'resize' method is
 a tuple
 - - gh-4163 ZeroDivisionError in scipy.sparse.linalg.lsqr
 - - gh-4164 Remove use of deprecated numpy API in lib/lapack/ f2py wrapper
 - - gh-4180 pil resize support tuple fix
 - - gh-4168 Address arpack test failures on windows 32 bits with numpy
 1.9.1
 - - gh-4218 make ndimage interpolation compatible with numpy relaxed
 strides
 - - gh-4225 off-by-one error in PPoly shape checks
 - - gh-4248 fix issue with incorrect use of closure for slsqp

 Source tarballs and binaries are available at
 https://sourceforge.net/projects/scipy/files/scipy/0.14.1rc1/

 Best regards,
 Pauli Virtanen
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1

 iEYEARECAAYFAlSOD0YACgkQ6BQxb7O0pWDO5ACfccLqMvZWfkHqSzDCkMSoRKAU
 n7cAni6XhWJRy7oJ757rlGeIi0e34HTn
 =9bB/
 -END PGP SIGNATURE-

 ___
 SciPy-Dev mailing list
 scipy-...@scipy.org
 http://mail.scipy.org/mailman/listinfo/scipy-dev

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] GDB macros for numpy

[David Cournapeau]

Hi there,

I remember having seen some numpy-aware gdb macros at some point, but
cannot find any reference. Does anyone know of any ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] GDB macros for numpy

[David Cournapeau]

On Sun, Nov 30, 2014 at 5:45 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Sun, Nov 30, 2014 at 4:54 AM, David Cournapeau courn...@gmail.com
 wrote:

 Hi there,

 I remember having seen some numpy-aware gdb macros at some point, but
 cannot find any reference. Does anyone know of any ?


 What would numpy aware gdb macros be? Never heard of them.


Python itself has some gdb macros (
https://wiki.python.org/moin/DebuggingWithGdb).

You can do things like:

# p is a PyObject * pointer to the list [1, 2]
 pyo p
[1, 2]

The idea would be to have a nice representation for arrays, dtypes, etc...

David


 Chuck


 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Setting up a newcomers label on the issue tracker ?

[David Cournapeau]

oups, I missed it. Will use that one then.

On Wed, Nov 26, 2014 at 9:01 AM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 On 11/26/2014 09:44 AM, David Cournapeau wrote:
  Hi,
 
  Would anybody mind if I create a label newcomers on GH, and start
  labelling simple issues ?
 
  This is in anticipation to the bloomberg lab event in London this WE. I
  will try to give a hand to people interested in numpy/scipy,
 
  David
 

 we have the easy-fix tag which we use for this purpose, though we have
 not been applying it very systematically, the bug list could probably do
 with a new sweep for these issues.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: Scipy 0.15.0 beta 1 release

[David Cournapeau]

Shall we consider https://github.com/scipy/scipy/issues/4168 to be a
blocker (the issue arises on scipy master as well as 0.14.1) ?

On Sun, Nov 23, 2014 at 11:13 PM, Pauli Virtanen p...@iki.fi wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Dear all,

 We have finally finished preparing the Scipy 0.15.0 beta 1 release.
 Please try it and report any issues on the scipy-dev mailing list,
 and/or on Github.

 If no surprises turn up, the final release is planned on Dec 20 in
 three weeks.

 Source tarballs and full release notes are available at
 https://sourceforge.net/projects/scipy/files/SciPy/0.15.0b1/
 Binary installers should also be up soon.

 Best regards,
 Pauli Virtanen


 - 

 SciPy 0.15.0 is the culmination of 6 months of hard work. It contains
 several new features, numerous bug-fixes, improved test coverage and
 better documentation.  There have been a number of deprecations and
 API changes in this release, which are documented below.  All users
 are encouraged to upgrade to this release, as there are a large number
 of bug-fixes and optimizations.  Moreover, our development attention
 will now shift to bug-fix releases on the 0.16.x branch, and on adding
 new features on the master branch.

 This release requires Python 2.6, 2.7 or 3.2-3.3 and NumPy 1.5.1 or
 greater.


 New features
 

 Linear Programming Interface
 - - 

 The new function ``scipy.optimize.linprog`` provides a generic
 linear programming similar to the way ``scipy.optimize.minimize``
 provides a generic interface to nonlinear programming optimizers.
 Currently the only method supported is *simplex* which provides
 a two-phase, dense-matrix-based simplex algorithm. Callbacks
 functions are supported,allowing the user to monitor the progress
 of the algorithm.

 Differential_evolution, a global optimizer
 - - --

 A new ``differential_evolution`` function is available in the
 ``scipy.optimize``
 module.  Differential Evolution is an algorithm used for finding the
 global
 minimum of multivariate functions. It is stochastic in nature (does
 not use
 gradient methods), and can search large areas of candidate space, but
 often
 requires larger numbers of function evaluations than conventional gradient
 based techniques.

 ``scipy.signal`` improvements
 - - -

 The function ``max_len_seq`` was added, which computes a Maximum
 Length Sequence (MLS) signal.

 ``scipy.integrate`` improvements
 - - 

 It is now possible to use ``scipy.integrate`` routines to integrate
 multivariate ctypes functions, thus avoiding callbacks to Python and
 providing better performance.

 ``scipy.linalg`` improvements
 - - -

 Add function ``orthogonal_procrustes`` for solving the procrustes
 linear algebra problem.

 ``scipy.sparse`` improvements
 - - -

 ``scipy.sparse.linalg.svds`` can now take a ``LinearOperator`` as its
 main input.

 ``scipy.special`` improvements
 - - --

 Values of ellipsoidal harmonic (i.e. Lame) functions and associated
 normalization constants can be now computed using ``ellip_harm``,
 ``ellip_harm_2``, and ``ellip_normal``.

 New convenience functions ``entr``, ``rel_entr`` ``kl_div``,
 ``huber``, and ``pseudo_huber`` were added.

 ``scipy.sparse.csgraph`` improvements
 - - -

 Routines ``reverse_cuthill_mckee`` and ``maximum_bipartite_matching``
 for computing reorderings of sparse graphs were added.

 ``scipy.stats`` improvements
 - - 

 Added a Dirichlet distribution as multivariate distribution.

 The new function ``scipy.stats.median_test`` computes Mood's median test.

 The new function ``scipy.stats.combine_pvalues`` implements Fisher's
 and Stouffer's methods for combining p-values.

 ``scipy.stats.describe`` returns a namedtuple rather than a tuple,
 allowing
 users to access results by index or by name.

 Deprecated features
 ===

 The ``scipy.weave`` module is deprecated.  It was the only module
 never ported
 to Python 3.x, and is not recommended to be used for new code - use Cython
 instead.  In order to support existing code, ``scipy.weave`` has been
 packaged
 separately: `https://github.com/scipy/weave`_.  It is a pure Python
 package, and
 can easily be installed with ``pip install weave``.

 ``scipy.special.bessel_diff_formula`` is deprecated.  It is a private
 function,
 and therefore will be removed from the public API in a following release.


 Backwards incompatible changes
 ==

 scipy.ndimage
 - - -

 The functions ``scipy.ndimage.minimum_positions``,
 ``scipy.ndimage.maximum_positions`` and ``scipy.ndimage.extrema`` return
 positions as ints instead of floats.

 scipy.integrate
 - - ---

 The format 

Re: [Numpy-discussion] Scipy 0.15.0 beta 1 release

[David Cournapeau]

On Tue, Nov 25, 2014 at 6:10 PM, Sturla Molden sturla.mol...@gmail.com
wrote:

 David Cournapeau courn...@gmail.com wrote:
  Shall we consider a
  href=https://github.com/scipy/scipy/issues/4168;
 https://github.com/scipy/scipy/issues/4168/a
  to be a
  blocker (the issue arises on scipy master as well as 0.14.1) ?
 

 It is really bad, but does anyone know what is really going on?


Yes, it is in the bug report.

David



 Which changes to NumPy set this off?

 Sturla

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Numpy 1.9.1, zeros and alignement

[David Cournapeau]

Hi,

I have not followed closely the changes that happen in 1.9.1, but was
surprised by the following:

x = np.zeros(12, d)
assert x.flags.aligned # fails

This is running numpy 1.9.1 built on windows with VS 2008. Is it expected
that zeros may return a non-aligned array ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[David Cournapeau]

It is on windows 32 bits, but I would need to make this work for complex
(pair of double) as well.

Is this a bug (I  assumed array creation methods would always create
aligned arrays for their type) ? Seems like quite a bit of code out there
would assume this (scipy itself does for example).

(the context is  100 test failures on scipy 0.14.x on top of numpy 1.9.,
because f2py intent(inout) fails on work arrays created by zeros, this is a
windows-32 only failure).

David

On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 On 18.11.2014 19:20, David Cournapeau wrote:
  Hi,
 
  I have not followed closely the changes that happen in 1.9.1, but was
  surprised by the following:
 
  x = np.zeros(12, d)
  assert x.flags.aligned # fails
 
  This is running numpy 1.9.1 built on windows with VS 2008. Is it
  expected that zeros may return a non-aligned array ?
 

 what is the real alignment of the array? Are you on 32 bit or 64 bit?
 What is the alignment of doubles in windows (linux its 4 byte on 32 bit
 8 byte on 64 bit (% special compiler flags)?
 print x.__array_interface__[data]

 there are problems with complex types but doubles should be aligned even
 on 32 bit.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[David Cournapeau]

Additional point: it seems to always return aligned data on 1.8.1 (same
platform/compiler/everything).

On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau courn...@gmail.com
wrote:

 It is on windows 32 bits, but I would need to make this work for complex
 (pair of double) as well.

 Is this a bug (I  assumed array creation methods would always create
 aligned arrays for their type) ? Seems like quite a bit of code out there
 would assume this (scipy itself does for example).

 (the context is  100 test failures on scipy 0.14.x on top of numpy 1.9.,
 because f2py intent(inout) fails on work arrays created by zeros, this is a
 windows-32 only failure).

 David

 On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor 
 jtaylor.deb...@googlemail.com wrote:

 On 18.11.2014 19:20, David Cournapeau wrote:
  Hi,
 
  I have not followed closely the changes that happen in 1.9.1, but was
  surprised by the following:
 
  x = np.zeros(12, d)
  assert x.flags.aligned # fails
 
  This is running numpy 1.9.1 built on windows with VS 2008. Is it
  expected that zeros may return a non-aligned array ?
 

 what is the real alignment of the array? Are you on 32 bit or 64 bit?
 What is the alignment of doubles in windows (linux its 4 byte on 32 bit
 8 byte on 64 bit (% special compiler flags)?
 print x.__array_interface__[data]

 there are problems with complex types but doubles should be aligned even
 on 32 bit.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.9.1, zeros and alignement

[David Cournapeau]

On Tue, Nov 18, 2014 at 6:40 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

  1.9 lies about alignment it doesn't actually check for new arrays.


When I do the following on 1.8.1 with win 32 bits:

x = np.zeros(12, D)
print x.aligned.flags  == (x.__array_interface__[data][0] % 16 == 0) #
always true
print x.aligned.flags # always true

but on 1.9.1:

x = np.zeros(12, D)
print x.aligned.flags == (x.__array_interface__[data][0] % 16 == 0) #
always true
print x.aligned.flags # not always true

I wonder why numpy 1.8.1 always returned 16 bytes aligned arrays in this
case (unlikely to be a coincidence, as I try quite a few times with
different sizes).



 is the array aligned?

 On 18.11.2014 19:37, David Cournapeau wrote:
  Additional point: it seems to always return aligned data on 1.8.1 (same
  platform/compiler/everything).
 
  On Tue, Nov 18, 2014 at 6:35 PM, David Cournapeau courn...@gmail.com
  mailto:courn...@gmail.com wrote:
 
  It is on windows 32 bits, but I would need to make this work for
  complex (pair of double) as well.
 
  Is this a bug (I  assumed array creation methods would always create
  aligned arrays for their type) ? Seems like quite a bit of code out
  there would assume this (scipy itself does for example).
 
  (the context is  100 test failures on scipy 0.14.x on top of numpy
  1.9., because f2py intent(inout) fails on work arrays created by
  zeros, this is a windows-32 only failure).
 
  David
 
  On Tue, Nov 18, 2014 at 6:26 PM, Julian Taylor
  jtaylor.deb...@googlemail.com
  mailto:jtaylor.deb...@googlemail.com wrote:
 
  On 18.11.2014 19:20, David Cournapeau wrote:
   Hi,
  
   I have not followed closely the changes that happen in 1.9.1,
  but was
   surprised by the following:
  
   x = np.zeros(12, d)
   assert x.flags.aligned # fails
  
   This is running numpy 1.9.1 built on windows with VS 2008. Is
 it
   expected that zeros may return a non-aligned array ?
  
 
  what is the real alignment of the array? Are you on 32 bit or 64
  bit?
  What is the alignment of doubles in windows (linux its 4 byte on
  32 bit
  8 byte on 64 bit (% special compiler flags)?
  print x.__array_interface__[data]
 
  there are problems with complex types but doubles should be
  aligned even
  on 32 bit.
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)

[David Cournapeau]

On Wed, Oct 29, 2014 at 9:48 AM, Eelco Hoogendoorn 
hoogendoorn.ee...@gmail.com wrote:

 My point isn't about speed; its about the scope of numpy. typing
 np.fft.fft isn't more or less convenient than using some other symbol from
 the scientific python stack.

 Numerical algorithms should be part of the stack, for sure; but should
 they be part of numpy? I think its cleaner to have them in a separate
 package. Id rather have us discuss how to facilitate the integration of as
 many possible fft libraries with numpy behind a maximally uniform
 interface, rather than having us debate which fft library is 'best'.


I would agree if it were not already there, but removing it (like
Blas/Lapack) is out of the question for backward compatibility reason. Too
much code depends on it.

David



 On Tue, Oct 28, 2014 at 6:21 PM, Sturla Molden sturla.mol...@gmail.com
 wrote:

 Eelco Hoogendoorn hoogendoorn.ee...@gmail.com wrote:

  Perhaps the 'batteries included' philosophy made sense in the early
 days of
  numpy; but given that there are several fft libraries with their own
 pros
  and cons, and that most numpy projects will use none of them at all, why
  should numpy bundle any of them?

 Because sometimes we just need to compute a DFT, just like we sometimes
 need to compute a sine or an exponential. It does that job perfectly well.
 It is not always about speed. Just typing np.fft.fft(x) is convinient.

 Sturla

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion



 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Choosing between NumPy and SciPy functions

[David Cournapeau]

On Tue, Oct 28, 2014 at 5:24 AM, Sturla Molden sturla.mol...@gmail.com
wrote:

 Matthew Brett matthew.br...@gmail.com wrote:

  Is this an option for us?  Aren't we a little behind the performance
  curve on FFT after we lost FFTW?

 It does not run on Windows because it uses POSIX to allocate executable
 memory for tasklets, as i understand it.

 By the way, why did we loose FFTW, apart from GPL? One thing to mention
 here is that MKL supports the FFTW APIs. If we can use MKL for linalg and
 numpy.dot I don't see why we cannot use it for FFT.


The problem is APIs: MKL, Accelerate, etc... all use a standard API
(BLAS/LAPACK), but for FFT, you need to reimplement pretty much the whole
thing. Unsurprisingly, this meant the code was not well maintained.

Wrapping non standard, non-BSD libraries makes much more sense in separate
libraries in general.

David



 On Mac there is also vDSP in Accelerate framework which has an insanely
 fast FFT (also claimed to be faster than FFTW). Since it is a system
 library there should be no license problems.

 There are clearly options if someone wants to work on it and maintain it.

 Sturla

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)

[David Cournapeau]

On Tue, Oct 28, 2014 at 9:19 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:



 On Tue, Oct 28, 2014 at 1:32 AM, Jerome Kieffer jerome.kief...@esrf.fr
 wrote:

 On Tue, 28 Oct 2014 04:28:37 +
 Nathaniel Smith n...@pobox.com wrote:

  It's definitely attractive. Some potential issues that might need
 dealing
  with, based on a quick skim:

 In my tests, numpy's FFTPACK isn't that bad considering
 * (virtually) no extra overhead for installation
 * (virtually) no plan creation time
 * not that slower for each transformation

 Because the plan creation was taking ages with FFTw, numpy's FFTPACK was
 often faster (overall)

 Cheers,


 Ondrej says that f90 fftpack (his mod) runs faster than fftw.


I would be interested to see the benchmarks for this.

The real issue with fftw (besides the license) is the need for plan
computation, which are expensive (but are not needed for each transform).
Handling this in a way that is user friendly while tweakable for advanced
users is not easy, and IMO more appropriate for a separate package.

The main thing missing from fftpack is the handling of transform sizes that
 are not products of 2,3,4,5.


Strickly speaking, it is handled, just not through an FFT (it goes back to
the brute force O(N**2)).

I made some experiments with the Bluestein transform to handle prime
transforms on fftpack, but the precision seemed to be an issue. Maybe I
should revive this work (if I still have it somewhere).

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)

[David Cournapeau]

I

On Tue, Oct 28, 2014 at 2:31 PM, Nathaniel Smith n...@pobox.com wrote:

 On 28 Oct 2014 07:32, Jerome Kieffer jerome.kief...@esrf.fr wrote:
 
  On Tue, 28 Oct 2014 04:28:37 +
  Nathaniel Smith n...@pobox.com wrote:
 
   It's definitely attractive. Some potential issues that might need
 dealing
   with, based on a quick skim:
 
  In my tests, numpy's FFTPACK isn't that bad considering
  * (virtually) no extra overhead for installation
  * (virtually) no plan creation time
  * not that slower for each transformation

 Well, this is what makes FFTS intriguing :-). It's BSD licensed, so we
 could distribute it by default like we do fftpack, it uses cache-oblivious
 algorithms so it has no planning step, and even without planning it
 benchmarks as faster than FFTW's most expensive planning mode (in the cases
 that FFTS supports, i.e. power-of-two transforms).

 The paper has lots of benchmark graphs, including measurements of setup
 time:
   http://anthonix.com/ffts/preprints/tsp2013.pdf


Nice. In this case, the solution may be to implement the Bluestein
transform to deal with prime/near-prime numbers on top of FFTS.

I did not look much, but it did not obviously support building on windows
as well ?

David


 -n

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)

[David Cournapeau]

On Tue, Oct 28, 2014 at 3:06 PM, David Cournapeau courn...@gmail.com
wrote:

 I

 On Tue, Oct 28, 2014 at 2:31 PM, Nathaniel Smith n...@pobox.com wrote:

 On 28 Oct 2014 07:32, Jerome Kieffer jerome.kief...@esrf.fr wrote:
 
  On Tue, 28 Oct 2014 04:28:37 +
  Nathaniel Smith n...@pobox.com wrote:
 
   It's definitely attractive. Some potential issues that might need
 dealing
   with, based on a quick skim:
 
  In my tests, numpy's FFTPACK isn't that bad considering
  * (virtually) no extra overhead for installation
  * (virtually) no plan creation time
  * not that slower for each transformation

 Well, this is what makes FFTS intriguing :-). It's BSD licensed, so we
 could distribute it by default like we do fftpack, it uses cache-oblivious
 algorithms so it has no planning step, and even without planning it
 benchmarks as faster than FFTW's most expensive planning mode (in the cases
 that FFTS supports, i.e. power-of-two transforms).

 The paper has lots of benchmark graphs, including measurements of setup
 time:
   http://anthonix.com/ffts/preprints/tsp2013.pdf


 Nice. In this case, the solution may be to implement the Bluestein
 transform to deal with prime/near-prime numbers on top of FFTS.

 I did not look much, but it did not obviously support building on windows
 as well ?


Ok, I took a quick look at it, and it will be a significant effort to be
able to make FFTS work at all with MSVC on windows:

- the code is not C89 compatible
- it uses code generation using POSIX library. One would need to port that
part to using Win32 API as well.
- the test suite looks really limited (roundtripping only).

The codebase does not seem particularly well written either (but neither is
FFTPACK to be fair).

Nothing impossible (looks like Sony at least uses this code on windows:
https://github.com/anthonix/ffts/issues/27#issuecomment-40204403), but not
a 2 hours thing either.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] npy_log2 undefined on Linux

[David Cournapeau]

Not exactly: if you build numpy with mingw (as is the official binary), you
need to build everything that uses numpy C API with it.

On Sun, Oct 26, 2014 at 1:22 AM, Matthew Brett matthew.br...@gmail.com
wrote:

 On Sat, Oct 25, 2014 at 2:15 PM, Matthew Brett matthew.br...@gmail.com
 wrote:
  On Fri, Oct 24, 2014 at 6:04 PM, Matthew Brett matthew.br...@gmail.com
 wrote:
  Hi,
 
  We (dipy developers) have a hit a new problem trying to use the
  ``npy_log`` C function in our code.
 
  Specifically, on Linux, but not on Mac or Windows, we are getting
  errors of form:
 
  ImportError: /path/to/extension/distances.cpython-34m.so: undefined
  symbol: npy_log2
 
  when compiling something like:
 
  eg_log.pyx
  import numpy as np
  cimport numpy as cnp
 
  cdef extern from numpy/npy_math.h nogil:
  double npy_log(double x)
 
 
  def use_log(double val):
  return npy_log(val)
  /eg_log.pyx
 
  See : https://github.com/matthew-brett/mincy/tree/npy_log_example for
  a self-contained example that replicates the failure with ``make``.
 
  I guess this means that the code referred to by ``npy_log`` is not on
  the ordinary runtime path on Linux?
 
  To answer my own question - npy_log is defined in ``libnpymath.a``, in
  numpy/core/lib.
 
  The hint I needed was in
 
 https://github.com/numpy/numpy/blob/master/doc/source/reference/c-api.coremath.rst
 
  The correct setup.py is:
 
  setup.py
  from distutils.core import setup
  from distutils.extension import Extension
  from Cython.Distutils import build_ext
 
  from numpy.distutils.misc_util import get_info
  npm_info = get_info('npymath')
 
  ext_modules = [Extension(eg_log, [eg_log.pyx],
   **npm_info)]
 
  setup(
name = 'eg_log',
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules
  )
  /setup.py

 Ah, except this doesn't work on Windows, compiling with Visual Studio:

 LINK : fatal error LNK1181: cannot open input file 'npymath.lib'

 Investigating, c:\Python27\Lib\site-packages\numpy\core\lib has only
 `libnpymath.a``; I guess this isn't going to work for Visual Studio.
 Is this a bug?

 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type

[David Cournapeau]

On Mon, Sep 22, 2014 at 4:31 AM, Nathaniel Smith n...@pobox.com wrote:

 On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer sho...@gmail.com wrote:
  pandas has some hacks to support custom types of data for which numpy
 can't
  handle well enough or at all. Examples include datetime and Categorical
 [1],
  and others like GeoArray [2] that haven't make it into pandas yet.
 
  Most of these look like numpy arrays but with custom dtypes and type
  specific methods/properties. But clearly nobody is particularly excited
  about writing the the C necessary to implement custom dtypes [3]. Nor is
 do
  we need the ndarray ABI.
 
  In many cases, writing C may not actually even be necessary for
 performance
  reasons, e.g., categorical can be fast enough just by wrapping an integer
  ndarray for the internal storage and using vectorized operations. And
 even
  if it is necessary, I think we'd all rather write Cython than C.
 
  It's great for pandas to write its own ndarray-like wrappers (*not*
  subclasses) that work with pandas, but it's a shame that there isn't a
  standard interface like the ndarray to make these arrays useable for the
  rest of the scientific Python ecosystem. For example, pandas has loads of
  fixes for np.datetime64, but nobody seems to be up for porting them to
 numpy
  (I doubt it would be easy).

 Writing them in the first place probably wasn't easy either :-). I
 don't really know why pandas spends so much effort on reimplementing
 stuff and papering over numpy limitations instead of fixing things
 upstream so that everyone can benefit. I assume they have reasons, and
 I could make some general guesses at what some of them might be, but
 if you want to know what they are -- which is presumably the first
 step in changing the situation -- you'll have to ask them, not us :-).

  I know these sort of concerns are not new, but I wish I had a sense of
 what
  the solution looks like. Is anyone actively working on these issues? Does
  the fix belong in numpy, pandas, blaze or a new project? I'd love to get
 a
  sense of where things stand and how I could help -- without writing any C
  :).

 I think there are there are three parts:

 For stuff that's literally just fixing bugs in stuff that numpy
 already has, then we'd certainly be happy to accept those bug fixes.
 Probably there are things we can do to make this easier, I dunno. I'd
 love to see some of numpy's internals moving into Cython to make them
 easier to hack on, but this won't be simple because right now using
 Cython to implement a module is really an all-or-nothing affair;
 making it possible to mix Cython with numpy's existing C code will
 require upstream changes in Cython.


 For cases where people genuinely want to implement a new array-like
 types (e.g. DataFrame or scipy.sparse) then numpy provides a fair
 amount of support for this already (e.g., the various hooks that allow
 things like np.asarray(mydf) or np.sin(mydf) to work), and we're
 working on adding more over time (e.g., __numpy_ufunc__).

 My feeling though is that in most of the cases you mention,
 implementing a new array-like type is huge overkill. ndarray's
 interface is vast and reimplementing even 90% of it is a huge effort.
 For most of the cases that people seem to run into in practice, the
 solution is to enhance numpy's dtype interface so that it's possible
 for mere mortals to implement new dtypes, e.g. by just subclassing
 np.dtype. This is totally doable and would enable a ton of
 awesomeness, but it requires someone with the time to sit down and
 work on it, and no-one has volunteered yet. Unfortunately it does
 require hacking on C code though.


While preparing my tutorial on NumPy C internals 1 year ago, I tried to get
a basic dtype implemented in cython, and there were various issues even
if you wanted to do all of it in cython (I can't remember the details now).
Solving this would be a good first step.

There were (are ?) also some issues regarding precedence in ufuncs
depending on the new dtype: numpy hardcodes that long double is the highest
precision floating point type, for example, and there were similar issues
regarding datetime handling. Does not matter for completely new types that
don't require interactions with others (categorical ?).

Would it help to prepare a set of implement your own dtype notebooks ? I
have a starting point from last year tutorial (the corresponding slides
were never shown for lack of time).

David




 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to install numpy on a box without hardware FPU

[David Cournapeau]

On Tue, Sep 2, 2014 at 8:29 AM, Emel Hasdal emel_has...@hotmail.com wrote:

 I am trying to run a python application which performs statistical
 calculations using Pandas which seem to depend on Numpy. Hence I have to
 install Numpy to get the app working.

 Do you mean I can change

numpy/core/src/npymath/ieee754.c.src

 such that the functions referencing exceptions (npy_get_floatstatus,
 npy_clear_floatstatus, npy_set_floatstatus_divbyzer,
 npy_set_floatstatus_overflow, npy_set_floatstatus_underflow,
 npy_set_floatstatus_invalid) do nothing?

 Could there be any implications of this on the numpy functionality?


AFAIK, few people have ever tried to run numpy on CPU without an FPU. The
generic answer is that we do not know, as this is not a supported platform,
so you are on your own.

I would suggest you just try adding stubs to see how far you can go, and
come back to this group with the result of your investigation.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Help - numpy / scipy binary compatibility

[David Cournapeau]

On Sat, Aug 9, 2014 at 10:41 AM, Matthew Brett matthew.br...@gmail.com
wrote:

 Hi,

 I would be very happy of some help trying to work out a numpy package
 binary incompatibility.

 I'm trying to work out what's happening for this ticket:

 https://github.com/scipy/scipy/issues/3863

 which I summarized at the end:

 https://github.com/scipy/scipy/issues/3863#issuecomment-51669861

 but basically, we're getting these errors:

 RuntimeWarning: numpy.dtype size changed, may indicate binary
 incompatibility

 I now realize I am lost in the world of numpy / scipy etc binary
 compatibility, I'd really like some advice.In this case

 numpy == 1.8.1
 scipy == 0.14.0 - compiled against numpy 1.5.1
 scikit-learn == 0.15.1 compiled against numpy 1.6.0

 Can y'all see any potential problem with those dependencies in binary
 builds?

 The relevant scipy Cython c files seem to guard against raising this
 error by doing not-strict checks of the e.g. numpy dtype, so I am
 confused how these errors come about.  Can anyone give any pointers?


Assuming the message is not bogus, I would try import von_mises with a venv
containing numpy 1.5.1, then 1.6.0, etc... to detect when the change
happened.

David


 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove numpy/compat/_inspect.py ?

[David Cournapeau]

On Fri, Aug 1, 2014 at 11:23 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:




 On Fri, Aug 1, 2014 at 7:59 AM, Robert Kern robert.k...@gmail.com wrote:

 On Fri, Aug 1, 2014 at 2:54 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:

  Importing inspect looks to take about  500 ns on my machine. Although
 It is
  hard to be exact, as I suspect the file is sitting in the file cache.
 Would
  probably be slower with hard disks.

 Or where site-packages is on NFS.

  But as the inspect module is already
  imported elsewhere, the python interpreter should also have it cached.

 Not on a normal import it's not.

  import numpy
  import sys
  sys.modules['inspect']
 Traceback (most recent call last):
   File stdin, line 1, in module
 KeyError: 'inspect'


 There are two lazy imports of inspect.



 You should feel free to remove whatever parts of `_inspect` are not
 being used and to move the parts that are closer to where they are
 used if you feel compelled to. Please do not replace the current uses
 of `_inspect` with `inspect`.


 It is used in just one place. Is importing inspect so much slower than all
 the other imports we do?


Yes, please look at the thread I referred to. The custom inspect cut
imports by 30 %, I doubt the ratio is much different today.

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove numpy/compat/_inspect.py ?

[David Cournapeau]

On my machine, if I use inspect instead of _inspect in
numpy.compat.__init__, the import time increases ~ 25 % (from 82 ms to 99
ms).

So the hack certainly still make sense, one just need to fix whatever needs
fixing (I am still not sure what's broken for the very specific usecase
that code was bundled for).

David


On Sat, Aug 2, 2014 at 5:11 AM, David Cournapeau courn...@gmail.com wrote:




 On Fri, Aug 1, 2014 at 11:23 PM, Charles R Harris 
 charlesr.har...@gmail.com wrote:




 On Fri, Aug 1, 2014 at 7:59 AM, Robert Kern robert.k...@gmail.com
 wrote:

 On Fri, Aug 1, 2014 at 2:54 PM, Charles R Harris
 charlesr.har...@gmail.com wrote:

  Importing inspect looks to take about  500 ns on my machine. Although
 It is
  hard to be exact, as I suspect the file is sitting in the file cache.
 Would
  probably be slower with hard disks.

 Or where site-packages is on NFS.

  But as the inspect module is already
  imported elsewhere, the python interpreter should also have it cached.

 Not on a normal import it's not.

  import numpy
  import sys
  sys.modules['inspect']
 Traceback (most recent call last):
   File stdin, line 1, in module
 KeyError: 'inspect'


 There are two lazy imports of inspect.



 You should feel free to remove whatever parts of `_inspect` are not
 being used and to move the parts that are closer to where they are
 used if you feel compelled to. Please do not replace the current uses
 of `_inspect` with `inspect`.


 It is used in just one place. Is importing inspect so much slower than
 all the other imports we do?


 Yes, please look at the thread I referred to. The custom inspect cut
 imports by 30 %, I doubt the ratio is much different today.

 David

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove numpy/compat/_inspect.py ?

[David Cournapeau]

On Sat, Aug 2, 2014 at 11:17 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:




 On Fri, Aug 1, 2014 at 8:01 PM, David Cournapeau courn...@gmail.com
 wrote:

 On my machine, if I use inspect instead of _inspect in
 numpy.compat.__init__, the import time increases ~ 25 % (from 82 ms to 99
 ms).

 So the hack certainly still make sense, one just need to fix whatever
 needs fixing (I am still not sure what's broken for the very specific
 usecase that code was bundled for).


 I'm not sure a one time hit of 17 ms is worth fighting for ;) The problems
 were that both the `string` and `dis` modules were used without importing
 them.


Don't fix what ain't broken ;)

The 17 ms is not what matters, the % is. People regularly complain about
import times, and 25 % increase in import time is significant (the above
timing are on my new macbook with SSD and 16 Gb RAM -- figures will easily
be 1 order of magnitude worse in common situations with slower computers,
slower HDD, NFS, etc...)

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove numpy/compat/_inspect.py ?

[David Cournapeau]

On Sat, Aug 2, 2014 at 11:36 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:




 On Fri, Aug 1, 2014 at 8:22 PM, David Cournapeau courn...@gmail.com
 wrote:




 On Sat, Aug 2, 2014 at 11:17 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:




 On Fri, Aug 1, 2014 at 8:01 PM, David Cournapeau courn...@gmail.com
 wrote:

 On my machine, if I use inspect instead of _inspect in
 numpy.compat.__init__, the import time increases ~ 25 % (from 82 ms to 99
 ms).

 So the hack certainly still make sense, one just need to fix whatever
 needs fixing (I am still not sure what's broken for the very specific
 usecase that code was bundled for).


 I'm not sure a one time hit of 17 ms is worth fighting for ;) The
 problems were that both the `string` and `dis` modules were used without
 importing them.


 Don't fix what ain't broken ;)

 The 17 ms is not what matters, the % is. People regularly complain about
 import times, and 25 % increase in import time is significant (the above
 timing are on my new macbook with SSD and 16 Gb RAM -- figures will easily
 be 1 order of magnitude worse in common situations with slower computers,
 slower HDD, NFS, etc...)


 Be interesting to compare times. Could you send along the code you used?
 My machine is similar except it is a desktop with 2 SSDs in raid 0.


I just hacked numpy.lib.__init__ to use inspect instead of _inspect:

diff --git a/numpy/compat/__init__.py b/numpy/compat/__init__.py
index 5b371f5..57f6d7f 100644
--- a/numpy/compat/__init__.py
+++ b/numpy/compat/__init__.py
@@ -10,11 +10,11 @@ extensions, which may be included for the following
reasons:
 
 from __future__ import division, absolute_import, print_function

-from . import _inspect
+import inspect as _inspect
 from . import py3k
-from ._inspect import getargspec, formatargspec
+from inspect import getargspec, formatargspec
 from .py3k import *

 __all__ = []
-__all__.extend(_inspect.__all__)
+__all__.extend([getargspec, formatargspec])
 __all__.extend(py3k.__all__)

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove numpy/compat/_inspect.py ?

[David Cournapeau]

The docstring at the beginning of the module is still relevant AFAIK: it
was about decreasing import times. See
http://mail.scipy.org/pipermail/numpy-discussion/2009-October/045981.html


On Fri, Aug 1, 2014 at 10:27 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Hi All,

 The _inspect.py function looks like a numpy version of the python inspect
 function. ISTR that is was a work around for problems with the early python
 versions, but that would have been back in 2009.

 Thoughts?

 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sun, Jul 6, 2014 at 2:24 AM, Julian Taylor jtaylor.deb...@googlemail.com
 wrote:

 On 05.07.2014 19:11, David Cournapeau wrote:
  On Sun, Jul 6, 2014 at 1:55 AM, Julian Taylor
  jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com
  wrote:
 
  On 05.07.2014 18:40, David Cournapeau wrote:
   The efforts are on average less demanding than this discussion. We
 are
   talking about adding entries to a list in most cases...
  
   Also, while adding the optimization support for bento, I've
  noticed that
   a lot of the related distutils code is broken, and does not work as
   expected on at least OS X + clang.
 
  It just spits out a lot of warnings but they are harmless.
 
 
  Adding lots of warnings are not harmless as they render the compiler
  warning system near useless (too many false alarms).
 

 true but until now we haven't received a single complaint nor fixes so
 probably not many developers are actually using macs/clang to work on
 numpy C code.
 But I do agree its bad and I have fixing that on my todo list, I didn't
 get around to it yet.


Here is an attempt: https://github.com/numpy/numpy/pull/4842

It uses a vile hack, but I did not see any other simple way. It fixes the
warnings on osx, once travis-ci confirms the tests pass ok on linux, I will
test it on msvc.

David



  I will fix the checks for both distutils and bento (using the autoconf
  macros setup, which should be more reliable than what we use for builtin
  and __attribute__-related checks)
 
  David
 
 
  We could remove them by using with -Werror=attribute for the
 conftests
  if it really bothers someone.
  Or do you mean something else?
 
  
   David
  
  
   On Sun, Jul 6, 2014 at 1:38 AM, Nathaniel Smith n...@pobox.com
  mailto:n...@pobox.com
   mailto:n...@pobox.com mailto:n...@pobox.com wrote:
  
   On Sat, Jul 5, 2014 at 3:21 PM, David Cournapeau
  courn...@gmail.com mailto:courn...@gmail.com
   mailto:courn...@gmail.com mailto:courn...@gmail.com
 wrote:
   
On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith
  n...@pobox.com mailto:n...@pobox.com
   mailto:n...@pobox.com mailto:n...@pobox.com wrote:
   
Maybe bento will revive and take over the new python
  packaging world!
Maybe not. Maybe something else will. I don't see how our
  support for
it will really affect these outcomes in any way. And I
  especially
don't see why it's important to spend time *now* on keeping
  bento
working, just in case it becomes useful *later*.
   
But it is working right now, so that argument is moot.
  
   My suggestion was that we should drop the rule that a patch
 has to
   keep bento working to be merged. We're talking about future
  breakages
   and future effort. The fact that it's working now doesn't say
  anything
   about whether it's worth continuing to invest time in it.
  
   --
   Nathaniel J. Smith
   Postdoctoral researcher - Informatics - University of Edinburgh
   http://vorpus.org
   ___
   NumPy-Discussion mailing list
   NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  mailto:NumPy-Discussion@scipy.org mailto:
 NumPy-Discussion@scipy.org
   http://mail.scipy.org/mailman/listinfo/numpy-discussion
  
  
  
  
   ___
   NumPy-Discussion mailing list
   NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
   http://mail.scipy.org/mailman/listinfo/numpy-discussion
  
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sat, Jul 5, 2014 at 11:25 AM, Charles R Harris charlesr.har...@gmail.com
 wrote:

 Ralf likes the speed of bento, but it is not currently maintained


What exactly is not maintained ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sat, Jul 5, 2014 at 5:23 PM, Ralf Gommers ralf.gomm...@gmail.com wrote:




 On Sat, Jul 5, 2014 at 10:13 AM, David Cournapeau courn...@gmail.com
 wrote:




 On Sat, Jul 5, 2014 at 11:25 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:

 Ralf likes the speed of bento, but it is not currently maintained


 What exactly is not maintained ?


 The issue is that Julian made some slightly nontrivial changes to
 core/setup.py and didn't want to update core/bscript. No one else has taken
 the time either to make those changes. That didn't bother me enough yet to
 go fix it, because they're all optional features and using Bento builds
 works just fine at the moment (and is part of the Travis CI test runs, so
 it'll keep working).


What are those changes ?

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith n...@pobox.com wrote:

 On Sat, Jul 5, 2014 at 2:32 PM, Ralf Gommers ralf.gomm...@gmail.com
 wrote:
 
  On Sat, Jul 5, 2014 at 1:54 PM, Nathaniel Smith n...@pobox.com wrote:
 
  On 5 Jul 2014 09:23, Ralf Gommers ralf.gomm...@gmail.com wrote:
  
   On Sat, Jul 5, 2014 at 10:13 AM, David Cournapeau courn...@gmail.com
 
   wrote:
  
   On Sat, Jul 5, 2014 at 11:25 AM, Charles R Harris
   charlesr.har...@gmail.com wrote:
  
   Ralf likes the speed of bento, but it is not currently maintained
  
  
   What exactly is not maintained ?
  
  
   The issue is that Julian made some slightly nontrivial changes to
   core/setup.py and didn't want to update core/bscript. No one else has
 taken
   the time either to make those changes. That didn't bother me enough
 yet to
   go fix it, because they're all optional features and using Bento
 builds
   works just fine at the moment (and is part of the Travis CI test
 runs, so
   it'll keep working).
 
  Perhaps a compromise would be to declare it officially unsupported and
  remove it from Travis CI, while leaving the files in place to be used
 on an
  at-your-own-risk basis? As long as it's in Travis, the default is that
  anyone who breaks it has to fix it. If it's not in Travis, then the
 default
  is that the people (person?) who use bento are responsible for keeping
 it
  working for their needs.
 
  -1 that just means that simple changes like adding a new extension will
 not
  get made before PRs get merged, and bento support will be in a broken
 state
  much more often.

 Yes, and then the handful of people who care about this would fix it
 or not. Your -1 is attempting to veto other people's *not* paying
 attention to this build system. I... don't think -1's work that way
 :-(

   I don't think the above is a good reason to remove Bento support. The
   much faster builds alone are a good reason to keep it. And the
 assertion
   that all numpy devs understand numpy.distutils is more than a little
   questionable:)
 
  They surely don't. But thousands of people use setup.py, and one or two
  use bento.
 
  I'm getting a little tired of these assertions. It's clear that David
 and I
  use it. A cursory search on Github reveals that Stefan, Fabian, Jonas and
  @aksarkar do (or did) as well:
 https://github.com/scipy/scipy/commit/74d823b3
 https://github.com/numpy/numpy/issues/2993
 https://github.com/numpy/numpy/pull/3606
 https://github.com/numpy/numpy/issues/3889
  For every user you can measure there's usually a number of users that you
  don't hear about.

 I apologize for forgetting before that you do use Bento, but these
 patches you're finding don't really change the overall picture. Let's
 assume that there are 100 people using Bento, who would be slightly
 inconvenienced if they had to use setup.py instead, or got stuck
 patching the bento build themselves to keep it working. 100 is
 probably an order of magnitude too high, but whatever. OTOH numpy has
 almost 7 million downloads on PyPI+sf.net, of which approximately
 every one used setup.py one way or another, plus all the people get it
 from alternative channels like distros, which also AFAIK universally
 use setup.py. Software development is all about trade-offs. Time that
 numpy developers spend messing about with bento to benefit those
 hundred users is time that could instead be spent on improvements that
 benefit many orders of magnitudes more users. Why do you want us to
 spend our time producing x units of value when we could instead be
 producing 100*x units of value for the same effort?

  Yet supporting both requires twice as much energy and attention as
  supporting just one.
 
  That's of course not true. For most changes the differences in where and
 how
  to update the build systems are small. Only for unusual changes like
 Julian
  patches to make use of optional GCC features, Bento and distutils may
  require very different changes.
 
  We've probably spent more person-hours talking about this, documenting
 the
  missing bscript bits, etc. than you've saved on those fast builds.
 
  Then maybe stop talking about it:)
 
  Besides the fast builds, which is only one example of why I like Bento
  better, there's also the fundamental question of what we do with build
 tools
  in the long term. It's clear that distutils is a dead end. All the PEPs
  related to packaging move in the direction of supporting tools like Bento
  better. If in the future we need significant new features in our build
 tool,
  Bento is a much better base to build on than numpy.distutils. It's
  unfortunate that at the moment there's no one that works on improving our
  build situation, but that is what it is. Removing Bento support is a
 step in
  the wrong direction imho.

 We must do something! This is something!

 Bento is pre-alpha software whose last upstream commit was in July
 2013. It's own CI tests have been failing since Feb. 2013, almost a
 year

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sat, Jul 5, 2014 at 11:51 PM, Charles R Harris charlesr.har...@gmail.com
 wrote:




 On Sat, Jul 5, 2014 at 8:28 AM, Matthew Brett matthew.br...@gmail.com
 wrote:

 On Sat, Jul 5, 2014 at 3:21 PM, David Cournapeau courn...@gmail.com
 wrote:
 
 
 
  On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith n...@pobox.com wrote:
 
  On Sat, Jul 5, 2014 at 2:32 PM, Ralf Gommers ralf.gomm...@gmail.com
  wrote:
  
   On Sat, Jul 5, 2014 at 1:54 PM, Nathaniel Smith n...@pobox.com
 wrote:
  
   On 5 Jul 2014 09:23, Ralf Gommers ralf.gomm...@gmail.com wrote:
   
On Sat, Jul 5, 2014 at 10:13 AM, David Cournapeau
courn...@gmail.com
wrote:
   
On Sat, Jul 5, 2014 at 11:25 AM, Charles R Harris
charlesr.har...@gmail.com wrote:
   
Ralf likes the speed of bento, but it is not currently
 maintained
   
   
What exactly is not maintained ?
   
   
The issue is that Julian made some slightly nontrivial changes to
core/setup.py and didn't want to update core/bscript. No one else
 has
taken
the time either to make those changes. That didn't bother me
 enough
yet to
go fix it, because they're all optional features and using Bento
builds
works just fine at the moment (and is part of the Travis CI test
runs, so
it'll keep working).
  
   Perhaps a compromise would be to declare it officially unsupported
 and
   remove it from Travis CI, while leaving the files in place to be
 used
   on an
   at-your-own-risk basis? As long as it's in Travis, the default is
 that
   anyone who breaks it has to fix it. If it's not in Travis, then the
   default
   is that the people (person?) who use bento are responsible for
 keeping
   it
   working for their needs.
  
   -1 that just means that simple changes like adding a new extension
 will
   not
   get made before PRs get merged, and bento support will be in a broken
   state
   much more often.
 
  Yes, and then the handful of people who care about this would fix it
  or not. Your -1 is attempting to veto other people's *not* paying
  attention to this build system. I... don't think -1's work that way
  :-(
 
I don't think the above is a good reason to remove Bento support.
 The
much faster builds alone are a good reason to keep it. And the
assertion
that all numpy devs understand numpy.distutils is more than a
 little
questionable:)
  
   They surely don't. But thousands of people use setup.py, and one or
 two
   use bento.
  
   I'm getting a little tired of these assertions. It's clear that David
   and I
   use it. A cursory search on Github reveals that Stefan, Fabian, Jonas
   and
   @aksarkar do (or did) as well:
  https://github.com/scipy/scipy/commit/74d823b3
  https://github.com/numpy/numpy/issues/2993
  https://github.com/numpy/numpy/pull/3606
  https://github.com/numpy/numpy/issues/3889
   For every user you can measure there's usually a number of users that
   you
   don't hear about.
 
  I apologize for forgetting before that you do use Bento, but these
  patches you're finding don't really change the overall picture. Let's
  assume that there are 100 people using Bento, who would be slightly
  inconvenienced if they had to use setup.py instead, or got stuck
  patching the bento build themselves to keep it working. 100 is
  probably an order of magnitude too high, but whatever. OTOH numpy has
  almost 7 million downloads on PyPI+sf.net, of which approximately
  every one used setup.py one way or another, plus all the people get it
  from alternative channels like distros, which also AFAIK universally
  use setup.py. Software development is all about trade-offs. Time that
  numpy developers spend messing about with bento to benefit those
  hundred users is time that could instead be spent on improvements that
  benefit many orders of magnitudes more users. Why do you want us to
  spend our time producing x units of value when we could instead be
  producing 100*x units of value for the same effort?
 
   Yet supporting both requires twice as much energy and attention as
   supporting just one.
  
   That's of course not true. For most changes the differences in where
 and
   how
   to update the build systems are small. Only for unusual changes like
   Julian
   patches to make use of optional GCC features, Bento and distutils may
   require very different changes.
  
   We've probably spent more person-hours talking about this,
 documenting
   the
   missing bscript bits, etc. than you've saved on those fast builds.
  
   Then maybe stop talking about it:)
  
   Besides the fast builds, which is only one example of why I like
 Bento
   better, there's also the fundamental question of what we do with
 build
   tools
   in the long term. It's clear that distutils is a dead end. All the
 PEPs
   related to packaging move in the direction of supporting tools like
   Bento
   better. If in the future we need significant new features in our
 build
   tool,
   Bento is a much better

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

The efforts are on average less demanding than this discussion. We are
talking about adding entries to a list in most cases...

Also, while adding the optimization support for bento, I've noticed that a
lot of the related distutils code is broken, and does not work as expected
on at least OS X + clang.

David


On Sun, Jul 6, 2014 at 1:38 AM, Nathaniel Smith n...@pobox.com wrote:

 On Sat, Jul 5, 2014 at 3:21 PM, David Cournapeau courn...@gmail.com
 wrote:
 
  On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith n...@pobox.com wrote:
 
  Maybe bento will revive and take over the new python packaging world!
  Maybe not. Maybe something else will. I don't see how our support for
  it will really affect these outcomes in any way. And I especially
  don't see why it's important to spend time *now* on keeping bento
  working, just in case it becomes useful *later*.
 
  But it is working right now, so that argument is moot.

 My suggestion was that we should drop the rule that a patch has to
 keep bento working to be merged. We're talking about future breakages
 and future effort. The fact that it's working now doesn't say anything
 about whether it's worth continuing to invest time in it.

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sun, Jul 6, 2014 at 1:55 AM, Julian Taylor jtaylor.deb...@googlemail.com
 wrote:

 On 05.07.2014 18:40, David Cournapeau wrote:
  The efforts are on average less demanding than this discussion. We are
  talking about adding entries to a list in most cases...
 
  Also, while adding the optimization support for bento, I've noticed that
  a lot of the related distutils code is broken, and does not work as
  expected on at least OS X + clang.

 It just spits out a lot of warnings but they are harmless.


Adding lots of warnings are not harmless as they render the compiler
warning system near useless (too many false alarms).

I will fix the checks for both distutils and bento (using the autoconf
macros setup, which should be more reliable than what we use for builtin
and __attribute__-related checks)

David


 We could remove them by using with -Werror=attribute for the conftests
 if it really bothers someone.
 Or do you mean something else?

 
  David
 
 
  On Sun, Jul 6, 2014 at 1:38 AM, Nathaniel Smith n...@pobox.com
  mailto:n...@pobox.com wrote:
 
  On Sat, Jul 5, 2014 at 3:21 PM, David Cournapeau courn...@gmail.com
  mailto:courn...@gmail.com wrote:
  
   On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith n...@pobox.com
  mailto:n...@pobox.com wrote:
  
   Maybe bento will revive and take over the new python packaging
 world!
   Maybe not. Maybe something else will. I don't see how our support
 for
   it will really affect these outcomes in any way. And I especially
   don't see why it's important to spend time *now* on keeping bento
   working, just in case it becomes useful *later*.
  
   But it is working right now, so that argument is moot.
 
  My suggestion was that we should drop the rule that a patch has to
  keep bento working to be merged. We're talking about future breakages
  and future effort. The fact that it's working now doesn't say
 anything
  about whether it's worth continuing to invest time in it.
 
  --
  Nathaniel J. Smith
  Postdoctoral researcher - Informatics - University of Edinburgh
  http://vorpus.org
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Remove bento from numpy

[David Cournapeau]

On Sun, Jul 6, 2014 at 2:24 AM, Julian Taylor jtaylor.deb...@googlemail.com
 wrote:

 On 05.07.2014 19:11, David Cournapeau wrote:
  On Sun, Jul 6, 2014 at 1:55 AM, Julian Taylor
  jtaylor.deb...@googlemail.com mailto:jtaylor.deb...@googlemail.com
  wrote:
 
  On 05.07.2014 18:40, David Cournapeau wrote:
   The efforts are on average less demanding than this discussion. We
 are
   talking about adding entries to a list in most cases...
  
   Also, while adding the optimization support for bento, I've
  noticed that
   a lot of the related distutils code is broken, and does not work as
   expected on at least OS X + clang.
 
  It just spits out a lot of warnings but they are harmless.
 
 
  Adding lots of warnings are not harmless as they render the compiler
  warning system near useless (too many false alarms).
 

 true but until now we haven't received a single complaint nor fixes so
 probably not many developers are actually using macs/clang to work on
 numpy C code.


Not many people are working on numpy C code period :)

FWIW, clang is now the standard OS X compiler since Maverick (Apple in all
its wisdom made gcc an alias to clang...).

David


 But I do agree its bad and I have fixing that on my todo list, I didn't
 get around to it yet.

  I will fix the checks for both distutils and bento (using the autoconf
  macros setup, which should be more reliable than what we use for builtin
  and __attribute__-related checks)
 
  David
 
 
  We could remove them by using with -Werror=attribute for the
 conftests
  if it really bothers someone.
  Or do you mean something else?
 
  
   David
  
  
   On Sun, Jul 6, 2014 at 1:38 AM, Nathaniel Smith n...@pobox.com
  mailto:n...@pobox.com
   mailto:n...@pobox.com mailto:n...@pobox.com wrote:
  
   On Sat, Jul 5, 2014 at 3:21 PM, David Cournapeau
  courn...@gmail.com mailto:courn...@gmail.com
   mailto:courn...@gmail.com mailto:courn...@gmail.com
 wrote:
   
On Sat, Jul 5, 2014 at 11:17 PM, Nathaniel Smith
  n...@pobox.com mailto:n...@pobox.com
   mailto:n...@pobox.com mailto:n...@pobox.com wrote:
   
Maybe bento will revive and take over the new python
  packaging world!
Maybe not. Maybe something else will. I don't see how our
  support for
it will really affect these outcomes in any way. And I
  especially
don't see why it's important to spend time *now* on keeping
  bento
working, just in case it becomes useful *later*.
   
But it is working right now, so that argument is moot.
  
   My suggestion was that we should drop the rule that a patch
 has to
   keep bento working to be merged. We're talking about future
  breakages
   and future effort. The fact that it's working now doesn't say
  anything
   about whether it's worth continuing to invest time in it.
  
   --
   Nathaniel J. Smith
   Postdoctoral researcher - Informatics - University of Edinburgh
   http://vorpus.org
   ___
   NumPy-Discussion mailing list
   NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  mailto:NumPy-Discussion@scipy.org mailto:
 NumPy-Discussion@scipy.org
   http://mail.scipy.org/mailman/listinfo/numpy-discussion
  
  
  
  
   ___
   NumPy-Discussion mailing list
   NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
   http://mail.scipy.org/mailman/listinfo/numpy-discussion
  
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org mailto:NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 
 
 
 
  ___
  NumPy-Discussion mailing list
  NumPy-Discussion@scipy.org
  http://mail.scipy.org/mailman/listinfo/numpy-discussion
 

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[David Cournapeau]

On Thu, Jun 5, 2014 at 11:41 PM, Nathaniel Smith n...@pobox.com wrote:

 On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris
 charlesr.har...@gmail.com wrote:
  @nathaniel IIRC, one of the objections to the missing values work was
 that
  it changed the underlying array object by adding a couple of variables to
  the structure. I'm willing to do that sort of thing, but it would be
 good to
  have general agreement that that is acceptable.

 I can't think of reason why adding new variables to the structure *per
 se* would be objectionable to anyone? IIRC the objection you're
 thinking of wasn't to the existence of new variables, but to their
 effect on compatibility: their semantics meant that every piece of
 legacy C code that worked with ndarrays had to be updated to check for
 the new variables before it could correctly interpret the -data
 field, and if it wasn't updated it would just return incorrect
 results. And there wasn't really a clear story for how we were going
 to detect and fix all this legacy code. This specific kind of
 compatibility break does seem pretty objectionable, but that's because
 of the details of its behaviour, not because variables in general are
 problematic, I think.

  As to blaze/dynd, I'd like to steal bits here and there, and maybe in the
  long term base numpy on top of it with a compatibility layer. There is a
 lot
  of thought and effort that has gone into those projects and we should use
  what we can. As is, I think numpy is good for another five to ten years
 and
  will probably hang on for fifteen, but it will be outdated by the end of
  that period. Like great whites, we need to keep swimming just to have
  oxygen. Software projects tend to be obligate ram ventilators.

 I worry a bit that this could become a self-fulfilling prophecy.
 Plenty of software survives longer than that; the Linux kernel hasn't
 had a real major number increase [1] since 2.6.0, more than 10 years
 ago, and it's still an extraordinarily vital project. Partly this is
 because they have resources we don't etc., but partly it's just
 because they've decided that incremental change is how they're going
 to do things, and approached each new feature with that in mind. And
 in ten years they haven't yet found any features that required a
 serious compatibility break.

 This is a pretty minor worry though -- we don't have to agree about
 what will happen in 10 years to agree about what to do now :-).

 [1] http://www.pcmag.com/article2/0,2817,2388926,00.asp

  The Python 3 experience is definitely something we want to avoid. And
 while
  blaze does big data and offers some nice features, I don't know that it
  offers compelling reasons to upgrade to the more ordinary user at this
 time,
  so I'd like to sort of slip it into numpy if possible.
 
  If we do start moving numpy forward in more radical steps, we should try
 to
  have some agreement beforehand as to what sort of changes are acceptable.
  For instance, to maintain backward compatibility, is it sufficient that a
  recompile will do the job, or do we require forward compatibility for
  extensions compiled against earlier releases?

 I find it hard to discuss these things in general, since specific
 compatibility issues usually involve complicated trade-offs -- will
 every package have to recompile or just some of them, if they don't
 will it be a nice error message or a segfault, is there some way we
 can issue warnings ahead of time for the offending behaviour, etc.
 etc.

 That said, my vote is that if there's a change that (a) can't be done
 some other way, (b) requires a recompile, (c) doesn't cause segfaults
 but rather produces some sensible error message like ABI mismatch
 please recompile, (d) is a change that's worth the bother (this
 determination to include at least canvassing the list to check that
 users in general agree that it's worth it), then yeah we should do it.
 I don't anticipate that this will happen very often given how far
 we've gotten without it, but yeah.


Changing the ABI 'safely' (i.e. raise a python exception if changed) is
already handled in numpy. We can always increase the ABI version if we
think it is worth it

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[David Cournapeau]

On Thu, Jun 5, 2014 at 11:48 PM, Nathaniel Smith n...@pobox.com wrote:

 On Thu, Jun 5, 2014 at 3:24 PM, David Cournapeau courn...@gmail.com
 wrote:
  On Thu, Jun 5, 2014 at 2:51 PM, Charles R Harris 
 charlesr.har...@gmail.com
  wrote:
  On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau courn...@gmail.com
  wrote:
  IMO, what is needed the most is refactoring the internal to extract the
  Python C API low level from the rest of the code, as I think that's
 the main
  bottleneck to get more contributors (or get new core features more
 quickly).
 
 
  What do you mean by extract the Python C API?
 
  Poor choice of words: I meant extracting the lower level part of
  array/ufunc/etc... from its wrapping into the python C API (with the idea
  that the latter could be done in Cython, modulo improvements in cython to
  manage the binary/code size explosion).
 
  IOW, split numpy into core and core-py (I think dynd benefits a lots from
  that, on top of its feature set).

 Can you give some examples of these benefits?


numpy.core is difficult to approach as a codebase: it is big, and quite
entangled. Why the concepts are sound, there is not much internal
architecture. I would love for numpy to have a proper internal C API. I'd
like to think my effort of splitting multiarray giant .c files into
multiple files somehow made everybody's like easier.

A lot of the current code is python C API, which nobody cares about, and
could be handled by e.g. cython (although again the feasibility of that
needs to be discussed with the cython team, as cython cannot
realisticallybe  used pervasively for numpy ATM, see e.g.
http://mail.scipy.org/pipermail/scipy-dev/2012-July/017717.html).

Such a separation would also be helpful for the pie-in-the-sky projects you
mentioned. I think Wes Mc Kinney and the pandas team experience about using
cython for the core vs just for the C-Python integration would be useful
there as well.

I'm kinda wary of
 refactoring-for-the-sake-of-it -- IME usually it's easier, more
 valuable, and more fun to refactor in the process of making some
 concrete improvement.


Sure, I am just suggesting there should be a conscious effort to not just
add features but also think about the internal consistency.

One concrete example is dtype and its pluggability: I would love to see
things like datetime in a separate extension. It would keep us honest to
allow people creating custom dtype (last time I've checked, there were some
hardcoding that made it hard to do).

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[David Cournapeau]

On Thu, Jun 5, 2014 at 9:44 AM, Todd toddr...@gmail.com wrote:


 On 5 Jun 2014 02:57, Nathaniel Smith n...@pobox.com wrote:
 
  On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant tra...@continuum.io
 wrote:
  And numpy will be much harder to replace than numeric --
  numeric wasn't the most-imported package in the pythonverse ;-).

 If numpy is really such a core part of  python ecosystem, does it really
 make sense to keep it as a stand-alone package?  Rather than thinking about
 a numpy 2, might it be better to be focusing on getting ndarray and dtype
 to a level of quality where acceptance upstream might be plausible?


There has been discussions about integrating numpy a long time ago (can't
find a reference right now), and the consensus was that this was possible
in its current shape nor advisable. The situation has not changed.

Putting something in the stdlib means it basically cannot change anymore:
API compatibility requirements would be stronger than what we provide even
now. NumPy is also a large codebase which would need some major clean up to
be accepted, etc...

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[David Cournapeau]

On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris charlesr.har...@gmail.com
wrote:




 On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant tra...@continuum.io
 wrote:

 Believe me, I'm all for incremental changes if it is actually possible
 and doesn't actually cost more.  It's also why I've been silent until now
 about anything we are doing being a candidate for a NumPy 2.0.  I
 understand the challenges of getting people to change.  But, features and
 solid improvements *will* get people to change --- especially if their new
 library can be used along with the old library and the transition can be
 done gradually. Python 3's struggle is the lack of features.

 At some point there *will* be a NumPy 2.0.   What features go into NumPy
 2.0, how much backward compatibility is provided, and how much porting is
 needed to move your code from NumPy 1.X to NumPy 2.X is the real user
 question --- not whether it is characterized as incremental change or
 re-write. What I call a re-write and what you call an
 incremental-change are two points on a spectrum and likely overlap
 signficantly if we really compared what we are thinking about.

 One huge benefit that came out of the numeric / numarray / numpy
 transition that we mustn't forget about was actually the extended buffer
 protocol and memory view objects.  This really does allow multiple array
 objects to co-exist and libraries to use the object that they prefer in a
 way that did not exist when Numarray / numeric / numpy came out.So, we
 shouldn't be afraid of that world.   The existence of easy package managers
 to update environments to try out new features and have applications on a
 single system that use multiple versions of the same library is also
 something that didn't exist before and that will make any transition easier
 for users.

 One thing I regret about my working on NumPy originally is that I didn't
 have the foresight, skill, and understanding to work more on a more
 extended and better designed multiple-dispatch system so that multiple
 array objects could participate together in an expression flow.   The
 __numpy_ufunc__ mechanism gives enough capability in that direction that it
 may be better now.

 Ultimately, I don't disagree that NumPy can continue to exist in
 incremental change mode ( though if you are swapping out whole swaths of
 C-code for Cython code --- it sounds a lot like a re-write) as long as
 there are people willing to put the effort into changing it.   I think this
 is actually benefited by the existence of other array objects that are
 pushing the feature envelope without the constraints --- in much the same
 way that the Python standard library is benefitted by many versions of
 different capabilities being tried out before moving into the standard
 library.

 I remain optimistic that things will continue to improve in multiple ways
 --- if a little messier than any of us would conceive individually.   It
 *is* great to see all the PR's coming from multiple people on NumPy and all
 the new energy around improving things whether great or small.


 @nathaniel IIRC, one of the objections to the missing values work was that
 it changed the underlying array object by adding a couple of variables to
 the structure. I'm willing to do that sort of thing, but it would be good
 to have general agreement that that is acceptable.



I think changing the ABI for some versions of numpy (2.0 , whatever) is
acceptable. There is little doubt that the ABI will need to change to
accommodate a better and more flexible architecture.

Changing the C API is more tricky: I am not up to date to the changes from
the last 2-3 years, but at that time, most things could have been changed
internally without breaking much, though I did not go far enough to
estimate what the performance impact could be (if any).



 As to blaze/dynd, I'd like to steal bits here and there, and maybe in the
 long term base numpy on top of it with a compatibility layer. There is a
 lot of thought and effort that has gone into those projects and we should
 use what we can. As is, I think numpy is good for another five to ten years
 and will probably hang on for fifteen, but it will be outdated by the end
 of that period. Like great whites, we need to keep swimming just to have
 oxygen. Software projects tend to be obligate ram ventilators.

 The Python 3 experience is definitely something we want to avoid. And
 while blaze does big data and offers some nice features, I don't know that
 it offers compelling reasons to upgrade to the more ordinary user at this
 time, so I'd like to sort of slip it into numpy if possible.

 If we do start moving numpy forward in more radical steps, we should try
 to have some agreement beforehand as to what sort of changes are
 acceptable. For instance, to maintain backward compatibility, is it
 sufficient that a recompile will do the job, or do we require forward
 compatibility for extensions compiled against earlier releases? Do we 

Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)

[David Cournapeau]

On Thu, Jun 5, 2014 at 2:51 PM, Charles R Harris charlesr.har...@gmail.com
wrote:




 On Thu, Jun 5, 2014 at 6:40 AM, David Cournapeau courn...@gmail.com
 wrote:




 On Thu, Jun 5, 2014 at 3:36 AM, Charles R Harris 
 charlesr.har...@gmail.com wrote:




 On Wed, Jun 4, 2014 at 7:29 PM, Travis Oliphant tra...@continuum.io
 wrote:

 Believe me, I'm all for incremental changes if it is actually possible
 and doesn't actually cost more.  It's also why I've been silent until now
 about anything we are doing being a candidate for a NumPy 2.0.  I
 understand the challenges of getting people to change.  But, features and
 solid improvements *will* get people to change --- especially if their new
 library can be used along with the old library and the transition can be
 done gradually. Python 3's struggle is the lack of features.

 At some point there *will* be a NumPy 2.0.   What features go into
 NumPy 2.0, how much backward compatibility is provided, and how much
 porting is needed to move your code from NumPy 1.X to NumPy 2.X is the real
 user question --- not whether it is characterized as incremental change
 or re-write. What I call a re-write and what you call an
 incremental-change are two points on a spectrum and likely overlap
 signficantly if we really compared what we are thinking about.

 One huge benefit that came out of the numeric / numarray / numpy
 transition that we mustn't forget about was actually the extended buffer
 protocol and memory view objects.  This really does allow multiple array
 objects to co-exist and libraries to use the object that they prefer in a
 way that did not exist when Numarray / numeric / numpy came out.So, we
 shouldn't be afraid of that world.   The existence of easy package managers
 to update environments to try out new features and have applications on a
 single system that use multiple versions of the same library is also
 something that didn't exist before and that will make any transition easier
 for users.

 One thing I regret about my working on NumPy originally is that I
 didn't have the foresight, skill, and understanding to work more on a more
 extended and better designed multiple-dispatch system so that multiple
 array objects could participate together in an expression flow.   The
 __numpy_ufunc__ mechanism gives enough capability in that direction that it
 may be better now.

 Ultimately, I don't disagree that NumPy can continue to exist in
 incremental change mode ( though if you are swapping out whole swaths of
 C-code for Cython code --- it sounds a lot like a re-write) as long as
 there are people willing to put the effort into changing it.   I think this
 is actually benefited by the existence of other array objects that are
 pushing the feature envelope without the constraints --- in much the same
 way that the Python standard library is benefitted by many versions of
 different capabilities being tried out before moving into the standard
 library.

 I remain optimistic that things will continue to improve in multiple
 ways --- if a little messier than any of us would conceive individually.
   It *is* great to see all the PR's coming from multiple people on NumPy
 and all the new energy around improving things whether great or small.


 @nathaniel IIRC, one of the objections to the missing values work was
 that it changed the underlying array object by adding a couple of variables
 to the structure. I'm willing to do that sort of thing, but it would be
 good to have general agreement that that is acceptable.



 I think changing the ABI for some versions of numpy (2.0 , whatever) is
 acceptable. There is little doubt that the ABI will need to change to
 accommodate a better and more flexible architecture.

 Changing the C API is more tricky: I am not up to date to the changes
 from the last 2-3 years, but at that time, most things could have been
 changed internally without breaking much, though I did not go far enough to
 estimate what the performance impact could be (if any).



 As to blaze/dynd, I'd like to steal bits here and there, and maybe in
 the long term base numpy on top of it with a compatibility layer. There is
 a lot of thought and effort that has gone into those projects and we should
 use what we can. As is, I think numpy is good for another five to ten years
 and will probably hang on for fifteen, but it will be outdated by the end
 of that period. Like great whites, we need to keep swimming just to have
 oxygen. Software projects tend to be obligate ram ventilators.

 The Python 3 experience is definitely something we want to avoid. And
 while blaze does big data and offers some nice features, I don't know that
 it offers compelling reasons to upgrade to the more ordinary user at this
 time, so I'd like to sort of slip it into numpy if possible.

 If we do start moving numpy forward in more radical steps, we should try
 to have some agreement beforehand as to what sort of changes are
 acceptable. For instance

Re: [Numpy-discussion] SciPy 2014 BoF NumPy Participation

[David Cournapeau]

I won't be able to make it at scipy this year sadly.

I concur with Nathaniel that we can do a lot of things without a full
rewrite -- it is all too easy to see what is gained with a rewrite and lose
sight of what is lost. I have yet to see a really strong argument for a
full rewrite. It may be easier to do a rewrite for a core when you have a
few full-time people, but that's a different story for a community effort
like numpy.

The main issue preventing new features in numpy is the lack of internal
architecture at the C level, but nothing that could not be done by
refactoring. Using cython to move away from the python C api would be
great, though we need to talk with the cython people so that we can share
common code between multiple extensions using cython, to avoid binary size
explosion.

There are things that may require some backward incompatible changes in the
C API, but that's much more acceptable than a significant break at the
python level.

David


On Wed, Jun 4, 2014 at 9:58 AM, Sebastian Berg sebast...@sipsolutions.net
wrote:

 On Mi, 2014-06-04 at 02:26 +0100, Nathaniel Smith wrote:
  On Wed, Jun 4, 2014 at 12:33 AM, Charles R Harris
  charlesr.har...@gmail.com wrote:
   On Tue, Jun 3, 2014 at 5:08 PM, Kyle Mandli kyle.man...@gmail.com
 wrote:
  
   Hello everyone,
  
   As one of the co-chairs in charge of organizing the birds-of-a-feather
   sesssions at the SciPy conference this year, I wanted to solicit
 through the
   NumPy list to see if we could get enough interest to hold a NumPy
 centered
   BoF this year.  The BoF format would be up to those who would lead the
   discussion, a couple of ideas used in the past include picking out a
 few of
   the lead devs to be on a panel and have a QA type of session or an
 open QA
   with perhaps audience guided list of topics.  I can help facilitate
   organization of something but we would really like to get something
   organized this year (last year NumPy was the only major project that
 was not
   really represented in the BoF sessions).
  
   I'll be at the conference, but I don't know who else will be there. I
 feel
   that NumPy has matured to the point where most of the current work is
   cleaning stuff up, making it run faster, and fixing bugs. A topic that
 I'd
   like to see discussed is where do we go from here. One option to look
 at is
   Blaze, which looks to have matured a lot in the last year. The problem
 with
   making it a NumPy replacement is that NumPy has become quite
 widespread,
   with downloads from PyPi running at about 3 million per year. With
 that much
   penetration it may be difficult for a new core like Blaze to gain
 traction.
   So I'd like to also discuss ways to bring the two branches of
 development
   together at some point and explore what NumPy can do to pave the way.
 Mind,
   there are definitely things that would be nice to add to NumPy, a
 better
   type system, missing values, etc., but doing that is difficult given
 the
   current design.
 
  I won't be at the conference unfortunately (I'm on the wrong continent
  and have family commitments then anyway), but I think there's lots of
  exciting stuff that can be done in numpy-land.
 

 I wouldn't like to come, but to be honest have not planned to yet and it
 doesn't fit too well with the stuff I work on mostly right now. So will
 have to see.

 - Sebastian

  We absolutely could rewrite the dtype system, and this would
  straightforwardly give us excellent support for missing values, units,
  categorical data, automatic differentiation, better datetimes, etc.
  etc. -- and make numpy much more friendly in general to third-party
  extensions.
 
  I'd like to see the ufunc system revisited in the light of all the
  things we know now, to make gufuncs more first-class, provide better
  support for user-defined types, more flexible loop selection (e.g.
  make it possible to implement np.add.reduce(a, type=kahan)), etc.;
  one goal would be to convert a lot of ufunc-like functions (np.mean
  etc.) into being real ufuncs, and then they'd automatically benefit
  from __numpy_ufunc__, which would also massively improve
  interoperability with alternative array types like blaze.
 
  I'd like to see support for extensible label-based indexing, like pandas.
 
  Internally, I'd like to see internal migrating out of C and into
  Cython -- we have hundreds of lines of code that could be replaced
  with a few lines of Cython and no-one would notice. (Combining this
  with a cffi cython backend and pypy would be pretty interesting
  too...)
 
  I'd like to see sparse ndarrays, with integration into the ufunc
  looping machinery so all ufuncs just work. Or even better, I'd like to
  see the right hooks added so that anyone can write a sparse ndarray
  package using only public APIs, and have all ufuncs just work. (I was
  going to put down deferred/loop-fused/out-of-core computation as a
  wishlist item too, but if we do it right then this too could be
  implemented 

Re: [Numpy-discussion] fftw supported?

[David Cournapeau]

FFTW is not used anymore in neither numpy or scipy (has not been for
several years). If you want to use fftw with numpy, there are 3rd party
extensions to do it, like pyfftw


On Mon, Jun 2, 2014 at 12:27 PM, Neal Becker ndbeck...@gmail.com wrote:

 I just d/l numpy-1.8.1 and try to build.  I uncomment:

 [fftw]
 libraries = fftw3

 This is fedora 20.  fftw3 (and devel) is installed as fftw.

 I see nothing written to stderr during the build that has any reference to
 fftw.

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing

[David Cournapeau]

On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.comwrote:

 Hi,

 On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau courn...@gmail.com
 wrote:
 
 
 
  On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.com
 
  wrote:
 
  Aha,
 
  On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com
 
  wrote:
   Hi,
  
   On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com
   wrote:
   A possible option is to install the toolchain inside site-packages
 and
   to
   deploy it as PYPI wheel or wininst packages. The PATH to the
 toolchain
   could
   be extended during import of the package. But I have no idea, whats
 the
   best
   strategy to additionaly install ATLAS or other third party libraries.
  
   Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the
   devkit package.  It sounds like OpenBLAS will be much easier to build,
   so we could start with ATLAS binaries as a default, expecting OpenBLAS
   to be built more often with the toolchain.  I think that's how numpy
   binary installers are built at the moment - using old binary builds of
   ATLAS.
  
   I'm happy to provide the builds of ATLAS - e.g. here:
  
   https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds
 
  I just found the official numpy binary builds of ATLAS:
 
  https://github.com/numpy/vendor/tree/master/binaries
 
  But - they are from an old version of ATLAS / Lapack, and only for
 32-bit.
 
  David - what say we update these to latest ATLAS stable?
 
 
  Fine by me (not that you need my approval !).
 
  How easy is it to build ATLAS targetting a specific CPU these days ? I
 think
  we need to at least support nosse and sse2 and above.

 I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I
 think Clint will have some time to help out next week.

 I did some analysis of SSE2 prevalence here:

 https://github.com/numpy/numpy/wiki/Window-versions

 Firefox crash reports now have about 1 percent of machines without
 SSE2.  I suspect that people running new installs of numpy will have
 slightly better machines on average than Firefox users, but it's only
 a guess.

 I wonder if we could add a CPU check on numpy import to give a polite
 'install from the exe' message for people without SSE2.


We could, although you unfortunately can't do it easily from ctypes only
(as you need some ASM).

I can take a quick look at a simple cython extension that could be imported
before anything else, and would raise an ImportError if the wrong arch is
detected.

David


 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing

[David Cournapeau]

On Fri, May 9, 2014 at 11:49 AM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 On 09.05.2014 12:42, David Cournapeau wrote:
 
 
 
  On Fri, May 9, 2014 at 1:51 AM, Matthew Brett matthew.br...@gmail.com
  mailto:matthew.br...@gmail.com wrote:
 
  Hi,
 
  On Mon, Apr 28, 2014 at 3:29 PM, David Cournapeau
  courn...@gmail.com mailto:courn...@gmail.com wrote:
  
  
  
   On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett
  matthew.br...@gmail.com mailto:matthew.br...@gmail.com
   wrote:
  
   Aha,
  
   On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett
  matthew.br...@gmail.com mailto:matthew.br...@gmail.com
   wrote:
Hi,
   
On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner
  cmkleff...@gmail.com mailto:cmkleff...@gmail.com
wrote:
A possible option is to install the toolchain inside
  site-packages and
to
deploy it as PYPI wheel or wininst packages. The PATH to the
  toolchain
could
be extended during import of the package. But I have no idea,
  whats the
best
strategy to additionaly install ATLAS or other third party
  libraries.
   
Maybe we could provide ATLAS binaries for 32 / 64 bit as part
  of the
devkit package.  It sounds like OpenBLAS will be much easier to
  build,
so we could start with ATLAS binaries as a default, expecting
  OpenBLAS
to be built more often with the toolchain.  I think that's how
  numpy
binary installers are built at the moment - using old binary
  builds of
ATLAS.
   
I'm happy to provide the builds of ATLAS - e.g. here:
   
https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds
  
   I just found the official numpy binary builds of ATLAS:
  
   https://github.com/numpy/vendor/tree/master/binaries
  
   But - they are from an old version of ATLAS / Lapack, and only
  for 32-bit.
  
   David - what say we update these to latest ATLAS stable?
  
  
   Fine by me (not that you need my approval !).
  
   How easy is it to build ATLAS targetting a specific CPU these days
  ? I think
   we need to at least support nosse and sse2 and above.
 
  I'm getting crashes trying to build SSE2-only ATLAS on 32-bits, I
  think Clint will have some time to help out next week.
 
  I did some analysis of SSE2 prevalence here:
 
  https://github.com/numpy/numpy/wiki/Window-versions
 
  Firefox crash reports now have about 1 percent of machines without
  SSE2.  I suspect that people running new installs of numpy will have
  slightly better machines on average than Firefox users, but it's only
  a guess.
 
  I wonder if we could add a CPU check on numpy import to give a polite
  'install from the exe' message for people without SSE2.
 
 
  We could, although you unfortunately can't do it easily from ctypes only
  (as you need some ASM).
 
  I can take a quick look at a simple cython extension that could be
  imported before anything else, and would raise an ImportError if the
  wrong arch is detected.
 

 assuming mingw is new enough

 #ifdef __SSE2___
 raise_if(!__builtin_cpu_supports(sse))
 #endof


We need to support it for VS as well, but it looks like win32 API has a
function to do it:
http://msdn.microsoft.com/en-us/library/ms724482%28VS.85%29.aspx

Makes it even easier.

David


 in import_array() should do it
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 64-bit windows numpy / scipy wheels for testing

[David Cournapeau]

On Sun, Apr 27, 2014 at 11:50 PM, Matthew Brett matthew.br...@gmail.comwrote:

 Aha,

 On Sun, Apr 27, 2014 at 3:19 PM, Matthew Brett matthew.br...@gmail.com
 wrote:
  Hi,
 
  On Sun, Apr 27, 2014 at 3:06 PM, Carl Kleffner cmkleff...@gmail.com
 wrote:
  A possible option is to install the toolchain inside site-packages and
 to
  deploy it as PYPI wheel or wininst packages. The PATH to the toolchain
 could
  be extended during import of the package. But I have no idea, whats the
 best
  strategy to additionaly install ATLAS or other third party libraries.
 
  Maybe we could provide ATLAS binaries for 32 / 64 bit as part of the
  devkit package.  It sounds like OpenBLAS will be much easier to build,
  so we could start with ATLAS binaries as a default, expecting OpenBLAS
  to be built more often with the toolchain.  I think that's how numpy
  binary installers are built at the moment - using old binary builds of
  ATLAS.
 
  I'm happy to provide the builds of ATLAS - e.g. here:
 
  https://nipy.bic.berkeley.edu/scipy_installers/atlas_builds

 I just found the official numpy binary builds of ATLAS:

 https://github.com/numpy/vendor/tree/master/binaries

 But - they are from an old version of ATLAS / Lapack, and only for 32-bit.

 David - what say we update these to latest ATLAS stable?


Fine by me (not that you need my approval !).

How easy is it to build ATLAS targetting a specific CPU these days ? I
think we need to at least support nosse and sse2 and above.

David


 Cheers,

 Matthew
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Slightly off-topic - accuracy of C exp function?

[David Cournapeau]

On Wed, Apr 23, 2014 at 9:43 AM, Nathaniel Smith n...@pobox.com wrote:

 On Wed, Apr 23, 2014 at 6:22 AM, Matthew Brett matthew.br...@gmail.com
 wrote:
  Hi,
 
  I'm exploring Mingw-w64 for numpy building, and I've found it gives a
  slightly different answer for 'exp' than - say - gcc on OSX.
 
  The difference is of the order of the eps value for the output number
  (2 * eps for a result of ~2.0).
 
  Is accuracy somewhere specified for C functions like exp?  Or is
  accuracy left as an implementation detail for the C library author?

 C99 says (sec 5.2.4.2.2) that The accuracy of the floating point
 operations ... and of the library functions in math.h and
 complex.h that return floating point results is implemenetation
 defined. The implementation may state that the accuracy is unknown.
 (This last sentence is basically saying that with regard to some
 higher up clauses that required all conforming implementations to
 document this stuff, saying eh, who knows counts as documenting it.
 Hooray for standards!)

 Presumably the accuracy in this case is a function of the C library
 anyway, not the compiler? Numpy has its own implementations for a
 bunch of the math functions, and it's been unclear in the past whether
 numpy or the libc implementations were better in any particular case.


In the case of MS runtime, at least 9 (as shipped in VS 2008), our
implementation is likely to be better (most of the code was taken from the
sun math library when the license allowed it).

David


 -n

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Test error with ATLAS, Windows 64 bit

[David Cournapeau]

On Mon, Apr 14, 2014 at 10:02 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 The official numpy mingw binaries do not have all these math issues.
 Only the VC builds do.
 As mingw is fine the functions must be somewhere in the windows API but
 no-one has contributed a fix for the VC builds to numpy yet.


Mingw generally has their own implementation, and do not use the C runtime
from MS for math functions:
https://github.com/mirror/mingw-w64/tree/master/trunk/mingw-w64-crt/math

David
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.8.1 release

[David Cournapeau]

On Wed, Apr 2, 2014 at 7:52 AM, Matthew Brett matthew.br...@gmail.comwrote:

 Hi,

 On Tue, Apr 1, 2014 at 4:46 PM, David Cournapeau courn...@gmail.com
 wrote:
 
 
 
  On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith n...@pobox.com wrote:
 
  On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau courn...@gmail.com
  wrote:
   On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith n...@pobox.com
 wrote:
  
   On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett 
 matthew.br...@gmail.com
   wrote:
I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a
 DLL
loaded via:
   
hDLL = LoadLibraryEx(pathname, NULL,
  LOAD_WITH_ALTERED_SEARCH_PATH);
   
will in turn (by default) search for its dependent DLLs in their
 own
directory.Or maybe in the directory of the first DLL to be
 loaded
with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the
documentation.  Looking forward to doing my tax return after this.
   
But - anyway - that means that any extensions in the DLLs directory
will get their dependencies from the DLLs directory, but that is
 only
true for extensions in that directory.
  
   So in conclusion, if we just drop our compiled dependencies next to
   the compiled module files then we're good, even on older Windows
   versions? That sounds much simpler than previous discussions, but
 good
   news if it's true...
  
  
   That does not work very well in my experience:
  
 - numpy has extension modules in multiple directories, so we would
   need to
   copy the dlls in multiple subdirectories
 - copying dlls means that windows will load that dll multiple times,
   with
   all the ensuing problems (I don't know for MKL/OpenBlas, but we've
 seen
   serious issues when doing something similar for hdf5 dll and
   pytables/h5py).
 
  We could just ship all numpy's extension modules in the same directory
  if we wanted. It would be pretty easy to stick some code at the top of
  numpy/__init__.py to load them from numpy/all_dlls/ and then slot them
  into the appropriate places in the package namespace.
 
  Of course scipy and numpy will still both have to ship BLAS etc., and
  so I guess it will get loaded at least twice in *any* binary install
  system. I'm not sure why this would be a problem (Windows, unlike
  Unix, carefully separates DLL namespaces, right?)
 
 
  It does not really matter here. For pure blas/lapack, that may be ok
 because
  the functions are stateless, but I would not count on it either.
 
  The cleanest solution I can think of is to have 'privately shared DLL',
 but
  that would AFAIK require patching python, so not really an option.

 David - do you know anything about private assemblies [1]?


I never managed to make that work properly.


 Might they work for our problem? How about AddDllDirectory [2]?


I don't think it is appropriate to use those functions in a C extension
module (as it impacts the whole process).

David


 Cheers,

 Matthew

 [1]
 http://msdn.microsoft.com/en-us/library/windows/desktop/ff951638(v=vs.85).aspx
 [2]
 http://msdn.microsoft.com/en-us/library/windows/desktop/hh310513(v=vs.85).asp
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.8.1 release

[David Cournapeau]

On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith n...@pobox.com wrote:

 On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett matthew.br...@gmail.com
 wrote:
  I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL
 loaded via:
 
  hDLL = LoadLibraryEx(pathname, NULL,  LOAD_WITH_ALTERED_SEARCH_PATH);
 
  will in turn (by default) search for its dependent DLLs in their own
  directory.Or maybe in the directory of the first DLL to be loaded
  with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the
  documentation.  Looking forward to doing my tax return after this.
 
  But - anyway - that means that any extensions in the DLLs directory
  will get their dependencies from the DLLs directory, but that is only
  true for extensions in that directory.

 So in conclusion, if we just drop our compiled dependencies next to
 the compiled module files then we're good, even on older Windows
 versions? That sounds much simpler than previous discussions, but good
 news if it's true...


That does not work very well in my experience:

  - numpy has extension modules in multiple directories, so we would need
to copy the dlls in multiple subdirectories
  - copying dlls means that windows will load that dll multiple times, with
all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen
serious issues when doing something similar for hdf5 dll and pytables/h5py).

David


 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: NumPy 1.8.1 release

[David Cournapeau]

On Wed, Apr 2, 2014 at 12:36 AM, Nathaniel Smith n...@pobox.com wrote:

 On Tue, Apr 1, 2014 at 11:58 PM, David Cournapeau courn...@gmail.com
 wrote:
  On Tue, Apr 1, 2014 at 6:43 PM, Nathaniel Smith n...@pobox.com wrote:
 
  On Tue, Apr 1, 2014 at 6:26 PM, Matthew Brett matthew.br...@gmail.com
  wrote:
   I'm guessing that the LOAD_WITH_ALTERED_SEARCH_PATH means that a DLL
   loaded via:
  
   hDLL = LoadLibraryEx(pathname, NULL,  LOAD_WITH_ALTERED_SEARCH_PATH);
  
   will in turn (by default) search for its dependent DLLs in their own
   directory.Or maybe in the directory of the first DLL to be loaded
   with LOAD_WITH_ALTERED_SEARCH_PATH, damned if I can follow the
   documentation.  Looking forward to doing my tax return after this.
  
   But - anyway - that means that any extensions in the DLLs directory
   will get their dependencies from the DLLs directory, but that is only
   true for extensions in that directory.
 
  So in conclusion, if we just drop our compiled dependencies next to
  the compiled module files then we're good, even on older Windows
  versions? That sounds much simpler than previous discussions, but good
  news if it's true...
 
 
  That does not work very well in my experience:
 
- numpy has extension modules in multiple directories, so we would
 need to
  copy the dlls in multiple subdirectories
- copying dlls means that windows will load that dll multiple times,
 with
  all the ensuing problems (I don't know for MKL/OpenBlas, but we've seen
  serious issues when doing something similar for hdf5 dll and
 pytables/h5py).

 We could just ship all numpy's extension modules in the same directory
 if we wanted. It would be pretty easy to stick some code at the top of
 numpy/__init__.py to load them from numpy/all_dlls/ and then slot them
 into the appropriate places in the package namespace.

 Of course scipy and numpy will still both have to ship BLAS etc., and
 so I guess it will get loaded at least twice in *any* binary install
 system. I'm not sure why this would be a problem (Windows, unlike
 Unix, carefully separates DLL namespaces, right?)


It does not really matter here. For pure blas/lapack, that may be ok
because the functions are stateless, but I would not count on it either.

The cleanest solution I can think of is to have 'privately shared DLL', but
that would AFAIK require patching python, so not really an option.


, but if it is a
 problem then it's a very fundamental one for any binaries we ship.

 Do the binaries we ship now have this problem? Or are we currently
 managing to statically link everything?


We currently statically link everything. The main challenge is that 'new'
(= 4) versions of mingw don't easily allow statically linking all the
mingw-related dependencies. While the options are there, everytime I tried
to do it with an official build of mingw, I had some weird, very hard to
track crashes. The other alternative that has been suggested is to build
one own's toolchain where everything is static by default. I am not sure
why that works, and that brings the risk of depending on a toolchain that
we can't really maintain.

David


 -n

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] f2py links extensions to incorrect python installation on OSX / Anaconda

[David Cournapeau]

On Thu, Mar 27, 2014 at 8:30 PM, Alex Goodman alex.good...@colostate.eduwrote:

 Hi all,

 I have used f2py in the past on a Linux machine with virtually no issues.
 However on my Mac, I get the following error when importing an f2py
 generated extension:

 Fatal Python error: PyThreadState_Get: no current thread
 Abort trap: 6

 After doing some research I found out that the extension is linked to the
 wrong python installation:
 otool -L add.so
 add.so:
 ./add.so (compatibility version 0.0.0, current version 0.0.0)
 /System/Library/Frameworks/Python.framework/Versions/2.7/Python
 (compatibility version 2.7.0, current version 2.7.2)
  /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
 169.3.0)

 This seems odd because I am using the f2py executable included in Anaconda
 1.9.1. I can easily fix this problem by manually using install_name_tool
 -change on the extension to link the correct library location, but this is
 really cumbersome. Is there an alternative solution, such as an additional
 command-line argument when invoking f2py?


This sounds like an issue specific to Anaconda, and you may get better
support on the Anaconda support ML.

David


 For what it is worth, I am also using Version 14.0.2 of the Intel Fortran
 Compiler.

 Thanks,
 Alex
 --
 Alex Goodman
 Graduate Research Assistant
 Department of Atmospheric Science
 Colorado State University

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy gsoc ideas (was: numpy gsoc topic idea: configurable algorithm precision and vector math library integration)

[David Cournapeau]

On Thu, Mar 6, 2014 at 1:59 PM, Nathaniel Smith n...@pobox.com wrote:

 On Thu, Mar 6, 2014 at 9:11 AM, David Cournapeau courn...@gmail.com
 wrote:
 
  On Wed, Mar 5, 2014 at 9:11 PM, Nathaniel Smith n...@pobox.com wrote:
  So this project would have the following goals, depending on how
  practical this turns out to be: (1) produce a hacky proof-of-concept
  system for doing the above, (2) turn the hacky proof-of-concept into
  something actually viable for use in real life (possibly this would
  require getting changes upstream into Cython, etc.), (3) use this
  system to actually port some interesting numpy code into cython.
 
 
  Having to synchronise two projects may be hard for a GSoC, no ?

 Yeah, if someone is interested in this it would be nice to get someone
 from Cython involved too. But that's why the primary goal is to
 produce a proof-of-concept -- even if all that comes out is that we
 learn that this cannot be done in an acceptable manner, then that's
 still a succesful (albeit disappointing) result.

  Otherwise, I am a bit worried about cython being used on the current C
 code
  as is, because core and python C API are so interwined (especially
  multiarray).

 I don't understand this objection. The whole advantage of Cython is
 that it makes it much, much easier to write code that involves
 intertwining complex algorithms and heavy use of the Python C API :-).

There's tons of bug-prone spaghetti in numpy for doing boring things
 like refcounting, exception passing, and argument parsing.


No argument there, doing refcounting, etc.. manually is a waste of time.

Ideally, cython would be used for the boring stuff, and we keep C for the
low-level machinery, but the current code don't cleanly separate those two
layers (there is simple C API for indexing, ufunc, etc...). I am concerned
about cython making that difference even more blurry.

David




 -n

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] numpy gsoc ideas (was: numpy gsoc topic idea: configurable algorithm precision and vector math library integration)

[David Cournapeau]

On Wed, Mar 5, 2014 at 9:11 PM, Nathaniel Smith n...@pobox.com wrote:

 On Mon, Mar 3, 2014 at 7:20 PM, Julian Taylor
 jtaylor.deb...@googlemail.com wrote:
  hi,
 
  as the numpy gsoc topic page is a little short on options I was thinking
  about adding two topics for interested students. But as I have no
  experience with gsoc or mentoring and the ideas are not very fleshed out
  yet I'd like to ask if it might make sense at all:
 
  1. configurable algorithm precision
 [...]
  with np.precmode(default=fast):
np.abs(complex_array)
 
  or fast everything except sum and hypot
 
  with np.precmode(default=fast, sum=kahan, hypot=standard):
np.sum(d)
 [...]

 Not a big fan of this one -- it seems like the biggest bulk of the
 effort would be in figuring out a non-horrible API for exposing these
 things and getting consensus around it, which is not a good fit to the
 SoC structure.

 I'm pretty nervous about the datetime proposal that's currently on the
 wiki, for similar reasons -- I'm not sure it's actually doable in the
 SoC context.

  2. vector math library integration

 This is a great suggestion -- clear scope, clear benefit.

 Two more ideas:

 3. Using Cython in the numpy core

 The numpy core contains tons of complicated C code implementing
 elaborate operations like indexing, casting, ufunc dispatch, etc. It
 would be really nice if we could use Cython to write some of these
 things. However, there is a practical problem: Cython assumes that
 each .pyx file generates a single compiled module with its own
 Cython-defined API. Numpy, however, contains a large number of .c
 files which are all compiled together into a single module, with its
 own home-brewed system for defining the public API. And we can't
 rewrite the whole thing. So for this to be viable, we would need some
 way to compile a bunch of .c *and .pyx* files together into a single
 module, and allow the .c and .pyx files to call each other. This might
 involve changes to Cython, some sort of clever post-processing or glue
 code to get existing cython-generated source code to play nicely with
 the rest of numpy, or something else.

 So this project would have the following goals, depending on how
 practical this turns out to be: (1) produce a hacky proof-of-concept
 system for doing the above, (2) turn the hacky proof-of-concept into
 something actually viable for use in real life (possibly this would
 require getting changes upstream into Cython, etc.), (3) use this
 system to actually port some interesting numpy code into cython.


Having to synchronise two projects may be hard for a GSoC, no ?

Otherwise, I am a bit worried about cython being used on the current C code
as is, because core and python C API are so interwined (especially
multiarray).

Maybe one could use cython on the non-core numpy parts that are still in C
? It is not as sexy of a project, though.




 4. Pythonic dtypes

 The current dtype system is klugey. It basically defines its own class
 system, in parallel to Python's, and unsurprisingly, this new class
 system is not as good. In particular, it has limitations around the
 storage of instance-specific data which rule out a large variety of
 interesting user-defined dtypes, and causes us to need some truly
 nasty hacks to support the built-in dtypes we do have. And it makes
 defining a new dtype much more complicated than defining a new Python
 class.

 This project would be to implement a new dtype system for numpy, in
 which np.dtype becomes a near-empty base class, different dtypes
 (e.g., float64, float32) are simply different subclasses of np.dtype,
 and dtype objects are simply instances of these classes. Further
 enhancements would be to make it possible to define new dtypes in pure
 Python by subclassing np.dtype and implementing special methods for
 the various dtype operations, and to make it possible for ufunc loops
 to see the dtype objects.

 This project would provide the key enabling piece for a wide variety
 of interesting new features: missing value support, better handling of
 strings and categorical data, unit handling, automatic
 differentiation, and probably a bunch more I'm forgetting right now.

 If we get someone who's up to handling the dtype thing then I can
 mentor or co-mentor.

 What do y'all think?

 (I don't think I have access to update that wiki page -- or maybe I'm
 just not clever enough to figure out how -- so it would be helpful if
 someone who can, could?)

 --
 Nathaniel J. Smith
 Postdoctoral researcher - Informatics - University of Edinburgh
 http://vorpus.org
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] allocated memory cache for numpy

[David Cournapeau]

On Mon, Feb 17, 2014 at 7:31 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 hi,
 I noticed that during some simplistic benchmarks (e.g.
 https://github.com/numpy/numpy/issues/4310) a lot of time is spent in
 the kernel zeroing pages.
 This is because under linux glibc will always allocate large memory
 blocks with mmap. As these pages can come from other processes the
 kernel must zero them for security reasons.


Do you have numbers for 'a lot of time' ? Is the above script the exact one
you used for benchmarking this issue ?


 For memory within the numpy process this unnecessary and possibly a
 large overhead for the many temporaries numpy creates.

 The behavior of glibc can be tuned to change the threshold at which it
 starts using mmap but that would be a platform specific fix.

 I was thinking about adding a thread local cache of pointers to of
 allocated memory.
 When an array is created it tries to get its memory from the cache and
 when its deallocated it returns it to the cache.
 The threshold and cached memory block sizes could be adaptive depending
 on the application workload.

 For simplistic temporary heavy benchmarks this eliminates the time spent
 in the kernel (system with time).


For this kind of setup, I would advise to look into perf on linux. It
should be much more precise than time.

If nobody beats me to it, I can try to look at this this WE,


 But I don't know how relevant this is for real world applications.
 Have you noticed large amounts of time spent in the kernel in your apps?


In my experience, more time is spent on figuring out how to spare memory
than speeding this kind of operations for 'real life applications' (TM).

What happens to your benchmark if you tune malloc to not use mmap at all ?

David


 I also found this paper which describes pretty much exactly what I'm
 proposing:
 pyhpc.org/workshop/papers/Doubling.pdf‎

 Someone know why their changes were never incorporated in numpy? I
 couldn't find a reference in the list archive.
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] windows and C99 math

[David Cournapeau]

On Mon, Jan 27, 2014 at 8:04 PM, Julian Taylor 
jtaylor.deb...@googlemail.com wrote:

 hi,
 numpys no-C99 fallback keeps turning up issues in corner cases, e.g.
 hypot https://github.com/numpy/numpy/issues/2385
 log1p https://github.com/numpy/numpy/issues/4225

 these only seem to happen on windows, on linux and mac it seems to use
 the C99 math library just fine.

 Are our binary builds for windows not correct or does windows just not
 support C99 math?


Ms compilers have not supported much of C  C89. Up to recently, they even
said publicly something close to we don't care about C, C is legacy and
you should use C++:
http://herbsutter.com/2012/05/03/reader-qa-what-about-vc-and-c99/

But it looks like they are finally changing their stance:
http://blogs.msdn.com/b/vcblog/archive/2013/07/19/c99-library-support-in-visual-studio-2013.aspx

Of course, it will be a while before we can rely on this, but hey, it only
took them 14 years !

David


 Hopefully it is the former. Any insight is appreciated (and patches to
 fix the build even more!)

 Cheers,
 Julian
 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] LLVM

[David Cournapeau]

On Tue, Jan 7, 2014 at 6:59 PM, Charles R Harris
charlesr.har...@gmail.comwrote:

 Has anyone tried using LLVM with Visual Studio? It is supposed to work
 with Visual Studio = 2010 and might provide an alternative to MinGw64.


Yes, I have. It is still pretty painful to use on windows beyond simple
examples, though I have not tried the new 3.4 version. See also that
discussion I had with one clang dev @ apple a couple of months ago:
https://twitter.com/cournape/status/381038514076655618

David


 Chuck

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Does NumPy support indirect memory views?

[David Cournapeau]

On Sat, Dec 14, 2013 at 7:39 PM, Alexander Belopolsky ndar...@mac.comwrote:

 PEP 3118 [1] allows exposing multi-dimensional data that is organized as
 array of pointers.  It appears, however that NumPy cannot consume such
 memory views.

 Looking at _array_from_buffer_3118() function [2], I don't see any attempt
 to process suboffsets.   The documentation [3] is also silent on this issue.

 What is the status of indirect memory views/buffers support in NumPy?


There is indeed no support in NumPy for this. Unfortunately, fixing this
would be a significant amount of work, as buffer management is not really
abstracted in NumPy ATM.

David



 [1] http://www.python.org/dev/peps/pep-3118/
 [2]
 https://github.com/numpy/numpy/blob/4050ac73af79ae8cc513648ff02e9a22041501c4/numpy/core/src/multiarray/ctors.c#L1253
 [3] http://docs.scipy.org/doc/numpy/reference/arrays.interface.html

 ___
 NumPy-Discussion mailing list
 NumPy-Discussion@scipy.org
 http://mail.scipy.org/mailman/listinfo/numpy-discussion


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion