Re: [Numpy-discussion] Default type for functions that accumulate integers

[Antoine Pitrou]

On Mon, 2 Jan 2017 18:46:08 -0800
Nathaniel Smith  wrote:
> 
> So some options include:
> - make the default integer precision 64-bits everywhere
> - make the default integer precision 32-bits on 32-bit systems, and
> 64-bits on 64-bit systems (including Windows)

Either of those two would be the best IMO.

Intuitively, I think people would expect 32-bit ints in 32-bit
processes by default, and 64-bit ints in 64-bit processes likewise. So
I would slightly favour the latter option.

> - leave the default integer precision the same, but make accumulators
> 64-bits everywhere
> - leave the default integer precision the same, but make accumulators
> 64-bits on 64-bit systems (including Windows)

Both of these options introduce a confusing discrepancy.

> - speed: there's probably some cost to using 64-bit integers on 32-bit
> systems; how big is the penalty in practice?

Ok, I have fired up a Windows VM to compare 32-bit and 64-bit builds.
Numpy version is 1.11.2, Python version is 3.5.2.  Keep in mind those
are Anaconda builds of Numpy, with MKL enabled for linear algebra;
YMMV.

For each benchmark, the first number is the result on the 32-bit build,
the second number on the 64-bit build.

Simple arithmetic
-

>>> v = np.ones(1024**2, dtype='int32')

>>> %timeit v + v# 1.73 ms per loop | 1.78 ms per loop
>>> %timeit v * v# 1.77 ms per loop | 1.79 ms per loop
>>> %timeit v // v   # 5.89 ms per loop | 5.39 ms per loop

>>> v = np.ones(1024**2, dtype='int64')

>>> %timeit v + v# 3.54 ms per loop | 3.54 ms per loop
>>> %timeit v * v# 5.61 ms per loop | 3.52 ms per loop
>>> %timeit v // v   # 17.1 ms per loop | 13.9 ms per loop

Linear algebra
--

>>> m = np.ones((1024,1024), dtype='int32')

>>> %timeit m @ m# 556 ms per loop  | 569 ms per loop

>>> m = np.ones((1024,1024), dtype='int64')

>>> %timeit m @ m# 3.81 s per loop  | 1.01 s per loop

Sorting
---

>>> v = np.random.RandomState(42).randint(1000, size=1024**2).astype('int32')

>>> %timeit np.sort(v)   # 43.4 ms per loop | 44 ms per loop

>>> v = np.random.RandomState(42).randint(1000, size=1024**2).astype('int64')

>>> %timeit np.sort(v)   # 61.5 ms per loop | 45.5 ms per loop

Indexing


>>> v = np.ones(1024**2, dtype='int32')

>>> %timeit v[v[::-1]]   # 2.38 ms per loop | 4.63 ms per loop

>>> v = np.ones(1024**2, dtype='int64')

>>> %timeit v[v[::-1]]   # 6.9 ms per loop  | 3.63 ms per loop



Quick summary:
- for very simple operations, 32b and 64b builds can have the same perf
  on each given bitwidth (though speed is uniformly halved on 64-bit
  integers when the given operation is SIMD-vectorized)
- for more sophisticated operations (such as element-wise
  multiplication or division, or quicksort, but much more so on the
  matrix product), 32b builds are competitive with 64b builds on 32-bit
  ints, but lag behind on 64-bit ints
- for indexing, it's desirable to use a "native" width integer,
  regardless of whether that means 32- or 64-bit

Of course the numbers will vary depend on the platform (read:
compiler), but some aspects of this comparison will probably translate
to other platforms.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] State-of-the-art to use a C/C++ library from Python

[Antoine Pitrou]

On Fri, 2 Sep 2016 02:16:35 -0700
Nathaniel Smith  wrote:
> >
> > Depending on how minimal and universal you want to keep things, I use
> > the ctypes approach quite often, i.e. treat your numpy inputs an
> > outputs as arrays of doubles etc using the ndpointer(...) syntax. I
> > find it works well if you have a small number of well-defined
> > functions (not too many options) which are numerically very heavy.
> > With this approach I usually wrap each method in python to check the
> > inputs for contiguity, pass in the sizes etc. and allocate the numpy
> > array for the result.  
> 
> FWIW, the broader Python community seems to have largely deprecated
> ctypes in favor of cffi.

I'm not sure about "largely deprecated". For sure, that's the notion
spreaded by a number of people.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

[Antoine Pitrou]

On Mon, 13 Jun 2016 10:49:44 -0400
josef.p...@gmail.com wrote:
> 
> My argument is that `**` is like integer division and sqrt where the domain
> where integer return are the correct numbers is too small to avoid
> headaches by users.

float64 has less integer precision than int64:

>>> math.pow(3, 39) == 3**39
False
>>> np.int64(3)**39 == 3**39
True


(as a sidenote, np.float64's equality operator seems to be slightly
broken:

>>> np.float64(3)**39 == 3**39
True
>>> int(np.float64(3)**39) == 3**39
False
>>> float(np.float64(3)**39) == 3**39
False
)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

[Antoine Pitrou]

On Mon, 13 Jun 2016 10:05:08 -0400
Alan Isaac  wrote:
> 
> That is a misunderstanding, which may be influencing the discussion.
> Examples of complications:
> 
>  >>> type(np.int8(2)**2)
> 
>  >>> type(np.uint64(2)**np.int8(2))
> 

The `uint64 x int8 -> float64` is IMHO an abberration in Numpy's
typing logic. Regardless, it's not specific to the power operator:

>>> np.int64(2) + np.int32(3)
5
>>> np.uint64(2) + np.int32(3)
5.0

The other complications have to do with the type width, which are less
annoying than changing the numeric kind altogether (as would be done by
mandating int x int -> float in all cases).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

[Antoine Pitrou]

On Fri, 10 Jun 2016 20:28:30 -0400
Allan Haldane  wrote:
> 
> Also, I like to think of numpy as having quite C-like behavior, allowing
> you to play with the lowlevel bits and bytes. (I actually wish its
> casting behavior was more C-like). I suspect that people working with
> uint8 arrays might be doing byte-fiddling hacks and actually *want*
> overflow/wraparound to occur, at least when multiplying/adding.

I agree. Currently, the choice is simple: if you want an int output,
have an int input; if you want a float output, have a float output.
This fidelity to the user's data type choice allows people to make
informed decisions.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Changing FFT cache to a bounded LRU cache

[Antoine Pitrou]

On Sat, 28 May 2016 20:19:27 +0200
Sebastian Berg  wrote:
> 
> The complexity addition is a bit annoying I must admit, on python 3
> functools.lru_cache could be another option, but only there.

You can backport the pure Python version of lru_cache for Python 2 (or
vendor the backport done here:
https://pypi.python.org/pypi/backports.functools_lru_cache/).
The advantage is that lru_cache is C-accelerated in Python 3.5 and
upwards...

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Integers to integer powers

[Antoine Pitrou]

On Thu, 19 May 2016 20:30:40 -0700
Nathaniel Smith  wrote:
> 
> Given these immutable and contradictory constraints, the last bad
> option IMHO would be that we make int ** (negative int) an error in
> all cases, and the error message can suggest that instead of writing
> 
> np.array(2) ** -2
> 
> they should instead write
> 
> np.array(2) ** -2.0
> 
> (And similarly for np.int64(2) ** -2 versus np.int64(2) ** -2.0.)
> 
> Definitely annoying, but all the other options seem even more
> inconsistent and confusing, and likely to encourage the writing of
> subtly buggy code...

That sounds like a good compromise, indeed.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Antoine Pitrou]

On Thu, 12 May 2016 23:14:36 + (UTC)
Sturla Molden <sturla.mol...@gmail.com> wrote:
> Antoine Pitrou <solip...@pitrou.net> wrote:
> 
> > Can you define "expensive"?
> 
> Slow enough to cause complaints on the Cython mailing list.

What kind of complaints? Please be specific.

> > Buffer acquisition itself only calls a single C callback and uses a
> > stack-allocated C structure. It shouldn't be "expensive".
> 
> I don't know the reason, only that buffer acquisition from NumPy arrays
> with typed memoryviews

Again, what have memoryviews to do with it?  "Acquiring a buffer" is
just asking the buffer provider (the Numpy array) to fill a Py_buffer
structure's contents.  That has nothing to do with memoryviews.

When writing C code to interact with buffer-providing objects, you
usually don't bother with memoryviews at all.  You just use a Py_buffer
structure.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy arrays shareable among related processes (PR #7533)

[Antoine Pitrou]

On Thu, 12 May 2016 06:27:43 + (UTC)
Sturla Molden  wrote:

> Allan Haldane  wrote:
> 
> > You probably already know this, but I just wanted to note that the
> > mpi4py module has worked around pickle too. They discuss how they
> > efficiently transfer numpy arrays in mpi messages here:
> > http://pythonhosted.org/mpi4py/usrman/overview.html#communicating-python-objects-and-array-data
> 
> Unless I am mistaken, they use the PEP 3118 buffer interface to support
> NumPy as well as a number of other Python objects. However, this protocol
> makes buffer aquisition an expensive operation.

Can you define "expensive"?

> You can see this in Cython
> if you use typed memory views. Assigning a NumPy array to a typed
> memoryview (i,e, buffer acqisition) is slow.

You're assuming this is the cost of "buffer acquisition", while most
likely it's the cost of creating the memoryview object itself.

Buffer acquisition itself only calls a single C callback and uses a
stack-allocated C structure. It shouldn't be "expensive".

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Calling C code that assumes SIMD aligned data.

[Antoine Pitrou]

Here's an obligatory plug for the two following PRs:
https://github.com/numpy/numpy/pull/5457
https://github.com/numpy/numpy/pull/5470

Regards

Antoine.


On Fri, 6 May 2016 15:01:32 +0200
Francesc Alted  wrote:
> 2016-05-05 22:10 GMT+02:00 Øystein Schønning-Johansen :
> 
> > Thanks for your answer, Francesc. Knowing that there is no numpy solution
> > saves the work of searching for this. I've not tried the solution described
> > at SO, but it looks like a real performance killer. I'll rather try to
> > override malloc with glibs malloc_hooks or LD_PRELOAD tricks. Do you think
> > that will do it? I'll try it and report back.
> >
> 
> I don't think you need that much weaponry.  Just create an array with some
> spare space for alignment.  Realize that you want a 64-byte aligned double
> precision array.  With that, create your desired array + 64 additional
> bytes (8 doubles):
> 
> In [92]: a = np.zeros(int(1e6) + 8)
> 
> In [93]: a.ctypes.data % 64
> Out[93]: 16
> 
> and compute the elements to shift this:
> 
> In [94]: shift = (64 / a.itemsize) - (a.ctypes.data % 64) / a.itemsize
> 
> In [95]: shift
> Out[95]: 6
> 
> now, create a view with the required elements less:
> 
> In [98]: b = a[shift:-((64 / a.itemsize)-shift)]
> 
> In [99]: len(b)
> Out[99]: 100
> 
> In [100]: b.ctypes.data % 64
> Out[100]: 0
> 
> and voila, b is now aligned to 64 bytes.  As the view is a copy-free
> operation, this is fast, and you only wasted 64 bytes.  Pretty cheap indeed.
> 
> Francesc
> 
> 
> >
> > Thanks,
> > -Øystein
> >
> > On Thu, May 5, 2016 at 1:55 PM, Francesc Alted  wrote:
> >
> >> 2016-05-05 11:38 GMT+02:00 Øystein Schønning-Johansen  >> >:
> >>
> >>> Hi!
> >>>
> >>> I've written a little code of numpy code that does a neural network
> >>> feedforward calculation:
> >>>
> >>> def feedforward(self,x):
> >>> for activation, w, b in zip( self.activations, self.weights,
> >>> self.biases ):
> >>> x = activation( np.dot(w, x) + b)
> >>>
> >>> This works fine when my activation functions are in Python, however I've
> >>> wrapped the activation functions from a C implementation that requires the
> >>> array to be memory aligned. (due to simd instructions in the C
> >>> implementation.) So I need the operation np.dot( w, x) + b to return a
> >>> ndarray where the data pointer is aligned. How can I do that? Is it
> >>> possible at all?
> >>>
> >>
> >> Yes.  np.dot() does accept an `out` parameter where you can pass your
> >> aligned array.  The way for testing if numpy is returning you an aligned
> >> array is easy:
> >>
> >> In [15]: x = np.arange(6).reshape(2,3)
> >>
> >> In [16]: x.ctypes.data % 16
> >> Out[16]: 0
> >>
> >> but:
> >>
> >> In [17]: x.ctypes.data % 32
> >> Out[17]: 16
> >>
> >> so, in this case NumPy returned a 16-byte aligned array which should be
> >> enough for 128 bit SIMD (SSE family).  This kind of alignment is pretty
> >> common in modern computers.  If you need 256 bit (32-byte) alignment then
> >> you will need to build your container manually.  See here for an example:
> >> http://stackoverflow.com/questions/9895787/memory-alignment-for-fast-fft-in-python-using-shared-arrrays
> >>
> >> Francesc
> >>
> >>
> >>>
> >>> (BTW: the function works  correctly about 20% of the time I run it, and
> >>> else it segfaults on the simd instruction in the the C function)
> >>>
> >>> Thanks,
> >>> -Øystein
> >>>
> >>> ___
> >>> NumPy-Discussion mailing list
> >>> NumPy-Discussion@scipy.org
> >>> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>>
> >>>
> >>
> >>
> >> --
> >> Francesc Alted
> >>
> >> ___
> >> NumPy-Discussion mailing list
> >> NumPy-Discussion@scipy.org
> >> https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >>
> >>
> >
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > https://mail.scipy.org/mailman/listinfo/numpy-discussion
> >
> >
> 
> 



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Floor divison on int returns float

[Antoine Pitrou]

On Wed, 13 Apr 2016 15:49:15 -0600
Charles R Harris  wrote:
> 
> I looked this up once, `C` returns unsigned in the scalar case when both
> operands have the same width. See Usual Arithmetic Conversions
> .
> I think that is not a bad choice, but there is the back compatibility
> problem, plus it is a bit exceptional.

It may be a worse choice for Python. In the original use case (indexing
with an integer), losing the sign is a bug since negative indices have
a well-defined meaning in Python. This is a far more likely issue than
magnitude loss on a 64-bit integer.

In Numba, we decided that combining signed and unsigned would return
signed (see
http://numba.pydata.org/numba-doc/dev/proposals/integer-typing.html#proposal-predictable-width-conserving-typing).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Multidimension array access in C via Python API

[Antoine Pitrou]

On Tue, 5 Apr 2016 08:39:39 -0700 (MST)
mpc  wrote:
> This is the reason I'm doing this in the first place, because I made a pure
> python version but it runs really slow for larger data sets, so I'm
> basically rewriting the same function but using the Python and Numpy C API,
> but if you're saying it won't run any faster then maybe I'm going at it the
> wrong way. (Why use the C function version if it's the same speed anyway?)

The Python and Numpy C API are generally not very user-friendly
compared to Python code, even hand-optimized.

Cython will let you write code that looks quite close to normal Python
code, but with additional annotations for better performance.  Or you
can try Numba, a just-in-time compiler for scientific code that
understand Numpy arrays:
http://numba.pydata.org/

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] tracemalloc + numpy?

[Antoine Pitrou]

On Tue, 08 Mar 2016 09:26:13 -0500
Neal Becker  wrote:
> I'm trying tracemalloc to find memory usage.  Will numpy array memory usage 
> be counted by tracemalloc?  (Doesn't seem to)

No, but together with something like
https://github.com/numpy/numpy/pull/5470 it could.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] How to check for memory leaks?

[Antoine Pitrou]

On Tue, 23 Feb 2016 12:36:00 -0700
Charles R Harris  wrote:
> Hi All,
> 
> I'm suspecting a possible memory leak in 1.11.x, what is the best way to
> check for that?

If that is due to a reference leak, you can use sys.getrefcount() or
weakref.ref().

Otherwise you may want to change Numpy to go through PyMem_RawMalloc /
PyMem_RawCalloc / PyMem_RawRealloc / PyMem_RawFree on recent Pythons,
so as to have Numpy-allocated memory accounted by the tracemalloc
module.

(https://github.com/numpy/numpy/pull/5470 may make it more
palatable ;-))

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy 1.11.0b2 released

[Antoine Pitrou]

On Wed, 3 Feb 2016 21:56:08 -0800
Nathaniel Smith  wrote:
> 
> An extra ~2 hours of tests / 6-way parallelism is not that big a deal
> in the grand scheme of things (and I guess it's probably less than
> that if we can take advantage of existing binary builds) -- certainly
> I can see an argument for enabling it by default on the
> maintenance/1.x branches. Running N extra test suites ourselves is not
> actually more expensive than asking N projects to run 1 more testsuite
> :-). The trickiest part is getting it to give actually-useful
> automated pass/fail feedback, as opposed to requiring someone to
> remember to look at it manually :-/

Yes, I think that's where the problem lies. Python had something called
"community buildbots" at a time (testing well-known libraries such as
Twisted against the Python trunk), but it suffered from lack of
attention and finally was dismantled. Apparently having the people
running it and the people most interested in it not being the same ones
ended up a bad idea :-)

That said, if you do something like that with Numpy, we would be
interested in having Numba be part of the tested packages.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Numpy intermittent seg fault

[Antoine Pitrou]

Hi,

On Fri, 11 Dec 2015 10:05:59 +1000
Jacopo Sabbatini  wrote:
> 
> I'm experiencing random segmentation faults from numpy. I have generated a
> core dumped and extracted a stack trace, the following:
> 
> #0  0x7f3a8d921d5d in getenv () from /lib64/libc.so.6
> #1  0x7f3a843bde21 in blas_set_parameter () from
> /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0
> #2  0x7f3a843bcd91 in blas_memory_alloc () from
> /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0
> #3  0x7f3a843bd4e5 in blas_thread_server () from
> /opt/apps/sidescananalysis-9.7.1-42-gdd3e068+dev/lib/python2.7/site-packages/numpy/core/../../../../libopenblas.so.0
> #4  0x7f3a8e09ff18 in start_thread () from /lib64/libpthread.so.0
> #5  0x7f3a8d9ceb2d in clone () from /lib64/libc.so.6
> 
> I have experience the segfault from several code paths but they all have
> the same stack trace.
> 
> I use conda to run python and numpy. The dump of the packages version is:

In addition to openblas, you should also submit a bug to Anaconda so
that they know of problems with that particular openblas version:
https://github.com/ContinuumIO/anaconda-issues

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 11:00:30 +0100
David Cournapeau  wrote:
> 
> Assuming one of the rumour is related to some comments I made some time
> (years ?) earlier, the context was the ability to hide exported symbols. As
> you know, the issue is not to build extensions w/ multiple compilation
> units, but sharing functionalities between them without sharing them
> outside the extension.

Can't you use the visibility attribute with gcc for this?
Other Unix compilers probably provide something similar. The issue
doesn't exist on Windows by construction.

https://gcc.gnu.org/onlinedocs/gcc-5.2.0/gcc/Function-Attributes.html#Function-Attributes

By the way, external packages may reuse the npy_* functions, so I would
like them not the be hidden :-)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 09:40:43 -0700
Nathaniel Smith  wrote:
> 
> If you need some npy_* function it'd be much better to let us know
> what it is and let us export it in an intentional way, instead of just
> relying on whatever stuff we accidentally exposed?

Ok, we seem to be using only the complex math functions (npy_cpow and
friends, I could make a complete list if required).

And, of course, we would also benefit from the CBLAS functions (or any
kind of C wrappers around them) :-)
https://github.com/numpy/numpy/issues/6324

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Tue, 6 Oct 2015 10:07:13 -0700
Nathaniel Smith  wrote:
> 
> And how are you getting at them? Are you just relying the way that on
> ELF systems, if two libraries are loaded into the same address space
> then they automatically get access to each other's symbols, even if
> they aren't linked to each other? What do you do on Windows?

Well it seems to work on Windows too, thanks to
numpy.distutils.misc_util.get_info('npymath').

Under Windows, I seem to have a
"\site-packages\numpy\core\lib\npymath.lib" static library,
and there's also a "npy-pkg-config" subdirectory there with some INI
files in it.  Hopefully you know better than me what this all is :-)

> > And, of course, we would also benefit from the CBLAS functions (or any
> > kind of C wrappers around them) :-)
> > https://github.com/numpy/numpy/issues/6324
> 
> This is difficult to do from NumPy itself -- we don't necessarily have
> access to a full BLAS or LAPACK API -- in some configurations we fall
> back on our minimal internal implementations that just have what we
> need.

I'm thinking about the functions exposed in
"numpy/core/src/private/npy_cblas.h".
My knowledge of the Numpy build system doesn't allow me to tell if it's
always available or not :-)

> There was an interesting idea that came up in some discussions here a
> few weeks ago -- we already know that we want to package up BLAS
> inside a Python package that (numpy / scipy / scikit-learn / ...) can
> depend on and assume is there to link against.
> 
> Maybe this new package would also be a good place for exposing these wrappers?

Yeah, why not - as long as there's something well-known and
well-supported to depend on.  But given Numpy is a hard dependency for
all the other packages you mentioned, it may make sense (and simplify
dependency management) to bundle it with Numpy.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Should we drop support for "one file" compilation mode?

[Antoine Pitrou]

On Mon, 5 Oct 2015 15:26:17 -0700
Nathaniel Smith  wrote:
> Hi all,
> 
> For a long time, NumPy has supported two different ways of being compiled:
> 
> "Separate compilation" mode: like most C projects, each .c file gets
> compiled to a .o file, and then the .o files get linked together to
> make a shared library. (This has been the default since 1.8.0.)
> 
> "One file" mode: first concatenate all the .c files together to make
> one monster .c file, and then compile that .c file to make a shared
> library. (This was the default before 1.8.0.)
> 
[...]
> 
> There are some rumors that "one file" mode might be needed on some
> obscure platform somewhere, or that it might be necessary for
> statically linking numpy into the CPython executable, but we can't
> continue supporting things forever based only on rumors.

If those rumors were true, CPython would not even be able to build
(the _io module in 3.x is linked from several C object files, for
example).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Sign of NaN

[Antoine Pitrou]

On Tue, 29 Sep 2015 09:13:15 -0600
Charles R Harris  wrote:
> 
> Due to a recent commit, Numpy master now raises an error when applying the
> sign function to an object array containing NaN. Other options may be
> preferable, returning NaN for instance, so I would like to open the topic
> for discussion on the list.

None for example? float('nan') may be a bit weird amongst e.g. an array
of Decimals.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Antoine Pitrou]

On Thu, 24 Sep 2015 00:20:23 -0700
Nathaniel Smith  wrote:
> > int PyUFunc_Identity(PyFuncObject *)
> >
> >   Replaces ufunc->identity.
> 
> Hmm, I can imagine cases where we might want to change how this works.
> (E.g. if np.dot were a ufunc then the existing identity settings
> wouldn't work very well... and I have some vague memory that there
> might already some delicate code in a few places because of
> difficulties in defining "zero" and "one" for arbitrary dtypes.)

Yes... As long as there is a way for us to set the identity value
(whatever the exact API) when constructing a ufunc, it should be ok.

> I assume the 'i' part isn't actually interesting here (since there's
> no longer any parallel vector of function pointers accessible), and
> the high-level semantics that you're looking for are "please give me
> the set of signatures that have a loop defined"?

Indeed.

> [Edit: Also, see the discussion below about integer type pointers. The
> consequences here are that we can certainly provide an operation like
> this, but if we do then we might be abandoning it in a few releases
> (e.g. it might start telling you about only a subset of defined
> signatures). So can you expand a bit on what you mean by "would be
> nice" above?]

"Would be nice" really means "we could make use of it" for letting the
user access ufunc metadata. We don't *need* it currently. But
generally being able to query the high-level properties of a ufunc,
from C, sounds like a good thing, and perhaps other people would be
interested.

> > PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *)
> >
> >   Sets the ufunc's "object" to the given object.  The object has no
> >   special semantics except that it is DECREF'ed when the ufunc is
> >   deallocated (this is today's ufunc->obj).  The DECREF should happen
> >   only after the ufunc has accessed any internal resources (since the
> >   DECREF could deallocate some of those resources).
> 
> I understand why you need a "base" object like this for individual
> loops, but if ufuncs start managing the ufunc-level memory buffers
> internally, then is this still useful? I guess I'm curious to see an
> example.

Well, for example, we dynamically allocate the ufunc's name (and
possibly its docstring), so we need to deallocate it when the ufunc is
destroyed.  Actually, we should probably deallocate more stuff that we
currently don't (such as the execution environment)...

> > PyObject *PyUFunc_GetObject(PyUFuncObject *)
> >
> >   Return the ufunc's current "object".
> 
> Oh, are you planning to actually use this to attach some arbitrary
> metadata, not just attach deallocation callbacks?

No, just deallocation callbacks. I was including the GetObject function
for completeness, I'm not sure we would need it (but it sounds trivial
to provide and maintain).

> Hmm, that's an interesting and tricky point, actually -- I think the
> way it will work eventually is that signatures will be specified in
> terms of "dtypetypes" (i.e., subclasses of dtype, rather than ints
> *or* instances of dtype = PyArray_Descrs).

Subclasses? I'm not sure what you mean by that, how would one specify
e.g. an int64 vs. an int32?

Are you referring to Travis' dtypes-as-classes project, or something
similar? In that case though, a dtype would still be an instance of a
"dtypetype" (metatype), not a subclass :-)

> But I guess that's just a
> challenge we'll have to think about when implementing this stuff --
> either it means that the new ufunc API will have to wait a bit for
> more of the new dtype machinery to be ready, or we'll have to
> temporarily bridge the gap with an loop registration API that takes
> new-style loop callbacks but uses int signatures (and then later turn
> it into a thin wrapper around the final API).

Well, as long as you keep the int typecodes in Numpy (and I guess
you'll do for quite some time, for compatibility), bridging should be
easy indeed.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Antoine Pitrou]

On Mon, 21 Sep 2015 21:38:36 -0700
Nathaniel Smith <n...@pobox.com> wrote:
> Hi Antoine,
> 
> On Mon, Sep 21, 2015 at 2:44 AM, Antoine Pitrou <solip...@pitrou.net> wrote:
> >
> > Hi Nathaniel,
> >
> > On Sun, 20 Sep 2015 21:13:30 -0700
> > Nathaniel Smith <n...@pobox.com> wrote:
> >> Given this, I propose that for 1.11 we:
> >> 1) go ahead and hide/disable the problematic parts of the ABI/API,
> >> 2) coordinate with the known affected projects to minimize disruption
> >> to their users (which is made easier since they are all projects that
> >> are almost exclusively distributed via conda, which enforces strict
> >> NumPy ABI versioning),
> >> 3) publicize these changes widely so as to give any private code that
> >> might be affected a chance to speak up or adapt, and
> >> 4) leave the "ABI version tag" as it is, so as not to force rebuilds
> >> of the vast majority of projects that will be unaffected by these
> >> changes.
> >
> > Thanks for a detailed and clear explanation of the proposed changes.
> > As far as Numba is concerned, making changes is ok for us provided
> > Numpy provides APIs to do what we want.
> 
> Good to hear, thanks!
> 
> Any interest in designing those new APIs that will do what you want?
> :-)

I'll take a look.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1 coming tomorrow, 22 Sept.

[Antoine Pitrou]

On Tue, 22 Sep 2015 04:43:18 -0500
Travis Oliphant  wrote:
> Absolutely it would be good if others can test.  All I was suggesting is
> that we do run a pretty decent set of tests upon build and that would be
> helpful.
> 
> If the numpy build recipes are not available, it is only because they have
> not been updated to use conda-build yet.  If somebody wants to volunteer to
> convert all of our internal recipes to conda-build recipes so they could be
> open source --- we would welcome the help.

Note there's a skeleton recipe for numpy here:
https://github.com/conda/conda-recipes/tree/master/numpy-openblas

If there's interest I could try to come up with a slightly better one,
although I can only promise to make it work on Linux.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Antoine Pitrou]

Hi,

This e-mail is an attempt at proposing an API to solve Numba's needs.

Attribute access


int PyUFunc_Nin(PyUFuncObject *)

  Replaces ufunc->nin.

int PyUFunc_Nout(PyUFuncObject *)

  Replaces ufunc->nout.

int PyUFunc_Nargs(PyUFuncObject *)

  Replaces ufunc->nargs.

PyObject *PyUFunc_Name(PyUFuncObject *)

  Replaces ufunc->name, returns a unicode object.
  (alternative: return a const char *)

For introspection, the following would be nice too:

int PyUFunc_Identity(PyFuncObject *)

  Replaces ufunc->identity.

const char *PyUFunc_Signature(PyUFuncObject *, int i)

  Gives a pointer to the types of the i'th signature.
  (equivalent today to >ntypes[i * ufunc->nargs])


Lifetime control


PyObject *PyUFunc_SetObject(PyUFuncObject *, PyObject *)

  Sets the ufunc's "object" to the given object.  The object has no
  special semantics except that it is DECREF'ed when the ufunc is
  deallocated (this is today's ufunc->obj).  The DECREF should happen
  only after the ufunc has accessed any internal resources (since the
  DECREF could deallocate some of those resources).

PyObject *PyUFunc_GetObject(PyUFuncObject *)

  Return the ufunc's current "object".


Loop registration
-

int PyUFunc_RegisterLoopForSignature(
PyUFuncObject* ufunc,
PyUFuncGenericFunction function, int *arg_types,
void *data, PyObject *obj)

  Register a loop implementation for the given arg_types (built-in
  types, presumably). This either appends the loop to the types and
  functions array (reallocating it if necessary), or replaces an
  existing one with the same signature.

  A copy of arg_types is done, such that the caller does not have to
  manage its lifetime. The optional "PyObject *obj" is an object which
  gets DECREF'ed when the loop is relinquished (for example when the
  ufunc is destroyed, or when the loop gets replaced with another by
  calling this function again).


I cannot say I'm 100% sure this is sufficient, but this seems it should
cover our current needs.

Note this is a minimal proposal. For example, Numpy could instead decide
to pass and return all argument types as PyArray_Descr pointers rather
than raw integers, and that would probably work for us too.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] draft NEP for breaking ufunc ABI in a controlled way

[Antoine Pitrou]

Hi Nathaniel,

On Sun, 20 Sep 2015 21:13:30 -0700
Nathaniel Smith  wrote:
> Given this, I propose that for 1.11 we:
> 1) go ahead and hide/disable the problematic parts of the ABI/API,
> 2) coordinate with the known affected projects to minimize disruption
> to their users (which is made easier since they are all projects that
> are almost exclusively distributed via conda, which enforces strict
> NumPy ABI versioning),
> 3) publicize these changes widely so as to give any private code that
> might be affected a chance to speak up or adapt, and
> 4) leave the "ABI version tag" as it is, so as not to force rebuilds
> of the vast majority of projects that will be unaffected by these
> changes.

Thanks for a detailed and clear explanation of the proposed changes.
As far as Numba is concerned, making changes is ok for us provided
Numpy provides APIs to do what we want.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] method to calculate the magnitude squared

[Antoine Pitrou]

On Fri, 18 Sep 2015 21:16:42 -0700
Phillip Feldman  wrote:
> In communications and signal processing, it is frequently necessary to
> calculate the power of a signal.  This can be done with a function like the
> following:
> 
> def magsq(z):
>"""
>Return the magnitude squared of the real- or complex-valued input.
>"""
>return z.real**2 + z.imag**2
> 
> A high percentage of the scripts that I write contain or import a function
> like this.  It would be great if there were a built-in method in NumPy,
> preferably with a name like `magsq`, `mag2`, or `msq`.

Are you asking for speed or convenience reasons?

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure?

[Antoine Pitrou]

On Mon, 14 Sep 2015 19:02:58 +0200
Sturla Molden <sturla.mol...@gmail.com> wrote:
> On 14/09/15 10:34, Antoine Pitrou wrote:
> 
> > If Numpy wanted to switch to a different generator, and if Numba wanted
> > to remain compatible with Numpy, one of the PCG functions would be an
> > excellent choice (also for CPU performance, incidentally).
> 
> Is Apache license ok in NumPy?

While I don't know Numpy's policy precisely, I have no idea why a
modern non-copyleft license such as Apache wouldn't be ok.

That said, PCG looks simple enough that you can reimplement it (or at
least the variants that are of interest to you) independently without
too much effort.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] OK to upload patched 1.9.2 for Python 3.5?

[Antoine Pitrou]

On Mon, 14 Sep 2015 01:32:13 -0700
Matthew Brett  wrote:
> >>
> >> Does anyone object to me uploading a wheel built from this patched
> >> version to pypi as 1.9.2 for Python 3.5 on OSX?   It would help to get
> >> the ball rolling for Python 3.5 binary wheels.
> >
> > Why not releasing this as 1.9.3 ? It does not need to be a full release
> > (with binaries and all), but having multiple sources for a given tag is
> > confusing.
> 
> Generally OK with me, but it's quite a bit of extra work for very
> little gain.  We'd have to tag, release a source tarball and OSX
> wheels, at least.

It's always bad to have two silent versions of a single release. People
can never know up front which version they got, since it's not written
anywhere. That's why the right policy is to bump the version number in
some way (be it by incrementing it or adding a ".patchXXX" at the end).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] [Python-ideas] Should our default random number generator be secure?

[Antoine Pitrou]

On Mon, 14 Sep 2015 09:26:58 +0100
Robert Kern  wrote:
> 
> Actually, I meant all of the crap *around* it, the platform-compatibility
> testing to see if you have such a hardware instruction or not, and C++
> template shenanigans in the surrounding code. It's possible that the
> complexity is only due to flexibility, but it was too complex for me to
> begin understanding *why* it's so complex before I succumbed to ennui and
> moved on to some other productive use of my free time. At least some of the
> complexity is due to needing software implementations of reduced-round
> crypto for decent performance in the absence of the hardware instruction.
> Performing well in the absence of the hardware instruction is very
> important to me as I do not seem to have the AES-NI instruction available
> on my mid-2012 Macbook Pro. Exposing counter-mode AES128 as a core PRNG is
> a nice idea, but it's just low on my wishlist. I want fast, multiple
> independent streams on my current hardware first, and PCG gives that to me.

Using AES also means emulating it on a GPU will be quite hard.

For the record, Numba is currently using a Mersenne Twister on the CPU,
to emulate Numpy's behaviour (although some of our distributions may be
different):

>>> def f(x):
... np.random.seed(x)
... l = []
... for i in range(10): l.append(np.random.random())
... return l
... 
>>> g = numba.jit(nopython=True)(f)
>>> f(10)
[0.771320643266746, 0.0207519493594015, 0.6336482349262754,
0.7488038825386119, 0.4985070123025904, 0.22479664553084766,
0.19806286475962398, 0.7605307121989587, 0.16911083656253545,
0.08833981417401027]
>>> g(10)
[0.771320643266746, 0.0207519493594015, 0.6336482349262754,
0.7488038825386119, 0.4985070123025904, 0.22479664553084766,
0.19806286475962398, 0.7605307121989587, 0.16911083656253545,
0.08833981417401027]


Currently we don't provide those APIs on the GPU, since MT is much too
costly there.

If Numpy wanted to switch to a different generator, and if Numba wanted
to remain compatible with Numpy, one of the PCG functions would be an
excellent choice (also for CPU performance, incidentally).

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] np.sign and object comparisons

[Antoine Pitrou]

On Mon, 31 Aug 2015 10:23:10 -0700
Stephan Hoyer  wrote:
> 
> My inclination is that return NaN would be the appropriate choice. It's
> certainly consistent with the behavior for float dtypes -- my expectation
> for object dtype behavior is that it works exactly like applying the
> np.sign ufunc to each element of the array individually.
> 
> On the other hand, I suppose there are other ways in which an object can
> fail all those comparisons (e.g., NaT?), so I suppose we could return None.

Currently:

>>> np.sign(np.timedelta64('nat'))
numpy.timedelta64(-1)

... probably because NaT is -2**63 under the hood. But in this case
returning NaT would sound better.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Antoine Pitrou]

Hi again,

The change seems to have possibly unforeseen consequences because some
ufuncs don't declare all possible types, e.g.:

 a = np.arange(10, dtype=np.int32)
 out = np.zeros_like(a)
 np.fabs(a, out=out)
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: ufunc 'fabs' output (typecode 'd') could not be coerced to provided 
output parameter (typecode 'i') according to the casting rule ''same_kind''
 np.fabs.types
['e-e', 'f-f', 'd-d', 'g-g', 'O-O']


(while fabs() wouldn't necessarily make sense on complex numbers, it
does make sense on integers... and, ah, I've just noticed that np.abs()
also exists with more input types, which is confusing...)

Regards

Antoine.


On Wed, 26 Aug 2015 07:52:09 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
 On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris charlesr.har...@gmail.com
  wrote:
 
 
 
  On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris 
  charlesr.har...@gmail.com wrote:
 
 
 
  On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
  wrote:
 
  On Tue, 25 Aug 2015 10:26:02 -0600
  Charles R Harris charlesr.har...@gmail.com wrote:
   Hi All,
  
   The silence after the 1.10 beta has been eerie. Consequently, I'm
  thinking
   of making a first release candidate this weekend. If you haven't yet
  tested
   the beta, please do so. It would be good to discover as many problems
  as we
   can before the first release.
 
  Has typing of ufunc parameters become much stricter? I can't find
  anything in the release notes, but see (1.10b1):
 
   arr = np.linspace(0, 5, 10)
   out = np.empty_like(arr, dtype=np.intp)
   np.round(arr, out=out)
  Traceback (most recent call last):
File stdin, line 1, in module
File
  /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
  line 2778, in round_
  return round(decimals, out)
  TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
  provided output parameter (typecode 'l') according to the casting rule
  ''same_kind''
 
 
  It used to work (1.9):
 
   arr = np.linspace(0, 5, 10)
   out = np.empty_like(arr, dtype=np.intp)
   np.round(arr, out=out)
  array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
   out
  array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 
 
  The default casting mode has been changed. I think this has been raising
  a warning since 1.7 and was mentioned as a future change in 1.10, but you
  are right, it needs to be mentioned in the 1.10  release notes.
 
 
  Make that warned of in the 1.9.0 release notes.
 
 
 Here it is in 1.9.0 with deprecation warning made visible.
 ```
 In [3]: import warnings
 
 In [4]: warnings.simplefilter('always')
 
 In [5]: arr = np.linspace(0, 5, 10)
 
 In [6]: out = np.empty_like(arr, dtype=np.intp)
 
 In [7]: np.round(arr, out=out)
 /home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
 DeprecationWarning: Implicitly casting between incompatible kinds. In a
 future numpy release, this will raise an error. Use casting=unsafe if
 this is intentional.
   return round(decimals, out)
 Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 ```
 
 Chuck
 


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Antoine Pitrou]

The change also seems to have made datetime64 computations stricter:

 np.datetime64('2010') - np.datetime64('2000-01-01')
numpy.timedelta64(3653,'D')

 np.datetime64('2010') - np.datetime64('2000-01-01T00:00:00Z')
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: Cannot cast ufunc subtract input from dtype('M8[Y]') to 
dtype('M8[s]') with casting rule 'same_kind'

Regards

Antoine.


On Wed, 26 Aug 2015 07:52:09 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
 On Wed, Aug 26, 2015 at 7:32 AM, Charles R Harris charlesr.har...@gmail.com
  wrote:
 
 
 
  On Wed, Aug 26, 2015 at 7:31 AM, Charles R Harris 
  charlesr.har...@gmail.com wrote:
 
 
 
  On Wed, Aug 26, 2015 at 7:11 AM, Antoine Pitrou solip...@pitrou.net
  wrote:
 
  On Tue, 25 Aug 2015 10:26:02 -0600
  Charles R Harris charlesr.har...@gmail.com wrote:
   Hi All,
  
   The silence after the 1.10 beta has been eerie. Consequently, I'm
  thinking
   of making a first release candidate this weekend. If you haven't yet
  tested
   the beta, please do so. It would be good to discover as many problems
  as we
   can before the first release.
 
  Has typing of ufunc parameters become much stricter? I can't find
  anything in the release notes, but see (1.10b1):
 
   arr = np.linspace(0, 5, 10)
   out = np.empty_like(arr, dtype=np.intp)
   np.round(arr, out=out)
  Traceback (most recent call last):
File stdin, line 1, in module
File
  /home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py,
  line 2778, in round_
  return round(decimals, out)
  TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to
  provided output parameter (typecode 'l') according to the casting rule
  ''same_kind''
 
 
  It used to work (1.9):
 
   arr = np.linspace(0, 5, 10)
   out = np.empty_like(arr, dtype=np.intp)
   np.round(arr, out=out)
  array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
   out
  array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 
 
  The default casting mode has been changed. I think this has been raising
  a warning since 1.7 and was mentioned as a future change in 1.10, but you
  are right, it needs to be mentioned in the 1.10  release notes.
 
 
  Make that warned of in the 1.9.0 release notes.
 
 
 Here it is in 1.9.0 with deprecation warning made visible.
 ```
 In [3]: import warnings
 
 In [4]: warnings.simplefilter('always')
 
 In [5]: arr = np.linspace(0, 5, 10)
 
 In [6]: out = np.empty_like(arr, dtype=np.intp)
 
 In [7]: np.round(arr, out=out)
 /home/charris/.local/lib/python2.7/site-packages/numpy/core/fromnumeric.py:2640:
 DeprecationWarning: Implicitly casting between incompatible kinds. In a
 future numpy release, this will raise an error. Use casting=unsafe if
 this is intentional.
   return round(decimals, out)
 Out[7]: array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 ```
 
 Chuck
 


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

[Antoine Pitrou]

On Wed, 26 Aug 2015 16:45:51 + (UTC)
Irwin Zaid iz...@continuum.io wrote:
 
 So, we see DyND is having a twofold purpose. The first is to expand upon the
 kinds of data that NumPy can represent and do computations upon. The second
 is to provide a standard array package that can cross the language barrier
 and easily interoperate between C++, Python, or whatever you want.

One possible limitation is that the lingua franca for language
interoperability is C, not C++. DyND doesn't have to be written in C,
but exposing a nice C API may help make it attractive to the various
language runtimes out there.

(even those languages whose runtime doesn't have a compile-time
interface to C generally have some kind of cffi or ctypes equivalent to
load external C routines at runtime)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] SHA256 mismatch on SourceForge downloads

[Antoine Pitrou]

Hello,

The SourceForge download page for 1.10.0b1 mentions:

 89e467cec774527dd254c1e039801726db1367433053801f0d8bc68deac74009
 numpy-1.10.0b1.tar.gz

But after downloading the file I get:

$ sha256sum numpy-1.10.0b1.tar.gz 
855695405092686264dc8ce7b3f5c939a6cf1a5639833e841a5bb6fb799cd6a8
numpy-1.10.0b1.tar.gz


Also, since SouceForge doesn't provide any HTTPS downloads (it
actually redirects HTTPS to HTTP (*)), this all looks a bit pointless.

(*) seems like SourceForge is becoming a poster child of worst
practices...

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] 1.10.0rc1

[Antoine Pitrou]

On Tue, 25 Aug 2015 10:26:02 -0600
Charles R Harris charlesr.har...@gmail.com wrote:
 Hi All,
 
 The silence after the 1.10 beta has been eerie. Consequently, I'm thinking
 of making a first release candidate this weekend. If you haven't yet tested
 the beta, please do so. It would be good to discover as many problems as we
 can before the first release.

Has typing of ufunc parameters become much stricter? I can't find
anything in the release notes, but see (1.10b1):

 arr = np.linspace(0, 5, 10)
 out = np.empty_like(arr, dtype=np.intp)
 np.round(arr, out=out)
Traceback (most recent call last):
  File stdin, line 1, in module
  File 
/home/antoine/np110/lib/python3.4/site-packages/numpy/core/fromnumeric.py, 
line 2778, in round_
return round(decimals, out)
TypeError: ufunc 'rint' output (typecode 'd') could not be coerced to provided 
output parameter (typecode 'l') according to the casting rule ''same_kind''


It used to work (1.9):

 arr = np.linspace(0, 5, 10)
 out = np.empty_like(arr, dtype=np.intp)
 np.round(arr, out=out)
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])
 out
array([0, 1, 1, 2, 2, 3, 3, 4, 4, 5])


Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] py2/py3 pickling

[Antoine Pitrou]

On Tue, 25 Aug 2015 19:12:30 +0300
Pauli Virtanen p...@iki.fi wrote:

 25.08.2015, 01:15, Chris Laumann kirjoitti:
  Would it be possible then (in relatively short order) to create
  a py2 - py3 numpy pickle converter? 
 
 You probably need to modify the pickle stream directly, replacing
 *STRING opcodes with *BYTES opcodes when it comes to objects that are
 needed for constructing Numpy arrays.
 
 https://hg.python.org/cpython/file/tip/Modules/_pickle.c#l82
 
 Or, use a custom pickler class that emits the new opcodes when it comes
 to data that is part of Numpy arrays, as Python 2 pickler doesn't know
 how to write bytes opcodes.
 
 It's probably doable, although likely annoying to implement. the pickles
 created won't be loadable on Py2, only Py3.

One could take a look at how the built-in bytearray type achieves
pickle compatibility between 2.x and 3.x. The solution is to serialize
the binary data as a latin-1 decoded unicode string, and to return the
right reconstructor from __reduce__.

The solution is less space-efficient than pure bytes pickling, since
the unicode string is serialized as utf-8 (so bytes  0x80 are
multibyte-encoded). There's also some CPU overhead, due to the
successive decoding and encoding steps.

You can take a look at the bytearray_reduce() function in
Objects/bytearrayobject.c, both for 2.x and 3.x.

(also note how the 3.x version does it only for protocols  3, to
achieve better efficiency on newer protocol versions)


Another possibility would be a custom Unpickler class for 3.x, dealing
specifically with 2.x-produced Numpy array pickles. That way the
pickles themselves could be cross-version.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

[Antoine Pitrou]

On Tue, 25 Aug 2015 03:03:41 -0700
Nathaniel Smith n...@pobox.com wrote:
 
 Supporting third-party dtypes
 ~
 
[...]
 
   Some features that would become straightforward to implement
   (e.g. even in third-party libraries) if this were fixed:
   - missing value support
   - physical unit tracking (meters / seconds - array of velocity;
 meters + seconds - error)
   - better and more diverse datetime representations (e.g. datetimes
 with attached timezones, or using funky geophysical or
 astronomical calendars)
   - categorical data
   - variable length strings
   - strings-with-encodings (e.g. latin1)
   - forward mode automatic differentiation (write a function that
 computes f(x) where x is an array of float64; pass that function
 an array with a special dtype and get out both f(x) and f'(x))
   - probably others I'm forgetting right now

It should also be the opportunity to streamline datetime64 and
timedelta64 dtypes. Currently the unit information is IIRC hidden in
some weird metadata thing called the PyArray_DatetimeMetaData.

Also, thanks the notes. It has been an interesting read.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Rationale for integer promotion rules

[Antoine Pitrou]

Hi,

We were discussion integer promotion rules amongst the Numba team, and
we were wondering about the rationale for Numpy's rules.  For example,
adding int8 and int8 will give int8 as result (with potential
magnitude loss), while adding int8 and uint8 will give int16 as result
(promoting to the smallest fitting type).  Furthermore, adding int64
and uint64 returns float64.

Is there a rationale somewhere documenting and explaining this behaviour
(sorry if I've missed it in a obvious location)? Is it the produce of
organic evolution?

Also, it is set to stay like this, or will it evolve in the future?

Thank you

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Rationale for integer promotion rules

[Antoine Pitrou]

On Thu, 16 Jul 2015 19:19:58 +0100
Robert Kern robert.k...@gmail.com wrote:
 On Thu, Jul 16, 2015 at 7:14 PM, Nathaniel Smith n...@pobox.com wrote:
 
  On Thu, Jul 16, 2015 at 9:18 AM, Antoine Pitrou solip...@pitrou.net
 wrote:
 
   while adding int8 and uint8 will give int16 as result
   (promoting to the smallest fitting type).
 
  I understand this to be a consequence of the previous rule (results
  should match inputs) combined with the need to find a common input
  type.
 
 Specifically, when combining signed and unsigned ints, we need to find a
 signed int type that can simultaneously hold both of the *inputs*.

I'm not sure that would be necessary for e.g. addition (as long as you
run your code on a 2's complement machine). But thanks for the
explanation.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Rationale for integer promotion rules

[Antoine Pitrou]

On Thu, 16 Jul 2015 11:14:10 -0700
Nathaniel Smith n...@pobox.com wrote:
 
  Also, it is set to stay like this, or will it evolve in the future?
 
 I don't know -- if you make a good case for something better then maybe?

No, I was just wondering if it would be a good use of our time to try to
use a scheme that looks remotely like Numpy's :-)

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] floats for indexing, reshape - too strict ?

[Antoine Pitrou]

On Thu, 02 Jul 2015 08:40:13 -0400
Neal Becker ndbeck...@gmail.com wrote:
 I'd be concerned that checking each index for exactness would be costly.
 I'm also concerned that using floats for an index is frequently a mistake 
 and that a warning is what I want.

Or just follow Python:

Python 3.4.3 |Continuum Analytics, Inc.| (default, Jun  4 2015,
15:29:08) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type help, copyright, credits or license for more information.
 [1][0.0]
Traceback (most recent call last):
  File stdin, line 1, in module
TypeError: list indices must be integers, not float


I don't think relaxing type checking here makes any good.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] ANN: numexpr 2.4.3 released

[Antoine Pitrou]

On Mon, 27 Apr 2015 19:35:51 -0400
Neil Girdhar mistersh...@gmail.com wrote:
 Also, FYI: http://numba.pydata.org/numba-doc/0.6/doc/modules/transforms.html
 
 It appears that numba does get the ast similar to pyautodiff and only get
 the ast from source code as a fallback?

That documentation is terribly obsolete (the latest Numba version is
0.18.2).
Modern Numba starts from the CPython bytecode, it doesn't look at the
AST.  We explain the architecture in some detail here:
http://numba.pydata.org/numba-doc/dev/developer/architecture.html

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Aligned / configurable memory allocation

[Antoine Pitrou]

Hello,

I apologize for pinging the list, but I was wondering if there was
interest in either of https://github.com/numpy/numpy/pull/5457 (make
array data aligned by default) or
https://github.com/numpy/numpy/pull/5470 (make the array data allocator
configurable)?

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Aligned array allocations

[Antoine Pitrou]

Hello,

In https://github.com/numpy/numpy/issues/5312 there's a request for an
aligned allocator in Numpy (more than the default alignment of the
platform's memory allocator). The reason is that on modern
vectorization instruction sets, a certain alignment is required for
optimal performance (even though unaligned data still works: it's just
that performance is degraded... by how much will depend on the CPU
micro-architecture). For example Intel recommends a 32-byte alignment
for AVX loads and stores.

In https://github.com/numpy/numpy/pull/5457 I have proposed a patch to
wrap the system allocator in an aligned allocator. The proposed scheme
makes the alignment configurable at runtime (through a Python API),
because different platforms may have different desirable alignments,
and it is not reasonable for Numpy to know about them all, nor for
users to recompile Numpy each time they have a different CPU.

By always using an aligned allocator there is some overhead:
- all arrays occupy a bit more memory by a small average amount
  (probably 16 bytes average on a 64-bit machine, for a 16 byte
  guaranteed alignment)
- array resizes can be more expensive in CPU time, when the physical
  start changes and its alignment changes too

There is also a limitation: while the physical start of an array will
always be aligned, this can be defeated when taking a view starting at
a non-zero index.

(note that to take advantage of certain instruction set features such
as AVX, Numpy may need to be compiled with specific compiler flags...
but Numpy's allocations also affect other packages such as Numba which
is able to generate code at runtime)

I would like to know if people are interested in this feature, and if
the proposed approach is acceptable.

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

[Numpy-discussion] Memory allocation of arrays and tracemalloc

[Antoine Pitrou]

Hello,

I see that the PyDataMem_* APIs call malloc()/free()/etc. directly,
instead of going through PyMem_Malloc, etc. This means the memory
allocated by those APIs won't be seen by tracemalloc. Is it deliberate?
Am I missing something?

Regards

Antoine.


___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion