Re: [Numpy-discussion] PyData Barcelona this May

"Taking numpy in stride, and the essential role of 0" ;-) -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Move scipy.org docs to Github?

Astropy uses readthedocs quite happily (auto-updates on merges to master too). -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] __numpy_ufunc__

HI Stephan, Indeed, `__array_ufunc__` is None would be for classes that interact with arrays only as if they were any other numeric type, and thus have no use for ufuncs, but may need normal operations (astropy's `Unit` class is a reasonably good example). Your example also makes clear that,

Re: [Numpy-discussion] __numpy_ufunc__

Hi All, I'd very much like to get `__array_ufunc__` in, and am willing to do some work, but fear we need to get past the last sticking point. As I noted in Chuck's PR [1], in python 3.6 there is now an explicit language change [2], which I think is relevant: ``` It is now possible to set a

Re: [Numpy-discussion] Could we simplify backporting?

Hi Ralf, Yes, good to think about other policies. For astropy, we do the decision by labelling with the bug-fix branch (with a policy that it really should fix a bug), and inserting text in that release's bug-fix notes (we really should automate that part...). Then, backporting is done shortly

[Numpy-discussion] Could we simplify backporting?

Hi All, In gh-8594, a question came up how to mark things that should be backported and Chuck commented [1]: > Our backport policy is still somewhat ad hoc, especially as I the only one > who has been doing release. What I currently do is set the milestone to the > earlier version, so I will

Re: [Numpy-discussion] ANN: NumExpr3 Alpha

Hi All, Just a side note that at a smaller scale some of the benefits of numexpr are coming to numpy: Julian Taylor has been working on identifying temporary arrays in https://github.com/numpy/numpy/pull/7997. Julian also commented (https://github.com/numpy/numpy/pull/7997#issuecomment-246118772)

Re: [Numpy-discussion] Proposal to support __format__

Hi Gustav, This is great! A few quick comments (mostly echo-ing Stephan's). 1. You basically have a NEP already! Making a PR from it allows to give line-by-line comments, so would help! 2. Don't worry about supporting python2 specifics; just try to ensure it doesn't break; I would not say more

Re: [Numpy-discussion] Deprecating matrices.

Hi All, It seems there are two steps that can be taken now and are needed no matter what: 1. Add numpy documentation describing the preferred way to handle matrices, extolling the virtues of @, and move np.matrix documentation to a deprecated section 2. Start on a new `sparse` class that is

Re: [Numpy-discussion] subclassing ndarray and keeping same ufunc behavior

Hi Stuart, It certainly seems correct behaviour to return the subclass you created: after all, you might want to keep the information on `columns` (e.g., consider doing nanmin along a given axis). Indeed, we certainly want to keep the unit in astropy's Quantity (which also is a subclass of

Re: [Numpy-discussion] __numpy_ufunc__

Hi Chuck, > We were pretty close. IIRC, the outstanding issue was some sort of override. Correct. With a general sentiment of those downstream that it would be great to merge in any form, as it will be really helpful! (Generic speedup of factor of 2 for computationally expensive ufuncs (sin,

Re: [Numpy-discussion] __numpy_ufunc__

Hi Chuck, I've revived my Quantity PRs that use __numpy_ufunc__ but is it correct that at present in *dev, one cannot use it? All the best, Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] __numpy_ufunc__

> The __numpy_ufunc__ functionality is the last bit I want for 1.12.0, the > rest of the remaining changes I can kick forward to 1.13.0. I will start > taking a look tomorrow, probably starting with Nathaniel's work. Great! I'll revive the Quantity PRs that implement __numpy_ufunc__! -- Marten

Re: [Numpy-discussion] padding options for diff

Matthew has made what looks like a very nice implementation of padding in np.diff in https://github.com/numpy/numpy/pull/8206. I raised two general questions about desired behaviour there that Matthew thought we should put out on the mailiing list as well. This indeed seemed a good opportunity to

Re: [Numpy-discussion] assert_allclose equal_nan default value.

Good, that means I can revert some changes to astropy, which made the tests less readable. -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Integers to negative integer powers, time for a decision.

I still strongly favour ending up at int**int -> float, and like Peter's suggestion of raising a general warning rather than an exception for negative powers. -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] automatically avoiding temporary arrays

Note that numpy does store some larger arrays already, in the fft module. (In fact, this was a cache of unlimited size until #7686.) It might not be bad if the same cache were used more generally. That said, if newer versions of python are offering ways of doing this better, maybe that is the

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

There remains the option to just let subclasses deal with new ndarray features... Certainly, for `Quantity`, I'll quite happily do that. And if it alllows the ndarray code to remain simple and efficient, it is probably the best solution. -- Marten ___

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

In a separate message, since perhaps a little less looney: yet another option would be work by analogy with np.ix_ and define pre-dispatch index preparation routines np.ox_ and np.vx_ (say), which would be used as in: ``` array[np.ox_[:, 10]] -- or -- array[np.vx_[:, 10]] ``` This could work if

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

I'd love to solve it with `__getitem__`... Since most subclasses will have that defined that with just a single argument, calling it from `oindex` with an extra mode argument set will properly fail, so good in that sense (although one better ensure a useful error message...). Another option would

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

p.s. Just to be clear: personally, I think we should have neither `__numpy_getitem__` nor a mixin; we should just get the quite wonderful new indexing methods! ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

Hi Nathan, The question originally posed is whether `ndarray` should provide that single method as a convenience already, even though it doesn't actually use it itself. Do you think that is useful, i.e., a big advantage over overwriting the new oindex, vindex, and another that I forget? My own

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

Actually, on those names: an alternative to your proposal would be to introduce only one new method which can do all types of indexing, depending on a keyword argument, i.e., something like ``` def getitem(self, item, mode='outer'): ... ``` -- Marten

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

Hi Sebastian, It would seem to me that any subclass has to keep up to date with new features in ndarray, and while I think ndarray has a responsibility not to break backward compatibility, I do not think it has to protect against new features possibly not working as expected in subclasses. In

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

Hi Sebastian, Indeed, having the scalar pass through `__array_wrap__` would have been useful (_finalize__ is too late, since one cannot change the class any more, just set attributes). But that is water under the bridge, since we're stuck with people not expecting that. I think the slightly

Re: [Numpy-discussion] New Indexing Methods Revival #N (subclasses!)

Hi Sebastian, I haven't given this as much thought as it deserves, but thought I would comment from the astropy perspective, where we both have direct subclasses of `ndarray` (`Quantity`, `Column`, `MaskedColumn`) and classes that store their data internally as ndarray (subclass) instances

Re: [Numpy-discussion] Views and Increments

Hi Anakim, The difference is really in the code path that gets taken: in the first case, you go through `a.__getitem__(np.array([1,6,5])`, in the second through `a.__setitem__(...)`. The increments would not work if you added an extra indexing to it, as in: ``` a[np.array([1,6,5])][:] += 1 ```

Re: [Numpy-discussion] Python 3 dict support (issue #5718)

I know it is slightly obnoxious to hold the "making a suggestion is to volunteer for it" -- but usually a PR to the docs is best made by someone who is trying to understand it rather than someone who already knows everything -- Marten ___

Re: [Numpy-discussion] Added atleast_nd, request for clarification/cleanup of atleast_3d

Hi All, I'm with Nathaniel here, in that I don't really see the point of these routines in the first place: broadcasting takes care of many of the initial use cases one might think of, and others are generally not all that well served by them: the examples from scipy to me do not really support

Re: [Numpy-discussion] Support of '@='?

> > Just if you are curious why it is an error at the moment. We can't have > it be filled in by python to be not in-place (`M = M @ P` meaning), but > copying over the result is a bit annoying and nobody was quite sure > about it, so it was delayed. The problem with using out in-place is clear

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

Hi All, ​I think we're getting a little off the rails, perhaps because two questions are being conflated: 1. What in principle is the best return type for int ** int (which Josef I think most properly rephrased as whether `**` should be thought of as a float operator, like `/` in python3 and

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

I do think one of the main arguments for returning float remains the analogy with division. I don't know about the rest of you, but it has been such a relief not to have to tell students any more "you should add a ".", otherwise it does integer division". For most purposes, it simply shouldn't

Re: [Numpy-discussion] ENH: compute many inner products quickly

There I was thinking vector-vector inner product was in fact covered by `np.inner`. Yikes, half inner, half outer. As for names, I think `matvecmul` and `vecmul` do seem quite OK (probably need `vecmatmul` as well, which does the same as `matmul` would for 1-D first argument). But as other

Re: [Numpy-discussion] Integers to integer powers, let's make a decision

Hi Chuck, I consider either proposal an improvement, but among the two I favour returning float for `**`, because, like for `/`, it ensures one gets closest to the (mathematically) true answer in most cases, and makes duck-typing that much easier -- I'd like to be able to do x** y without having

Re: [Numpy-discussion] Changing FFT cache to a bounded LRU cache

> > A lot of us use NumPy linked with MKL or Accelerate, both of which have > some really nifty FFTs. And the license issue is hardly any worse than > linking with them for BLAS and LAPACK, which we do anyway. We could extend > numpy.fft to use MKL or Accelerate when they are available. > That

Re: [Numpy-discussion] Changing FFT cache to a bounded LRU cache

Hi, I did a few simple timing tests (see comment in PR), which suggests it is hardly worth having the cache. Indeed, if one really worries about speed, one should probably use pyFFTW (scipy.fft is a bit faster too, but at least for me the way real FFT values are stored is just too inconvenient).

Re: [Numpy-discussion] Integers to integer powers

The error on int ** (-int) is OK with me too (even though I prefer just returning floats). Having a `floor_pow` function may still be good with this solution too. -- Marten ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] Behavior of .reduceat()

Hi Jaime, Very belated reply, but only with the semester over I seem to have regained some time to think. The behaviour of reduceat always has seemed a bit odd to me, logical for dividing up an array into irregular but contiguous pieces, but illogical for more random ones (where one effectively

Re: [Numpy-discussion] Integers to integer powers

Hi All, As a final desired state, always returning float seems the best idea. It seems quite similar to division in this regard, where integer division works for some values, but not for all. This means not being quite consistent with python, but as Nathan pointed out, one cannot have

Re: [Numpy-discussion] Numpy 1.11.0b2 released

Hi Julian, While the numpy 1.10 situation was bad, I do want to clarify that the problems we had in astropy were a consequence of *good* changes in `recarray`, which solved many problems, but also broke the work-arounds that had been created in `astropy.io.fits` quite a long time ago (possibly

Re: [Numpy-discussion] Behavior of np.random.uniform

For what it is worth, the current behaviour seems the most logical to me, i.e., that the first limit is always the one that is included in the interval, and the second is not. -- Marten ​ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org

Re: [Numpy-discussion] Get rid of special scalar arithmetic

Just thought I would add here a general comment I made in the thread: replacing scalars everywhere with array scalars (i.e., ndim=0) would be great also from the perspective of ndarray subclasses; as is, it is quite annoying to have to special-case, e.g., getting a single subclass element, and

[Numpy-discussion] Subclass awareness of outer, inner, dot, ...

Hi All, Having just had a look at functions where astropy's quantities are silently converted to ndarray, I came across some that, in principle, are easy to solve, yet for which, as always, there is a worry about backward compatibility. So, the question arises what to do. As a specific example,

Re: [Numpy-discussion] 1.10.3 release tomorrow, 1.11.x branch this month.

Hi Chuck, others, A propos __numpy_ufunc__, what is the current status? Is it still the undetermined result of the monster-thread ( https://github.com/numpy/numpy/issues/5844 -- just found it again by sorting by number of comments...)? As noted by Stephan and myself when the decision was made to

Re: [Numpy-discussion] Fast vectorized arithmetic with ~32 significant digits under Numpy

Hi All, astropy `Time` indeed using two doubles internally, but is very limited in the operations it allows: essentially only addition/subtraction, and multiplication with/division by a normal double. It would be great to have better support within numpy; it is a pity to have a float128 type

Re: [Numpy-discussion] deprecate fromstring() for text reading?

I think it would be good to keep the usage to read binary data at least. Or is there a good alternative to `np.fromstring(, dtype=...)`? -- Marten On Thu, Oct 22, 2015 at 1:03 PM, Chris Barker wrote: > There was just a question about a bug/issue with scipy.fromstring

Re: [Numpy-discussion] Making datetime64 timezone naive

> > > However, numpy datetime is optimized for compact storage and fast > computation of absolute deltas (actual hours, minutes, seconds... not > calendar units like "the next day" ). > > Except that ironically it actually can't compute absolute deltas > accurately with one second resolution,

Re: [Numpy-discussion] method to calculate the magnitude squared

Hi Nils, I think performance will actually be better than I indicated, especially for larger arrays, since `r.real**2 + r.imag**2` makes a quite unnecessary intermediate arrays. With a `ufunc`, this can be done much faster. Indeed, it should be no slower than `np.square` (which does more

Re: [Numpy-discussion] composition of the steering council (was Re: Governance model request)

Hi Nathaniel, Piping in again from the outside: being on council would certainly seem to be a job not a privilege, and I suspect it will be hard enough to find people willing to actually put in work to worry about restricting membership overly. Given this, my suggestion would be to have a

Re: [Numpy-discussion] Governance model request

Hi All, I've been reading this thread with amazement and a bit of worry. It seems Nathaniel's proposal is clearly an improvement, even if it is not perfect. But it is in the end for a project where, at least as seen from the outside, the main challenge is not in governance, but rather in having

Re: [Numpy-discussion] method to calculate the magnitude squared

> > Is that not the same as > np.abs(z)**2 ? > It is, but since that involves taking sqrt, it is *much* slower. Even now, ``` In [32]: r = np.arange(1)*(1+1j) In [33]: %timeit np.abs(r)**2 1000 loops, best of 3: 213 µs per loop In [34]: %timeit r.real**2 + r.imag**2 1 loops, best of

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

Hi Nathaniel, Thanks for the detailed reply; it helped a lot to understand how one could, indeed, have dtypes contain units. And if one had not just on-the-fly conversion from int to float as part of an internal loop, but also on-the-fly multiplication, then it would even be remarkably fast. Will

Re: [Numpy-discussion] Notes from the numpy dev meeting at scipy 2015

Hi Nathaniel, others, I read the discussion of plans with interest. One item that struck me is that while there are great plans to have a proper extensible and presumably subclassable dtype, it is discouraged to subclass ndarray itself (rather, it is encouraged to use a broader array interface).

Re: [Numpy-discussion] Changes to np.digitize since NumPy 1.9?

For what it's worth, also from my astropy perspective I think hat any index array should be a base ndarray! -- Marten On Fri, Aug 14, 2015 at 7:11 AM, Jaime Fernández del Río jaime.f...@gmail.com wrote: On Thu, Aug 13, 2015 at 9:57 AM, Jaime Fernández del Río jaime.f...@gmail.com wrote: On

Re: [Numpy-discussion] Bug in np.nonzero / Should index returning functions return ndarray subclasses?

Agreed that indexing functions should return bare `ndarray`. Note that in Jaime's PR one can override it anyway by defining __nonzero__. -- Marten On Sat, May 9, 2015 at 9:53 PM, Stephan Hoyer sho...@gmail.com wrote: With regards to np.where -- shouldn't where be a ufunc, so subclasses or