Re: [Numpy-discussion] Advanced indexing: fancy vs. orthogonal
On So, 2015-04-05 at 14:13 +0200, Sebastian Berg wrote: On So, 2015-04-05 at 00:45 -0700, Jaime Fernández del Río wrote: On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río snip A PR it is, #5749 to be precise. I think it has all the bells and whistles: integers, boolean and integer 1-D arrays, slices, ellipsis, and even newaxis, both for getting and setting. No tests yet, so correctness of the implementation is dubious at best. As a small example: Looks neat, I am sure there will be some details. Just a quick thought, I wonder if it might make sense to even introduce a context manager. Not sure how easy it is to make sure that it is thread safe, etc? Also wondering, because while I think that actually changing numpy is probably impossible, I do think we can talk about something like: np.enable_outer_indexing() or along the lines of: from numpy.future import outer_indexing or some such, to do a module wide switch and maybe also allow at some point to make it easier to write code that is compatible between a possible followup such as blaze (or also pandas I guess), that uses incompatible indexing. I have no clue if this is technically feasible, though. The python equivalent would be teaching someone to use: from __future__ import division even though you don't even tell them that python 3 exists ;), just because you like the behaviour more. snip a = np.arange(60).reshape(3, 4, 5) a.ix_ snip Jaime -- (\__/) ( O.o) ( ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing: fancy vs. orthogonal
On So, 2015-04-05 at 00:45 -0700, Jaime Fernández del Río wrote: On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río snip A PR it is, #5749 to be precise. I think it has all the bells and whistles: integers, boolean and integer 1-D arrays, slices, ellipsis, and even newaxis, both for getting and setting. No tests yet, so correctness of the implementation is dubious at best. As a small example: Looks neat, I am sure there will be some details. Just a quick thought, I wonder if it might make sense to even introduce a context manager. Not sure how easy it is to make sure that it is thread safe, etc? If the code is not too difficult, maybe it can even be moved to C. Though I have to think about it, I think currently we parse from first index to last, maybe it would be plausible to parse from last to first so that adding dimensions could be done easily inside the preparation function. The second axis remapping is probably reasonably easy (if, like the first thing, tedious). - Sebastian PS: One side comment about the discussion. I don't think anyone suggests that we should not/do not even consider proposals as such, even if it might looks like that. Not that I can compare, but my guess is that numpy is actually very open (though no idea if it appears like that, too). But also to me it does seem like a lost cause to try to actually change indexing itself. So maybe that does not sound diplomatic, but without a specific reasoning about how the change does not wreak havoc, talking about switching indexing behaviour seems a waste time to me. Please try to surprise me, but until then a = np.arange(60).reshape(3, 4, 5) a.ix_ snip Jaime -- (\__/) ( O.o) ( ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
On Sun, Apr 5, 2015 at 4:37 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote: Hi All, Just to raise the question if these two options should be removed at some point? The current default value for both is 0, so we have separate compilation and relaxed strides checking by default. I still have some small doubts that leaving relaxed strides as default will work out for 1.10, plus we will have to make debugging mode switchable (default off), and abusing the flag with different values for it is probably simplest. So my guess is, we should wait at least one version with it. Agree, I'm thinking one or two release down the road. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
On Apr 5, 2015 3:09 AM, David Cournapeau courn...@gmail.com wrote: On Sat, Apr 4, 2015 at 4:25 AM, Nathaniel Smith n...@pobox.com wrote: IIRC there allegedly exist platforms where separate compilation doesn't work right? I'm happy to get rid of it if no one speaks up to defend such platforms, though, we can always add it back later. One case was for statically linking numpy into the interpreter, but I'm skeptical about how much we should care about that case, since that's already a hacky kind of process and there are simple alternative hacks that could be used to strip the offending symbols. Depends on how much it lets us simplify things, I guess. Would we get to remove all the no-export attributes on everything? No, the whole point of the no-export is to support the separate compilation use case. Oog, on further checking I guess this is still true as long as we are using our heirloom mingw compiler on Windows. AFAIK all other compilers we care about support -fvisibility=hidden or equivalent. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
On So, 2015-04-05 at 07:08 -0600, Charles R Harris wrote: On Sun, Apr 5, 2015 at 4:37 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote: Hi All, Just to raise the question if these two options should be removed at some point? The current default value for both is 0, so we have separate compilation and relaxed strides checking by default. I still have some small doubts that leaving relaxed strides as default will work out for 1.10, plus we will have to make debugging mode switchable (default off), and abusing the flag with different values for it is probably simplest. So my guess is, we should wait at least one version with it. Agree, I'm thinking one or two release down the road. Ah ok, misunderstood it. I suppose it will depend on whether the debug feature of messing up strides will be used. I don't think we will have to support it for disabling relaxed strides at that point though (so maybe a rename makes sense by then). Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote: Hi All, Just to raise the question if these two options should be removed at some point? The current default value for both is 0, so we have separate compilation and relaxed strides checking by default. I still have some small doubts that leaving relaxed strides as default will work out for 1.10, plus we will have to make debugging mode switchable (default off), and abusing the flag with different values for it is probably simplest. So my guess is, we should wait at least one version with it. - Sebastian Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing: fancy vs. orthogonal
On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río jaime.f...@gmail.com wrote: I have an all-Pyhton implementation of an OrthogonalIndexer class, loosely based on Stephan's code plus some axis remapping, that provides all the needed functionality for getting and setting with orthogonal indices. Would those interested rather see it as a gist to play around with, or as a PR adding an orthogonally indexable `.ix_` argument to ndarray? A PR it is, #5749 https://github.com/numpy/numpy/pull/5749 to be precise. I think it has all the bells and whistles: integers, boolean and integer 1-D arrays, slices, ellipsis, and even newaxis, both for getting and setting. No tests yet, so correctness of the implementation is dubious at best. As a small example: a = np.arange(60).reshape(3, 4, 5) a.ix_ numpy.core._indexer.OrthogonalIndexer at 0x1027979d0 a.ix_[[0, 1], :, [True, False, True, False, True]] array([[[ 0, 2, 4], [ 5, 7, 9], [10, 12, 14], [15, 17, 19]], [[20, 22, 24], [25, 27, 29], [30, 32, 34], [35, 37, 39]]]) a.ix_[[0, 1], :, [True, False, True, False, True]] = 0 a array([[[ 0, 1, 0, 3, 0], [ 0, 6, 0, 8, 0], [ 0, 11, 0, 13, 0], [ 0, 16, 0, 18, 0]], [[ 0, 21, 0, 23, 0], [ 0, 26, 0, 28, 0], [ 0, 31, 0, 33, 0], [ 0, 36, 0, 38, 0]], [[40, 41, 42, 43, 44], [45, 46, 47, 48, 49], [50, 51, 52, 53, 54], [55, 56, 57, 58, 59]]]) Jaime -- (\__/) ( O.o) ( ) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING
On Sat, Apr 4, 2015 at 4:25 AM, Nathaniel Smith n...@pobox.com wrote: IIRC there allegedly exist platforms where separate compilation doesn't work right? I'm happy to get rid of it if no one speaks up to defend such platforms, though, we can always add it back later. One case was for statically linking numpy into the interpreter, but I'm skeptical about how much we should care about that case, since that's already a hacky kind of process and there are simple alternative hacks that could be used to strip the offending symbols. Depends on how much it lets us simplify things, I guess. Would we get to remove all the no-export attributes on everything? No, the whole point of the no-export is to support the separate compilation use case. David On Apr 3, 2015 8:01 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Apr 3, 2015 at 9:00 PM, Charles R Harris charlesr.har...@gmail.com wrote: Hi All, Just to raise the question if these two options should be removed at some point? The current default value for both is 0, so we have separate compilation and relaxed strides checking by default. Oops, default value is 1, not 0. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing: fancy vs. orthogonal
On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith n...@pobox.com wrote: On Apr 4, 2015 4:12 AM, Todd toddr...@gmail.com wrote: On Apr 4, 2015 10:54 AM, Nathaniel Smith n...@pobox.com wrote: Core python broke backcompat on a regular basis throughout the python 2 series, and almost certainly will again -- the bar to doing so is *very* high, and they use elaborate mechanisms to ease the way (__future__, etc.), but they do it. A few months ago there was even some serious consideration given to changing py3 bytestring indexing to return bytestrings instead of integers. (Consensus was unsurprisingly that this was a bad idea, but there were core devs seriously exploring it, and no-one complained about the optics.) There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
