Re: [Numpy-discussion] Advanced indexing: "fancy" vs. orthogonal

[Sebastian Berg]

On So, 2015-04-05 at 14:13 +0200, Sebastian Berg wrote:
> On So, 2015-04-05 at 00:45 -0700, Jaime Fernández del Río wrote:
> > On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río 
> 
> > 
> > 
> > A PR it is, #5749 to be precise. I think it has all the bells and
> > whistles: integers, boolean and integer 1-D arrays, slices, ellipsis,
> > and even newaxis, both for getting and setting. No tests yet, so
> > correctness of the implementation is dubious at best. As a small
> > example:
> > 
> 
> Looks neat, I am sure there will be some details. Just a quick thought,
> I wonder if it might make sense to even introduce a context manager. Not
> sure how easy it is to make sure that it is thread safe, etc?

Also wondering, because while I think that actually changing numpy is
probably impossible, I do think we can talk about something like:

np.enable_outer_indexing()
or along the lines of:
from numpy.future import outer_indexing

or some such, to do a module wide switch and maybe also allow at some
point to make it easier to write code that is compatible between a
possible followup such as blaze (or also pandas I guess), that uses
incompatible indexing.
I have no clue if this is technically feasible, though.

The python equivalent would be teaching someone to use:

from __future__ import division

even though you don't even tell them that python 3 exists ;), just
because you like the behaviour more.


> 

> > >>> a = np.arange(60).reshape(3, 4, 5)
> > >>> a.ix_
> 
> > 
> > Jaime
> > 
> > 
> > -- 
> > (\__/)
> > ( O.o)
> > ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
> > planes de dominación mundial.
> > ___
> > NumPy-Discussion mailing list
> > NumPy-Discussion@scipy.org
> > http://mail.scipy.org/mailman/listinfo/numpy-discussion
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[Sebastian Berg]

On So, 2015-04-05 at 07:08 -0600, Charles R Harris wrote:
> 
> 
> On Sun, Apr 5, 2015 at 4:37 AM, Sebastian Berg
>  wrote:
> On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote:
> > Hi All,
> >
> >
> > Just to raise the question if these two options should be
> removed at
> > some point? The current default value for both is 0, so we
> have
> > separate compilation and relaxed strides checking by
> default.
> >
> 
> 
> I still have some small doubts that leaving relaxed strides as
> default
> will work out for 1.10, plus we will have to make "debugging
> mode"
> switchable (default off), and abusing the flag with different
> values for
> it is probably simplest.
> So my guess is, we should wait at least one version with it.
> 
> 
> Agree, I'm thinking one or two release down the road.
> 

Ah ok, misunderstood it. I suppose it will depend on whether the debug
feature of messing up strides will be used. I don't think we will have
to support it for disabling relaxed strides at that point though (so
maybe a rename makes sense by then).

> 
> Chuck 
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[Charles R Harris]

On Sun, Apr 5, 2015 at 4:37 AM, Sebastian Berg 
wrote:

> On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote:
> > Hi All,
> >
> >
> > Just to raise the question if these two options should be removed at
> > some point? The current default value for both is 0, so we have
> > separate compilation and relaxed strides checking by default.
> >
>
> I still have some small doubts that leaving relaxed strides as default
> will work out for 1.10, plus we will have to make "debugging mode"
> switchable (default off), and abusing the flag with different values for
> it is probably simplest.
> So my guess is, we should wait at least one version with it.
>

Agree, I'm thinking one or two release down the road.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Advanced indexing: "fancy" vs. orthogonal

[Sebastian Berg]

On So, 2015-04-05 at 00:45 -0700, Jaime Fernández del Río wrote:
> On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río 

> 
> 
> A PR it is, #5749 to be precise. I think it has all the bells and
> whistles: integers, boolean and integer 1-D arrays, slices, ellipsis,
> and even newaxis, both for getting and setting. No tests yet, so
> correctness of the implementation is dubious at best. As a small
> example:
> 

Looks neat, I am sure there will be some details. Just a quick thought,
I wonder if it might make sense to even introduce a context manager. Not
sure how easy it is to make sure that it is thread safe, etc?

If the code is not too difficult, maybe it can even be moved to C.
Though I have to think about it, I think currently we parse from first
index to last, maybe it would be plausible to parse from last to first
so that adding dimensions could be done easily inside the preparation
function. The second axis remapping is probably reasonably easy (if,
like the first thing, tedious).

- Sebastian


PS: One side comment about the discussion. I don't think anyone suggests
that we should not/do not even consider proposals as such, even if it
might looks like that. Not that I can compare, but my guess is that
numpy is actually very open (though no idea if it appears like that,
too).

But also to me it does seem like a lost cause to try to actually change
indexing itself. So maybe that does not sound diplomatic, but without a
specific reasoning about how the change does not wreak havoc, talking
about switching indexing behaviour seems a waste time to me. Please try
to surprise me, but until then


> 
> >>> a = np.arange(60).reshape(3, 4, 5)
> >>> a.ix_

> 
> Jaime
> 
> 
> -- 
> (\__/)
> ( O.o)
> ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus
> planes de dominación mundial.
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[Nathaniel Smith]

On Apr 5, 2015 3:09 AM, "David Cournapeau"  wrote:
>
> On Sat, Apr 4, 2015 at 4:25 AM, Nathaniel Smith  wrote:
>>
>> IIRC there allegedly exist platforms where separate compilation doesn't
work right? I'm happy to get rid of it if no one speaks up to defend such
platforms, though, we can always add it back later. One case was for
statically linking numpy into the interpreter, but I'm skeptical about how
much we should care about that case, since that's already a hacky kind of
process and there are simple alternative hacks that could be used to strip
the offending symbols.
>>
>> Depends on how much it lets us simplify things, I guess. Would we get to
remove all the no-export attributes on everything?
>
>
> No, the whole point of the no-export is to support the separate
compilation use case.
>

Oog, on further checking I guess this is still true as long as we are using
our heirloom mingw compiler on Windows. AFAIK all other compilers we care
about support -fvisibility=hidden or equivalent.

-n
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Advanced indexing: "fancy" vs. orthogonal

[Robert Kern]

On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith  wrote:
>
> On Apr 4, 2015 4:12 AM, "Todd"  wrote:
> >
> >
> > On Apr 4, 2015 10:54 AM, "Nathaniel Smith"  wrote:
> > >
> > > Core python broke backcompat on a regular basis throughout the python
> > > 2 series, and almost certainly will again -- the bar to doing so is
> > > *very* high, and they use elaborate mechanisms to ease the way
> > > (__future__, etc.), but they do it. A few months ago there was even
> > > some serious consideration given to changing py3 bytestring indexing
> > > to return bytestrings instead of integers. (Consensus was
> > > unsurprisingly that this was a bad idea, but there were core devs
> > > seriously exploring it, and no-one complained about the optics.)
> >
> > There was no break as large as this. In fact I would say this is even a
larger change than any individual change we saw in the python 2 to 3
switch.  The basic mechanics of indexing are just too fundamental and touch
on too many things to make this sort of change feasible.
>
> I'm afraid I'm not clever enough to know how large or feasible a change
is without even seeing the proposed change.

It doesn't take any cleverness. The change in question was to make the
default indexing semantics to orthogonal indexing. No matter the details of
the ultimate proposal to achieve that end, it has known minimum
consequences, at least in the broad outline. Current documentation and
books become obsolete for a fundamental operation. Current code must be
modified by some step to continue working. These are consequences inherent
in the end, not just the means to the end; we don't need a concrete
proposal in front of us to know what they are. There are ways to mitigate
these consequences, but there are no silver bullets that eliminate them.
And we can compare those consequences to approaches like Jaime's that
achieve a majority of the benefits of such a change without any of the
negative consequences. That comparison does not bode well for any proposal.

--
Robert Kern
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[Sebastian Berg]

On Fr, 2015-04-03 at 21:00 -0600, Charles R Harris wrote:
> Hi All,
> 
> 
> Just to raise the question if these two options should be removed at
> some point? The current default value for both is 0, so we have
> separate compilation and relaxed strides checking by default.
> 

I still have some small doubts that leaving relaxed strides as default
will work out for 1.10, plus we will have to make "debugging mode"
switchable (default off), and abusing the flag with different values for
it is probably simplest.
So my guess is, we should wait at least one version with it.

- Sebastian

> 
> Chuck
> 
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion



signature.asc
Description: This is a digitally signed message part
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] NPY_SEPARATE_COMPILATION and RELAXED_STRIDES_CHECKING

[David Cournapeau]

On Sat, Apr 4, 2015 at 4:25 AM, Nathaniel Smith  wrote:

> IIRC there allegedly exist platforms where separate compilation doesn't
> work right? I'm happy to get rid of it if no one speaks up to defend such
> platforms, though, we can always add it back later. One case was for
> statically linking numpy into the interpreter, but I'm skeptical about how
> much we should care about that case, since that's already a hacky kind of
> process and there are simple alternative hacks that could be used to strip
> the offending symbols.
>
> Depends on how much it lets us simplify things, I guess. Would we get to
> remove all the no-export attributes on everything?
>

No, the whole point of the no-export is to support the separate compilation
use case.

David


> On Apr 3, 2015 8:01 PM, "Charles R Harris" 
> wrote:
>
>>
>>
>> On Fri, Apr 3, 2015 at 9:00 PM, Charles R Harris <
>> charlesr.har...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> Just to raise the question if these two options should be removed at
>>> some point? The current default value for both is 0, so we have separate
>>> compilation and relaxed strides checking by default.
>>>
>>>
>> Oops, default value is 1, not 0.
>>
>> Chuck
>>
>> ___
>> NumPy-Discussion mailing list
>> NumPy-Discussion@scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>>
>>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] Advanced indexing: "fancy" vs. orthogonal

[Jaime Fernández del Río]

On Fri, Apr 3, 2015 at 10:59 AM, Jaime Fernández del Río <
jaime.f...@gmail.com> wrote:

> I have an all-Pyhton implementation of an OrthogonalIndexer class, loosely
> based on Stephan's code plus some axis remapping, that provides all the
> needed functionality for getting and setting with orthogonal indices.
>
> Would those interested rather see it as a gist to play around with, or as
> a PR adding an orthogonally indexable `.ix_` argument to ndarray?
>

A PR it is, #5749  to be precise.
I think it has all the bells and whistles: integers, boolean and integer
1-D arrays, slices, ellipsis, and even newaxis, both for getting and
setting. No tests yet, so correctness of the implementation is dubious at
best. As a small example:

>>> a = np.arange(60).reshape(3, 4, 5)
>>> a.ix_

>>> a.ix_[[0, 1], :, [True, False, True, False, True]]
array([[[ 0,  2,  4],
[ 5,  7,  9],
[10, 12, 14],
[15, 17, 19]],

   [[20, 22, 24],
[25, 27, 29],
[30, 32, 34],
[35, 37, 39]]])
>>> a.ix_[[0, 1], :, [True, False, True, False, True]] = 0
>>> a
array([[[ 0,  1,  0,  3,  0],
[ 0,  6,  0,  8,  0],
[ 0, 11,  0, 13,  0],
[ 0, 16,  0, 18,  0]],

   [[ 0, 21,  0, 23,  0],
[ 0, 26,  0, 28,  0],
[ 0, 31,  0, 33,  0],
[ 0, 36,  0, 38,  0]],

   [[40, 41, 42, 43, 44],
[45, 46, 47, 48, 49],
[50, 51, 52, 53, 54],
[55, 56, 57, 58, 59]]])

Jaime

-- 
(\__/)
( O.o)
( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes
de dominación mundial.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion