Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 10:20 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Apr 8, 2015 at 8:40 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: To address in detail the list of Matthew you mention above: * implement orthogonal indexing as a method arr.sensible_index[...] That's basically Jaime's PR. * implement the current non-boolean fancy indexing behavior as a method - arr.crazy_index[...] Not that harmful, but only makes sense in combination with the next steps. Well, since we got the peanut butter in our meta, I might as well join in here. Yeah, sorry about that. I don't even like peanut butter. I think this step is useful even without the deprecation steps. Your arguments make sense. leaves for non-meta land Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 2:09 PM, Alan G Isaac alan.is...@gmail.com wrote: That analogy fails because it suggests a private conversation. This list is extremely public. For example, I am just a user, and I am on it. I can tell you that as a long-time numpy user my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM and then (ii) take a breath; this too will fade away. It is very reasonable to worry that some users will start at the same place but them move in a different direction, and that worry should affect how such proposals are floated and discussed. I am personally grateful that the idea's reception has been so chilly; it's very reassuring. Thanks, this is really useful feedback. I can totally understand that panic flare and it sucks. Do you think there's anything we could be doing to reduce this kind of adrenaline reaction while still allowing for relaxed discussion about out-there ideas? In my mind the relaxed part is actually a huge part of the goal: reaching the point where everyone can be confident that their voice will be heard, etc., so that things become less fraught and it becomes easier to focus on ideas. -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Apr 5, 2015 7:04 AM, Robert Kern robert.k...@gmail.com wrote: On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith n...@pobox.com wrote: On Apr 4, 2015 4:12 AM, Todd toddr...@gmail.com wrote: There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of you that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the majority of benefits of such a change is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. Plus, I dunno, our current approach to discussions just seems to make things hostile and shouty and unpleasant. If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? I think things will work better if the next time something like this comes up, *one* person just says hmm, interesting idea, but the backcompat issues seem pretty severe; do you have any ideas about how to mitigate that?, and then we let that point be taken as having been made and see where the discussion goes. Maybe we can all give it a try? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimensional Indexing
I think the rationale is to allow selection of whole rows / columns. If you want to choose a single element from each row/column, then, yes, you have to pass np.arange(...). There is also np.choose function, but not recommended to use for such cases as far as I understand. I'm not an expert, though. Nikolay. From: misno...@gmail.com Date: Tue, 7 Apr 2015 00:49:34 +0100 To: numpy-discussion@scipy.org Subject: [Numpy-discussion] Multidimensional Indexing With the indexing example from the documentation: y = np.arange(35).reshape(5,7) Why does selecting an item from explicitly every row work as I’d expect: y[np.array([0,1,2,3,4]),np.array([0,0,0,0,0])] array([ 0, 7, 14, 21, 28]) But doing so from a full slice (which, I would naively expect to mean “Every Row”) has some…other… behaviour: y[:,np.array([0,0,0,0,0])] array([[ 0, 0, 0, 0, 0], [ 7, 7, 7, 7, 7], [14, 14, 14, 14, 14], [21, 21, 21, 21, 21], [28, 28, 28, 28, 28]]) What is going on in this example, and how do I get what I expect? By explicitly passing in an extra array with value===index? What is the rationale for this difference in behaviour? Thanks, Nick ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 2:06 AM, Nathaniel Smith n...@pobox.com wrote: On Apr 5, 2015 7:04 AM, Robert Kern robert.k...@gmail.com wrote: On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith n...@pobox.com wrote: On Apr 4, 2015 4:12 AM, Todd toddr...@gmail.com wrote: There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of you that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the majority of benefits of such a change is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. And we can get all of that discussion from discussing Jaime's proposal. I would argue that we will get better, more focused discussion from it since it is actually a concrete proposal and not just a wish that numpy's indexing semantics were something else. I think that a full airing and elaboration of Jaime's proposal (as the final PR should look quite different than the initial one to incorporate the what is found in the discussion) will give us a satisficing solution. I certainly think that that is *more likely* to arrive at a satisficing solution than an attempt to change the default indexing semantics. I can name specific improvements that would specifically address the concerns you named if you would like. Maybe it won't be *quite* as good (for some parties) than if Numeric chose orthogonal indexing from the get-go, but it will likely be much better for everyone than if numpy broke backward compatibility on this feature now. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. That is not the issue. Ralf asked you not to invite such PRs in the first place. No one thinks that such a PR would get snuck in.
[Numpy-discussion] Research position in the Brazilian Research Institute for Science and Neurotechnology - BRAINN
Research position in the Brazilian Research Institute for Science and Neurotechnology – BRAINN Postdoc researcher to work with software development for medical imaging The Brazilian Research Institute for Neuroscience and Neurotechnology (BRAINN) (www.brainn.org.br) focuses on the investigation of basic mechanisms leading to epilepsy and stroke, and the injury mechanisms that follow disease onset and progression. This research has important applications related to prevention, diagnosis, treatment and rehabilitation and will serve as a model for better understanding normal and abnormal brain function. The BRAINN Institute is composed of 10 institutions from Brazil and abroad and hosted by State University of Campinas (UNICAMP). Among the associated institutions is Renato Archer Information Technology Center (CTI) that has a specialized team in open-source software development for medical imaging (www.cti.gov.br/invesalius) and 3D printing applications for healthcare. CTI is located close the UNICAMP in the city of Campinas, State of São Paulo in a very technological region of Brazil and is looking for a postdoc researcher to work with software development for medical imaging related to the imaging analysis, diagnosis and treatment of brain diseases. The postdoc position is for two years with the possibility of being renovated for more two years. Education - PhD in computer science, computer engineering, mathematics, physics or related. Requirements - Digital image processing (Medical imaging) - Computer graphics (basic) Benefits 6.143,40 Reais per month free of taxes (about US$ 2.000,00); 15% technical reserve for conferences participation and specific materials acquisition; Interested Send curriculum to: jorge.si...@cti.gov.br with subject “Postdoc position” Applications reviews will begin April 30, 2015 and continue until the position is filled. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 1:38 PM, Robert Kern robert.k...@gmail.com wrote: On Wed, Apr 8, 2015 at 2:06 AM, Nathaniel Smith n...@pobox.com wrote: On Apr 5, 2015 7:04 AM, Robert Kern robert.k...@gmail.com wrote: On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith n...@pobox.com wrote: On Apr 4, 2015 4:12 AM, Todd toddr...@gmail.com wrote: There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of you that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the majority of benefits of such a change is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. And we can get all of that discussion from discussing Jaime's proposal. I would argue that we will get better, more focused discussion from it since it is actually a concrete proposal and not just a wish that numpy's indexing semantics were something else. I think that a full airing and elaboration of Jaime's proposal (as the final PR should look quite different than the initial one to incorporate the what is found in the discussion) will give us a satisficing solution. I certainly think that that is *more likely* to arrive at a satisficing solution than an attempt to change the default indexing semantics. I can name specific improvements that would specifically address the concerns you named if you would like. Maybe it won't be *quite* as good (for some parties) than if Numeric chose orthogonal indexing from the get-go, but it will likely be much better for everyone than if numpy broke backward compatibility on this feature now. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one
Re: [Numpy-discussion] FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
On Apr 8, 2015 2:16 PM, Andreas Hilboll li...@hilboll.de wrote: Hi all, I'm commonly using function signatures like def myfunc(a, b, c=None): if c is None: # do something ... ... where c is an optional array argument. For some time now, I'm getting a FutureWarning: comparison to `None` will result in an elementwise object comparison in the future from the c is None comparison. I'm wondering what would be the best way to do this check in a future-proof way? As far as I know, you should be getting the warning when you write c == None and the fix should be to write c is None instead. (And this is definitely an important fix -- it's basically a bug in numpy that the == form ever worked.) Are you certain that you're getting warnings from 'c is None'? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimensional Indexing
On Mon, Apr 6, 2015 at 4:49 PM, Nicholas Devenish misno...@gmail.com wrote: With the indexing example from the documentation: y = np.arange(35).reshape(5,7) Why does selecting an item from explicitly every row work as I’d expect: y[np.array([0,1,2,3,4]),np.array([0,0,0,0,0])] array([ 0, 7, 14, 21, 28]) But doing so from a full slice (which, I would naively expect to mean “Every Row”) has some…other… behaviour: y[:,np.array([0,0,0,0,0])] array([[ 0, 0, 0, 0, 0], [ 7, 7, 7, 7, 7], [14, 14, 14, 14, 14], [21, 21, 21, 21, 21], [28, 28, 28, 28, 28]]) What is going on in this example, and how do I get what I expect? By explicitly passing in an extra array with value===index? What is the rationale for this difference in behaviour? To understand this example, it is important to understand that for multi-dimensional arrays, Numpy attempts to make the index array along each dimension the same size, using broadcasting. So in your original example, y[np.array([0,1,2,3,4]),np.array([0,0,0,0,0])], the arrays are the same size, and the behavior is as you'd expect. In the second case, the first index is a slice, and the second index is an array. Documentation for this case can be found in the indexing docs under Combining index arrays with slices. Here's the relevant portion: In effect, the slice is converted to an [new] index array ... that is broadcast with the [other] index array So in your case, the slice : is *first* being converted to np.arange(5), *then* is broadcast across the shape of the [other] index array so that it is ultimately transformed into something like np.repeat(np.arange(5)[:,np.newaxis], 5, axis=1), giving you: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) Now at this point you have converted your slice to an [new] index array of shape (5,5), and your [other] index array is shaped (5,). So now numpy applies broadcasting rules to the second array to get it into shape 5. This operation is identical to what just occurred, so your [other] index array *also* looks like: array([[0, 0, 0, 0, 0], [1, 1, 1, 1, 1], [2, 2, 2, 2, 2], [3, 3, 3, 3, 3], [4, 4, 4, 4, 4]]) Which then gives the result you saw. Now, you may say: once the slice was converted to np.arange(5), why was it then broadcast to shape (5,5) rather than kept at shape (5,) which would work. The reason (I suspect at least) is to keep it consistent with other types of slices. Consider if you did something like: y[1:3, np.array([0,0,0,0,0])] Then the same operation would apply as above, except that when the slice was converted to an array, it would be converted to np.arange(1,3) which has shape (2,). Obviously this isn't compatible with the second index array of shape (5,), so it *has* to be broadcast. One final note: in this case, you can instead use either of the following: y[np.array([0,1,2,3,4]), 0] or y[:, 0] using the same steps above, the slice is converted to an np.arange(5), and then the shapes are compared, (5,) versus (). Then the integer index is broadcast to shape (5,) which gives you what you want. Hope that helps. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
On 08.04.2015 20:30, Nathaniel Smith wrote: On Apr 8, 2015 2:16 PM, Andreas Hilboll li...@hilboll.de mailto:li...@hilboll.de wrote: Hi all, I'm commonly using function signatures like def myfunc(a, b, c=None): if c is None: # do something ... ... where c is an optional array argument. For some time now, I'm getting a FutureWarning: comparison to `None` will result in an elementwise object comparison in the future from the c is None comparison. I'm wondering what would be the best way to do this check in a future-proof way? As far as I know, you should be getting the warning when you write c == None and the fix should be to write c is None instead. (And this is definitely an important fix -- it's basically a bug in numpy that the == form ever worked.) Are you certain that you're getting warnings from 'c is None'? My mistake; I was actually doing if p1 is None == h1 is None: instead of if (p1 is None) == (h1 is None): Sorry for the noise. -- Andreas. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] IDE's for numpy development?
Spyder supports C. Thanks for correcting this. I wasn't aware of it. How was your experience with it? Best, -m ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On 2015/04/08 8:09 AM, Alan G Isaac wrote: That analogy fails because it suggests a private conversation. This list is extremely public. For example, I am just a user, and I am on it. I can tell you that as a long-time numpy user my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM and then (ii) take a breath; this too will fade away. It is very reasonable to worry that some users will start at the same place but them move in a different direction, and that worry should affect how such proposals are floated and discussed. I am personally grateful that the idea's reception has been so chilly; it's very reassuring. OK, so I was not sufficiently tactful when I tried to illustrate the real practical problem associated with a *core* aspect of numpy. My intent was not to alarm users, and I apologize if I have done so. I'm glad you have been reassured. I know perfectly well that back-compatibility and stability are highly important. What I wanted to do was to stimulate thought about how to handle a serious challenge to numpy's future--short-term, and long-term. Jaime's PR is a very welcome response to that challenge, but it might not be the end of the story. Matthew nicely sketched out one possible scenario, or actually a range of scenarios. Now, can we please get back to consideration of reasonable options? What sequence of steps might reduce the disconnect between numpy and the rest of the array-handling world? And make it a little friendlier for students? Are there *any* changes to indexing, whether by default or as an option, that would help? Consider the example I started with, in which indexing with [1, :, array] gives results that many find surprising and hard to understand. Might it make sense to *slowly* deprecate this? Or are such indexing expressions actually useful? If they are, would it be out of the question to have them *optionally* trigger a warning, so that numpy could be configured to be a little less likely to trip up a non-expert user? Eric fwiw, Alan On 4/7/2015 9:06 PM, Nathaniel Smith wrote: If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Behavior of np.random.multivariate_normal with bad covariance matrices
I like your idea Josef, I'll add it to the PR. Just to be clear, we should have something like: Have a single check_valid keyword arg, which will default to warn, since that is the current behavior. It will check approximate symmetry, PSDness, and for NaN infs. Other options on the check_valid keyword arg will be ignore, and raise. What should happen when fix is passed for check_valid? Set negative eigenvalues to 0 and symmetrize the matrix? On Mon, Mar 30, 2015 at 8:34 AM, josef.p...@gmail.com wrote: On Sun, Mar 29, 2015 at 7:39 PM, Blake Griffith blake.a.griff...@gmail.com wrote: I have an open PR which lets users control the checks on the input covariance matrix. The matrix is required to be symmetric and positve semi-definite (PSD). The current behavior is that NumPy raises a warning if the matrix is not PSD, and does not even check for symmetry. I added a symmetry check, which raises a warning when the input is not symmetric. And added two keyword args which users can use to turn off the checks/warnings when the matrix is ill formed. So this would only cause another new warning to be raised in existing code. This is needed because sometimes the covariance matrix is only *almost* symmetric or PSD due to roundoff error. Thoughts? My only question is why is **exact** symmetry relevant? AFAIU A empirical covariance matrix might not be exactly symmetric unless we specifically force it to be. But I don't see why some roundoff errors that violate symmetry should be relevant. use allclose with floating point rtol or equivalent? Some user code might suddenly get irrelevant warnings. BTW: neg = (np.sum(u.T * v, axis=1) 0) (s 0) doesn't need to be calculated if cov_psd is false. - some more: svd can hang if the values are not finite, i.e. nan or infs counter proposal would be to add a `check_valid` keyword with option ignore. warn, raise, and fix and raise an error if there are nans and check_valid is not ignore. - aside: np.random.multivariate_normal is only relevant if you have a new cov each call (or don't mind repeated possibly expensive calculations), so, I guess, adding checks by default won't upset many users. Josef PR: https://github.com/numpy/numpy/pull/5726 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
That analogy fails because it suggests a private conversation. This list is extremely public. For example, I am just a user, and I am on it. I can tell you that as a long-time numpy user my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM and then (ii) take a breath; this too will fade away. It is very reasonable to worry that some users will start at the same place but them move in a different direction, and that worry should affect how such proposals are floated and discussed. I am personally grateful that the idea's reception has been so chilly; it's very reassuring. fwiw, Alan On 4/7/2015 9:06 PM, Nathaniel Smith wrote: If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
Hi all, I'm commonly using function signatures like def myfunc(a, b, c=None): if c is None: # do something ... ... where c is an optional array argument. For some time now, I'm getting a FutureWarning: comparison to `None` will result in an elementwise object comparison in the future from the c is None comparison. I'm wondering what would be the best way to do this check in a future-proof way? Best, -- Andreas. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] IDE's for numpy development?
I think spyder supports code highlighting in C and that's all... There's no way to compile in Spyder, is there? Shawn On Tue, Apr 7, 2015 at 2:46 AM, Suzen, Mehmet msu...@gmail.com wrote: Spyder supports C. Thanks for correcting this. I wasn't aware of it. How was your experience with it? Best, -m ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Yuxiang Shawn Wang Gerling Research Lab University of Virginia yw...@virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 9:05 PM, Eric Firing efir...@hawaii.edu wrote: On 2015/04/08 8:09 AM, Alan G Isaac wrote: That analogy fails because it suggests a private conversation. This list is extremely public. For example, I am just a user, and I am on it. I can tell you that as a long-time numpy user my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM and then (ii) take a breath; this too will fade away. It is very reasonable to worry that some users will start at the same place but them move in a different direction, and that worry should affect how such proposals are floated and discussed. I am personally grateful that the idea's reception has been so chilly; it's very reassuring. OK, so I was not sufficiently tactful when I tried to illustrate the real practical problem associated with a *core* aspect of numpy. My intent was not to alarm users, and I apologize if I have done so. I'm glad you have been reassured. I know perfectly well that back-compatibility and stability are highly important. What I wanted to do was to stimulate thought about how to handle a serious challenge to numpy's future--short-term, and long-term. Jaime's PR is a very welcome response to that challenge, but it might not be the end of the story. Matthew nicely sketched out one possible scenario, or actually a range of scenarios. Now, can we please get back to consideration of reasonable options? Well, in many people's definition of reasonable, that's basically Jaime's proposal and maybe the original __orthogonal_indexing__ one. Those both have a chance of actually being implemented, and presumably the original proposers have a use for the latter. Their proposal is not being discussed; instead that potentially useful discussion is being completely derailed by insisting on wanting to talk about changes to numpy's indexing behavior. To address in detail the list of Matthew you mention above: * implement orthogonal indexing as a method arr.sensible_index[...] That's basically Jaime's PR. * implement the current non-boolean fancy indexing behavior as a method - arr.crazy_index[...] Not that harmful, but only makes sense in combination with the next steps. * deprecate non-boolean fancy indexing as standard arr[...] indexing; No, see negative reaction by many people. * wait a long time; * remove non-boolean fancy indexing as standard arr[...] (errors are preferable to change in behavior) Most definitely no. Ralf P.S. random thought: maybe we should have a numpy-ideas list, just like python-dev has python-ideas ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 8:05 PM, Eric Firing efir...@hawaii.edu wrote: Now, can we please get back to consideration of reasonable options? Sure, but I recommend going back to the actually topical thread (or a new one), as this one is meta. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On 2015/04/08 9:40 AM, Ralf Gommers wrote: Their proposal is not being discussed; instead that potentially useful discussion is being completely derailed by insisting on wanting to talk about changes to numpy's indexing behavior. Good point. That was an unintended consequence of my message. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
1. I use numpy in teaching. I have never heard a complaint about its indexing behavior. Have you heard such complaints? 2. One reason I use numpy in teaching is its indexing behavior. What specific language provides a better indexing model, in your opinion? 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? Cheers, Alan On 4/8/2015 3:05 PM, Eric Firing wrote: What sequence of steps might reduce the disconnect between numpy and the rest of the array-handling world? And make it a little friendlier for students? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On 2015/04/08 10:02 AM, Alan G Isaac wrote: 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? Yes, one attempted to, essentially by accident. That was in my original message. Please refer back to that. The earlier part of this thread, under its original name, is also relevant to your other questions. I'm not going to discuss this further. The thread is now closed as far as I am concerned. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
Trying to bring the meta back into this thread (sorry for Robert's PB :)... The only thing I'd like to add, is that it's perhaps worth messaging that: a PR is just (as the Github folks like to say) a conversation based on code. It is NOT necessarily something intended explicitly for merging. In IPython, we do sometimes create PRs that we mark explicitly as *Not for merging*, just so that we can play with a concrete implementation of an idea, even when we know in advance it's not going to go in. But it may be a useful way to explore the problem with code everyone can easily grab and run, to have a thread of discussion right there next to the code, to evolve the code together with the discussions as insights arise, and to finally document anything learned, all in one place. So, with a bit of messaging, encouraging PRs doesn't need to be seen as the numpy core devs would like to see every last crazy idea you have in mind to see what we can merge in our next drunken stupor. But rather, some ideas, even crazy ones, are easier to understand when accompanied by code, could you send a PR for what you have in mind, knowing we're nearly certain we won't merge it, but it will make it easier for us to have a fruitful discussion with you. Back to lurking ;) f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 8:40 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: To address in detail the list of Matthew you mention above: * implement orthogonal indexing as a method arr.sensible_index[...] That's basically Jaime's PR. * implement the current non-boolean fancy indexing behavior as a method - arr.crazy_index[...] Not that harmful, but only makes sense in combination with the next steps. Well, since we got the peanut butter in our meta, I might as well join in here. I think this step is useful even without the deprecation steps. First, allow me to rename things in a less judgy fashion: * arr.ortho_ix is a property that allows for orthogonal indexing, a la Jaime's PR. * arr.fancy_ix is a property that implements the current numpy-standard fancy indexing. Even though arr.fancy_ix only replicating the default semantics, having it opens up some possibilities. Other array-like objects can implement both of these with the same names. Thus, to write generic code that exploits the particular behaviors of one of these semantics, you just make sure to use the property that behaves the way you want. Then you don't care what the default syntax does on that object. You don't have to test if an object has arr.__orthogonal_indexing__ and have two different code paths; you just use the property that behaves the way you want. It also allows us to issue warnings when the default indexing syntax is used for some of the behaviors that are weird corner cases, like [1, :, array]. This is one of those corner cases where the behavior is probably not what anyone actually *wants*; it was just the only thing we could do that is consistent with the desired semantics of the rest of the cases. I think it would be reasonable to issue a warning if the default indexing syntax was used with it. It's probably a sign that the user thought that indexing worked like orthogonal indexing. The warning would *not* be issued if the arr.fancy_ix property was used, since that is an explicit signal that the user is specifically requesting a particular set of behaviors. I probably won't want to ever *deprecate* the behavior for the default syntax, but a warning is easy to deal with even with old code that you don't want to modify directly. Lastly, I would appreciate having some signal to tell readers pay attention; this is nontrivial index manipulation; here is a googleable term so you can look up what this means. I almost always use the default fancy indexing, but I'd use the arr.fancy_ix property for the nontrivial cases just for this alone. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Multidimensional Indexing
On Di, 2015-04-07 at 00:49 +0100, Nicholas Devenish wrote: With the indexing example from the documentation: y = np.arange(35).reshape(5,7) Why does selecting an item from explicitly every row work as I’d expect: y[np.array([0,1,2,3,4]),np.array([0,0,0,0,0])] array([ 0, 7, 14, 21, 28]) But doing so from a full slice (which, I would naively expect to mean “Every Row”) has some…other… behaviour: y[:,np.array([0,0,0,0,0])] array([[ 0, 0, 0, 0, 0], [ 7, 7, 7, 7, 7], [14, 14, 14, 14, 14], [21, 21, 21, 21, 21], [28, 28, 28, 28, 28]]) What is going on in this example, and how do I get what I expect? By explicitly passing in an extra array with value===index? What is the rationale for this difference in behaviour? The rationale is historic. Indexing with arrays (advanced indexing) works different from slicing. So two arrays will be iterated together, while slicing is not (we sometimes call it outer/orthogonal indexing for that matter, there is just a big discussion about this). These are different beasts, you can basically get the slicing like behaviour by adding appropriate axes to your indexing arrays: y[np.array([[0],[1],[2],[3],[4]]),np.array([0,0,0,0,0])] The other way around is not possible. Note that if it was the case: y[:, :] would give the diagonal (if possible) and not the full array as you would probably also expect. One warning: If you index with more then one array (scalars are also arrays in this sense -- so `[0, :, array]` is an example) in combination with slices, the result can be transposed in a confusing way (it is not that difficult, but usually unexpected). - Sebastian Thanks, Nick ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
