Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Chris Barker
On 6/27/11 9:53 AM, Charles R Harris wrote: Some discussion of disk storage might also help. I don't see how the rules can be enforced if two files are used, one for the mask and another for the data, but that may just be something we need to live with. It seems it wouldn't be too big deal to

[Numpy-discussion] Fitting Log normal or truncated log normal distibution to three points

2011-06-30 Thread Sachin Kumar Sharma
Hi, I have three points 10800, 81100, 582000. What is the easiest way of fitting a log normal and truncated log normal distribution to these three points using numpy. I would appreciate your reply for the same. Cheers Sachin

Re: [Numpy-discussion] Fitting Log normal or truncated log normal distibution to three points

2011-06-30 Thread Christoph Deil
On Jun 30, 2011, at 10:03 AM, Sachin Kumar Sharma wrote: Hi, I have three points 10800, 81100, 582000. What is the easiest way of fitting a log normal and truncated log normal distribution to these three points using numpy. The lognormal and maximum likelihood fit is available in

Re: [Numpy-discussion] Fitting Log normal or truncated log normal distibution to three points

2011-06-30 Thread josef . pktd
On Thu, Jun 30, 2011 at 5:25 AM, Christoph Deil deil.christ...@googlemail.com wrote: On Jun 30, 2011, at 10:03 AM, Sachin Kumar Sharma wrote: Hi, I have three points 10800, 81100, 582000. What is the easiest way of fitting a log normal and truncated log normal distribution to these three

[Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: Anyway, it's pretty clear that in this particular case, there are two distinct features that different people want: the missing data feature, and the masked array feature. The more I think about it, the less I see how

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Pierre GM
On Jun 30, 2011, at 3:31 PM, Matthew Brett wrote: ### A alternative-NEP on masking and missing values ### I like the idea of two different special values, np.NA for missing values, np.IGNORE for masked

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Charles R Harris
On Thu, Jun 30, 2011 at 7:31 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: Anyway, it's pretty clear that in this particular case, there are two distinct features that different people want: the missing data

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Dag Sverre Seljebotn
On 06/30/2011 04:17 PM, Charles R Harris wrote: On Thu, Jun 30, 2011 at 7:31 AM, Matthew Brett matthew.br...@gmail.com mailto:matthew.br...@gmail.com wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com mailto:n...@pobox.com wrote: Anyway, it's

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Charles R Harris
On Thu, Jun 30, 2011 at 8:17 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Jun 30, 2011 at 7:31 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: Anyway, it's pretty clear that in this particular

Re: [Numpy-discussion] feedback request: proposal to add masks to the core ndarray

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 1:51 PM, Lluís xscr...@gmx.net wrote: Mark Wiebe writes: [...] I think that deciding on the value of NA signal values boils down to this question: should 3rd party code be able to interpret missing data information stored in the separate mask array?

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 1:07 PM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no wrote: On 06/29/2011 07:38 PM, Mark Wiebe wrote: On Wed, Jun 29, 2011 at 9:35 AM, Dag Sverre Seljebotn d.s.seljeb...@astro.uio.no mailto:d.s.seljeb...@astro.uio.no wrote: On 06/29/2011 03:45 PM, Matthew

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 2:32 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Wed, Jun 29, 2011 at 6:22 PM, Mark Wiebe mwwi...@gmail.com wrote: On Wed, Jun 29, 2011 at 8:20 AM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: Maybe instead of np.NA, we could say np.IGNORE,

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 1:20 PM, Lluís xscr...@gmx.net wrote: Mark Wiebe writes: There seems to be a general idea that masks and NA bit patterns imply particular differing semantics, something which I think is simply false. Well, my example contained a difference (the need for the

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 4:21 PM, Eric Firing efir...@hawaii.edu wrote: On 06/29/2011 09:32 AM, Matthew Brett wrote: Hi, [...] Clearly there are some overlaps between what masked arrays are trying to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Wed, Jun 29, 2011 at 5:42 PM, Nathaniel Smith n...@pobox.com wrote: On Wed, Jun 29, 2011 at 2:40 PM, Lluís xscr...@gmx.net wrote: I'm for the option of having a single API when you want to have NA elements, regardless of whether it's using masks or bit patterns. I understand the desire

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Thu, Jun 30, 2011 at 1:49 AM, Chris Barker chris.bar...@noaa.gov wrote: On 6/27/11 9:53 AM, Charles R Harris wrote: Some discussion of disk storage might also help. I don't see how the rules can be enforced if two files are used, one for the mask and another for the data, but that may

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 2:58 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 30, 2011, at 3:31 PM, Matthew Brett wrote: ### A alternative-NEP on masking and missing values ### I like the idea of

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Pierre GM
On Jun 30, 2011, at 5:38 PM, Matthew Brett wrote: Hi, On Thu, Jun 30, 2011 at 2:58 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 30, 2011, at 3:31 PM, Matthew Brett wrote: ### A alternative-NEP on masking and missing values

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Gary Strangman
Clearly there are some overlaps between what masked arrays are trying to achieve and what Rs NA mechanisms are trying to achieve.  Are they really similar enough that they should function using the same API? Yes. And if so, won't that be confusing? No, I don't

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Lluís
Mark Wiebe writes: On Wed, Jun 29, 2011 at 1:20 PM, Lluís xscr...@gmx.net wrote: [...] As far as I can tell, the only required difference between them is that NA bit patterns must destroy the data. Nothing else. Everything on top of that is a choice of API and interface mechanisms. I

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Thu, Jun 30, 2011 at 11:04 AM, Gary Strangman str...@nmr.mgh.harvard.edu wrote: Clearly there are some overlaps between what masked arrays are trying to achieve and what Rs NA mechanisms are trying to achieve. Are they really similar enough that they should function using

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 5:03 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 30, 2011, at 5:38 PM, Matthew Brett wrote: Hi, On Thu, Jun 30, 2011 at 2:58 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 30, 2011, at 3:31 PM, Matthew Brett wrote:

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 5:13 PM, Mark Wiebe mwwi...@gmail.com wrote: On Thu, Jun 30, 2011 at 11:04 AM, Gary Strangman str...@nmr.mgh.harvard.edu wrote:      Clearly there are some overlaps between what masked arrays are      trying to achieve and what Rs NA mechanisms are trying to

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Lluís
Mark Wiebe writes: Why is one magic and the other real? All of this is already sitting on 100 layers of abstraction above electrons and atoms. If we're talking about real, maybe we should be programming in machine code or using breadboards with individual transistors. M-x butterfly RET

[Numpy-discussion] broacasting question

2011-06-30 Thread Thomas K Gamble
I'm trying to convert some IDL code to python/numpy and i'm having some trouble understanding the rules for boradcasting during some operations. example: given the following arrays: a = array((2048,3577), dtype=float) b = array((256,25088), dtype=float) c = array((2048,3136), dtype=float) d =

Re: [Numpy-discussion] broacasting question

2011-06-30 Thread Olivier Delalleau
2011/6/30 Thomas K Gamble tkgam...@windstream.net I'm trying to convert some IDL code to python/numpy and i'm having some trouble understanding the rules for boradcasting during some operations. example: given the following arrays: a = array((2048,3577), dtype=float) b = array((256,25088),

[Numpy-discussion] missing data: semantics

2011-06-30 Thread Lluís
Ok, I think it's time to step back and reformulate the problem by completely ignoring the implementation. Here we have 2 generic concepts (i.e., applicable to R), plus another extra concept that is exclusive to numpy: * Assigning np.NA to an array, cannot be undone unless through explicit

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Nathaniel Smith
On Thu, Jun 30, 2011 at 6:31 AM, Matthew Brett matthew.br...@gmail.com wrote: In the interest of making the discussion as concrete as possible, here is my draft of an alternative proposal for NAs and masking, based on Nathaniel's comments.  Writing it, it seemed to me that Nathaniel is right,

Re: [Numpy-discussion] missing data: semantics

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 6:46 PM, Lluís xscr...@gmx.net wrote: Ok, I think it's time to step back and reformulate the problem by completely ignoring the implementation. Here we have 2 generic concepts (i.e., applicable to R), plus another extra concept that is exclusive to numpy: *

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 6:51 PM, Nathaniel Smith n...@pobox.com wrote: On Thu, Jun 30, 2011 at 6:31 AM, Matthew Brett matthew.br...@gmail.com wrote: In the interest of making the discussion as concrete as possible, here is my draft of an alternative proposal for NAs and masking, based on

Re: [Numpy-discussion] broacasting question

2011-06-30 Thread Derek Homeier
On 30.06.2011, at 7:32PM, Thomas K Gamble wrote: I'm trying to convert some IDL code to python/numpy and i'm having some trouble understanding the rules for boradcasting during some operations. example: given the following arrays: a = array((2048,3577), dtype=float) b =

Re: [Numpy-discussion] missing data: semantics

2011-06-30 Thread Charles R Harris
On Thu, Jun 30, 2011 at 11:46 AM, Lluís xscr...@gmx.net wrote: Ok, I think it's time to step back and reformulate the problem by completely ignoring the implementation. Here we have 2 generic concepts (i.e., applicable to R), plus another extra concept that is exclusive to numpy: *

Re: [Numpy-discussion] missing data: semantics

2011-06-30 Thread Charles R Harris
On Thu, Jun 30, 2011 at 11:51 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Jun 30, 2011 at 6:46 PM, Lluís xscr...@gmx.net wrote: Ok, I think it's time to step back and reformulate the problem by completely ignoring the implementation. Here we have 2 generic concepts

[Numpy-discussion] review request: introductory datetime documentation

2011-06-30 Thread Mark Wiebe
https://github.com/numpy/numpy/pull/101 Thanks, Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Lluís
Matthew Brett writes: [...] I'm afraid, like you, I'm a little lost in the world of masking, because I only need the NAs. I was trying to see if I could come up with an API that picked up some of the syntactic convenience of NAs, without conflating NAs with IGNOREs. I guess we need some

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Thu, Jun 30, 2011 at 11:42 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Jun 30, 2011 at 5:13 PM, Mark Wiebe mwwi...@gmail.com wrote: On Thu, Jun 30, 2011 at 11:04 AM, Gary Strangman str...@nmr.mgh.harvard.edu wrote: Clearly there are some overlaps between what

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Mark Wiebe
On Thu, Jun 30, 2011 at 11:54 AM, Lluís xscr...@gmx.net wrote: Mark Wiebe writes: Why is one magic and the other real? All of this is already sitting on 100 layers of abstraction above electrons and atoms. If we're talking about real, maybe we should be programming in machine code or

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Nathaniel Smith
On Thu, Jun 30, 2011 at 11:27 AM, Lluís xscr...@gmx.net wrote: As I tried to convey on my other mail, separating both will force you to either: * Make a copy of the array before passing it to another routine (because  the routine will assign np.NA but you still want the original data) To

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Nathaniel Smith
On Wed, Jun 29, 2011 at 2:21 PM, Eric Firing efir...@hawaii.edu wrote: In addition, for new code, the full-blown masked array module may not be needed.  A convenience it adds, however, is the automatic masking of invalid values: In [1]: np.ma.log(-1) Out[1]: masked I'm sure this horrifies

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 7:27 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: [...] I'm afraid, like you, I'm a little lost in the world of masking, because I only need the NAs.  I was trying to see if I could come up with an API that picked up some of the syntactic convenience of

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Lluís
Nathaniel Smith writes: On Thu, Jun 30, 2011 at 11:27 AM, Lluís xscr...@gmx.net wrote: As I tried to convey on my other mail, separating both will force you to either: * Make a copy of the array before passing it to another routine (because  the routine will assign np.NA but you still want

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Eric Firing
On 06/30/2011 08:53 AM, Nathaniel Smith wrote: On Wed, Jun 29, 2011 at 2:21 PM, Eric Firingefir...@hawaii.edu wrote: In addition, for new code, the full-blown masked array module may not be needed. A convenience it adds, however, is the automatic masking of invalid values: In [1]:

Re: [Numpy-discussion] missing data discussion round 2

2011-06-30 Thread Nathaniel Smith
On Thu, Jun 30, 2011 at 12:27 PM, Eric Firing efir...@hawaii.edu wrote: On 06/30/2011 08:53 AM, Nathaniel Smith wrote: On Wed, Jun 29, 2011 at 2:21 PM, Eric Firingefir...@hawaii.edu  wrote: In addition, for new code, the full-blown masked array module may not be needed.  A convenience it adds,

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Lluís
Matthew Brett writes: Hi, On Thu, Jun 30, 2011 at 7:27 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: [...] I'm afraid, like you, I'm a little lost in the world of masking, because I only need the NAs.  I was trying to see if I could come up with an API that picked up some of the

Re: [Numpy-discussion] broacasting question

2011-06-30 Thread Thomas K Gamble
On 30.06.2011, at 7:32PM, Thomas K Gamble wrote: I'm trying to convert some IDL code to python/numpy and i'm having some trouble understanding the rules for boradcasting during some operations. example: given the following arrays: a = array((2048,3577), dtype=float) b =

Re: [Numpy-discussion] broacasting question

2011-06-30 Thread Derek Homeier
On 30.06.2011, at 11:57PM, Thomas K Gamble wrote: np.add(b.reshape(2048,3136) * c, d, out=a[:,:3136]) But to say whether this is really the equivalent result to what IDL does, one would have to study the IDL manual in detail or directly compare the output (e.g. check what happens to the

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Matthew Brett
Hi, On Thu, Jun 30, 2011 at 9:01 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: Hi, On Thu, Jun 30, 2011 at 7:27 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: [...] I'm afraid, like you, I'm a little lost in the world of masking, because I only need the NAs.  I was trying

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Gary Strangman
It seems to me, that what ``func`` should do, if it wants you to be able to unmask the NAs, is to make a masked array view of ``arr``, and return that.   And indeed the simplicity of the separated API immediately makes that clear - in my view at least. I agree on this example. My only concern

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Charles R Harris
On Thu, Jun 30, 2011 at 6:02 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Jun 30, 2011 at 9:01 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: Hi, On Thu, Jun 30, 2011 at 7:27 PM, Lluís xscr...@gmx.net wrote: Matthew Brett writes: [...] I'm afraid, like you,

Re: [Numpy-discussion] alterNEP - was: missing data discussion round 2

2011-06-30 Thread Keith Goodman
On Thu, Jun 30, 2011 at 10:51 AM, Nathaniel Smith n...@pobox.com wrote: On Thu, Jun 30, 2011 at 6:31 AM, Matthew Brett matthew.br...@gmail.com wrote: In the interest of making the discussion as concrete as possible, here is my draft of an alternative proposal for NAs and masking, based on