[Numpy-discussion] Clarifications in numpy.ma module
Hi all, I have recently been trying out various functions in masked array module of numpy. I have got confused at a places in the *core.py *of *ma *module. 1. In the *masked_equal *method, the docstring doesn't suggest that the *fill_value *gets updated by the *value *parameter of the function, but this line ( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L1978 ) sets the *fill_value* as *value. * 2. The outputs of following functions - *any *( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4327) - *all* ( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4280) are similar, they return *np.ma.masked *if all the elements have masks in the array, else return *True*. 3. _*MaskedBinaryOperation : *Used for multiply, add, subtract ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
Guys, sorry for the incomplete message, *_DomainedBinaryOperation *for divide remainder Related to issue 5354, where the docstring for _*MaskedBinaryOperation *says that invalid values are pre-masked*, but for **_DomainedBinaryOperation *where the invalid values are masked in result, even if they are not masked in the input. 4. Also, I had a doubt regarding the working of a%b and np.ma.remainder(a,b), whether they are analogous to the way functions like add, divide work. Since, the changes done to the above BinaryOperation classes are visible to a/b, a*b, np.ma.multiply, np.ma.divide, np.ma.remainder, np.ma.mod but not a%b. Please do correct me if I am wrong about *mod, **remainder and % *use. 5. The *mean* function doesn't take care of the edge case where array is empty. np.mean(np.array([])) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice. warnings.warn(Mean of empty slice., RuntimeWarning) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) nan np.ma.mean(np.ma.array([])) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:69: RuntimeWarning: invalid value encountered in true_divide ret, rcount, out=ret, casting='unsafe', subok=False) masked_array(data = nan, mask = False, fill_value = 1e+20) Thanks , Maniteja. On Tue, Dec 30, 2014 at 4:19 PM, Maniteja Nandana maniteja.modesty...@gmail.com wrote: Hi all, I have recently been trying out various functions in masked array module of numpy. I have got confused at a places in the *core.py *of *ma *module. 1. In the *masked_equal *method, the docstring doesn't suggest that the *fill_value *gets updated by the *value *parameter of the function, but this line ( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L1978 ) sets the *fill_value* as *value. * 2. The outputs of following functions - *any *( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4327) - *all* (https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4280) are similar, they return *np.ma.masked *if all the elements have masks in the array, else return *True*. 3. _*MaskedBinaryOperation : *Used for multiply, add, subtract ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Why is numpy.ma.extras.clump_masked() not in main documentation?
Wow, thanks for pointing these out! I have been using masked arrays for quite a while now, but I never noticed these before! Cheers! Ben Root On Sun, Dec 28, 2014 at 3:35 PM, Yuxiang Wang yw...@virginia.edu wrote: Hi Ralf, Thanks for the quick response! I will submit a PR soon. I noticed there are some other functions with outdated examples, such as numpy.ma.notmasked_contiguous() (still using the numpy.ma.extras.notmasked_contiguous). -Shawn On Sun, Dec 28, 2014 at 3:04 PM, Ralf Gommers ralf.gomm...@gmail.com wrote: On Sun, Dec 28, 2014 at 8:48 PM, Yuxiang Wang yw...@virginia.edu wrote: Dear all, I am really glad to find out a very useful function called numpy.ma.extras.clump_masked(), and it is indeed well documented if you look into the source. However, may I ask why does it not show up in the main documentation website (http://docs.scipy.org/doc/numpy/reference/routines.ma.html)? Because they (there's also clump_unmasked) weren't added to the function list in doc/source/reference/routines.ma.rst. Care to send a PR for that? Other todo there is to fix up the examples, they should be used as np.ma.clump_masked not np.ma.extras.clump_masked. Cheers, Ralf Not a big deal, just being curious. Shawn -- Yuxiang Shawn Wang Gerling Research Lab University of Virginia yw...@virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Yuxiang Shawn Wang Gerling Research Lab University of Virginia yw...@virginia.edu +1 (434) 284-0836 https://sites.google.com/a/virginia.edu/yw5aj/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
What do you mean that the mean function doesn't take care of the case where the array is empty? In the example you provided, they both end up being NaN, which is exactly correct. Ben Root On Tue, Dec 30, 2014 at 6:10 AM, Maniteja Nandana maniteja.modesty...@gmail.com wrote: Guys, sorry for the incomplete message, *_DomainedBinaryOperation *for divide remainder Related to issue 5354, where the docstring for _*MaskedBinaryOperation *says that invalid values are pre-masked*, but for **_DomainedBinaryOperation *where the invalid values are masked in result, even if they are not masked in the input. 4. Also, I had a doubt regarding the working of a%b and np.ma.remainder(a,b), whether they are analogous to the way functions like add, divide work. Since, the changes done to the above BinaryOperation classes are visible to a/b, a*b, np.ma.multiply, np.ma.divide, np.ma.remainder, np.ma.mod but not a%b. Please do correct me if I am wrong about *mod, **remainder and % *use. 5. The *mean* function doesn't take care of the edge case where array is empty. np.mean(np.array([])) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice. warnings.warn(Mean of empty slice., RuntimeWarning) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) nan np.ma.mean(np.ma.array([])) /home/maniteja/FOSS/numpy/numpy/core/_methods.py:69: RuntimeWarning: invalid value encountered in true_divide ret, rcount, out=ret, casting='unsafe', subok=False) masked_array(data = nan, mask = False, fill_value = 1e+20) Thanks , Maniteja. On Tue, Dec 30, 2014 at 4:19 PM, Maniteja Nandana maniteja.modesty...@gmail.com wrote: Hi all, I have recently been trying out various functions in masked array module of numpy. I have got confused at a places in the *core.py *of *ma * module. 1. In the *masked_equal *method, the docstring doesn't suggest that the *fill_value *gets updated by the *value *parameter of the function, but this line ( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L1978 ) sets the *fill_value* as *value. * 2. The outputs of following functions - *any *( https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4327) - *all* (https://github.com/numpy/numpy/blob/master/numpy/ma/core.py#L4280) are similar, they return *np.ma.masked *if all the elements have masks in the array, else return *True*. 3. _*MaskedBinaryOperation : *Used for multiply, add, subtract ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
On Tue, Dec 30, 2014 at 1:45 PM, Benjamin Root ben.r...@ou.edu wrote: What do you mean that the mean function doesn't take care of the case where the array is empty? In the example you provided, they both end up being NaN, which is exactly correct. Operations on masked arrays should not produce NaNs. They should produce ma.masked. For example, np.ma.array(0)/0 masked The fact that the user sees runtime warnings also suggests that the edge case was not thought out. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
Where does it say that operations on masked arrays should not produce NaNs? Operations on masked arrays should ignore masked data. If I have NaNs in my masked array, but are not masked out for some reason, I expect it to give me NaNs. The mask is not the same as NaNs. Having np.mean([]) return the same thing as np.ma.mean([]) makes complete sense. Now, the fun comes with the whole controversy over np.nanmean([]) and np.nanmean([np.nan])... As for the rest of your points in your original post, I do not have the knowledge to know whether or not they are actual issues, but they do look like something that should be address in some form. Ben Root On Tue, Dec 30, 2014 at 2:39 PM, Alexander Belopolsky ndar...@mac.com wrote: On Tue, Dec 30, 2014 at 1:45 PM, Benjamin Root ben.r...@ou.edu wrote: What do you mean that the mean function doesn't take care of the case where the array is empty? In the example you provided, they both end up being NaN, which is exactly correct. Operations on masked arrays should not produce NaNs. They should produce ma.masked. For example, np.ma.array(0)/0 masked The fact that the user sees runtime warnings also suggests that the edge case was not thought out. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: Scipy 0.14.1 release
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dear all, We are pleased to announce the Scipy 0.14.1 release. The 0.14.1 release is a bugfix-only release, addressing the following issues: - - gh-3630 NetCDF reading results in a segfault - - gh-3631 SuperLU object not working as expected for complex matrices - - gh-3733 Segfault from map_coordinates - - gh-3780 Segfault when using CSR/CSC matrix and uint32/uint64 - - gh-3781 Fix omitted types in sparsetools typemaps - - gh-3802 0.14.0 API breakage: _gen generators are missing from scipy.stats.distributions API - - gh-3805 Ndimge test failures with numpy 1.10 - - gh-3812 == sometimes wrong on csr_matrix - - gh-3853 Many scipy.sparse test errors/failures with numpy 1.9.0b2 - - gh-4084 Fix exception declarations for Cython 0.21.1 compatibility - - gh-4093 Avoid a memory error in splev(x, tck, der=k) - - gh-4104 Workaround SGEMV segfault in Accelerate (maintenance 0.14.x) - - gh-4143 Fix ndimage functions for large data - - gh-4149 Bug in expm for integer arrays - - gh-4154 Ensure that the 'size' argument of PIL's 'resize' method is a tuple - - gh-4163 ZeroDivisionError in scipy.sparse.linalg.lsqr - - gh-4164 Remove use of deprecated numpy API in lib/lapack/ f2py wrapper - - gh-4180 PIL resize support tuple fix - - gh-4168 Address arpack test failures on windows 32 bits with numpy 1.9.1 - - gh-4203 Sparse matrix multiplication in 0.14.x slower compared to 0.13.x - - gh-4218 Make ndimage interpolation compatible with numpy relaxed strides - - gh-4225 Off-by-one error in PPoly shape checks - - gh-4248 Fix issue with incorrect use of closure for slsqp Source tarballs and binaries are available at https://sourceforge.net/projects/scipy/files/SciPy/0.14.1/ Best regards, Pauli Virtanen -BEGIN PGP SIGNATURE- Version: GnuPG v1 iEYEARECAAYFAlSjCcYACgkQ6BQxb7O0pWBxcwCfcnd4uva5hzMHQlHmWxlfbja3 T0AAn2QQmhcotDRB2c2p41Xzjb4MJ13f =yBxH -END PGP SIGNATURE- ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
On Tue, Dec 30, 2014 at 2:49 PM, Benjamin Root ben.r...@ou.edu wrote: Where does it say that operations on masked arrays should not produce NaNs? Masked arrays were invented with the specific goal to avoid carrying NaNs in computations. Back in the days, NaNs were not available on some platforms and had significant performance issues on others. These days NaN support for floating point types is nearly universal, but numpy types are not limited by floating point. Having np.mean([]) return the same thing as np.ma.mean([]) makes complete sense. Does the following make sense as well? import numpy numpy.ma.masked_values([0, 0], 0).mean() masked numpy.ma.masked_values([0], 0).mean() masked numpy.ma.masked_values([], 0).mean() * Two warnings * masked_array(data = nan, mask = False, fill_value = 0.0) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
On Tue, Dec 30, 2014 at 3:29 PM, Alexander Belopolsky ndar...@mac.com wrote: On Tue, Dec 30, 2014 at 2:49 PM, Benjamin Root ben.r...@ou.edu wrote: Where does it say that operations on masked arrays should not produce NaNs? Masked arrays were invented with the specific goal to avoid carrying NaNs in computations. Back in the days, NaNs were not available on some platforms and had significant performance issues on others. These days NaN support for floating point types is nearly universal, but numpy types are not limited by floating point. From the numpy.ma docstring: Arrays sometimes contain invalid or missing data. When doing operations on such arrays, we wish to suppress invalid values, which is the purpose masked arrays fulfill (an example of typical use is given below). A few lines down: Here, we construct a masked array that suppress all ``NaN`` values. We may now proceed to calculate the mean of the other values Note the repeated usage of the term suppress in the context of the input arrays. The phrase We may now proceed to calculate the mean of the other values implies that the mean of a masked array is taken to be the mean of everything but the masked values. If there are no values remaining, then I expect it to give me the equivalent of np.mean([]). Having np.mean([]) return the same thing as np.ma.mean([]) makes complete sense. Does the following make sense as well? import numpy numpy.ma.masked_values([0, 0], 0).mean() masked numpy.ma.masked_values([0], 0).mean() masked numpy.ma.masked_values([], 0).mean() * Two warnings * masked_array(data = nan, mask = False, fill_value = 0.0) No, I would consider the first two to be bugs. And actually, returning a masked array in the third one is also incorrect in this case. The result should be a scalar. This is now veering to the same issues discussed in the np.nanmean([]) vs. np.nanmean([np.nan]) discussion. Cheers! Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
I was just referring to the exception raised in the case where the length of the array is zero. I have not thought if the example provided by @Alexander. I was also wondering if the automatic masking of NaN should be done or not, which is why I asked about the difference in the operating named arrays upon division, where they are automatically masked and multiplication, where they aren't.( point 3) Cheers, N.Maniteja ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http:// http://mail.scipy.org/mailman/listinfo/numpy-discussion mail.scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion /mailman/ http://mail.scipy.org/mailman/listinfo/numpy-discussionlistinfo http://mail.scipy.org/mailman/listinfo/numpy-discussion/ http://mail.scipy.org/mailman/listinfo/numpy-discussionnumpy-discussion http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
exception? Did you mean warning? If warning, I recall some discussion recently to figure out a way to hide that, but only for masked values (I would want to see the warning if I do bad calculations in the unmasked portions of my array). Now I see your point 3 much more clearly. I had never noticed that the divide could produce new masked elements. It is presumptuous to assume that NaNs are what I want masked. Division (and exponential) are the only two binary operations I can imagine where two valid floats could produce a NaN or Inf, so that is probably why the division was different from the others. This confusion probably came about in conflating valid-ness with NaN and Inf as concepts. In small parts of the codebase, it seems to operate with the concept that NaN === invalid, while other parts strictly works within the framework of masked === invalid. Of course, fixing any of this would be potentially a significant change in behavior. I am certainly not one to make any sort of determination on this. I am just a heavy user of masked arrays. Cheers! Ben Root On Tue, Dec 30, 2014 at 5:37 PM, Maniteja Nandana maniteja.modesty...@gmail.com wrote: I was just referring to the exception raised in the case where the length of the array is zero. I have not thought if the example provided by @Alexander. I was also wondering if the automatic masking of NaN should be done or not, which is why I asked about the difference in the operating named arrays upon division, where they are automatically masked and multiplication, where they aren't.( point 3) Cheers, N.Maniteja ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http:// http://mail.scipy.org/mailman/listinfo/numpy-discussion mail.scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion /mailman/ http://mail.scipy.org/mailman/listinfo/numpy-discussion listinfo http://mail.scipy.org/mailman/listinfo/numpy-discussion/ http://mail.scipy.org/mailman/listinfo/numpy-discussionnumpy-discussion http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Access dtype kind from cython
* Eric Moore e...@redtetrahedron.org [2014-12-30]:
On Monday, December 29, 2014, Valentin Haenel valen...@haenel.co wrote:
Hi,
how do I access the kind of the data from cython, i.e. the single
character string:
'b' boolean
'i' (signed) integer
'u' unsigned integer
'f' floating-point
'c' complex-floating point
'O' (Python) objects
'S', 'a' (byte-)string
'U' Unicode
'V' raw data (void)
In regular Python I can do:
In [7]: d = np.dtype('S')
In [8]: d.kind
Out[8]: 'S'
Looking at the definition of dtype that comes with cython, I see:
ctypedef class numpy.dtype [object PyArray_Descr]:
# Use PyDataType_* macros when possible, however there are no macros
# for accessing some of the fields, so some are defined. Please
# ask on cython-dev if you need more.
cdef int type_num
cdef int itemsize elsize
cdef char byteorder
cdef object fields
cdef tuple names
I.e. no kind.
Also, i looked for an appropriate PyDataType_* macro but couldn't find one.
Perhaps there is something simple I could use?
best,
V-
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org javascript:;
http://mail.scipy.org/mailman/listinfo/numpy-discussion
From C or cython I'd just use the typenum. Compare against the appropriate
macros, NPY_DOUBLE e.g.
That would be nice, but I am refactoring existing code and it is
specifically asking for the kind character.
V-
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Access dtype kind from cython
On Tue, Dec 30, 2014 at 11:03 PM, Valentin Haenel valen...@haenel.co wrote:
* Eric Moore e...@redtetrahedron.org [2014-12-30]:
On Monday, December 29, 2014, Valentin Haenel valen...@haenel.co wrote:
Hi,
how do I access the kind of the data from cython, i.e. the single
character string:
'b' boolean
'i' (signed) integer
'u' unsigned integer
'f' floating-point
'c' complex-floating point
'O' (Python) objects
'S', 'a' (byte-)string
'U' Unicode
'V' raw data (void)
In regular Python I can do:
In [7]: d = np.dtype('S')
In [8]: d.kind
Out[8]: 'S'
Looking at the definition of dtype that comes with cython, I see:
ctypedef class numpy.dtype [object PyArray_Descr]:
# Use PyDataType_* macros when possible, however there are no macros
# for accessing some of the fields, so some are defined. Please
# ask on cython-dev if you need more.
cdef int type_num
cdef int itemsize elsize
cdef char byteorder
cdef object fields
cdef tuple names
I.e. no kind.
The problem is just that whoever wrote numpy.pxd was feeling a bit
lazy that day and only filled in the fields they felt were most
important :-). There are a bunch of public fields in PyArray_Descr
that are just being left out of the Cython file you quote:
https://github.com/numpy/numpy/blob/master/numpy/core/include/numpy/ndarraytypes.h#L566
In particular, there's a 'char kind' field.
The quick workaround is
cdef extern from *:
cdef struct my_numpy_dtype [object PyArray_Descr]:
cdef char kind
# ... whatever other fields you might need
and then cast to my_numpy_dtype when you need to get at the kind field
from Cython.
If feeling generous, then submit a PR to Cython adding 'cdef char
kind' to the definition above. If feeling extra generous, it would be
awesome if someone systematically went through and added all the
missing fields that are in the numpy header but not cython -- I've run
into these missing field issues annoyingly often myself, and it's
silly that we should all keep making our own individual workarounds
for numpy.pxd's limitations...
-n
--
Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
On Tue, Dec 30, 2014 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: exception? Did you mean warning? If warning, I recall some discussion recently to figure out a way to hide that, but only for masked values (I would want to see the warning if I do bad calculations in the unmasked portions of my array). Now I see your point 3 much more clearly. I had never noticed that the divide could produce new masked elements. It is presumptuous to assume that NaNs are what I want masked. Division (and exponential) are the only two binary operations I can imagine where two valid floats could produce a NaN or Inf, so that is probably why the division was different from the others. This confusion probably came about in conflating valid-ness with NaN and Inf as concepts. In small parts of the codebase, it seems to operate with the concept that NaN === invalid, while other parts strictly works within the framework of masked === invalid. Of course, fixing any of this would be potentially a significant change in behavior. I am certainly not one to make any sort of determination on this. I am just a heavy user of masked arrays. Unfortunately, as we discovered during the NA debate, it turns out that there are several different ways to think about masked/missing values, and np.ma kinda can't decide which one it wants to implement so it implements a mix of all of them. This makes it difficult to know whether it's working correctly or not :-). @Maniteja: Also unfortunately (and probably not unrelatedly) the np.ma code is mostly pretty old and receives only minimal maintenance. So if you don't receive answers to your original questions it may just be that there's no-one around who knows. It works that way because that's the way it works... (My personal recommendation is to steer clear of using np.ma entirely, but reasonable people can disagree.) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
On 31-Dec-2014 4:53 am, Nathaniel Smith n...@pobox.com wrote: On Tue, Dec 30, 2014 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: exception? Did you mean warning? If warning, I recall some discussion recently to figure out a way to hide that, but only for masked values (I would want to see the warning if I do bad calculations in the unmasked portions of my array). I was referring to the warning. I thought it could be handled elegantly. Yeah I do get the warning serves as reminder to the user. Now I see your point 3 much more clearly. I had never noticed that the divide could produce new masked elements. It is presumptuous to assume that NaNs are what I want masked. Division (and exponential) are the only two binary operations I can imagine where two valid floats could produce a NaN or Inf, so that is probably why the division was different from the others. This confusion probably came about in conflating valid-ness with NaN and Inf as concepts. In small parts of the codebase, it seems to operate with the concept that NaN === invalid, while other parts strictly works within the framework of masked === invalid. Of course, fixing any of this would be potentially a significant change in behavior. I am certainly not one to make any sort of determination on this. I am just a heavy user of masked arrays. I was just thinking if there was a uniform policy for handling NaN and inf. Unfortunately, as we discovered during the NA debate, it turns out that there are several different ways to think about masked/missing values, and np.ma kinda can't decide which one it wants to implement so it implements a mix of all of them. This makes it difficult to know whether it's working correctly or not :-). I actually got the last point since I was not sure about the operator overloading,for eg, whether a/b would be equal to np.ma.divide (a, b) or np.divide (a, b). @Maniteja: Also unfortunately (and probably not unrelatedly) the np.ma code is mostly pretty old and receives only minimal maintenance. So if you don't receive answers to your original questions it may just be that there's no-one around who knows. It works that way because that's the way it works... (My personal recommendation is to steer clear of using np.ma entirely, but reasonable people can disagree.) Thanks for the info, but I was just trying to get a idea of the source code and I have had some exposure previously to np.ma, but never found these issues until I looked at the core. :-) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Clarifications in numpy.ma module
Maniteja, Careful with advertising that you are reading up on any under-maintained codebases. I did that for mplot3d four years ago and the previous maintainer said tag! you're it! I haven't been able to tag anyone since then... Cheers! Ben Root On Tue, Dec 30, 2014 at 6:34 PM, Maniteja Nandana maniteja.modesty...@gmail.com wrote: On 31-Dec-2014 4:53 am, Nathaniel Smith n...@pobox.com wrote: On Tue, Dec 30, 2014 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: exception? Did you mean warning? If warning, I recall some discussion recently to figure out a way to hide that, but only for masked values (I would want to see the warning if I do bad calculations in the unmasked portions of my array). I was referring to the warning. I thought it could be handled elegantly. Yeah I do get the warning serves as reminder to the user. Now I see your point 3 much more clearly. I had never noticed that the divide could produce new masked elements. It is presumptuous to assume that NaNs are what I want masked. Division (and exponential) are the only two binary operations I can imagine where two valid floats could produce a NaN or Inf, so that is probably why the division was different from the others. This confusion probably came about in conflating valid-ness with NaN and Inf as concepts. In small parts of the codebase, it seems to operate with the concept that NaN === invalid, while other parts strictly works within the framework of masked === invalid. Of course, fixing any of this would be potentially a significant change in behavior. I am certainly not one to make any sort of determination on this. I am just a heavy user of masked arrays. I was just thinking if there was a uniform policy for handling NaN and inf. Unfortunately, as we discovered during the NA debate, it turns out that there are several different ways to think about masked/missing values, and np.ma kinda can't decide which one it wants to implement so it implements a mix of all of them. This makes it difficult to know whether it's working correctly or not :-). I actually got the last point since I was not sure about the operator overloading,for eg, whether a/b would be equal to np.ma.divide (a, b) or np.divide (a, b). @Maniteja: Also unfortunately (and probably not unrelatedly) the np.ma code is mostly pretty old and receives only minimal maintenance. So if you don't receive answers to your original questions it may just be that there's no-one around who knows. It works that way because that's the way it works... (My personal recommendation is to steer clear of using np.ma entirely, but reasonable people can disagree.) Thanks for the info, but I was just trying to get a idea of the source code and I have had some exposure previously to np.ma, but never found these issues until I looked at the core. :-) -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
