Re: [Numpy-discussion] read-only or immutable masked array
Hi Pierre, I'm a bit surprised, though. Here's what I tried np.version.version 1.7.0 x = np.ma.array([1,2,3], mask=[0,1,0]) x.flags.writeable=False x[0]=-1 ValueError: assignment destination is read-only Thanks, it works perfectly =) Sorry, probably have overlooked this simple solution, tried to set x.data and x.mask directly. I noticed that this only protects the data, so mask also has to be set to read-only or be hardened to avoid accidental (un)masking. Gregorio ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
On Jul 15, 2013, at 10:04 , Gregorio Bastardo gregorio.basta...@gmail.com wrote: Hi Pierre, I'm a bit surprised, though. Here's what I tried np.version.version 1.7.0 x = np.ma.array([1,2,3], mask=[0,1,0]) x.flags.writeable=False x[0]=-1 ValueError: assignment destination is read-only Thanks, it works perfectly =) Sorry, probably have overlooked this simple solution, tried to set x.data and x.mask directly. I noticed that this only protects the data, so mask also has to be set to read-only or be hardened to avoid accidental (un)masking. Well, yes and no. Settings the flags of `x` doesn't set (yet) the flags of the mask, that's true. Still, `.writeable=False` should prevent you to unmask data, provided you're not trying to modify the mask directly but use basic assignment like `x[…]=…`. However, assigning `np.ma.masked` to array items does modify the mask and only the mask, hence the absence of error if the array is not writeable. Note as well that hardening the mask only prevents unmasking: you can still grow the mask, which may not be what you want. Use `x.mask.flags.writeable=False` to make the mask really read-only. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
Hi Pierre, Note as well that hardening the mask only prevents unmasking: you can still grow the mask, which may not be what you want. Use `x.mask.flags.writeable=False` to make the mask really read-only. I ran into an unmasking problem with the suggested approach: np.version.version '1.7.0' x = np.ma.masked_array(xrange(4), [0,1,0,1]) x masked_array(data = [0 -- 2 --], mask = [False True False True], fill_value = 99) x.flags.writeable = False x.mask.flags.writeable = False x.mask[1] = 0 # ok Traceback (most recent call last): ... ValueError: assignment destination is read-only x[1] = 0 # ok Traceback (most recent call last): ... ValueError: assignment destination is read-only x.mask[1] = 0 # ?? x masked_array(data = [0 1 2 --], mask = [False False False True], fill_value = 99) I noticed that sharedmask attribute changes (from True to False) after x[1] = 0. Also, some of the ma operations result mask identity of the new ma, which causes ValueError when the new ma mask is modified: x = np.ma.masked_array(xrange(4), [0,1,0,1]) x.flags.writeable = False x.mask.flags.writeable = False x1 = x 0 x1.mask is x.mask # ok False x2 = x != 0 x2.mask is x.mask # ?? True x2.mask[1] = 0 Traceback (most recent call last): ... ValueError: assignment destination is read-only which is a bit confusing. And I experienced that *_like operations give mask identity too: y = np.ones_like(x) y.mask is x.mask True but for that I found a recent discussion (empty_like for masked arrays) on the mailing list: http://mail.scipy.org/pipermail/numpy-discussion/2013-June/066836.html I might be missing something but could you clarify these issues? Thanks, Gregorio ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
On Jul 15, 2013, at 14:40 , Gregorio Bastardo gregorio.basta...@gmail.com wrote: Hi Pierre, Note as well that hardening the mask only prevents unmasking: you can still grow the mask, which may not be what you want. Use `x.mask.flags.writeable=False` to make the mask really read-only. I ran into an unmasking problem with the suggested approach: np.version.version '1.7.0' x = np.ma.masked_array(xrange(4), [0,1,0,1]) x masked_array(data = [0 -- 2 --], mask = [False True False True], fill_value = 99) x.flags.writeable = False x.mask.flags.writeable = False x.mask[1] = 0 # ok Traceback (most recent call last): ... ValueError: assignment destination is read-only x[1] = 0 # ok Traceback (most recent call last): ... ValueError: assignment destination is read-only x.mask[1] = 0 # ?? x masked_array(data = [0 1 2 --], mask = [False False False True], fill_value = 99) Ouch… Quick workaround: use `x.harden_mask()` *then* `x.mask.flags.writeable=False` [Longer explanation] I noticed that sharedmask attribute changes (from True to False) after x[1] = 0. Indeed, indeed… When setting items, the mask is unshared to limit some issues (like propagation to the other masked_arrays sharing the mask). Unsharing the mask involves a copy, which unfortunately doesn't copy the flags. In other terms, when you try `x[1]=0`, the mask becomes rewritable. That hurts… But! This call to `unshare_mask` is performed only when the mask is 'soft' hence the quick workaround… Note to self (or whomever will fix the issue before I can do it): * We could make sure that copying a mask copies some of its flags to (like the `writeable` one, which other ones?) * The call to `unshare_mask` is made *before* we try to call `__setitem__` on the `_data` part: that's silly, if we called `__setitem__(_data,index,dval)` before, the `ValueError: assignment destination is read-only` would be raised before the mask could get unshared… TLD;DR: move L3073 of np.ma.core to L3068 * There should be some simpler ways to make a masked_array read-only, this little dance is rapidly tiring. Also, some of the ma operations result mask identity of the new ma, which causes ValueError when the new ma mask is modified: x = np.ma.masked_array(xrange(4), [0,1,0,1]) x.flags.writeable = False x.mask.flags.writeable = False x1 = x 0 x1.mask is x.mask # ok False x2 = x != 0 x2.mask is x.mask # ?? True x2.mask[1] = 0 Traceback (most recent call last): ... ValueError: assignment destination is read-only which is a bit confusing. Ouch again. [TL;DR] No workaround, sorry [Long version] The inconsistency comes from the fact that '!=' or '==' call the `__ne__` or `__eq__` methods while other comparison operators call their own function. In the first case, because we're comparing with a non-masked scalar, no copy of the mask is made; in the second case, a copy is systematically made. As pointed out earlier, copies of a mask don't preserve its flags… [Note to self] * Define a factory for __lt__/__le__/__gt__/__ge__ based on __eq__ : MaskedArray.__eq__ and __ne__ already have almost the same code.. (but what about filling? Is it an issue?) And I experienced that *_like operations give mask identity too: y = np.ones_like(x) y.mask is x.mask True This may change in the future, depending on a yet-to-be-achieved consensus on the definition of 'least-surprising behaviour'. Right now, the *-like functions return an array that shares the mask with the input, as you've noticed. Some people complained about it, what's your take on that? I might be missing something but could you clarify these issues? You were not missing anything, np.ma isn't the most straightforward module: plenty of corner cases, and the implementation is pretty naive at times (but hey, it works). My only advice is to never lose hope. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
Ouch… Quick workaround: use `x.harden_mask()` *then* `x.mask.flags.writeable=False` Thanks for the update and the detailed explanation. I'll try this trick. This may change in the future, depending on a yet-to-be-achieved consensus on the definition of 'least-surprising behaviour'. Right now, the *-like functions return an array that shares the mask with the input, as you've noticed. Some people complained about it, what's your take on that? I already took part in the survey (possibly out of thread): http://mail.scipy.org/pipermail/numpy-discussion/2013-July/067136.html You were not missing anything, np.ma isn't the most straightforward module: plenty of corner cases, and the implementation is pretty naive at times (but hey, it works). My only advice is to never lose hope. I agree there are plenty of hard-to-define cases, and I came accross a hot debate on missing data representation in python: https://github.com/njsmith/numpy/wiki/NA-discussion-status but still I believe np.ma is very usable when compression is not strongly needed. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
On Jul 13, 2013, at 13:36 , Gregorio Bastardo gregorio.basta...@gmail.com wrote: Hi Stéfan, Thanks for the suggestion, but it does not protect the array: Thinking about it, it can't: when `x` is a MaskedArray, `x.data` is just a view of the underlying array as a regular ndarray. As far as I understand, changing the `.flags` of a view doesn't affect the original. I'm a bit surprised, though. Here's what I tried np.version.version 1.7.0 x = np.ma.array([1,2,3], mask=[0,1,0]) x.flags.writeable=False x[0]=-1 ValueError: assignment destination is read-only What did you mean by array.flags.writeable = False is perfectly fine, but it does not work on ma-s. ? Could you post what you did and what you got? Moreover, mask hardening only protects masked elements, and does not raise error (as I'd expect). Yes, that's how it supposed to work. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
Hi Stéfan, Thanks for the suggestion, but it does not protect the array: x = np.ma.masked_array(xrange(4), [0,1,0,1]) x masked_array(data = [0 -- 2 --], mask = [False True False True], fill_value = 99) x.mask.flags.writeable = False x.data.flags.writeable = False x.data.flags.writeable True x.mask.flags.writeable False x[0] = -1 x masked_array(data = [-1 -- 2 --], mask = [False True False True], fill_value = 99) Is there a working solution for this problem? Thanks, Gregorio 2013/7/12 Stéfan van der Walt ste...@sun.ac.za: On Fri, Jul 12, 2013 at 4:41 PM, Gregorio Bastardo gregorio.basta...@gmail.com wrote: array.flags.writeable = False is perfectly fine, but it does not work on ma-s. Moreover, mask hardening only protects masked elements, and does not raise error (as I'd expect). You probably have to modify the underlying array and mask: x = np.ma.array(...) x.mask.flags.writeable = False x.data.flags.writeable = False Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] read-only or immutable masked array
Hi, I use masked arrays to mark missing values in data and found it very convenient, although sometimes counterintuitive. I'd like to make a pool of masked arrays (shared between several processing steps) read-only (both data and mask property) to protect the arrays from accidental modification (and the array users from hours of debugging). The regular ndarray trick array.flags.writeable = False is perfectly fine, but it does not work on ma-s. Moreover, mask hardening only protects masked elements, and does not raise error (as I'd expect). Could you recommend an easy way to set an ma read-only? Thanks, Gregorio ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] read-only or immutable masked array
On Fri, Jul 12, 2013 at 4:41 PM, Gregorio Bastardo gregorio.basta...@gmail.com wrote: array.flags.writeable = False is perfectly fine, but it does not work on ma-s. Moreover, mask hardening only protects masked elements, and does not raise error (as I'd expect). You probably have to modify the underlying array and mask: x = np.ma.array(...) x.mask.flags.writeable = False x.data.flags.writeable = False Stéfan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
