Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 11:38 PM, Travis Oliphant tra...@continuum.iowrote: On May 20, 2012, at 12:15 AM, Charles R Harris wrote: On Sat, May 19, 2012 at 10:48 PM, Travis Oliphant tra...@continuum.iowrote: My own plan for the near term would be as follows: 1) Put in the experimental option and get the 1.7 release out. This gets us through the next couple of months and keeps things moving. The experimental option does not solve the problem which is that the ndarray object now has masked fields which changes the fundamental nature of an ndarray for a lot of downstream users that really have no idea what has just happened.I don't see how this has been addressed by any proposal except for the one I have suggested which allows a masked array object and a regular ndarray to co-exist for a time.I doubt that the proposal actually helps get 1.7 out any faster either as there are multiple experimental APIs that would have to be created to pull it off on both the C and Python level. So, remove them in 1.8 and try something else. With experimental (say in site.cfg), the base array could even be different. I don't see the problem here. Think big. I don't think I understand your mental model of this.Are you saying add an experimental flag at the C-level (essentially a #define that eliminates any code involving masked arrays unless the define is made at compile time?) It seems like just applying Nathaniel's patch would be a better approach. Do so then. Otherwise I am going to fork. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. Yes, I'm sufficiently geared up about that that I'd like to try to do something; perhaps try to put the support in the C API for masks in a vtable (so that the functions can take happily take PyObject* rather than PyArrayObject*, and thus be used for other ways of associating masks with array data down the line, so that I could for instance make a class outside of numpy with a 'shared mask'). My available time is an extremely stochastic quantity at the moment though; so I really can't say how relevant that is to 1.7 yet. You'll see. (My discussion was really for the long term, as in, I'd like to do something over the coming year...) Dag One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. Best regards, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.iowrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 3:17 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.iowrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Agreed. 1.6.0 was released one year ago already, let's focus on polishing what's in there *now*. I have not followed closely what the decision was for a LTS release, but if 1.7 is supposed to be it, that's another argument about changing anything there for 1.7. David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 4:21 PM, Mark Wiebe mwwi...@gmail.com wrote: The motivation behind splitting the mask out into a separate ndmasked is primarily so that pre-existing code will not silently function on NA-masked arrays and produce incorrect results. This centres around using PyArray_DATA to get at the data after manually checking flags, instead of calling PyArray_FromAny. Maybe a reasonable solution is to tweak the behavior of PyArray_DATA? It could work as follows: - If an ndarray has no mask, PyArray_DATA returns the data pointer as it does currently. - If the ndarray has an NA-mask, PyArray_DATA sets an exception and returns NULL - Create a new accessor, PyArray_DATAPTR or PyArray_RAWDATA, which returns the array data under all circumstances. This way, code which currently uses the data pointer through PyArray_DATA will fail instead of silently working with the wrong interpretation of the data. What do people feel about this idea? By fail you mean, specifically, segfault, right? PyArray_DATA currently cannot fail or return NULL, so I doubt any existing code checks for exceptions before dereferencing the pointer it returns. - N ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 9:21 AM, Mark Wiebe mwwi...@gmail.com wrote: On Sat, May 19, 2012 at 10:00 AM, David Cournapeau courn...@gmail.comwrote: On Sat, May 19, 2012 at 3:17 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.iowrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Agreed. 1.6.0 was released one year ago already, let's focus on polishing what's in there *now*. I have not followed closely what the decision was for a LTS release, but if 1.7 is supposed to be it, that's another argument about changing anything there for 1.7. The motivation behind splitting the mask out into a separate ndmasked is primarily so that pre-existing code will not silently function on NA-masked arrays and produce incorrect results. This centres around using PyArray_DATA to get at the data after manually checking flags, instead of calling PyArray_FromAny. Maybe a reasonable solution is to tweak the behavior of PyArray_DATA? It could work as follows: - If an ndarray has no mask, PyArray_DATA returns the data pointer as it does currently. - If the ndarray has an NA-mask, PyArray_DATA sets an exception and returns NULL - Create a new accessor, PyArray_DATAPTR or PyArray_RAWDATA, which returns the array data under all circumstances. This way, code which currently uses the data pointer through PyArray_DATA will fail instead of silently working with the wrong interpretation of the data. What do people feel about this idea? Code working with the wrong interpretation of the data doesn't bother me much at this point in development. Long term it matters, but in the short term we can't expect code not explicitly written to work with masked arrays to do the right thing. I think we are looking at a period of several years before things settle out and get accepted. First, the implementation and its interface needs to get close to final form, and then the long slow process of adoption into things like matplotlib needs to take place. I'd quess three to five years for that process. That said, my main concern is to move forward and not spend the next year waiting. I see splitting the masked code out as rather like the python types having pointers to sequence/numerical/etc methods, i.e., ndarray then looks something like an abstract class. I don't have
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 10:02 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Sat, May 19, 2012 at 9:21 AM, Mark Wiebe mwwi...@gmail.com wrote: On Sat, May 19, 2012 at 10:00 AM, David Cournapeau courn...@gmail.comwrote: On Sat, May 19, 2012 at 3:17 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.iowrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Agreed. 1.6.0 was released one year ago already, let's focus on polishing what's in there *now*. I have not followed closely what the decision was for a LTS release, but if 1.7 is supposed to be it, that's another argument about changing anything there for 1.7. The motivation behind splitting the mask out into a separate ndmasked is primarily so that pre-existing code will not silently function on NA-masked arrays and produce incorrect results. This centres around using PyArray_DATA to get at the data after manually checking flags, instead of calling PyArray_FromAny. Maybe a reasonable solution is to tweak the behavior of PyArray_DATA? It could work as follows: - If an ndarray has no mask, PyArray_DATA returns the data pointer as it does currently. - If the ndarray has an NA-mask, PyArray_DATA sets an exception and returns NULL - Create a new accessor, PyArray_DATAPTR or PyArray_RAWDATA, which returns the array data under all circumstances. This way, code which currently uses the data pointer through PyArray_DATA will fail instead of silently working with the wrong interpretation of the data. What do people feel about this idea? Code working with the wrong interpretation of the data doesn't bother me much at this point in development. Long term it matters, but in the short term we can't expect code not explicitly written to work with masked arrays to do the right thing. I think we are looking at a period of several years before things settle out and get accepted. First, the implementation and its interface needs to get close to final form, and then the long slow process of adoption into things like matplotlib needs to take place. I'd quess three to five years for that process. That said, my main concern is to move forward and not spend the next year waiting. I see splitting the masked code out as rather like the python types having
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On Sat, May 19, 2012 at 5:45 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Sat, May 19, 2012 at 10:02 AM, Charles R Harris charlesr.har...@gmail.com wrote: On Sat, May 19, 2012 at 9:21 AM, Mark Wiebe mwwi...@gmail.com wrote: On Sat, May 19, 2012 at 10:00 AM, David Cournapeau courn...@gmail.com wrote: On Sat, May 19, 2012 at 3:17 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us). If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Agreed. 1.6.0 was released one year ago already, let's focus on polishing what's in there *now*. I have not followed closely what the decision was for a LTS release, but if 1.7 is supposed to be it, that's another argument about changing anything there for 1.7. The motivation behind splitting the mask out into a separate ndmasked is primarily so that pre-existing code will not silently function on NA-masked arrays and produce incorrect results. This centres around using PyArray_DATA to get at the data after manually checking flags, instead of calling PyArray_FromAny. Maybe a reasonable solution is to tweak the behavior of PyArray_DATA? It could work as follows: - If an ndarray has no mask, PyArray_DATA returns the data pointer as it does currently. - If the ndarray has an NA-mask, PyArray_DATA sets an exception and returns NULL - Create a new accessor, PyArray_DATAPTR or PyArray_RAWDATA, which returns the array data under all circumstances. This way, code which currently uses the data pointer through PyArray_DATA will fail instead of silently working with the wrong interpretation of the data. What do people feel about this idea? Code working with the wrong interpretation of the data doesn't bother me much at this point in development. Long term it matters, but in the short term we can't expect code not explicitly written to work with masked arrays to do the right thing. I think we are looking at a period of several years before things settle out and get accepted. First, the implementation and its interface needs to get close to final form, and then the long slow process of adoption into things like matplotlib needs to take place. I'd quess three to five years for that process. That said, my main concern is to move forward and not spend the
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On May 19, 2012, at 9:17 AM, Charles R Harris wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. I don't see how option 2 gets 1.7 out the door any easier as it does not address the actual problems with the changes to the ndarray object that are currently in master and it introduces a new experimental flag concept in a hurried way just to get a release out.This helps the Python side of things, but does nothing to address the C-side of things. Fundamentally, I don't see how we can put 1.7 out with the masked array fields on the ndarray object itself. This is clear from everything I've seen. -Travis Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On May 19, 2012, at 10:00 AM, David Cournapeau wrote: On Sat, May 19, 2012 at 3:17 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, May 18, 2012 at 3:47 PM, Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. This will be a difficult thing for others to help with since the concept is vague, the design decisions seem to be in your and Mark's hands, and you say you don't have much time. It looks to me like 1.7 will keep slipping and I don't think that is a good thing. Why not go for option 2, which will get 1.7 out there and push the new masked array work in to 1.8? Breaking the flow of development and release has consequences, few of them good. Agreed. 1.6.0 was released one year ago already, let's focus on polishing what's in there *now*. I have not followed closely what the decision was for a LTS release, but if 1.7 is supposed to be it, that's another argument about changing anything there for 1.7. This won't work, because what is in there *now* really shouldn't be there. The only way this would work is if we remove the masked array from the ndarray object. I think I saw that Nathaniel has a patch for this, but I have not had time to review this patch. There has been no agreement on an LTS release. The best candidate for that is 1.6.2, but I don't think it makes sense to talk about an LTS release until the masked array questions are resolved from what is there in master right now. -Travis David ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
My own plan for the near term would be as follows: 1) Put in the experimental option and get the 1.7 release out. This gets us through the next couple of months and keeps things moving. The experimental option does not solve the problem which is that the ndarray object now has masked fields which changes the fundamental nature of an ndarray for a lot of downstream users that really have no idea what has just happened. I don't see how this has been addressed by any proposal except for the one I have suggested which allows a masked array object and a regular ndarray to co-exist for a time.I doubt that the proposal actually helps get 1.7 out any faster either as there are multiple experimental APIs that would have to be created to pull it off on both the C and Python level. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
My own plan for the near term would be as follows: 1) Put in the experimental option and get the 1.7 release out. This gets us through the next couple of months and keeps things moving. +1 on not blocking the release while we invent+implement yet another experimental API. Nobody has suggested inventing or implementing yet another anything. All I want to do is separate out the ndmasked array as a separate object and leave the ndarray object alone. 2) Look at what hooks/low level functions would let us reimplement np.ma. Because there are so many different mask uses out there, this would be a good way to discover what low level support is likely to provide a good basis for others to build on. 3) Revisit the idea of making all ndarrays masked by default, but do so with the experience and feedback from current mask users. I like this plan. I think points 2 and 3 are fine as long as the 1.7 release does not have masked array notions attached to all ndarray objects. -Travis -- Nathaniel ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
On May 20, 2012, at 12:15 AM, Charles R Harris wrote: On Sat, May 19, 2012 at 10:48 PM, Travis Oliphant tra...@continuum.io wrote: My own plan for the near term would be as follows: 1) Put in the experimental option and get the 1.7 release out. This gets us through the next couple of months and keeps things moving. The experimental option does not solve the problem which is that the ndarray object now has masked fields which changes the fundamental nature of an ndarray for a lot of downstream users that really have no idea what has just happened.I don't see how this has been addressed by any proposal except for the one I have suggested which allows a masked array object and a regular ndarray to co-exist for a time.I doubt that the proposal actually helps get 1.7 out any faster either as there are multiple experimental APIs that would have to be created to pull it off on both the C and Python level. So, remove them in 1.8 and try something else. With experimental (say in site.cfg), the base array could even be different. I don't see the problem here. Think big. I don't think I understand your mental model of this.Are you saying add an experimental flag at the C-level (essentially a #define that eliminates any code involving masked arrays unless the define is made at compile time?) It seems like just applying Nathaniel's patch would be a better approach. -Travis snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
Travis, We have a significant user base for masked arrays, with a lot of real-life experience, use-cases and data. We would really like to get involved on this, please keep us in the loop. C. On 5/18/12 2:47 PM, Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. Best regards, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Masked Array for NumPy 1.7
The best way to keep in the loop is to comment on this list and pay attention to threads that discuss it.Thank you for speaking up, as I was aware of your significant use of the current masked array in NumPy, but it is good when you can articulate your use-cases and APIs that are helpful or annoying to you. -Travis On May 18, 2012, at 4:54 PM, Doutriaux, Charles wrote: Travis, We have a significant user base for masked arrays, with a lot of real-life experience, use-cases and data. We would really like to get involved on this, please keep us in the loop. C. On 5/18/12 2:47 PM, Travis Oliphant tra...@continuum.io wrote: Hey all, After reading all the discussion around masked arrays and getting input from as many people as possible, it is clear that there is still disagreement about what to do, but there have been some fruitful discussions that ensued. This isn't really new as there was significant disagreement about what to do when the masked array code was initially checked in to master. So, in order to move forward, Mark and I are going to work together with whomever else is willing to help with an effort that is in the spirit of my third proposal but has a few adjustments. The idea will be fleshed out in more detail as it progresses, but the basic concept is to create an (experimental) ndmasked object in NumPy 1.7 and leave the actual ndarray object unchanged. While the details need to be worked out here, a goal is to have the C-API work with both ndmasked arrays and arrayobjects (possibly by defining a base-class C-level structure that both ndarrays inherit from). This might also be a good way for Dag to experiment with his ideas as well but that is not an explicit goal. One way this could work, for example is to have PyArrayObject * be the base-class array (essentially the same C-structure we have now with a HASMASK flag). Then, the ndmasked object could inherit from PyArrayObject * as well but add more members to the C-structure. I think this is the easiest thing to do and requires the least amount of code-change. It is also possible to define an abstract base-class PyArrayObject * that both ndarray and ndmasked inherit from. That way ndarray and ndmasked are siblings even though the ndarray would essentially *be* the PyArrayObject * --- just with a different type-hierarchy on the python side. This work will take some time and, therefore, I don't expect 1.7 to be released prior to SciPy Austin with an end of June target date. The timing will largely depend on what time is available from people interested in resolving the situation. Mark and I will have some availability for this work in June but not a great deal (about 2 man-weeks total between us).If there are others who can step in and help, it will help accelerate the process. Best regards, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
