Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On 02/13/2012 08:07 PM, Charles R Harris wrote: Let it go, Travis. It's a waste of time. (Off-list) Chuck, I really appreciate your consistent good sense; this is just one of many examples. Thank you for all your numpy work. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Tue, Feb 14, 2012 at 12:05 AM, Eric Firing efir...@hawaii.edu wrote: On 02/13/2012 08:07 PM, Charles R Harris wrote: Let it go, Travis. It's a waste of time. (Off-list) Chuck, I really appreciate your consistent good sense; this is just one of many examples. Thank you for all your numpy work. Not off list. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Tue, Feb 14, 2012 at 12:58 AM, Travis Oliphant tra...@continuum.iowrote: The lack of commutativity wasn't in precision, it was in the typecodes, and was there from the beginning. That caused confusion. A current cause of confusion is the many to one relation of, say, int32 and long, longlong which varies platform to platform. I think that confusion is a more significant problem. Having some types derived from Python types, a correspondence that also varies platform to platform is another source of inconsistent behavior that can be confusing. So there are still plenty of issues to deal with I didn't think it was in the precision. I knew what you meant. However, I'm still hoping for an example of what you mean by lack of commutativity in the typecodes. I made a table back around 1.3 and the lack of symmetry was readily apparent. The confusion of long and longlong varying from platform to platform comes from C. The whole point of having long and longlong is to ensure that you can specify the same types in Python that you would in C. They should not be used if you don't care about that. Deriving from Python types for some array-scalars is an issue. I don't like that either. However, Python itself special-cases it's scalars in ways that necessitated it to have some use-cases not fall-over.This shows a limitation of Python. I would prefer that all array-scalars were recognized appropriately by the Python type system. Most of the concerns that you mention here are mis-understandings. Maybe there are solutions that fix the problem without just educating people. I am open to them. I do think that it was a mistake to have the intp and uintp dtypes as *separate* dtypes. They should have just mapped to the right one. I think it was also a mistake to have dtypes for all the C-spellings instead of just a dtype for each different bit-length with an alias for the C-spellings. We should change that in NumPy 2.0. About the behavior in question, I would frame this as a specific case with argument for and against like so: *The Current Behavior* In [1]: array([127], int8) + 127 Out[1]: array([-2], dtype=int8) In [2]: array([127], int8) + 128 Out[2]: array([255], dtype=int16) *Arguments for Old Behavior* Predictable, explicit output type. This is a good thing, in that no one wants their 8GB int8 array turning into a 16GB int16 array. Backward compatibility. *Arguments for New Behavior* Fewer overflow problems. But no cure. Put that way I think you can make a solid argument for a tweak to restore old behavior. Overflow can be a problem, but partial cures are not going to solve it. I think we do need a way to deal with overflow. Maybe in two ways. 1) saturated operations, i.e., 127 + 128 - 127. This might be good for images. 2) raise an error. We could make specific ufuncs for these behaviors. snip Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, 2012-02-13 at 22:56 -0600, Travis Oliphant wrote: But, I am also aware of *a lot* of users who never voice their opinion on this list, and a lot of features that they want and need and are currently working around the limitations of NumPy to get.These are going to be my primary focus for the rest of the 1.X series. Is that a prompt for feedback? :) (btw, whilst the back slapping is going on, I think Python and Numpy in conjunction with Cython is just how software should be developed nowadays. It truly is a wonderful and winning combo. So a huge thanks to all the developers for making the world a better place.) Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Feb 14, 2012, at 1:47 PM, Charles R Harris wrote: clip About the behavior in question, I would frame this as a specific case with argument for and against like so: The Current Behavior In [1]: array([127], int8) + 127 Out[1]: array([-2], dtype=int8) In [2]: array([127], int8) + 128 Out[2]: array([255], dtype=int16) Good point. Arguments for Old Behavior Predictable, explicit output type. This is a good thing, in that no one wants their 8GB int8 array turning into a 16GB int16 array. Exactly, IIRC this is the main reason why the old behavior was decided. Backward compatibility. Arguments for New Behavior Fewer overflow problems. But no cure. Put that way I think you can make a solid argument for a tweak to restore old behavior. Overflow can be a problem, but partial cures are not going to solve it. I think we do need a way to deal with overflow. Maybe in two ways. 1) saturated operations, i.e., 127 + 128 - 127. This might be good for images. 2) raise an error. We could make specific ufuncs for these behaviors. Hmm, I'm thinking that it would be nice if NumPy could actually support both behaviors. I just wonder whether that should be implemented as a property of each array or as global attribute for the whole NumPy package. While the latter should be easier to implement (what to do when different behaved arrays are being operated?), the former would give more flexibility. I know, this will introduce more complexity in the code base, but anyway, I think that would be a nice thing to support for NumPy 2.0. Just a thought, -- Francesc Alted ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Feb 14, 2012, at 7:04 AM, Henry Gomersall wrote: On Mon, 2012-02-13 at 22:56 -0600, Travis Oliphant wrote: But, I am also aware of *a lot* of users who never voice their opinion on this list, and a lot of features that they want and need and are currently working around the limitations of NumPy to get.These are going to be my primary focus for the rest of the 1.X series. Is that a prompt for feedback? :) Absolutely. That's the reason I'm getting more active on this list.But, at the same time, we all need to be aware of the tens of thousands of users of NumPy who don't use the mailing list and who need a better way to communicate their voice. Even while I have not been active on the mailing list, I have had a chance to communicate with, work with, and collaborate with hundreds of those users and hear about their needs, use-cases, and requirements.It has given me a fairly broad perspective on where particular corners cut in delivery of NumPy 1.0 need to be smoothed off, and where easy-to-add, but essential missing features could be proposed. Some of these have been proposed already, and others will be proposed throughout this year. I look forward to the discussion. -Travis (btw, whilst the back slapping is going on, I think Python and Numpy in conjunction with Cython is just how software should be developed nowadays. It truly is a wonderful and winning combo. So a huge thanks to all the developers for making the world a better place.) Cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Tue, 2012-02-14 at 14:14 -0600, Travis Oliphant wrote: Is that a prompt for feedback? :) Absolutely. That's the reason I'm getting more active on this list. But, at the same time, we all need to be aware of the tens of thousands of users of NumPy who don't use the mailing list and who need a better way to communicate their voice. Great, I'll have a think. I have been pushing quite hard at the boundaries of Numpy in my projects. cheers, Henry ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
Hi,
I recently noticed a change in the upcasting rules in numpy 1.6.0 /
1.6.1 and I just wanted to check it was intentional.
For all versions of numpy I've tested, we have:
import numpy as np
Adata = np.array([127], dtype=np.int8)
Bdata = np.int16(127)
(Adata + Bdata).dtype
dtype('int8')
That is - adding an integer scalar of a larger dtype does not result
in upcasting of the output dtype, if the data in the scalar type fits
in the smaller.
For numpy 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int8')
That is - even if the data in the scalar does not fit in the dtype of
the array to which it is being added, there is no upcasting.
For numpy = 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int16')
There is upcasting...
I can see why the numpy 1.6.0 way might be preferable but it is an API
change I suppose.
Best,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
Hmmm. This seems like a regression. The scalar casting API was fairly
intentional.
What is the reason for the change?
--
Travis Oliphant
(on a mobile)
512-826-7480
On Feb 13, 2012, at 6:25 PM, Matthew Brett matthew.br...@gmail.com wrote:
Hi,
I recently noticed a change in the upcasting rules in numpy 1.6.0 /
1.6.1 and I just wanted to check it was intentional.
For all versions of numpy I've tested, we have:
import numpy as np
Adata = np.array([127], dtype=np.int8)
Bdata = np.int16(127)
(Adata + Bdata).dtype
dtype('int8')
That is - adding an integer scalar of a larger dtype does not result
in upcasting of the output dtype, if the data in the scalar type fits
in the smaller.
For numpy 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int8')
That is - even if the data in the scalar does not fit in the dtype of
the array to which it is being added, there is no upcasting.
For numpy = 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int16')
There is upcasting...
I can see why the numpy 1.6.0 way might be preferable but it is an API
change I suppose.
Best,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 5:00 PM, Travis Oliphant tra...@continuum.iowrote:
Hmmm. This seems like a regression. The scalar casting API was fairly
intentional.
What is the reason for the change?
In order to make 1.6 ABI-compatible with 1.5, I basically had to rewrite
this subsystem. There were virtually no tests in the test suite specifying
what the expected behavior should be, and there were clear inconsistencies
where for example a+b could result in a different type than b+a. I
recall there being some bugs in the tracker related to this as well, but I
don't remember those details.
This change felt like an obvious extension of an existing behavior for
eliminating overflow, where the promotion changed unsigned - signed based
on the value of the scalar. This change introduced minimal upcasting only
in a set of cases where an overflow was guaranteed to happen without that
upcasting.
During the 1.6 beta period, I signaled that this subsystem had changed, as
the bullet point starting The ufunc uses a more consistent algorithm for
loop selection.:
http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
The behavior Matthew has observed is a direct result of how I designed the
minimization function mentioned in that bullet point, and the algorithm for
it is documented in the 'Notes' section of the result_type page:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
Hopefully that explains it well enough. I made the change intentionally and
carefully, tested its impact on SciPy and other projects, and advocated for
it during the release cycle.
Cheers,
Mark
--
Travis Oliphant
(on a mobile)
512-826-7480
On Feb 13, 2012, at 6:25 PM, Matthew Brett matthew.br...@gmail.com
wrote:
Hi,
I recently noticed a change in the upcasting rules in numpy 1.6.0 /
1.6.1 and I just wanted to check it was intentional.
For all versions of numpy I've tested, we have:
import numpy as np
Adata = np.array([127], dtype=np.int8)
Bdata = np.int16(127)
(Adata + Bdata).dtype
dtype('int8')
That is - adding an integer scalar of a larger dtype does not result
in upcasting of the output dtype, if the data in the scalar type fits
in the smaller.
For numpy 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int8')
That is - even if the data in the scalar does not fit in the dtype of
the array to which it is being added, there is no upcasting.
For numpy = 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int16')
There is upcasting...
I can see why the numpy 1.6.0 way might be preferable but it is an API
change I suppose.
Best,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
The problem is that these sorts of things take a while to emerge. The original
system was more consistent than I think you give it credit. What you are
seeing is that most people get NumPy from distributions and are relying on us
to keep things consistent.
The scalar coercion rules were deterministic and based on the idea that a
scalar does not determine the output dtype unless it is of a different kind.
The new code changes that unfortunately.
Another thing I noticed is that I thought that int16 op scalar float would
produce float32 originally. This seems to have changed, but I need to check on
an older version of NumPy.
Changing the scalar coercion rules is an unfortunate substantial change in
semantics and should not have happened in the 1.X series.
I understand you did not get a lot of feedback and spent a lot of time on the
code which we all appreciate. I worked to stay true to the Numeric casting
rules incorporating the changes to prevent scalar upcasting due to the absence
of single precision Numeric literals in Python.
We will need to look in detail at what has changed. I will write a test to do
that.
Thanks,
Travis
--
Travis Oliphant
(on a mobile)
512-826-7480
On Feb 13, 2012, at 7:58 PM, Mark Wiebe mwwi...@gmail.com wrote:
On Mon, Feb 13, 2012 at 5:00 PM, Travis Oliphant tra...@continuum.io wrote:
Hmmm. This seems like a regression. The scalar casting API was fairly
intentional.
What is the reason for the change?
In order to make 1.6 ABI-compatible with 1.5, I basically had to rewrite this
subsystem. There were virtually no tests in the test suite specifying what
the expected behavior should be, and there were clear inconsistencies where
for example a+b could result in a different type than b+a. I recall there
being some bugs in the tracker related to this as well, but I don't remember
those details.
This change felt like an obvious extension of an existing behavior for
eliminating overflow, where the promotion changed unsigned - signed based on
the value of the scalar. This change introduced minimal upcasting only in a
set of cases where an overflow was guaranteed to happen without that
upcasting.
During the 1.6 beta period, I signaled that this subsystem had changed, as
the bullet point starting The ufunc uses a more consistent algorithm for
loop selection.:
http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html
The behavior Matthew has observed is a direct result of how I designed the
minimization function mentioned in that bullet point, and the algorithm for
it is documented in the 'Notes' section of the result_type page:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html
Hopefully that explains it well enough. I made the change intentionally and
carefully, tested its impact on SciPy and other projects, and advocated for
it during the release cycle.
Cheers,
Mark
--
Travis Oliphant
(on a mobile)
512-826-7480
On Feb 13, 2012, at 6:25 PM, Matthew Brett matthew.br...@gmail.com wrote:
Hi,
I recently noticed a change in the upcasting rules in numpy 1.6.0 /
1.6.1 and I just wanted to check it was intentional.
For all versions of numpy I've tested, we have:
import numpy as np
Adata = np.array([127], dtype=np.int8)
Bdata = np.int16(127)
(Adata + Bdata).dtype
dtype('int8')
That is - adding an integer scalar of a larger dtype does not result
in upcasting of the output dtype, if the data in the scalar type fits
in the smaller.
For numpy 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int8')
That is - even if the data in the scalar does not fit in the dtype of
the array to which it is being added, there is no upcasting.
For numpy = 1.6.0 we have this:
Bdata = np.int16(128)
(Adata + Bdata).dtype
dtype('int16')
There is upcasting...
I can see why the numpy 1.6.0 way might be preferable but it is an API
change I suppose.
Best,
Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
I believe the main lessons to draw from this are just how incredibly important a complete test suite and staying on top of code reviews are. I'm of the opinion that any explicit design choice of this nature should be reflected in the test suite, so that if someone changes it years later, they get immediate feedback that they're breaking something important. NumPy has gradually increased its test suite coverage, and when I dealt with the type promotion subsystem, I added fairly extensive tests: https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345 Another subsystem which is in a similar state as what the type promotion subsystem was, is the subscript operator and how regular/fancy indexing work. What this means is that any attempt to improve it that doesn't coincide with the original intent years ago can easily break things that were originally intended without them being caught by a test. I believe this subsystem needs improvement, and the transition to new/improved code will probably be trickier to manage than for the dtype promotion case. Let's try to learn from the type promotion case as best we can, and use it to improve NumPy's process. I believe Charles and Ralph have been doing a great job of enforcing high standards in new NumPy code, and managing the release process in a way that has resulted in very few bugs and regressions in the release. Most of these quality standards are still informal, however, and it's probably a good idea to write them down in a canonical location. It will be especially helpful for newcomers, who can treat the standards as a checklist before submitting pull requests. Thanks, -Mark On Mon, Feb 13, 2012 at 7:11 PM, Travis Oliphant tra...@continuum.iowrote: The problem is that these sorts of things take a while to emerge. The original system was more consistent than I think you give it credit. What you are seeing is that most people get NumPy from distributions and are relying on us to keep things consistent. The scalar coercion rules were deterministic and based on the idea that a scalar does not determine the output dtype unless it is of a different kind. The new code changes that unfortunately. Another thing I noticed is that I thought that int16 op scalar float would produce float32 originally. This seems to have changed, but I need to check on an older version of NumPy. Changing the scalar coercion rules is an unfortunate substantial change in semantics and should not have happened in the 1.X series. I understand you did not get a lot of feedback and spent a lot of time on the code which we all appreciate. I worked to stay true to the Numeric casting rules incorporating the changes to prevent scalar upcasting due to the absence of single precision Numeric literals in Python. We will need to look in detail at what has changed. I will write a test to do that. Thanks, Travis -- Travis Oliphant (on a mobile) 512-826-7480 On Feb 13, 2012, at 7:58 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Feb 13, 2012 at 5:00 PM, Travis Oliphant tra...@continuum.iowrote: Hmmm. This seems like a regression. The scalar casting API was fairly intentional. What is the reason for the change? In order to make 1.6 ABI-compatible with 1.5, I basically had to rewrite this subsystem. There were virtually no tests in the test suite specifying what the expected behavior should be, and there were clear inconsistencies where for example a+b could result in a different type than b+a. I recall there being some bugs in the tracker related to this as well, but I don't remember those details. This change felt like an obvious extension of an existing behavior for eliminating overflow, where the promotion changed unsigned - signed based on the value of the scalar. This change introduced minimal upcasting only in a set of cases where an overflow was guaranteed to happen without that upcasting. During the 1.6 beta period, I signaled that this subsystem had changed, as the bullet point starting The ufunc uses a more consistent algorithm for loop selection.: http://mail.scipy.org/pipermail/numpy-discussion/2011-March/055156.html The behavior Matthew has observed is a direct result of how I designed the minimization function mentioned in that bullet point, and the algorithm for it is documented in the 'Notes' section of the result_type page: http://docs.scipy.org/doc/numpy/reference/generated/numpy.result_type.html Hopefully that explains it well enough. I made the change intentionally and carefully, tested its impact on SciPy and other projects, and advocated for it during the release cycle. Cheers, Mark -- Travis Oliphant (on a mobile) 512-826-7480 On Feb 13, 2012, at 6:25 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, I recently noticed a change in the upcasting rules in numpy 1.6.0 / 1.6.1 and I just wanted to check it was intentional. For all versions of numpy
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 9:04 PM, Travis Oliphant tra...@continuum.iowrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. Well, the typecasting wasn't perfect and, as Mark points out, it wasn't commutative. The addition of float16 also complicated the picture, and user types is going to do more in that direction. And I don't see how a new developer should be responsible for tests enforcing old traditions, the original developers should be responsible for those. But history is history, it didn't happen that way, and here we are. That said, I think we need to show a little flexibility in the corner cases. And going forward I think that typecasting is going to need a rethink. Chuck On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just how incredibly important a complete test suite and staying on top of code reviews are. I'm of the opinion that any explicit design choice of this nature should be reflected in the test suite, so that if someone changes it years later, they get immediate feedback that they're breaking something important. NumPy has gradually increased its test suite coverage, and when I dealt with the type promotion subsystem, I added fairly extensive tests: https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345 Another subsystem which is in a similar state as what the type promotion subsystem was, is the subscript operator and how regular/fancy indexing work. What this means is that any attempt to improve it that doesn't coincide with the original intent years ago can easily break things that were originally intended without them being caught by a test. I believe this subsystem needs improvement, and the transition to new/improved code will probably be trickier to manage than for the dtype promotion case. Let's try to learn from the type promotion case as best we can, and use it to improve NumPy's process. I believe Charles and Ralph have been doing a great job of enforcing high standards in new NumPy code, and managing the release process in a way that has resulted in very few bugs and regressions in the release. Most of these quality standards are still informal, however, and it's probably a good idea to write them down in a canonical location. It will be especially helpful for newcomers, who can treat the standards as a checklist before submitting pull requests. Thanks, -Mark On Mon, Feb 13, 2012 at 7:11 PM, Travis Oliphant tra...@continuum.iowrote: The problem is that these sorts of things take a while to emerge. The original system was more consistent than I think you give it credit. What you are seeing is that most people get NumPy from distributions and are relying on us to keep things consistent. The scalar coercion rules were deterministic and based on the idea that a scalar does not determine the output dtype unless it is of a different kind. The new code changes that unfortunately. Another thing I noticed is that I thought that int16 op scalar float
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
It hasn't changed: since float is of a fundamentally different kind of data, it's expected to upcast the result. However, if I may add a personal comment on numpy's casting rules: until now, I've found them confusing and somewhat inconsistent. Some of the inconsistencies I've found were bugs, while others were unintuitive behavior (or, you may say, me not having the correct intuition ;) In particular the rule about mixed scalar / array operations is currently only described in the doc by a rather vague sentence. Also, the fact that the result's dtype can depend on the actual numerical values can be confusing when you work with variable whose values can span a wide range. So I think if you could come up with a table that says an operation involving two arrays of dtype1 dtype2 always returns an output of dtype3, and a similar table for mixed scalar / array operations, that would be great! My 2 cents, -=- Olivier Le 13 février 2012 23:08, Travis Oliphant tra...@continuum.io a écrit : I can also confirm that at least on NumPy 1.5.1: integer array * (literal Python float scalar) --- creates a double result. So, my memory was incorrect on that (unless it changed at an earlier release, but I don't think so). -Travis On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just how incredibly important a complete test suite and staying on top of code reviews are. I'm of the opinion that any explicit design choice of this nature should be reflected in the test suite, so that if someone changes it years later, they get immediate feedback that they're breaking something important. NumPy has gradually increased its test suite coverage, and when I dealt with the type promotion subsystem, I added fairly extensive tests: https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345 Another subsystem which is in a similar state as what the type promotion subsystem was, is the subscript operator and how regular/fancy indexing work. What this means is that any attempt to improve it that doesn't coincide with the original intent years ago can easily break things that were originally intended without them being caught by a test. I believe this subsystem needs improvement, and the transition to new/improved code will probably be trickier to manage than for the dtype promotion case. Let's try to learn from the type promotion case as best we can, and use it to improve NumPy's process. I believe Charles and Ralph have been doing a great job of enforcing high standards in new NumPy code, and managing the release process in a way that has resulted in very few bugs and regressions in the release. Most of these quality standards are still informal, however, and it's probably a good idea to write them down in a canonical location. It will be especially helpful for newcomers, who can treat the standards as a checklist before submitting pull requests. Thanks, -Mark On Mon, Feb 13, 2012 at 7:11 PM, Travis Oliphant tra...@continuum.iowrote: The problem is that these sorts of things take a while to emerge. The original system was more consistent than I think you give it credit. What you are seeing is that most people get NumPy from distributions and are relying on us to keep things consistent. The scalar coercion rules were deterministic and based on the idea that a scalar does not determine the output dtype unless it is of a different kind. The new code changes that unfortunately. Another thing I noticed is that I thought that int16 op scalar float would produce float32 originally. This seems to have changed, but I need to check on an older version of NumPy. Changing the scalar coercion rules is an unfortunate substantial change in semantics and should not have happened in the 1.X series. I understand you did not get a lot of feedback and spent a lot of time on the code which we all appreciate. I worked to stay true to the Numeric casting rules incorporating the changes to prevent scalar upcasting due to the absence of single precision Numeric literals in Python. We will need to look in detail at what has changed. I will write a test to do that. Thanks, Travis -- Travis Oliphant (on a mobile) 512-826-7480 On Feb 13, 2012, at 7:58 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Feb 13, 2012 at 5:00 PM, Travis Oliphant tra...@continuum.iowrote: Hmmm. This seems like a regression. The scalar casting API was fairly intentional. What is the reason for the change? In order to make 1.6 ABI-compatible with 1.5, I basically had to rewrite this subsystem. There were virtually no tests in the test suite specifying what the expected behavior should be, and there were clear inconsistencies where for example a+b could result in a different type than b+a. I recall there being some bugs in the tracker related to this as well, but I don't remember those details. This change felt
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Monday, February 13, 2012, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Feb 13, 2012 at 9:04 PM, Travis Oliphant tra...@continuum.io wrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. Well, the typecasting wasn't perfect and, as Mark points out, it wasn't commutative. The addition of float16 also complicated the picture, and user types is going to do more in that direction. And I don't see how a new developer should be responsible for tests enforcing old traditions, the original developers should be responsible for those. But history is history, it didn't happen that way, and here we are. That said, I think we need to show a little flexibility in the corner cases. And going forward I think that typecasting is going to need a rethink. Chuck On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just how incredibly important a complete test suite and staying on top of code reviews are. I'm of the opinion that any explicit design choice of this nature should be reflected in the test suite, so that if someone changes it years later, they get immediate feedback that they're breaking something important. NumPy has gradually increased its test suite coverage, and when I dealt with the type promotion subsystem, I added fairly extensive tests: https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345 Another subsystem which is in a similar state as what the type promotion subsystem was, is the subscript operator and how regular/fancy indexing work. What this means is that any attempt to improve it that doesn't coincide with the original intent years ago can easily break things that were originally intended without them being caught by a test. I believe this subsystem needs improvement, and the transition to new/improved code will probably be trickier to manage than for the dtype promotion case. Let's try to learn from the type promotion case as best we can, and use it to improve NumPy's process. I believe Charles and Ralph have been doing a great job of enforcing high standards in new NumPy code, and managing the release process in a way that has resulted in very few bugs and regressions in the release. Most of these quality standards are still informal, however, and it's probably a good idea to write them down in a canonical location. It will be especially helpful for newcomers, who can treat the standards as a checklist before submitting pull requests. Thanks, -Mark On Mon, Feb 13, 2012 at 7:11 PM, Travis Oliphant tra...@continuum.io wrote: The problem is that these sorts of things take a while to emerge. The original system was more consistent than I think you give it credit. What you are seeing is that most people get NumPy from distributions and are relying on us to keep things consistent. The scalar coercion rules were deterministic and based on the idea that a scalar does not determine the output dtype unless it is of a different kind. The new code changes that unfortunately. Another thing I noticed is that I thought
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 8:04 PM, Travis Oliphant tra...@continuum.iowrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). Likely the only way we will be able to know for certain the extent to which our opinions are accurate is to actually dig into the code. I think we can agree, however, that at the very least it could use some performance improvement. :) I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. I did put quite a bit of effort into maintaining compatibility, and was incredibly careful about the change we're discussing. I used something I suspect you created, the can cast safely table here: http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules I extended it to more cases including scalar/array combinations of type promotion, and validated that 1.5 and 1.6 produced the same outputs. The script I used is here: https://github.com/numpy/numpy/blob/master/numpy/testing/print_coercion_tables.py I definitely didn't jump into the change blind, but I did approach it from a clean perspective with the willingness to try and make things better. I understand this is a delicate balance to walk, and I'd like to stress that I didn't take any of the changes I made here lightly. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Well, everything I did for 1.6 that we're discussing here was volunteer work too. :) You and Enthought have all the credit for the later bit where I did get paid a little bit to do the datetime64 and NA stuff! Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. It's great to have you back and active in the community again too. I'm sure this is improving the moods of many NumPy and SciPy users. -Mark -Travis On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just how incredibly important a complete test suite and staying on top of code reviews are. I'm of the opinion that any explicit design choice of this nature should be reflected in the test suite, so that if someone changes it years later, they get immediate feedback that they're breaking something important. NumPy has gradually increased its test suite coverage, and when I dealt with the type promotion subsystem, I added fairly extensive tests: https://github.com/numpy/numpy/blob/master/numpy/core/tests/test_numeric.py#L345 Another subsystem which is in a similar state as what the type promotion subsystem was, is the subscript operator and how regular/fancy indexing work. What this means is that any attempt to improve it that doesn't coincide with the original intent years ago can easily break things that were originally intended without them being caught by a test. I believe this subsystem needs improvement, and the transition to new/improved code will probably be trickier to manage than for the dtype promotion case. Let's try to learn from the type promotion case as best we can, and use it to improve NumPy's process. I believe Charles and Ralph have been doing a great job of enforcing high standards in new NumPy code, and managing the release process in a way that has resulted in very few bugs and regressions in the release. Most of these quality standards are still informal,
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
These are great suggestions. I am happy to start digging into the code. I'm also happy to re-visit any and all design decisions for NumPy 2.0 (with a strong-eye towards helping people migrate and documenting the results). Mark, I think you have done an excellent job of working with a stodgy group and pushing things forward. That is a rare talent, and the world is a better place because you jumped in. There is a lot of cruft all over the place, I know. I also know a lot more now than I did 6 years ago about software design :-)I'm very excited about what we are going to be able to do with NumPy together --- and with the others in the community. But, I am also aware of *a lot* of users who never voice their opinion on this list, and a lot of features that they want and need and are currently working around the limitations of NumPy to get.These are going to be my primary focus for the rest of the 1.X series. I see at least a NumPy 1.8 at this point with maybe even a NumPy 1.9. At the same time, I am looking forward to working with you and others in the community as you lead the push toward NumPy 2.0 (which I hope is not delayed too long with all the possible discussions that can take place :-) ) Best regards, -Travis On Feb 13, 2012, at 10:31 PM, Mark Wiebe wrote: On Mon, Feb 13, 2012 at 8:04 PM, Travis Oliphant tra...@continuum.io wrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). Likely the only way we will be able to know for certain the extent to which our opinions are accurate is to actually dig into the code. I think we can agree, however, that at the very least it could use some performance improvement. :) I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. I did put quite a bit of effort into maintaining compatibility, and was incredibly careful about the change we're discussing. I used something I suspect you created, the can cast safely table here: http://docs.scipy.org/doc/numpy/reference/ufuncs.html#casting-rules I extended it to more cases including scalar/array combinations of type promotion, and validated that 1.5 and 1.6 produced the same outputs. The script I used is here: https://github.com/numpy/numpy/blob/master/numpy/testing/print_coercion_tables.py I definitely didn't jump into the change blind, but I did approach it from a clean perspective with the willingness to try and make things better. I understand this is a delicate balance to walk, and I'd like to stress that I didn't take any of the changes I made here lightly. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Well, everything I did for 1.6 that we're discussing here was volunteer work too. :) You and Enthought have all the credit for the later bit where I did get paid a little bit to do the datetime64 and NA stuff! Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. It's great to have you back and active in the community again too. I'm sure this is improving the moods of many NumPy and SciPy users. -Mark -Travis On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Feb 13, 2012, at 10:14 PM, Charles R Harris wrote: On Mon, Feb 13, 2012 at 9:04 PM, Travis Oliphant tra...@continuum.io wrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. Well, the typecasting wasn't perfect and, as Mark points out, it wasn't commutative. The addition of float16 also complicated the picture, and user types is going to do more in that direction. And I don't see how a new developer should be responsible for tests enforcing old traditions, the original developers should be responsible for those. But history is history, it didn't happen that way, and here we are. That said, I think we need to show a little flexibility in the corner cases. And going forward I think that typecasting is going to need a rethink. No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. New developers are awesome, and the life-blood of a project. But, you have to respect the history of a code-base and if you are re-factoring code that might create a change in corner-cases, then you are absolutely responsible for writing the tests if they aren't there already.That is a pretty simple rule. If you are changing semantics and are not doing a new major version number that you can document the changes in, then any re-factor needs to have tests written *before* the re-factor to ensure behavior does not change. That might be annoying, for sure, and might make you curse the original author for not writing the tests you wish were already written --- but it doesn't change the fact that a released code has many, many tests already written for it in the way of applications and users. All of these are outside of the actual code-base, and may rely on behavior that you can't just change even if you think it needs to change. Bug-fixes are different, of course, but it can sometimes be difficult to discern what is a bug and what is just behavior that seems inappropriate. Type-coercion, in particular, can be a difficult nut to crack because NumPy doesn't always control what happens and is trying to work-within Python's stunted type-system. I've often thought that it might be easier if NumPy were more tightly integrated into Python. For example, it would be great if NumPy's Int-scalar was the same thing as Python's int. Same for float and complex.It would also be nice if you could specify scalar literals with different precisions in Python directly.I've often wished that NumPy developers had more access to all the great language people who have spent their time on IronPython, Jython, and PyPy instead. -Travis Chuck On Feb 13, 2012, at 9:40 PM, Mark Wiebe wrote: I believe the main lessons to draw from this are just how incredibly important
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Monday, February 13, 2012, Travis Oliphant tra...@continuum.io wrote: On Feb 13, 2012, at 10:14 PM, Charles R Harris wrote: On Mon, Feb 13, 2012 at 9:04 PM, Travis Oliphant tra...@continuum.io wrote: I disagree with your assessment of the subscript operator, but I'm sure we will have plenty of time to discuss that. I don't think it's correct to compare the corner cases of the fancy indexing and regular indexing to the corner cases of type coercion system.If you recall, I was quite nervous about all the changes you made to the coercion rules because I didn't believe you fully understood what had been done before and I knew there was not complete test coverage. It is true that both systems have emerged from a long history and could definitely use fresh perspectives which we all appreciate you and others bringing. It is also true that few are aware of the details of how things are actually implemented and that there are corner cases that are basically defined by the algorithm used (this is more true of the type-coercion system than fancy-indexing, however). I think it would have been wise to write those extensive tests prior to writing new code. I'm curious if what you were expecting for the output was derived from what earlier versions of NumPy produced.NumPy has never been in a state where you could just re-factor at will and assume that tests will catch all intended use cases. Numeric before it was not in that state either. This is a good goal, and we always welcome new tests.It just takes a lot of time and a lot of tedious work that the volunteer labor to this point have not had the time to do. Very few of us have ever been paid to work on NumPy directly and have often been trying to fit in improvements to the code base between other jobs we are supposed to be doing.Of course, you and I are hoping to change that this year and look forward to the code quality improving commensurately. Thanks for all you are doing. I also agree that Rolf and Charles have-been and are invaluable in the maintenance and progress of NumPy and SciPy. They deserve as much praise and kudos as anyone can give them. Well, the typecasting wasn't perfect and, as Mark points out, it wasn't commutative. The addition of float16 also complicated the picture, and user types is going to do more in that direction. And I don't see how a new developer should be responsible for tests enforcing old traditions, the original developers should be responsible for those. But history is history, it didn't happen that way, and here we are. That said, I think we need to show a little flexibility in the corner cases. And going forward I think that typecasting is going to need a rethink. No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... First of all, I don't recall the broken commutibility issue --- nor how long it had actually been in the code-base. So, I'm not sure how much weight to give that problem The problem I see with the weighting of these issues that is being implied is that 1) Requiring a re-compile is getting easier and easier as more and more people get their NumPy from distributions and not from downloads of NumPy itself. They just wait until the distribution upgrades and everything is re-compiled. 2) That same trend means that changes to run-time code (like those that can occur when type-coercion is changed) is likely to affect people much later after the discussions have taken place on the list and everyone who was involved in the discussion assumes all is fine. This sort of change should be signaled by a version change.I would like to understand what the bugginess was and where it was better because I think we are painting a wide-brush. Some-things I will probably agree with you were buggy, but others are likely just different preferences. I have a script that documents the old-behavior. I will compare it to the new behavior and we can go from there.Certainly, there is precedent for using something like a __future__ statement to move forward which your boolean switch implies. -Travis Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? Ben Root ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? I think we just leave it as is. If it was a big problem we would have heard screams of complaint long ago. The post that started this off wasn't even a complaint, more of a see this. Spending time reverting or whatever would be a waste of resources, IMHO. Chuck You might be right, Chuck. I would like to investigate more, however. What I fear is that there are *a lot* of users still on NumPy 1.3 and NumPy 1.5. The fact that we haven't heard any complaints, yet, does not mean to me that we aren't creating headache for people later who have just not had time to try things. However, I can believe that the specifics of minor casting rules are probably not relied upon by a lot of codes out there. Still, as Robert Kern often reminds us well --- our intuitions about this are usually not worth much. I may be making more of this then it's worth, I realize. I was just sensitive to it at the time things were changing (even though I didn't have time to be vocal), and now hearing this users experience, it confirms my bias... Believe me, I do not want to revert if at all possible.There is plenty of more work to do, and I'm very much in favor of the spirit of the work Mark was and is doing. Best regards, -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 11:00 PM, Travis Oliphant tra...@continuum.iowrote: No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... First of all, I don't recall the broken commutibility issue --- nor how long it had actually been in the code-base. So, I'm not sure how much weight to give that problem The problem I see with the weighting of these issues that is being implied is that 1) Requiring a re-compile is getting easier and easier as more and more people get their NumPy from distributions and not from downloads of NumPy itself. They just wait until the distribution upgrades and everything is re-compiled. 2) That same trend means that changes to run-time code (like those that can occur when type-coercion is changed) is likely to affect people much later after the discussions have taken place on the list and everyone who was involved in the discussion assumes all is fine. This sort of change should be signaled by a version change.I would like to understand what the bugginess was and where it was better because I think we are painting a wide-brush. Some-things I will probably agree with you were buggy, but others are likely just different preferences. I have a script that documents the old-behavior. I will compare it to the new behavior and we can go from there.Certainly, there is precedent for using something like a __future__ statement to move forward which your boolean switch implies. Let it go, Travis. It's a waste of time. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 11:07 PM, Travis Oliphant tra...@continuum.iowrote: No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? I think we just leave it as is. If it was a big problem we would have heard screams of complaint long ago. The post that started this off wasn't even a complaint, more of a see this. Spending time reverting or whatever would be a waste of resources, IMHO. Chuck You might be right, Chuck. I would like to investigate more, however. What I fear is that there are *a lot* of users still on NumPy 1.3 and NumPy 1.5. The fact that we haven't heard any complaints, yet, does not mean to me that we aren't creating headache for people later who have just not had time to try things. However, I can believe that the specifics of minor casting rules are probably not relied upon by a lot of codes out there. Still, as Robert Kern often reminds us well --- our intuitions about this are usually not worth much. I may be making more of this then it's worth, I realize. I was just sensitive to it at the time things were changing (even though I didn't have time to be vocal), and now hearing this users experience, it confirms my bias... Believe me, I do not want to revert if at all possible.There is plenty of more work to do, and I'm very much in favor of the spirit of the work Mark was and is doing. I think writing tests would be more productive. The current coverage is skimpy in that we typically don't cover *all* the combinations. Sometimes we don't cover any of them ;) I know you are sensitive to the typecasting, it was one of your babies. Nevertheless, I don't think it is that big an issue at the moment. If you can think of ways to *improve* it I think everyone will be interested in that. The lack of commutativity wasn't in precision, it was in the typecodes, and was there from the beginning. That caused confusion. A current cause of confusion is the many to one relation of, say, int32 and long, longlong which varies platform to platform. I think that confusion is a more significant problem. Having some types derived from Python types, a correspondence that also varies platform to platform is another source of inconsistent behavior that can be confusing. So there are still plenty of issues to deal with. I'd like to point out that the addition of float16 necessitated a certain amount of rewriting, as well as the addition of datetime. It was only through Mark's work that we were able to include the latter in the 1.* series at all. Before, we always had to remove datetime before a release, a royal PITA, while waiting on the ever receding 2.0. So there were very good reasons to deal with the type system. That isn't to say that typecasting can't use some tweaks here and there, I think we are all open to discussion along those lines. But it should about specific cases. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 10:38 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Feb 13, 2012 at 11:07 PM, Travis Oliphant tra...@continuum.iowrote: No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? I think we just leave it as is. If it was a big problem we would have heard screams of complaint long ago. The post that started this off wasn't even a complaint, more of a see this. Spending time reverting or whatever would be a waste of resources, IMHO. Chuck You might be right, Chuck. I would like to investigate more, however. What I fear is that there are *a lot* of users still on NumPy 1.3 and NumPy 1.5. The fact that we haven't heard any complaints, yet, does not mean to me that we aren't creating headache for people later who have just not had time to try things. However, I can believe that the specifics of minor casting rules are probably not relied upon by a lot of codes out there. Still, as Robert Kern often reminds us well --- our intuitions about this are usually not worth much. I may be making more of this then it's worth, I realize. I was just sensitive to it at the time things were changing (even though I didn't have time to be vocal), and now hearing this users experience, it confirms my bias... Believe me, I do not want to revert if at all possible.There is plenty of more work to do, and I'm very much in favor of the spirit of the work Mark was and is doing. I think writing tests would be more productive. The current coverage is skimpy in that we typically don't cover *all* the combinations. Sometimes we don't cover any of them ;) I know you are sensitive to the typecasting, it was one of your babies. Nevertheless, I don't think it is that big an issue at the moment. If you can think of ways to *improve* it I think everyone will be interested in that. The lack of commutativity wasn't in precision, it was in the typecodes, and was there from the beginning. That caused confusion. A current cause of confusion is the many to one relation of, say, int32 and long, longlong which varies platform to platform. I think that confusion is a more significant problem. Having some types derived from Python types, a correspondence that also varies platform to platform is another source of inconsistent behavior that can be confusing. So there are still plenty of issues to deal with. This reminds me of something that it would be really nice for the bug tracker to have - user votes. This might be a particularly good way to draw in some more of the users who don't want to stick their neck out with emails and comments, put are comfortable adding a vote to a bug. Something like this: http://code.google.com/p/googleappengine/issues/detail?id=190 where it says that 566 people have starred the issue. -Mark I'd like to point out that the addition of float16 necessitated a certain amount of rewriting, as well as the addition of datetime. It was only through Mark's work that we were able to include the latter in the 1.* series at all. Before, we always had to remove datetime before a release, a royal PITA, while waiting on the ever receding 2.0. So there were very good reasons to deal with the type system. That isn't to say that typecasting can't use some tweaks here and there, I think we are all open to discussion along those lines. But it should about specific cases. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
On Mon, Feb 13, 2012 at 10:48 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Feb 13, 2012 at 10:38 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Feb 13, 2012 at 11:07 PM, Travis Oliphant tra...@continuum.iowrote: No argument on any of this. It's just that this needs to happen at NumPy 2.0, not in the NumPy 1.X series. I think requiring a re-compile is far-less onerous than changing the type-coercion subtly in a 1.5 to 1.6 release. That's my major point, and I'm surprised others are more cavalier about this. I thought the whole datetime debacle was the impetus for binary compatibility? Also, I disagree with your cavalier charge here. When we looked at the rationale for the changes Mark made, the old behavior was not documented, broke commutibility, and was unexpected. So, if it walks like a duck... Now we are in an odd situation. We have undocumented old behavior, and documented new behavior. What do we do? I understand the drive to revert, but I hate the idea of putting back what I see as buggy, especially when new software may fail with old behavior. Maybe a Boolean switch defaulting to new behavior? Anybody having issues with old software could just flip the switch? I think we just leave it as is. If it was a big problem we would have heard screams of complaint long ago. The post that started this off wasn't even a complaint, more of a see this. Spending time reverting or whatever would be a waste of resources, IMHO. Chuck You might be right, Chuck. I would like to investigate more, however. What I fear is that there are *a lot* of users still on NumPy 1.3 and NumPy 1.5. The fact that we haven't heard any complaints, yet, does not mean to me that we aren't creating headache for people later who have just not had time to try things. However, I can believe that the specifics of minor casting rules are probably not relied upon by a lot of codes out there. Still, as Robert Kern often reminds us well --- our intuitions about this are usually not worth much. I may be making more of this then it's worth, I realize. I was just sensitive to it at the time things were changing (even though I didn't have time to be vocal), and now hearing this users experience, it confirms my bias... Believe me, I do not want to revert if at all possible.There is plenty of more work to do, and I'm very much in favor of the spirit of the work Mark was and is doing. I think writing tests would be more productive. The current coverage is skimpy in that we typically don't cover *all* the combinations. Sometimes we don't cover any of them ;) I know you are sensitive to the typecasting, it was one of your babies. Nevertheless, I don't think it is that big an issue at the moment. If you can think of ways to *improve* it I think everyone will be interested in that. The lack of commutativity wasn't in precision, it was in the typecodes, and was there from the beginning. That caused confusion. A current cause of confusion is the many to one relation of, say, int32 and long, longlong which varies platform to platform. I think that confusion is a more significant problem. Having some types derived from Python types, a correspondence that also varies platform to platform is another source of inconsistent behavior that can be confusing. So there are still plenty of issues to deal with. This reminds me of something that it would be really nice for the bug tracker to have - user votes. This might be a particularly good way to draw in some more of the users who don't want to stick their neck out with emails and comments, put are comfortable adding a vote to a bug. Something like this: http://code.google.com/p/googleappengine/issues/detail?id=190 where it says that 566 people have starred the issue. Here's how this feature looks in YouTrack: http://youtrack.jetbrains.net/issues?q=sort+by%3Avotes -Mark -Mark I'd like to point out that the addition of float16 necessitated a certain amount of rewriting, as well as the addition of datetime. It was only through Mark's work that we were able to include the latter in the 1.* series at all. Before, we always had to remove datetime before a release, a royal PITA, while waiting on the ever receding 2.0. So there were very good reasons to deal with the type system. That isn't to say that typecasting can't use some tweaks here and there, I think we are all open to discussion along those lines. But it should about specific cases. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
You might be right, Chuck. I would like to investigate more, however. What I fear is that there are *a lot* of users still on NumPy 1.3 and NumPy 1.5. The fact that we haven't heard any complaints, yet, does not mean to me that we aren't creating headache for people later who have just not had time to try things. However, I can believe that the specifics of minor casting rules are probably not relied upon by a lot of codes out there. Still, as Robert Kern often reminds us well --- our intuitions about this are usually not worth much. I may be making more of this then it's worth, I realize. I was just sensitive to it at the time things were changing (even though I didn't have time to be vocal), and now hearing this users experience, it confirms my bias... Believe me, I do not want to revert if at all possible.There is plenty of more work to do, and I'm very much in favor of the spirit of the work Mark was and is doing. I think writing tests would be more productive. The current coverage is skimpy in that we typically don't cover *all* the combinations. Sometimes we don't cover any of them ;) I know you are sensitive to the typecasting, it was one of your babies. Nevertheless, I don't think it is that big an issue at the moment. If you can think of ways to *improve* it I think everyone will be interested in that. First of all, I would hardly call it one of my babies. I care far more for my actual babies than for this.It was certainly one of my headaches that I had to deal with and write code for (and take into account previous behavior with). I certainly spent a lot of time wrestling with type-coercion and integrating numerous opinions as quickly as I could with it --- even in Numeric with the funny down_casting arrays. At best the resulting system was a compromise (with an implementation that you could reason about with the right perspective despite claims to the contrary). This discussion is not about me being sensitive because I wrote some code or had a hand in a design that needed changing. I hope we replace all the code I've written with something better. I expect that eventually. This just has to be done in an appropriate way. I'm sensitive because I understand where the previous code came from and *why it was written* and am concerned about changing things out from under users in ways that are subtle. I continue to affirm that breaking ABI compatibility is much preferable to changing type-casting behavior. I know people disagree with me. But, distributions help solve the ABI compatibility problem, but nothing solves required code changes due to subtle type-casting issues. I would just expect this sort of change at NumPy 2.0. We could have waited for half-float until then. I will send the result of my analysis shortly on what changed between 1.5.1 and 1.6.1 -Travis ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Change in scalar upcasting rules for 1.6.x?
The lack of commutativity wasn't in precision, it was in the typecodes, and was there from the beginning. That caused confusion. A current cause of confusion is the many to one relation of, say, int32 and long, longlong which varies platform to platform. I think that confusion is a more significant problem. Having some types derived from Python types, a correspondence that also varies platform to platform is another source of inconsistent behavior that can be confusing. So there are still plenty of issues to deal with I didn't think it was in the precision. I knew what you meant. However, I'm still hoping for an example of what you mean by lack of commutativity in the typecodes. The confusion of long and longlong varying from platform to platform comes from C. The whole point of having long and longlong is to ensure that you can specify the same types in Python that you would in C. They should not be used if you don't care about that. Deriving from Python types for some array-scalars is an issue. I don't like that either. However, Python itself special-cases it's scalars in ways that necessitated it to have some use-cases not fall-over.This shows a limitation of Python. I would prefer that all array-scalars were recognized appropriately by the Python type system. Most of the concerns that you mention here are mis-understandings. Maybe there are solutions that fix the problem without just educating people. I am open to them. I do think that it was a mistake to have the intp and uintp dtypes as *separate* dtypes. They should have just mapped to the right one. I think it was also a mistake to have dtypes for all the C-spellings instead of just a dtype for each different bit-length with an alias for the C-spellings. We should change that in NumPy 2.0. -Travis I'd like to point out that the addition of float16 necessitated a certain amount of rewriting, as well as the addition of datetime. It was only through Mark's work that we were able to include the latter in the 1.* series at all. Before, we always had to remove datetime before a release, a royal PITA, while waiting on the ever receding 2.0. So there were very good reasons to deal with the type system. That isn't to say that typecasting can't use some tweaks here and there, I think we are all open to discussion along those lines. But it should about specific cases. Chuck ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
