To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
np.ones((2,N)).mean(1) produce different results with the implementation in
the current master? If so, I think that would be very much regrettable; and
if this is a minority opinion, I do hope that at least this gets documented
On Mo, 2014-07-28 at 14:37 +0200, Eelco Hoogendoorn wrote:
To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
np.ones((2,N)).mean(1) produce different results with the
implementation in the current master? If so, I think that would be
very much regrettable; and if this is a
On Mon, Jul 28, 2014 at 8:46 AM, Sebastian Berg
sebast...@sipsolutions.net wrote:
On Mo, 2014-07-28 at 14:37 +0200, Eelco Hoogendoorn wrote:
To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
np.ones((2,N)).mean(1) produce different results with the
implementation in the
On 28 July 2014 14:46, Sebastian Berg sebast...@sipsolutions.net wrote:
To rephrase my most pressing question: may np.ones((N,2)).mean(0) and
np.ones((2,N)).mean(1) produce different results with the
implementation in the current master? If so, I think that would be
very much regrettable;
On 28/07/14 15:21, alex wrote:
Are you sure they always give different results? Notice that
np.ones((N,2)).mean(0)
np.ones((2,N)).mean(1)
compute means of different axes on transposed arrays so these
differences 'cancel out'.
They will be if different algorithms are used.
On 28.07.2014 15:30, Daπid wrote:
An example using float16 on Numpy 1.8.1 (I haven't seen diferences with
float32):
Why aren't there differences between float16 and float32 ?
Could this be related to my earlier post in this thread where I
mentioned summation problems occurring much earlier in
On Mo, 2014-07-28 at 15:35 +0200, Sturla Molden wrote:
On 28/07/14 15:21, alex wrote:
Are you sure they always give different results? Notice that
np.ones((N,2)).mean(0)
np.ones((2,N)).mean(1)
compute means of different axes on transposed arrays so these
differences 'cancel out'.
On Mo, 2014-07-28 at 15:50 +0200, Fabien wrote:
On 28.07.2014 15:30, Daπid wrote:
An example using float16 on Numpy 1.8.1 (I haven't seen diferences with
float32):
Why aren't there differences between float16 and float32 ?
float16 calculations are actually float32 calculations. If done
Sebastian:
Those are good points. Indeed iteration order may already produce different
results, even though the semantics of numpy suggest identical operations.
Still, I feel this different behavior without any semantical clues is
something to be minimized.
Indeed copying might have large speed
On Mo, 2014-07-28 at 16:31 +0200, Eelco Hoogendoorn wrote:
Sebastian:
Those are good points. Indeed iteration order may already produce
different results, even though the semantics of numpy suggest
identical operations. Still, I feel this different behavior without
any semantical clues is
I see, thanks for the clarification. Just for the sake of argument, since
unfortunately I don't have the time to go dig in the guts of numpy myself:
a design which always produces results of the same (high) accuracy, but
only optimizes the common access patterns in a hacky way, and may be
On 28.07.2014 23:32, Eelco Hoogendoorn wrote:
I see, thanks for the clarification. Just for the sake of argument,
since unfortunately I don't have the time to go dig in the guts of numpy
myself: a design which always produces results of the same (high)
accuracy, but only optimizes the common
On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden sturla.mol...@gmail.com
wrote:
Robert Kern robert.k...@gmail.com wrote:
It would presumably require a global threading.RLock for protecting the
global state.
We would use thread-local storage like we currently do with the
np.errstate()
On Sun, Jul 27, 2014 at 7:04 AM, josef.p...@gmail.com wrote:
On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden sturla.mol...@gmail.com
wrote:
Robert Kern robert.k...@gmail.com wrote:
It would presumably require a global threading.RLock for protecting the
global state.
We would use
On Sun, Jul 27, 2014 at 4:24 AM, Robert Kern robert.k...@gmail.com wrote:
On Sun, Jul 27, 2014 at 7:04 AM, josef.p...@gmail.com wrote:
On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden sturla.mol...@gmail.com
wrote:
Robert Kern robert.k...@gmail.com wrote:
It would presumably
On Sun, Jul 27, 2014 at 9:56 AM, josef.p...@gmail.com wrote:
On Sun, Jul 27, 2014 at 4:24 AM, Robert Kern robert.k...@gmail.com wrote:
On Sun, Jul 27, 2014 at 7:04 AM, josef.p...@gmail.com wrote:
On Sat, Jul 26, 2014 at 5:19 PM, Sturla Molden sturla.mol...@gmail.com
wrote:
Robert
At 02:04 AM 7/27/2014, you wrote:
You won't be able to do it by accident or omission or a lack of
discipline. It's not a tempting public target like, say, np.seterr().
BTW, why not throw an overflow error in the large float32 sum() case?
Is it too expensive to check while accumulating?
- Ray
On Sun, Jul 27, 2014 at 3:16 PM, RayS r...@blue-cove.com wrote:
At 02:04 AM 7/27/2014, you wrote:
You won't be able to do it by accident or omission or a lack of
discipline. It's not a tempting public target like, say, np.seterr().
BTW, why not throw an overflow error in the large float32
Thanks for the clarification, but how is the numpy rounding directed?
Round to nearest, ties to even?
http://en.wikipedia.org/wiki/IEEE_floating_point#Rounding_rules
Just curious, as I couldn't find a reference.
- Ray
At 07:44 AM 7/27/2014, you wrote:
On Sun, Jul 27, 2014 at 3:16 PM, RayS
Nathaniel Smith n...@pobox.com wrote:
The problem here is that when summing up the values, the sum gets
large enough that after rounding, x + 1 = x and the sum stops
increasing.
Interesting. That explains why the divide-and-conquer reduction is much
more robust.
Thanks :)
Sturla
-
From: Julian Taylor jtaylor.deb...@googlemail.com
Sent: 26-7-2014 00:58
To: Discussion of Numerical Python numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
On 25.07.2014 23:51, Eelco Hoogendoorn wrote:
Ray: I'm not working with Hubble
On Fr, 2014-07-25 at 21:23 +0200, Eelco Hoogendoorn wrote:
It need not be exactly representable as such; take the mean of [1, 1
+eps] for instance. Granted, there are at most two number in the range
of the original dtype which are closest to the true mean; but im not
sure that computing them
Sebastian Berg sebast...@sipsolutions.net wrote:
chose more stable algorithms for such statistical functions. The
pairwise summation that is in master now is very awesome, but it is not
secure enough in the sense that a new user will have difficulty
understanding when he can be sure it is
I was wondering the same thing. Are there any known tradeoffs to this
method of reduction?
On Sat, Jul 26, 2014 at 12:39 PM, Sturla Molden sturla.mol...@gmail.com
wrote:
Sebastian Berg sebast...@sipsolutions.net wrote:
chose more stable algorithms for such statistical functions. The
On 26.07.2014 15:38, Eelco Hoogendoorn wrote:
Why is it not always used?
for 1d reduction the iterator blocks by 8192 elements even when no
buffering is required. There is a TODO in the source to fix that by
adding additional checks. Unfortunately nobody knows hat these
additional tests would
I could get behind the context manager approach. It would help keep
backwards compatibility, while providing a very easy (and clean) way of
consistently using the same reduction operation. Adding kwargs is just a
road to hell.
Cheers!
Ben Root
On Sat, Jul 26, 2014 at 9:53 AM, Julian Taylor
On Sa, 2014-07-26 at 15:38 +0200, Eelco Hoogendoorn wrote:
I was wondering the same thing. Are there any known tradeoffs to this
method of reduction?
Yes, it is much more complicated and incompatible with naive ufuncs if
you want your memory access to be optimized. And optimizing that is very
A context manager makes sense.
I very much appreciate the time constraints and the effort put in this far,
but if we can not make something work uniformly, I wonder if we should
include it in the master at all. I don't have a problem with customizing
algorithms where fp accuracy demands it; I
Sebastian Berg sebast...@sipsolutions.net wrote:
Yes, it is much more complicated and incompatible with naive ufuncs if
you want your memory access to be optimized. And optimizing that is very
much worth it speed wise...
Why? Couldn't we just copy the data chunk-wise to a temporary buffer of
Sturla Molden sturla.mol...@gmail.com wrote:
Sebastian Berg sebast...@sipsolutions.net wrote:
Yes, it is much more complicated and incompatible with naive ufuncs if
you want your memory access to be optimized. And optimizing that is very
much worth it speed wise...
Why? Couldn't we just
On Sat, Jul 26, 2014 at 9:57 AM, Benjamin Root ben.r...@ou.edu wrote:
I could get behind the context manager approach. It would help keep
backwards compatibility, while providing a very easy (and clean) way of
consistently using the same reduction operation. Adding kwargs is just a
road to
That is one way of doing it, and probably the cleanest way. Or else you
have to pass in the context object everywhere anyway. But I am not so
concerned about that (we do that for other things as well). Bigger concerns
would be nested contexts. For example, what if one of the scikit functions
use
On Sat, Jul 26, 2014 at 2:44 PM, Benjamin Root ben.r...@ou.edu wrote:
That is one way of doing it, and probably the cleanest way. Or else you
have to pass in the context object everywhere anyway. But I am not so
concerned about that (we do that for other things as well). Bigger concerns
would
Benjamin Root ben.r...@ou.edu wrote:
My other concern would be with multi-threaded code (which is where a global
state would be bad).
It would presumably require a global threading.RLock for protecting the
global state.
Sturla
___
NumPy-Discussion
Perhaps I in turn am missing something; but I would suppose that any
algorithm that requires multiple passes over the data is off the table?
Perhaps I am being a little old fashioned and performance oriented here,
but to make the ultra-majority of use cases suffer a factor two performance
penalty
josef.p...@gmail.com wrote:
statsmodels still has avoided anything that smells like a global state that
changes calculation.
If global states are stored in a stack, as in OpenGL, it is not so bad. A
context manager could push a state in __enter__ and pop the state in
__exit__. This is actually
I completely agree with Eelco. I expect numpy.mean to do something
simple and straightforward. If the naive method is not well suited for
my data, I can deal with it and have my own ad hoc method.
On Sat, Jul 26, 2014 at 3:19 PM, Eelco Hoogendoorn
hoogendoorn.ee...@gmail.com wrote:
Perhaps I in
On Sat, Jul 26, 2014 at 8:04 PM, Sturla Molden sturla.mol...@gmail.com wrote:
Benjamin Root ben.r...@ou.edu wrote:
My other concern would be with multi-threaded code (which is where a global
state would be bad).
It would presumably require a global threading.RLock for protecting the
global
Robert Kern robert.k...@gmail.com wrote:
It would presumably require a global threading.RLock for protecting the
global state.
We would use thread-local storage like we currently do with the
np.errstate() context manager. Each thread will have its own global
state.
That sounds like a
: 25-7-2014 00:10
To: Discussion of Numerical Python numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for
largefloat32arrays
On 7/24/2014 4:42 PM, Eelco Hoogendoorn wrote:
This isn't a bug report, but rather a feature request.
I'm not sure statement
At 01:22 AM 7/25/2014, you wrote:
Actually the maximum precision I am not so
sure of, as I personally prefer to make an
informed decision about precision used, and get
an error on a platform that does not support
the specified precision, rather than obtain
subtly or horribly broken
On Fri, Jul 25, 2014 at 3:11 PM, RayS r...@blue-cove.com wrote:
At 01:22 AM 7/25/2014, you wrote:
Actually the maximum precision I am not so
sure of, as I personally prefer to make an
informed decision about precision used, and get
an error on a platform that does not support
the specified
At 07:22 AM 7/25/2014, you wrote:
We were talking on this in the office, as we
realized it does affect a couple of lines dealing
with large arrays, including complex64.
As I expect Python modules to work uniformly
cross platform unless documented otherwise, to me
that includes 32 vs 64 bit
: RayS r...@blue-cove.com
Sent: 25-7-2014 19:56
To: Discussion of Numerical Python numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
At 07:22 AM 7/25/2014, you wrote:
We were talking on this in the office, as we
realized it does affect
On 7/25/2014 1:40 PM, Eelco Hoogendoorn wrote:
At the risk of repeating myself: explicit is better than implicit
This sounds like an argument for renaming the `mean` function `naivemean`
rather than `mean`. Whatever numpy names `mean`, shouldn't it
implement an algorithm that produces the
On Fri, Jul 25, 2014 at 5:56 PM, RayS r...@blue-cove.com wrote:
The important point was that it would be best if all of the methods affected
by summing 32 bit floats with 32 bit accumulators had the same Notes as
numpy.mean(). We went through a lot of code yesterday, assuming that any
numpy or
It need not be exactly representable as such; take the mean of [1, 1+eps]
for instance. Granted, there are at most two number in the range of the
original dtype which are closest to the true mean; but im not sure that
computing them exactly is a tractable problem for arbitrary input.
Im not sure
At 11:29 AM 7/25/2014, you wrote:
On Fri, Jul 25, 2014 at 5:56 PM, RayS r...@blue-cove.com wrote:
The important point was that it would be best if all of the
methods affected
by summing 32 bit floats with 32 bit accumulators had the same Notes as
numpy.mean(). We went through a lot of code
On Fri, Jul 25, 2014 at 4:25 PM, RayS r...@blue-cove.com wrote:
At 11:29 AM 7/25/2014, you wrote:
On Fri, Jul 25, 2014 at 5:56 PM, RayS r...@blue-cove.com wrote:
The important point was that it would be best if all of the
methods affected
by summing 32 bit floats with 32 bit
-cove.com
Sent: 25-7-2014 23:26
To: Discussion of Numerical Python numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
At 11:29 AM 7/25/2014, you wrote:
On Fri, Jul 25, 2014 at 5:56 PM, RayS r...@blue-cove.com wrote:
The important point
).
From: RayS mailto:r...@blue-cove.com
Sent: 25-7-2014 23:26
To: Discussion of Numerical Python mailto:numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for
largefloat32arrays
At 11:29 AM 7/25/2014, you wrote:
On Fri
At 02:36 PM 7/25/2014, you wrote:
But it doesn't compensate for users to be aware of the problems. I
think the docstring and the description of the dtype argument is pretty clear.
Most of the docs for the affected functions do not have a Note with
the same warning as mean()
- Ray
alan.is...@gmail.com
Sent: 25-7-2014 00:10
To: Discussion of Numerical Python numpy-discussion@scipy.org
Subject: Re: [Numpy-discussion] numpy.mean still broken for largefloat32arrays
On 7/24/2014 4:42 PM, Eelco Hoogendoorn wrote:
This isn't a bug report, but rather a feature request.
I'm
53 matches
Mail list logo
